The direction for this work has been set in the discussion thread:
"About creating machines on the command line" in January/February 2021:
https://lists.gnu.org/archive/html/qemu-devel/2021-01/msg01839.html
https://lists.gnu.org/archive/html/qemu-devel/2021-02/msg01070.html

To customize a machine via QMP we need the ability to stop QEMU at a specific
machine initialization phase.

Currently, machine initialization phases are:
1) no-machine: machine does not exist yet (current_machine == NULL)
2) machine-created: machine exists, but its accelerator does not
   (current_machine->accelerator == NULL)
3) accel-created: machine's accelerator is configured
   (current_machine->accelerator != NULL), but machine class's init() has not
   been called (no properties validated, machine_init_done notifiers not
   registered, no sysbus, etc.)
4) initialized: machine class's init() has been called, thus machine properties
   are validated, machine_init_done notifiers registered, sysbus realized, etc.
   Devices added at this phase are considered to be cold-plugged.
5) ready: machine_init_done notifiers are called, then QEMU is ready to start
   CPUs. Devices added at this phase are considered to be hot-plugged.

QEMU can be stopped today using the -preconfig CLI option at phase 3
(accel-created). This option was introduced to enable the QMP configuration of
parameters that affect the machine initialization. We cannot add devices at
this point because the machine class's init() has not been called, thus sysbus
does not exist yet (a device cannot be added because there is no bus to attach
it to).

QEMU can be also stopped using the -S CLI option at the machine ready phase.
However, it is too late to add devices at this phase because the machine is
already configured, and any devices added at this point are considered to be
hot-plugged.

Since the existing -preconfig CLI option stops QEMU too early, and the -S option
stops too late, we need a way to stop QEMU in between (after the machine is
initialized and before it becomes ready).

We propose to add QMP commands to step through machine phases starting from the
initial CLI's '-preconfig' early stop point. With this addition, we may now
execute QMP commands at any machine phase.

The 'next-machine-phase' command would trigger QEMU to execute initialization
steps that are needed to enter the next phase. If it's more convenient to jump
to an initialization phase than to single-step through phases, the
'advance-machine-phase' command should be used instead of 'next-machine-phase'.
Additionally, we propose to add the command to query the current machine phase,
namely 'query-machine-phase'.

With this patch it would be possible to add devices via QMP. For example,
by running QEMU with:
$ qemu-system-riscv32 \
  -M sifive_dt \
  -qmp unix:./qmp-sock,server \
  -preconfig \
  ...

and scripts/qmp/qmp-shell as the QMP client:
$ qemu/scripts/qmp/qmp-shell ./qmp-sock
Welcome to the QMP low-level shell!
Connected to QEMU 6.0.0

(QEMU) query-machine-phase
{"return": {"phase": "accel-created"}}
(QEMU) next-machine-phase
{"return": {}}
(QEMU) query-machine-phase
{"return": {"phase": "initialized"}}
(QEMU) device_add driver=...
{"return": {}}
(QEMU) next-machine-phase
{"return": {}}
(QEMU) query-machine-phase
{"return": {"phase": "ready"}}

Note that with the introduced changes, devices can still be added via CLI, i.e.
we support a mixed configuration approach (CLI, QMP, or CLI/QMP). Any device
specified via CLI will be added before QEMU waits for the QMP configuration
in the machine 'initialized' phase.

Mirela Grujic (9):
  vl: Allow finer control in advancing machine through phases
  replace machine phase_check with machine_is_initialized/ready calls
  rename MachineInitPhase enumeration constants
  qapi: Implement 'query-machine-phase' command
  qapi: Implement 'next-machine-phase' command
  qapi: Implement 'advance-machine-phase' command
  qdev-monitor: Restructure and fix the check for command availability
  qapi: Introduce 'allow-init-config' option
  qapi: Allow some commands to be executed in machine 'initialized'
    phase

 docs/sphinx/qapidoc.py      |   2 +-
 qapi/machine.json           | 105 ++++++++++++++++++++++++++++++++++++
 qapi/qdev.json              |   3 +-
 include/hw/qdev-core.h      |  32 ++---------
 include/qapi/qmp/dispatch.h |   1 +
 include/sysemu/sysemu.h     |   3 ++
 hw/core/machine-qmp-cmds.c  |  33 +++++++++++-
 hw/core/machine.c           |   6 +--
 hw/core/qdev.c              |  17 +++++-
 hw/pci/pci.c                |   2 +-
 hw/usb/core.c               |   2 +-
 hw/virtio/virtio-iommu.c    |   2 +-
 monitor/hmp.c               |   2 +-
 monitor/misc.c              |   2 +-
 softmmu/qdev-monitor.c      |  31 +++++++----
 softmmu/vl.c                |  94 ++++++++++++++++++++------------
 ui/console.c                |   2 +-
 scripts/qapi/commands.py    |  10 ++--
 scripts/qapi/expr.py        |   5 +-
 scripts/qapi/introspect.py  |   3 +-
 scripts/qapi/schema.py      |  10 ++--
 21 files changed, 274 insertions(+), 93 deletions(-)

-- 
2.25.1

Hi Mirela, this is very interesting!

It's unfortunate that I completely missed the discussions in 
January/February.  You might have noticed that in the 5.2/6.0 timeframe 
I worked on cleaning up the machine initialization phases and qemu_init. 
  The idea behind the cleanup was to identify clearly the steps from one 
phase to the next.  I am very happy that you are already benefitting 
from that work in this series and you were able to make a prototype with 
so little code.

My plan was a bit more ambitious though :) and it is laid out at 
https://wiki.qemu.org/User:Paolo_Bonzini/Machine_init_sequence.

My plan was to have completely different binaries than the current 
qemu-system-*.  These would only have a handful of command line options 
(such as -name, -sandbox, -trace, -L) and would open a QMP connection on 
stdio.

All other command line option would be either considered legacy or 
adjusted to be part of two new QMP commands, machine-set and accel-set, 
which would advance through the phases like this:

PHASE_NO_MACHINE
    -> machine-set -> PHASE_MACHINE_CREATED ->
    -> accel-set -> PHASE_ACCEL_CREATED -> PHASE_MACHINE_INITIALIZED ->
    -> finish-machine-init -> PHASE_MACHINE_READY
    -> cont

I think this idea would work well with your plan below, because the 
preconfiguration that you need to do happens mostly at 
PHASE_MACHINE_INITIALIZED.

Of course, the disadvantage of my approach is that, in the initial 
version, a lot of capabilities of QEMU are not available (especially 
with respect to the UI, since there's no "display-add" command). 
However, the basic implementation of machine-set and accel-set should 
not really be *that* much more work, and it should be doable in parallel 
with the conversion efforts for those options.  For example, today I 
posted a series that maps -smp to -M (and then, SMP configuration would 
automatically become available in machine-set).

I have placed the skeleton of the work I was doing at 
https://gitlab.com/bonzini/qemu/ in the branch qemu-qmp-targets.  You 
can see a sample execution at 
https://asciinema.org/a/TXMX8EZh8Md0fZnuE7uhfv6cO.  I have not touched 
some of the patches in a long time, but I plan to give them a quick test 
tomorrow.  Starting from the code in that branch, it should not be hard 
to implement the machine-set and accel-set commands in the same fashion 
as QEMU 5.2's implementation of object-add.

Thanks for posting these patches, I have started a light review of them.

Paolo

On 13/05/21 10:25, Mirela Grujic wrote:
> The direction for this work has been set in the discussion thread:
> "About creating machines on the command line" in January/February 2021:
> https://lists.gnu.org/archive/html/qemu-devel/2021-01/msg01839.html
> https://lists.gnu.org/archive/html/qemu-devel/2021-02/msg01070.html
> 
> To customize a machine via QMP we need the ability to stop QEMU at a specific
> machine initialization phase.
> 
> Currently, machine initialization phases are:
> 1) no-machine: machine does not exist yet (current_machine == NULL)
> 2) machine-created: machine exists, but its accelerator does not
>     (current_machine->accelerator == NULL)
> 3) accel-created: machine's accelerator is configured
>     (current_machine->accelerator != NULL), but machine class's init() has not
>     been called (no properties validated, machine_init_done notifiers not
>     registered, no sysbus, etc.)
> 4) initialized: machine class's init() has been called, thus machine properties
>     are validated, machine_init_done notifiers registered, sysbus realized, etc.
>     Devices added at this phase are considered to be cold-plugged.
> 5) ready: machine_init_done notifiers are called, then QEMU is ready to start
>     CPUs. Devices added at this phase are considered to be hot-plugged.
> 
> QEMU can be stopped today using the -preconfig CLI option at phase 3
> (accel-created). This option was introduced to enable the QMP configuration of
> parameters that affect the machine initialization. We cannot add devices at
> this point because the machine class's init() has not been called, thus sysbus
> does not exist yet (a device cannot be added because there is no bus to attach
> it to).
> 
> QEMU can be also stopped using the -S CLI option at the machine ready phase.
> However, it is too late to add devices at this phase because the machine is
> already configured, and any devices added at this point are considered to be
> hot-plugged.
> 
> Since the existing -preconfig CLI option stops QEMU too early, and the -S option
> stops too late, we need a way to stop QEMU in between (after the machine is
> initialized and before it becomes ready).

Hi Paolo,


Thanks for the feedback!


On 5/13/21 7:52 PM, Paolo Bonzini wrote:
> Hi Mirela, this is very interesting!
>
> It's unfortunate that I completely missed the discussions in 
> January/February.  You might have noticed that in the 5.2/6.0 
> timeframe I worked on cleaning up the machine initialization phases 
> and qemu_init.  The idea behind the cleanup was to identify clearly 
> the steps from one phase to the next.  I am very happy that you are 
> already benefitting from that work in this series and you were able to 
> make a prototype with so little code.


I was really happy to see your changes in 6.0, they simplified the 
implementation a lot. It looked like you're up to something bigger, and 
I'm glad that we can sync up now.


>
> My plan was a bit more ambitious though :) and it is laid out at 
> https://wiki.qemu.org/User:Paolo_Bonzini/Machine_init_sequence.
>
> My plan was to have completely different binaries than the current 
> qemu-system-*.  These would only have a handful of command line 
> options (such as -name, -sandbox, -trace, -L) and would open a QMP 
> connection on stdio.
>
> All other command line option would be either considered legacy or 
> adjusted to be part of two new QMP commands, machine-set and 
> accel-set, which would advance through the phases like this:
>
> PHASE_NO_MACHINE
>    -> machine-set -> PHASE_MACHINE_CREATED ->
>    -> accel-set -> PHASE_ACCEL_CREATED -> PHASE_MACHINE_INITIALIZED ->
>    -> finish-machine-init -> PHASE_MACHINE_READY
>    -> cont
>
> I think this idea would work well with your plan below, because the 
> preconfiguration that you need to do happens mostly at 
> PHASE_MACHINE_INITIALIZED.
>
> Of course, the disadvantage of my approach is that, in the initial 
> version, a lot of capabilities of QEMU are not available (especially 
> with respect to the UI, since there's no "display-add" command). 
> However, the basic implementation of machine-set and accel-set should 
> not really be *that* much more work, and it should be doable in 
> parallel with the conversion efforts for those options.  For example, 
> today I posted a series that maps -smp to -M (and then, SMP 
> configuration would automatically become available in machine-set).


With our approach, transitioning to the QMP configuration suppose to 
happen gradually, i.e. we still specify some configuration options via 
command line. For your approach to be applicable to our use case we 
would at least need a QMP equivalent for the following:

qemu-system-riscv64 \
     -M sifive_dt \
     -cpu 
rv64,i=true,g=false,m=true,a=true,f=true,d=true,c=true,s=false,u=false,x-b=true,pmp=true,mmu=false,num-pmp-regions=8 
\
     -smp 1 \
     -device ...

AFAIU from the materials you shared, we would need to add -cpu and 
-device, but I don't see any reason why we wouldn't do this.


>
> I have placed the skeleton of the work I was doing at 
> https://gitlab.com/bonzini/qemu/ in the branch qemu-qmp-targets. You 
> can see a sample execution at 
> https://asciinema.org/a/TXMX8EZh8Md0fZnuE7uhfv6cO.  I have not touched 
> some of the patches in a long time, but I plan to give them a quick 
> test tomorrow.  Starting from the code in that branch, it should not 
> be hard to implement the machine-set and accel-set commands in the 
> same fashion as QEMU 5.2's implementation of object-add.
>

Ok, please let me know once you test, then I would run your code and 
play with it to better understand what needs to be done. Then I might 
come back with a couple of questions, so that we align on the TODOs. Is 
that ok with you?

Btw, when (in which version) did you plan to integrate the qemu-qmp-* 
support? I guess once machine-set/accel-set is implemented, but maybe 
I'm wrong...


> Thanks for posting these patches, I have started a light review of them.
>

If we would add the support for our use case to your approach, then this 
series would likely be split into a couple of patches that are 
applicable and the rest that is obsolete.


In summary, we believe it would be great to join efforts, please let us 
know how can we help.


Thanks,

Mirela


> Paolo
>
> On 13/05/21 10:25, Mirela Grujic wrote:
>> The direction for this work has been set in the discussion thread:
>> "About creating machines on the command line" in January/February 2021:
>> https://lists.gnu.org/archive/html/qemu-devel/2021-01/msg01839.html
>> https://lists.gnu.org/archive/html/qemu-devel/2021-02/msg01070.html
>>
>> To customize a machine via QMP we need the ability to stop QEMU at a 
>> specific
>> machine initialization phase.
>>
>> Currently, machine initialization phases are:
>> 1) no-machine: machine does not exist yet (current_machine == NULL)
>> 2) machine-created: machine exists, but its accelerator does not
>>     (current_machine->accelerator == NULL)
>> 3) accel-created: machine's accelerator is configured
>>     (current_machine->accelerator != NULL), but machine class's 
>> init() has not
>>     been called (no properties validated, machine_init_done notifiers 
>> not
>>     registered, no sysbus, etc.)
>> 4) initialized: machine class's init() has been called, thus machine 
>> properties
>>     are validated, machine_init_done notifiers registered, sysbus 
>> realized, etc.
>>     Devices added at this phase are considered to be cold-plugged.
>> 5) ready: machine_init_done notifiers are called, then QEMU is ready 
>> to start
>>     CPUs. Devices added at this phase are considered to be hot-plugged.
>>
>> QEMU can be stopped today using the -preconfig CLI option at phase 3
>> (accel-created). This option was introduced to enable the QMP 
>> configuration of
>> parameters that affect the machine initialization. We cannot add 
>> devices at
>> this point because the machine class's init() has not been called, 
>> thus sysbus
>> does not exist yet (a device cannot be added because there is no bus 
>> to attach
>> it to).
>>
>> QEMU can be also stopped using the -S CLI option at the machine ready 
>> phase.
>> However, it is too late to add devices at this phase because the 
>> machine is
>> already configured, and any devices added at this point are 
>> considered to be
>> hot-plugged.
>>
>> Since the existing -preconfig CLI option stops QEMU too early, and 
>> the -S option
>> stops too late, we need a way to stop QEMU in between (after the 
>> machine is
>> initialized and before it becomes ready).
>

On 14/05/21 14:48, Mirela Grujic wrote:
> 
> With our approach, transitioning to the QMP configuration suppose to 
> happen gradually, i.e. we still specify some configuration options via 
> command line. For your approach to be applicable to our use case we 
> would at least need a QMP equivalent for the following:
> 
> qemu-system-riscv64 \
>      -M sifive_dt \
>      -cpu 
> rv64,i=true,g=false,m=true,a=true,f=true,d=true,c=true,s=false,u=false,x-b=true,pmp=true,mmu=false,num-pmp-regions=8 
> \
>      -smp 1 \
>      -device ...
> 
> AFAIU from the materials you shared, we would need to add -cpu and 
> -device, but I don't see any reason why we wouldn't do this.

-cpu is indeed the big one that I had not looked at so far, while 
-device should be mostly covered by the existing device_add command.

One possibility for -cpu is to make it an argument of machine-set too. 
For example the above would be

{ 'execute': 'machine-set', arguments: {
     'type': 'sifive_dt',
     'smp': { 'cpus': 1 },
     'cpu': { 'model': 'rv64', 'i': true, 'g': false, ... }
}

> Ok, please let me know once you test, then I would run your code and
> play with it to better understand what needs to be done. Then I might
> come back with a couple of questions, so that we align on the TODOs. Is
> that ok with you?

Yes, of course.  I pushed something that at least compiles and passes a 
basic smoke test.

> Btw, when (in which version) did you plan to integrate the
> qemu-qmp-* support? I guess once machine-set/accel-set is implemented,  > but maybe I'm wrong...

Well, the right answer is "when somebody needs it".  The things that I 
was mostly interested in (e.g. compound properties for machines, such as 
smp in the example above) were all enablers for qemu-qmp-* but I was not 
really interested in the new binaries.  I did the qemu-qmp-* patches 
mostly to validate that the 5.2/6.0 refactoring of preconfig was going 
in the right direction.

However, if there is indeed somebody that needs it I'll contribute where 
our interests overlap.  In particular I can take care of converting the 
command line options to properties.

Paolo

On Fri, May 14, 2021 at 06:00:37PM +0200, Paolo Bonzini wrote:
> On 14/05/21 14:48, Mirela Grujic wrote:
> > 
> > With our approach, transitioning to the QMP configuration suppose to
> > happen gradually, i.e. we still specify some configuration options via
> > command line. For your approach to be applicable to our use case we
> > would at least need a QMP equivalent for the following:
> > 
> > qemu-system-riscv64 \
> >      -M sifive_dt \
> >      -cpu rv64,i=true,g=false,m=true,a=true,f=true,d=true,c=true,s=false,u=false,x-b=true,pmp=true,mmu=false,num-pmp-regions=8
> > \
> >      -smp 1 \
> >      -device ...
> > 
> > AFAIU from the materials you shared, we would need to add -cpu and
> > -device, but I don't see any reason why we wouldn't do this.
> 
> -cpu is indeed the big one that I had not looked at so far, while -device
> should be mostly covered by the existing device_add command.
> 
> One possibility for -cpu is to make it an argument of machine-set too. For
> example the above would be
> 
> { 'execute': 'machine-set', arguments: {
>     'type': 'sifive_dt',
>     'smp': { 'cpus': 1 },
>     'cpu': { 'model': 'rv64', 'i': true, 'g': false, ... }
> }

CPUs are a little complex because they have association with both machine
types and accelerator backends.  You have 'accel-set' being issued after
'machine-set', but the 'host' CPU model is only valid if you have set
the KVM accelerator, not TCG.

It is desirable that you get an error report about bad CPU at the time
you specify the CPU config, rather than have that error delayed to when
a later command runs and invalidates your CPU choice.

This is a long winded way of saying either 'accel-set' should be
run before 'machine-set', or 'cpu' would have to be set as its own
command or as part of 'accel-set'.

My gut feeling though is accel-set would be more logical being done
first, as that also influences the set of features available in other
areas of QEMU configuration. Was there a reason you listed it after
machine-set ?

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Il ven 14 mag 2021, 18:20 Daniel P. Berrangé <berrange@redhat.com> ha
scritto:

> My gut feeling though is accel-set would be more logical being done
> first, as that also influences the set of features available in other
> areas of QEMU configuration. Was there a reason you listed it after
> machine-set ?
>

That was also my initial gut feeling, but actually right now the machine
influences the accelerator more than the other way round. For example the
initialization of the accelerator takes a machine so that for example on
x86 the per-architecture KVM knows whether to set up SMM. Also different
machines could use different modes for KVM (HV vs PR for ppc), and some
machines may not be virtualizable at all so they require TCG.

The host CPU these days is really a virtualization-only synonym for -cpu
max, which works for TCG as well. But you're right that x86 CPU flags are
dictated by the accelerator rather than the machine, so specifying it in
machine-set would be clumsy. On the other hand on ARM it's a bit of both:
for KVM it's basically always -cpu host so the accelerator is important;
but some machines may have an M profile CPU and some may have an A.

I don't have the sources at hand to check in which phase CPUs are created,
but it's definitely after ACCEL_CREATED. Adding a third command
cpu-model-set is probably the easiest way to proceed.

Paolo


> Regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>

On Fri, 14 May 2021 20:32:22 +0200
Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il ven 14 mag 2021, 18:20 Daniel P. Berrangé <berrange@redhat.com> ha
> scritto:
> 
> > My gut feeling though is accel-set would be more logical being done
> > first, as that also influences the set of features available in other
> > areas of QEMU configuration. Was there a reason you listed it after
> > machine-set ?
> >  
> 
> That was also my initial gut feeling, but actually right now the machine
> influences the accelerator more than the other way round. For example the
> initialization of the accelerator takes a machine so that for example on
> x86 the per-architecture KVM knows whether to set up SMM. Also different
> machines could use different modes for KVM (HV vs PR for ppc), and some
> machines may not be virtualizable at all so they require TCG.
> 
> The host CPU these days is really a virtualization-only synonym for -cpu
> max, which works for TCG as well. But you're right that x86 CPU flags are
> dictated by the accelerator rather than the machine, so specifying it in
> machine-set would be clumsy. On the other hand on ARM it's a bit of both:
> for KVM it's basically always -cpu host so the accelerator is important;
> but some machines may have an M profile CPU and some may have an A.
and some do not support -cpu/-smp/or rather ignore it and go ahead with hardcoded one/


> I don't have the sources at hand to check in which phase CPUs are created,
> but it's definitely after ACCEL_CREATED. Adding a third command
> cpu-model-set is probably the easiest way to proceed.


a naive question,
why not ditch -cpu completely and instantiate CPUs with

device_add cpu-foo,flagX=on/off

the former is just a kluge for making CLI based -smp initial_nr_cpu/-cpu foo, work.
I'd move that logic to CLI compat wrapper translating that to a series of device_add calls
for QMP based QEMU.

Also I maybe wrong, but I thought that -cpu nowadays does nothing except of translating
legacy cpu model name to cpu-type and flags to a bunch of '-global', which are applied
automatically when CPUa are created at board_init() time or later on when device_add is used.

> Paolo
> 
> 
> > Regards,
> > Daniel
> > --
> > |: https://berrange.com      -o-
> > https://www.flickr.com/photos/dberrange :|
> > |: https://libvirt.org         -o-
> > https://fstop138.berrange.com :|
> > |: https://entangle-photo.org    -o-
> > https://www.instagram.com/dberrange :|
> >
> >  

On Mon, 24 May 2021 19:20:37 +0200
Igor Mammedov <imammedo@redhat.com> wrote:

> On Fri, 14 May 2021 20:32:22 +0200
> Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> > Il ven 14 mag 2021, 18:20 Daniel P. Berrangé <berrange@redhat.com> ha
> > scritto:
> >   
> > > My gut feeling though is accel-set would be more logical being done
> > > first, as that also influences the set of features available in other
> > > areas of QEMU configuration. Was there a reason you listed it after
> > > machine-set ?
> > >    
> > 
> > That was also my initial gut feeling, but actually right now the machine
> > influences the accelerator more than the other way round. For example the
> > initialization of the accelerator takes a machine so that for example on
> > x86 the per-architecture KVM knows whether to set up SMM. Also different
> > machines could use different modes for KVM (HV vs PR for ppc), and some
> > machines may not be virtualizable at all so they require TCG.
> > 
> > The host CPU these days is really a virtualization-only synonym for -cpu
> > max, which works for TCG as well. But you're right that x86 CPU flags are
> > dictated by the accelerator rather than the machine, so specifying it in
> > machine-set would be clumsy. On the other hand on ARM it's a bit of both:
> > for KVM it's basically always -cpu host so the accelerator is important;
> > but some machines may have an M profile CPU and some may have an A.  
> and some do not support -cpu/-smp/or rather ignore it and go ahead with hardcoded one/
> 
> 
> > I don't have the sources at hand to check in which phase CPUs are created,
> > but it's definitely after ACCEL_CREATED. Adding a third command
> > cpu-model-set is probably the easiest way to proceed.  
> 
> 
> a naive question,
> why not ditch -cpu completely and instantiate CPUs with
> 
> device_add cpu-foo,flagX=on/off
while writing my another reply to this thread, I realized we might need cpu type
at least for 3 reasons:
 1. It would be hard/impossible to convert every board to create CPUs with
    device_add after board_init(), many boards depend on CPUs being available
    at that time to wire up another devices created at the same time.
 2. If we can postpone cpu creation after board_init() time, user would need
    query-hotpluggable-cpus command to get which CPU type and topology properties
    to use with device_add. (that's what libvirt uses curently)
 3. Recent work on ECPY cpu topology issues, showed that the topology, that board
    generates from -smp might depends on CPU type (we got rid of that eventually but
    I wouldn't rule out possibility in the future).
    
What I would remove from -cpu behavior is resolving cpu-model to CPU type.
QMP command would use only cpu-type so it would be consistent with device_add
which already uses cpu-type, which user gets from query-hotpluggble-cpus.

All legacy cpu-model conversion can be pushed out into CLI compat binary or
up the stack to keep existing deployments running, and new VMs can use cpu types
directly.

Question is how user can query available (for given target/machine/accel) CPU types
using QMP, is there such command already?

> the former is just a kluge for making CLI based -smp initial_nr_cpu/-cpu foo, work.
> I'd move that logic to CLI compat wrapper translating that to a series of device_add calls
> for QMP based QEMU.
> 
> Also I maybe wrong, but I thought that -cpu nowadays does nothing except of translating
> legacy cpu model name to cpu-type and flags to a bunch of '-global', which are applied
> automatically when CPUa are created at board_init() time or later on when device_add is used.
> 
> > Paolo
> > 
> >   
> > > Regards,
> > > Daniel
> > > --
> > > |: https://berrange.com      -o-
> > > https://www.flickr.com/photos/dberrange :|
> > > |: https://libvirt.org         -o-
> > > https://fstop138.berrange.com :|
> > > |: https://entangle-photo.org    -o-
> > > https://www.instagram.com/dberrange :|
> > >
> > >    
> 
> 

Paolo Bonzini <pbonzini@redhat.com> writes:

> Hi Mirela, this is very interesting!
>
> It's unfortunate that I completely missed the discussions in
> January/February.  You might have noticed that in the 5.2/6.0
> timeframe I worked on cleaning up the machine initialization phases
> and qemu_init.   The idea behind the cleanup was to identify clearly
> the steps from one phase to the next.  I am very happy that you are
> already benefitting from that work in this series and you were able to
> make a prototype with so little code.
>
> My plan was a bit more ambitious though :) and it is laid out at
> https://wiki.qemu.org/User:Paolo_Bonzini/Machine_init_sequence.
>
> My plan was to have completely different binaries than the current
> qemu-system-*.

Now you have 2x problems :)

>                 These would only have a handful of command line
> options (such as -name, -sandbox, -trace, -L) and would open a QMP
> connection on stdio.
>
> All other command line option would be either considered legacy or
> adjusted to be part of two new QMP commands, machine-set and
> accel-set, which would advance through the phases like this:
>
> PHASE_NO_MACHINE
>    -> machine-set -> PHASE_MACHINE_CREATED ->
>    -> accel-set -> PHASE_ACCEL_CREATED -> PHASE_MACHINE_INITIALIZED ->
>    -> finish-machine-init -> PHASE_MACHINE_READY
>    -> cont

Is machine-set one big command, or a sequence of commands, where each
command configures just one thing?

Same for accel-set.

Permit me to go off on a tangent: how much and what kind of magic do we
want in the initialization sequence?

The existing order is a confusing mess grown out of a half-hearted
attempt to make things just work.  We've talked about it a few times,
e.g. here:

    Message-ID: <87poomg77m.fsf@dusky.pond.sub.org>
    https://lists.nongnu.org/archive/html/qemu-devel/2016-09/msg00163.html

Are you aiming for "leave ordering to the user", or for "do the right
thing" (the user's order doesn't matter)", or for "neither; confusing
messes are fun"?

Another tangent:

1. A command line option is like a QMP command that returns nothing.
Generating CLI options from a QAPI schema is no harder than generating
QMP commands, it's just work.

2. A configuration file is like a command line, only easier to work with
when configuration becomes non-trivial.  Generating configuration file
language from a QAPI schema is no harder than generating QMP commands /
CLI options, it's just work.

3. QMP is strictly more powerful than CLI or comfiguration file.  It's
also less convenient for ad hoc use by humans: compare -readconfig
vm123.cfg to feeding the equivalent QMP commands with socat.

4. We'll see whether the convenience for humans can motivate us to put
in the work.

> I think this idea would work well with your plan below, because the
> preconfiguration that you need to do happens mostly at 
> PHASE_MACHINE_INITIALIZED.
>
> Of course, the disadvantage of my approach is that, in the initial
> version, a lot of capabilities of QEMU are not available (especially 
> with respect to the UI, since there's no "display-add"
> command). However, the basic implementation of machine-set and
> accel-set should not really be *that* much more work, and it should be
> doable in parallel with the conversion efforts for those options.  For
> example, today I posted a series that maps -smp to -M (and then, SMP
> configuration would automatically become available in machine-set).
>
> I have placed the skeleton of the work I was doing at
> https://gitlab.com/bonzini/qemu/ in the branch qemu-qmp-targets.  You 
> can see a sample execution at
> https://asciinema.org/a/TXMX8EZh8Md0fZnuE7uhfv6cO.  I have not touched 
> some of the patches in a long time, but I plan to give them a quick
> test tomorrow.  Starting from the code in that branch, it should not
> be hard to implement the machine-set and accel-set commands in the
> same fashion as QEMU 5.2's implementation of object-add.
>
> Thanks for posting these patches, I have started a light review of them.

Please cc: on future work in this area.

I'm about to drop off for two weeks of vacation, though :)

On 21/05/21 13:32, Markus Armbruster wrote:
> PHASE_NO_MACHINE
>    -> machine-set -> PHASE_MACHINE_CREATED ->
>    -> accel-set -> PHASE_ACCEL_CREATED -> PHASE_MACHINE_INITIALIZED ->
>    -> finish-machine-init -> PHASE_MACHINE_READY
>    -> cont
>
> Is machine-set one big command, or a sequence of commands, where each
> command configures just one thing?
> 
> Same for accel-set.

They would be almost 1:1 mappings with -M and -accel.  If we add a third 
command for the CPU model, machine-set and accel-set would be basically 
as big as device_add or object-add.

So the full flow would be

  PHASE_NO_MACHINE
     -> machine-set -> PHASE_MACHINE_CREATED ->
     -> accel-set -> PHASE_ACCEL_CREATED ->
     -> cpu-model-set -> PHASE_MACHINE_INITIALIZED ->
     -> device_add...
     -> finish-machine-init -> PHASE_MACHINE_READY
     -> cont

> Permit me to go off on a tangent: how much and what kind of magic do we
> want in the initialization sequence?

No magic at all, because the QMP configuration would be entirely 
-nodefaults.  Default devices, for boards that need them, can be created 
by setting properties such as serial0, netdev0 in machine-set (and in no 
other way).

Paolo

I cc-ed Igor because he worked on preconfig support.


On 5/13/21 7:52 PM, Paolo Bonzini wrote:
> Hi Mirela, this is very interesting!
>
> It's unfortunate that I completely missed the discussions in 
> January/February.  You might have noticed that in the 5.2/6.0 
> timeframe I worked on cleaning up the machine initialization phases 
> and qemu_init.  The idea behind the cleanup was to identify clearly 
> the steps from one phase to the next.  I am very happy that you are 
> already benefitting from that work in this series and you were able to 
> make a prototype with so little code.
>
> My plan was a bit more ambitious though :) and it is laid out at 
> https://wiki.qemu.org/User:Paolo_Bonzini/Machine_init_sequence.
>
> My plan was to have completely different binaries than the current 
> qemu-system-*.  These would only have a handful of command line 
> options (such as -name, -sandbox, -trace, -L) and would open a QMP 
> connection on stdio.
>
> All other command line option would be either considered legacy or 
> adjusted to be part of two new QMP commands, machine-set and 
> accel-set, which would advance through the phases like this:
>
> PHASE_NO_MACHINE
>    -> machine-set -> PHASE_MACHINE_CREATED ->
>    -> accel-set -> PHASE_ACCEL_CREATED -> PHASE_MACHINE_INITIALIZED ->


My understanding is that an equivalent of previously supported 
'preconfig' state is PHASE_ACCEL_CREATED, from the perspective of the 
QMP configuration that Igor implemented. In other words, I believe that 
when -preconfig CLI option was passed, QEMU was waiting for the QMP 
configuration in PHASE_ACCEL_CREATED phase. Now, if accel-set advances 
the machine directly to PHASE_MACHINE_INITIALIZED, there will be no 
chance to configure what Igor did with -preconfig.

Is this something you don't want to support anymore, or it can be 
configured in another way? Or is this something that we haven't thought 
of yet, but we should?


Another point I'm trying to make here - sooner or later I believe we 
will run again into the issue similar to the one described above. I 
truly believe that the commands you proposed (machine-set and accel-set) 
should be kept as is, except that they wouldn't automatically advance 
the machine initialization phase. I think the best approach would be to 
combine your proposal with mine: the machine-set and accel-set commands 
would configure stuff as you proposed, but there will be separate 
commands to advance machine phases (next-machine-phase and 
target/set-machine-phase). I also believe such a separation would be 
extremely important for troubleshooting configurations.


Btw, we already have such a separation because we will reuse device_add 
command, that only does the configuration. I believe it will be much 
cleaner and flexible for future development to separate QMP commands 
into 1) those that configure stuff, and 2) those that control advancing 
of the machine init phases.

Moreover, once it turns out that a command receives too many arguments 
and should be split into few, it will be easier to do so if a single 
command did not include both configuration and control-flow. Similar 
holds for adding new commands into the flow.


Please let me know what you think about the proposal.


Thanks,

Mirela


> -> finish-machine-init -> PHASE_MACHINE_READY
>    -> cont
>
> I think this idea would work well with your plan below, because the 
> preconfiguration that you need to do happens mostly at 
> PHASE_MACHINE_INITIALIZED.
>
> Of course, the disadvantage of my approach is that, in the initial 
> version, a lot of capabilities of QEMU are not available (especially 
> with respect to the UI, since there's no "display-add" command). 
> However, the basic implementation of machine-set and accel-set should 
> not really be *that* much more work, and it should be doable in 
> parallel with the conversion efforts for those options.  For example, 
> today I posted a series that maps -smp to -M (and then, SMP 
> configuration would automatically become available in machine-set).
>
> I have placed the skeleton of the work I was doing at 
> https://gitlab.com/bonzini/qemu/ in the branch qemu-qmp-targets. You 
> can see a sample execution at 
> https://asciinema.org/a/TXMX8EZh8Md0fZnuE7uhfv6cO.  I have not touched 
> some of the patches in a long time, but I plan to give them a quick 
> test tomorrow.  Starting from the code in that branch, it should not 
> be hard to implement the machine-set and accel-set commands in the 
> same fashion as QEMU 5.2's implementation of object-add.
>
> Thanks for posting these patches, I have started a light review of them.
>
> Paolo
>
> On 13/05/21 10:25, Mirela Grujic wrote:
>> The direction for this work has been set in the discussion thread:
>> "About creating machines on the command line" in January/February 2021:
>> https://lists.gnu.org/archive/html/qemu-devel/2021-01/msg01839.html
>> https://lists.gnu.org/archive/html/qemu-devel/2021-02/msg01070.html
>>
>> To customize a machine via QMP we need the ability to stop QEMU at a 
>> specific
>> machine initialization phase.
>>
>> Currently, machine initialization phases are:
>> 1) no-machine: machine does not exist yet (current_machine == NULL)
>> 2) machine-created: machine exists, but its accelerator does not
>>     (current_machine->accelerator == NULL)
>> 3) accel-created: machine's accelerator is configured
>>     (current_machine->accelerator != NULL), but machine class's 
>> init() has not
>>     been called (no properties validated, machine_init_done notifiers 
>> not
>>     registered, no sysbus, etc.)
>> 4) initialized: machine class's init() has been called, thus machine 
>> properties
>>     are validated, machine_init_done notifiers registered, sysbus 
>> realized, etc.
>>     Devices added at this phase are considered to be cold-plugged.
>> 5) ready: machine_init_done notifiers are called, then QEMU is ready 
>> to start
>>     CPUs. Devices added at this phase are considered to be hot-plugged.
>>
>> QEMU can be stopped today using the -preconfig CLI option at phase 3
>> (accel-created). This option was introduced to enable the QMP 
>> configuration of
>> parameters that affect the machine initialization. We cannot add 
>> devices at
>> this point because the machine class's init() has not been called, 
>> thus sysbus
>> does not exist yet (a device cannot be added because there is no bus 
>> to attach
>> it to).
>>
>> QEMU can be also stopped using the -S CLI option at the machine ready 
>> phase.
>> However, it is too late to add devices at this phase because the 
>> machine is
>> already configured, and any devices added at this point are 
>> considered to be
>> hot-plugged.
>>
>> Since the existing -preconfig CLI option stops QEMU too early, and 
>> the -S option
>> stops too late, we need a way to stop QEMU in between (after the 
>> machine is
>> initialized and before it becomes ready).
>

On 21/05/21 16:06, Mirela Grujic wrote:
>>
>> PHASE_NO_MACHINE
>>    -> machine-set -> PHASE_MACHINE_CREATED ->
>>    -> accel-set -> PHASE_ACCEL_CREATED -> PHASE_MACHINE_INITIALIZED ->
> 
> 
> My understanding is that an equivalent of previously supported 
> 'preconfig' state is PHASE_ACCEL_CREATED, from the perspective of the 
> QMP configuration that Igor implemented. In other words, I believe that 
> when -preconfig CLI option was passed, QEMU was waiting for the QMP 
> configuration in PHASE_ACCEL_CREATED phase. Now, if accel-set advances 
> the machine directly to PHASE_MACHINE_INITIALIZED, there will be no 
> chance to configure what Igor did with -preconfig.

Right, that was only NUMA.  I have to check, but I think it can be moved 
to PHASE_MACHINE_CREATED.

Apart from that, if we add a third command for the CPU model, that third 
command would run from PHASE_ACCEL_CREATED so the pre-existing preconfig 
state would be accessible.

Paolo

> Is this something you don't want to support anymore, or it can be 
> configured in another way? Or is this something that we haven't thought 
> of yet, but we should?

On Fri, 21 May 2021 18:57:36 +0200
Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 21/05/21 16:06, Mirela Grujic wrote:
> >>
> >> PHASE_NO_MACHINE
> >>    -> machine-set -> PHASE_MACHINE_CREATED ->
> >>    -> accel-set -> PHASE_ACCEL_CREATED -> PHASE_MACHINE_INITIALIZED ->  
> > 
> > 
> > My understanding is that an equivalent of previously supported 
> > 'preconfig' state is PHASE_ACCEL_CREATED, from the perspective of the 
> > QMP configuration that Igor implemented. In other words, I believe that 
> > when -preconfig CLI option was passed, QEMU was waiting for the QMP 
> > configuration in PHASE_ACCEL_CREATED phase. Now, if accel-set advances 
> > the machine directly to PHASE_MACHINE_INITIALIZED, there will be no 
> > chance to configure what Igor did with -preconfig.  
> 
> Right, that was only NUMA.  I have to check, but I think it can be moved 
> to PHASE_MACHINE_CREATED.
Dependency for NUMA were:
  1: -smp/-cpu being parsed before set_numa_options QMP command is called
        it's necessary to for machine being able to provide topology for
        given -smp combination.

        -cpu is not must have dependency (currently), it was just conveniently
        available when building topology in possible_cpu_arch_ids(), setting
        cpu-type there could be deffered to the later time (actual user for
        CPU type is QMP command query-hotpluggable-cpus, so that user could know
        what cpu type and what properties to use with device_add at hot-add time).

  2: memory backends depended on accel (TCG)
     (I think, Paolo has removed that dependency)


> Apart from that, if we add a third command for the CPU model, that third 
> command would run from PHASE_ACCEL_CREATED so the pre-existing preconfig 
> state would be accessible.
> 
> Paolo
> 
> > Is this something you don't want to support anymore, or it can be 
> > configured in another way? Or is this something that we haven't thought 
> > of yet, but we should?  
> 
>