Hi,

The goal of this work is to bring dynamic machine creation to QEMU:
we want to setup a machine without compiling a specific machine C
code. It would ease supporting highly configurable platforms (for
example resulting from an automated design flow). The requirements
for such configuration include begin able to specify the number of
cores, available peripherals, emmory mapping, IRQ mapping, etc.

This series focuses on the first step: populating a machine with
devices during its creation. We propose patches to support this
using QMP commands. This is a working set of patches and improves
over the earlier rfc (posted in May):
https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg03706.html

Although it is working and could be merged, it is tag as an RFC:
we probably need to discuss the conditions for allowing a device to
be created at an early stage. Patches 6, 10 and 13, 15 and 16 depend
on such conditions and are subject to change. Other patches are
unrelated to this point.

We address several issues in this series. They are detailed below.

## 1. Stoping QEMU to populate the machine with devices

QEMU goes through several steps (called _machine phases_) when
creating the machine: 'no-machine', 'machine-created',
'accel-created', 'initialized', and finally 'ready'. At 'ready'
phase, QEMU is ready to start (see Paolo's page
https://wiki.qemu.org/User:Paolo_Bonzini/Machine_init_sequence for
more details).

Using the -preconfig CLI option, QEMU can be stopped today during
the 'accel-created' phase. Then the 'x-exit-preconfig' QMP command
triggers QEMU moving forwards to the completion of the machine
creation ('ready' phase).

The devices are created during the 'initialized' phase. 
In this phase the machine init() method has been executed and thus
machine properties have been handled. Although the sysbus exists and
the machine may have been populated by the init(), 
_machine_init_done_ notifiers have not been called yet. At this point
we can add more devices to a machine.

We propose to add 2 QMP commands:
+ The 'query-machine-phase' command would return the current machine
  phase.
+ The 'x-machine-init' command would advance the machine phase to
  'initialized'. 'x-exit-preconfig' could then still be used to
  advance to the last phase.

## 2. Adding devices

Right now, the user can create devices in 2 ways: using '-device' CLI
option or 'device_add' QMP command. Both are executed after the
machine is ready: such devices are hot-plugged. We propose to allow
'device_add' QMP command to be used during the 'initialized' phase.

In this series, we keep the constraint that the device must be
'user-creatable' (this is a device class flag). We do not see any
reason why a device the user can hot-plug could not be created at an
earlier stage.

This part is still RFC because, as Peter mentioned it (in this thread
https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg01933.html),
we may want additional or distinct conditions for:
+ device we can hot-plug
+ device we can add in '-preconfig' (cold-plug)
We are open to suggestions. We could for example add a
'preconfig-creatable' or 'init-creatable' flag to device class, which
can identify a set of devices we can create this way.

The main addition is how we handle the case of sysbus devices. Sysbus
devices are particular because unlike, for example, pci devices, you
have to manually handle the memory mapping and interrupts wiring. So
right now, a sysbus device is dynamically creatable (using -device
CLI option or device_add QMP command) only if:
+ it is 'user_creatable' (like any other device),
+ and it is in the current machine sysbus device allow list.

In this series, we propose to relax the second constraint during the
earlier phases of machine creation so that when using -preconfig we
can create any 'user-creatable' sysbus device. When the machine
progresses to the 'ready' phase, sysbus devices creation will come
back to the legacy behavior: it will be possible only based on the
per-machine authorization basis.

For sysbus devices, wiring interrupts is not a problem as we can use
the 'qom-set' QMP command, but memory mapping is.

## 3. Memory mapping

There is no point allowing the creation sysbus devices if we cannot
map them onto the memory bus (the 'sysbus').

As far as we know, right now, there is no way to add memory mapping
for sysbus device using QMP commands. We propose a 'x-sysbus-mmio-map'
command to do this. This command would only be allowed during the
'initialized' phase when using -preconfig.

## 4. Working example

The last patches of the series add and modify devices in order to
build a working machine starting from the 'none' machine.

We add a new sysbus device modeling a simple memory (ram or rom). We
also set 'user-creatable' flag of some sysbus devices. These are
trivial patches, but they depends on the conditions we choose to allow
creating devices with -preconfig. Therefore, there is really no need
to review them until we settled on the device conditions first.

With these devices (memory, ibex_uart, ibex_plic) we can dynamically
configure a part (we did not add the timer, but we could) the
opentitan machine very easily and run firmwares which demonstrates
interrupts and memory-mapping are working.

We use the existing qmp-shell script to issue machine devices
from a qmp commands script file which contains qmp commands listed in
a file.

The following qmp commands add some memories, an interrupt controller
and an uart with an interrupt.

cat > opentitan.qmp <<EOF
x-machine-init

# ROM 0x00008000
device_add        driver=sysbus-memory id=rom size=0x4000 readonly=true
x-sysbus-mmio-map device=rom addr=32768

# FLASH 0x20000000
device_add        driver=sysbus-memory id=flash size=0x80000 readonly=true
x-sysbus-mmio-map device=flash addr=536870912

# RAM 0x10000000
device_add        driver=sysbus-memory id=ram size=0x10000
x-sysbus-mmio-map device=ram addr=268435456

# PLIC 0x41010000
device_add        driver=ibex-plic id=plic
x-sysbus-mmio-map device=plic addr=1090584576

# UART 0x40000000
device_add        driver=ibex-uart id=uart chardev=serial0
x-sysbus-mmio-map device=uart addr=1073741824
qom-set path=uart property=sysbus-irq[1] value=plic/unnamed-gpio-in[2]

x-exit-preconfig
EOF

We've put the opentitan.qmp and a firmware opentitan-echo.elf here
(among some other qmp machine files we are working on):
https://github.com/GreenSocs/qemu-qmp-machines
This firmware is just a small interrupt-based program echoing back
whatever is sent in the uart.

QEMU should be run using the following command:
qemu-system-riscv32 -preconfig -qmp unix:/tmp/qmp-socket,server \
    -display none \
    -M none -cpu lowrisc-ibex \
    -serial mon:stdio \
    -device loader,addr=0x8090,cpu-num=0 \
    -device loader,file=opentitan-hello.elf \

and in other terminal to do the configuration (grep is just here to
remove comments):
grep -v -e '^#' opentitan.qmp | qmp-shell -v /tmp/qmp-socket

Alternatively we can load the firmware on the existing machine and
observe the same behavior:
qemu-system-riscv32 -display none \
     -M opentitan \
     -serial mon:stdio \
     -kernel opentitan-echo.elf

We chose this example because it is very simple and does not need a
lot of devices.

This work has still a lot of limitations. Cpus config is done the
normal way (the C machine does that): in our example we used the
'none' machine. We have work to do for handling backend
connection (for example net/nic are complicated) because the way it
is done in machine C code does not translate easily in QMP commands.
Firmware loading is also a bit tricky. We plan to work on this in
follow-up series.

The series is organized as follows:
- Patches 1 to 3 add qmp support to stop QEMU at an early phase
  to populate the machine with devices.
- Patches 4 to 6 prepare and allow issuing device_add during this phase.
- Patches 7 to 10 prepare and allow creating sysbus device during this phase.
- Patches 11 and 12 add the x-sysbus-mmio-map QMP command
- Patch 13 add the memory sysbus device to model ram and rom 
- Patch 14 adds some documentation
- Patches 15 and 16 set 'user_creatable' flag of ibex_uart and ibex_plic.

This work is supported by Greensocs, Sifive and Xilinx.

Thanks,
--
Damien

Damien Hedde (12):
  softmmu/qdev-monitor: add error handling in qdev_set_id
  qdev-monitor: prevent conflicts between qmp/device_add and cli/-device
  hw/core/machine: add machine_class_is_dynamic_sysbus_dev_allowed
  qdev-monitor: Check sysbus device type before creating it
  hw/core/machine: Remove the dynamic sysbus devices type check
  qdev-monitor: allow adding any sysbus device before machine is ready
  softmmu/memory: add memory_region_try_add_subregion function
  add x-sysbus-mmio-map qmp command
  hw/mem/system-memory: add a memory sysbus device
  docs/system: add doc about the initialized machine phase and an
    example
  hw/char/ibex_uart: set user_creatable
  hw/intc/ibex_plic: set user_creatable

Mirela Grujic (4):
  rename MachineInitPhase enum constants for QAPI compatibility
  qapi: Implement query-machine-phase QMP command
  qapi: Implement x-machine-init QMP command
  qapi: Allow device_add to execute in machine initialized phase

 docs/system/managed-startup.rst | 77 +++++++++++++++++++++++++++++
 qapi/machine.json               | 79 ++++++++++++++++++++++++++++++
 qapi/qdev.json                  | 24 ++++++++-
 include/exec/memory.h           | 22 +++++++++
 include/hw/boards.h             | 18 ++++++-
 include/hw/mem/sysbus-memory.h  | 32 ++++++++++++
 include/hw/qdev-core.h          | 30 +-----------
 include/monitor/qdev.h          | 25 +++++++++-
 hw/char/ibex_uart.c             |  1 +
 hw/core/machine-qmp-cmds.c      | 11 ++++-
 hw/core/machine.c               | 48 ++++++------------
 hw/core/qdev.c                  |  7 ++-
 hw/core/sysbus.c                | 41 ++++++++++++++++
 hw/intc/ibex_plic.c             |  1 +
 hw/mem/sysbus-memory.c          | 83 +++++++++++++++++++++++++++++++
 hw/pci/pci.c                    |  2 +-
 hw/usb/core.c                   |  2 +-
 hw/virtio/virtio-iommu.c        |  2 +-
 hw/xen/xen-legacy-backend.c     |  3 +-
 monitor/hmp.c                   |  2 +-
 monitor/misc.c                  |  2 +-
 softmmu/memory.c                | 22 ++++++---
 softmmu/qdev-monitor.c          | 86 +++++++++++++++++++++++++++------
 softmmu/vl.c                    | 23 ++++++---
 ui/console.c                    |  3 +-
 hw/mem/meson.build              |  2 +
 26 files changed, 547 insertions(+), 101 deletions(-)
 create mode 100644 include/hw/mem/sysbus-memory.h
 create mode 100644 hw/mem/sysbus-memory.c

-- 
2.33.0

On Thu, Sep 23, 2021 at 2:22 AM Damien Hedde <damien.hedde@greensocs.com> wrote:
>
> Hi,
>
> The goal of this work is to bring dynamic machine creation to QEMU:
> we want to setup a machine without compiling a specific machine C
> code. It would ease supporting highly configurable platforms (for
> example resulting from an automated design flow). The requirements
> for such configuration include begin able to specify the number of
> cores, available peripherals, emmory mapping, IRQ mapping, etc.
>
> This series focuses on the first step: populating a machine with
> devices during its creation. We propose patches to support this
> using QMP commands. This is a working set of patches and improves
> over the earlier rfc (posted in May):
> https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg03706.html
>
> Although it is working and could be merged, it is tag as an RFC:
> we probably need to discuss the conditions for allowing a device to
> be created at an early stage. Patches 6, 10 and 13, 15 and 16 depend
> on such conditions and are subject to change. Other patches are
> unrelated to this point.
>
> We address several issues in this series. They are detailed below.
>
> ## 1. Stoping QEMU to populate the machine with devices
>
> QEMU goes through several steps (called _machine phases_) when
> creating the machine: 'no-machine', 'machine-created',
> 'accel-created', 'initialized', and finally 'ready'. At 'ready'
> phase, QEMU is ready to start (see Paolo's page
> https://wiki.qemu.org/User:Paolo_Bonzini/Machine_init_sequence for
> more details).
>
> Using the -preconfig CLI option, QEMU can be stopped today during
> the 'accel-created' phase. Then the 'x-exit-preconfig' QMP command
> triggers QEMU moving forwards to the completion of the machine
> creation ('ready' phase).
>
> The devices are created during the 'initialized' phase.
> In this phase the machine init() method has been executed and thus
> machine properties have been handled. Although the sysbus exists and
> the machine may have been populated by the init(),
> _machine_init_done_ notifiers have not been called yet. At this point
> we can add more devices to a machine.
>
> We propose to add 2 QMP commands:
> + The 'query-machine-phase' command would return the current machine
>   phase.
> + The 'x-machine-init' command would advance the machine phase to
>   'initialized'. 'x-exit-preconfig' could then still be used to
>   advance to the last phase.
>
> ## 2. Adding devices
>
> Right now, the user can create devices in 2 ways: using '-device' CLI
> option or 'device_add' QMP command. Both are executed after the
> machine is ready: such devices are hot-plugged. We propose to allow
> 'device_add' QMP command to be used during the 'initialized' phase.
>
> In this series, we keep the constraint that the device must be
> 'user-creatable' (this is a device class flag). We do not see any
> reason why a device the user can hot-plug could not be created at an
> earlier stage.
>
> This part is still RFC because, as Peter mentioned it (in this thread
> https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg01933.html),
> we may want additional or distinct conditions for:
> + device we can hot-plug
> + device we can add in '-preconfig' (cold-plug)
> We are open to suggestions. We could for example add a
> 'preconfig-creatable' or 'init-creatable' flag to device class, which
> can identify a set of devices we can create this way.
>
> The main addition is how we handle the case of sysbus devices. Sysbus
> devices are particular because unlike, for example, pci devices, you
> have to manually handle the memory mapping and interrupts wiring. So
> right now, a sysbus device is dynamically creatable (using -device
> CLI option or device_add QMP command) only if:
> + it is 'user_creatable' (like any other device),
> + and it is in the current machine sysbus device allow list.
>
> In this series, we propose to relax the second constraint during the
> earlier phases of machine creation so that when using -preconfig we
> can create any 'user-creatable' sysbus device. When the machine
> progresses to the 'ready' phase, sysbus devices creation will come
> back to the legacy behavior: it will be possible only based on the
> per-machine authorization basis.
>
> For sysbus devices, wiring interrupts is not a problem as we can use
> the 'qom-set' QMP command, but memory mapping is.
>
> ## 3. Memory mapping
>
> There is no point allowing the creation sysbus devices if we cannot
> map them onto the memory bus (the 'sysbus').
>
> As far as we know, right now, there is no way to add memory mapping
> for sysbus device using QMP commands. We propose a 'x-sysbus-mmio-map'
> command to do this. This command would only be allowed during the
> 'initialized' phase when using -preconfig.
>
> ## 4. Working example
>
> The last patches of the series add and modify devices in order to
> build a working machine starting from the 'none' machine.
>
> We add a new sysbus device modeling a simple memory (ram or rom). We
> also set 'user-creatable' flag of some sysbus devices. These are
> trivial patches, but they depends on the conditions we choose to allow
> creating devices with -preconfig. Therefore, there is really no need
> to review them until we settled on the device conditions first.
>
> With these devices (memory, ibex_uart, ibex_plic) we can dynamically
> configure a part (we did not add the timer, but we could) the
> opentitan machine very easily and run firmwares which demonstrates
> interrupts and memory-mapping are working.
>
> We use the existing qmp-shell script to issue machine devices
> from a qmp commands script file which contains qmp commands listed in
> a file.
>
> The following qmp commands add some memories, an interrupt controller
> and an uart with an interrupt.
>
> cat > opentitan.qmp <<EOF
> x-machine-init
>
> # ROM 0x00008000
> device_add        driver=sysbus-memory id=rom size=0x4000 readonly=true
> x-sysbus-mmio-map device=rom addr=32768
>
> # FLASH 0x20000000
> device_add        driver=sysbus-memory id=flash size=0x80000 readonly=true
> x-sysbus-mmio-map device=flash addr=536870912
>
> # RAM 0x10000000
> device_add        driver=sysbus-memory id=ram size=0x10000
> x-sysbus-mmio-map device=ram addr=268435456
>
> # PLIC 0x41010000
> device_add        driver=ibex-plic id=plic
> x-sysbus-mmio-map device=plic addr=1090584576
>
> # UART 0x40000000
> device_add        driver=ibex-uart id=uart chardev=serial0
> x-sysbus-mmio-map device=uart addr=1073741824
> qom-set path=uart property=sysbus-irq[1] value=plic/unnamed-gpio-in[2]
>
> x-exit-preconfig
> EOF
>
> We've put the opentitan.qmp and a firmware opentitan-echo.elf here
> (among some other qmp machine files we are working on):
> https://github.com/GreenSocs/qemu-qmp-machines

I am unable to access this repo, maybe it's not public?

Alistair

Fixed
Cheers
Mark.


> On 13 Oct 2021, at 00:16, Alistair Francis <alistair23@gmail.com> wrote:
> 
> On Thu, Sep 23, 2021 at 2:22 AM Damien Hedde <damien.hedde@greensocs.com> wrote:
>> 
>> Hi,
>> 
>> The goal of this work is to bring dynamic machine creation to QEMU:
>> we want to setup a machine without compiling a specific machine C
>> code. It would ease supporting highly configurable platforms (for
>> example resulting from an automated design flow). The requirements
>> for such configuration include begin able to specify the number of
>> cores, available peripherals, emmory mapping, IRQ mapping, etc.
>> 
>> This series focuses on the first step: populating a machine with
>> devices during its creation. We propose patches to support this
>> using QMP commands. This is a working set of patches and improves
>> over the earlier rfc (posted in May):
>> https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg03706.html
>> 
>> Although it is working and could be merged, it is tag as an RFC:
>> we probably need to discuss the conditions for allowing a device to
>> be created at an early stage. Patches 6, 10 and 13, 15 and 16 depend
>> on such conditions and are subject to change. Other patches are
>> unrelated to this point.
>> 
>> We address several issues in this series. They are detailed below.
>> 
>> ## 1. Stoping QEMU to populate the machine with devices
>> 
>> QEMU goes through several steps (called _machine phases_) when
>> creating the machine: 'no-machine', 'machine-created',
>> 'accel-created', 'initialized', and finally 'ready'. At 'ready'
>> phase, QEMU is ready to start (see Paolo's page
>> https://wiki.qemu.org/User:Paolo_Bonzini/Machine_init_sequence for
>> more details).
>> 
>> Using the -preconfig CLI option, QEMU can be stopped today during
>> the 'accel-created' phase. Then the 'x-exit-preconfig' QMP command
>> triggers QEMU moving forwards to the completion of the machine
>> creation ('ready' phase).
>> 
>> The devices are created during the 'initialized' phase.
>> In this phase the machine init() method has been executed and thus
>> machine properties have been handled. Although the sysbus exists and
>> the machine may have been populated by the init(),
>> _machine_init_done_ notifiers have not been called yet. At this point
>> we can add more devices to a machine.
>> 
>> We propose to add 2 QMP commands:
>> + The 'query-machine-phase' command would return the current machine
>>  phase.
>> + The 'x-machine-init' command would advance the machine phase to
>>  'initialized'. 'x-exit-preconfig' could then still be used to
>>  advance to the last phase.
>> 
>> ## 2. Adding devices
>> 
>> Right now, the user can create devices in 2 ways: using '-device' CLI
>> option or 'device_add' QMP command. Both are executed after the
>> machine is ready: such devices are hot-plugged. We propose to allow
>> 'device_add' QMP command to be used during the 'initialized' phase.
>> 
>> In this series, we keep the constraint that the device must be
>> 'user-creatable' (this is a device class flag). We do not see any
>> reason why a device the user can hot-plug could not be created at an
>> earlier stage.
>> 
>> This part is still RFC because, as Peter mentioned it (in this thread
>> https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg01933.html),
>> we may want additional or distinct conditions for:
>> + device we can hot-plug
>> + device we can add in '-preconfig' (cold-plug)
>> We are open to suggestions. We could for example add a
>> 'preconfig-creatable' or 'init-creatable' flag to device class, which
>> can identify a set of devices we can create this way.
>> 
>> The main addition is how we handle the case of sysbus devices. Sysbus
>> devices are particular because unlike, for example, pci devices, you
>> have to manually handle the memory mapping and interrupts wiring. So
>> right now, a sysbus device is dynamically creatable (using -device
>> CLI option or device_add QMP command) only if:
>> + it is 'user_creatable' (like any other device),
>> + and it is in the current machine sysbus device allow list.
>> 
>> In this series, we propose to relax the second constraint during the
>> earlier phases of machine creation so that when using -preconfig we
>> can create any 'user-creatable' sysbus device. When the machine
>> progresses to the 'ready' phase, sysbus devices creation will come
>> back to the legacy behavior: it will be possible only based on the
>> per-machine authorization basis.
>> 
>> For sysbus devices, wiring interrupts is not a problem as we can use
>> the 'qom-set' QMP command, but memory mapping is.
>> 
>> ## 3. Memory mapping
>> 
>> There is no point allowing the creation sysbus devices if we cannot
>> map them onto the memory bus (the 'sysbus').
>> 
>> As far as we know, right now, there is no way to add memory mapping
>> for sysbus device using QMP commands. We propose a 'x-sysbus-mmio-map'
>> command to do this. This command would only be allowed during the
>> 'initialized' phase when using -preconfig.
>> 
>> ## 4. Working example
>> 
>> The last patches of the series add and modify devices in order to
>> build a working machine starting from the 'none' machine.
>> 
>> We add a new sysbus device modeling a simple memory (ram or rom). We
>> also set 'user-creatable' flag of some sysbus devices. These are
>> trivial patches, but they depends on the conditions we choose to allow
>> creating devices with -preconfig. Therefore, there is really no need
>> to review them until we settled on the device conditions first.
>> 
>> With these devices (memory, ibex_uart, ibex_plic) we can dynamically
>> configure a part (we did not add the timer, but we could) the
>> opentitan machine very easily and run firmwares which demonstrates
>> interrupts and memory-mapping are working.
>> 
>> We use the existing qmp-shell script to issue machine devices
>> from a qmp commands script file which contains qmp commands listed in
>> a file.
>> 
>> The following qmp commands add some memories, an interrupt controller
>> and an uart with an interrupt.
>> 
>> cat > opentitan.qmp <<EOF
>> x-machine-init
>> 
>> # ROM 0x00008000
>> device_add        driver=sysbus-memory id=rom size=0x4000 readonly=true
>> x-sysbus-mmio-map device=rom addr=32768
>> 
>> # FLASH 0x20000000
>> device_add        driver=sysbus-memory id=flash size=0x80000 readonly=true
>> x-sysbus-mmio-map device=flash addr=536870912
>> 
>> # RAM 0x10000000
>> device_add        driver=sysbus-memory id=ram size=0x10000
>> x-sysbus-mmio-map device=ram addr=268435456
>> 
>> # PLIC 0x41010000
>> device_add        driver=ibex-plic id=plic
>> x-sysbus-mmio-map device=plic addr=1090584576
>> 
>> # UART 0x40000000
>> device_add        driver=ibex-uart id=uart chardev=serial0
>> x-sysbus-mmio-map device=uart addr=1073741824
>> qom-set path=uart property=sysbus-irq[1] value=plic/unnamed-gpio-in[2]
>> 
>> x-exit-preconfig
>> EOF
>> 
>> We've put the opentitan.qmp and a firmware opentitan-echo.elf here
>> (among some other qmp machine files we are working on):
>> https://github.com/GreenSocs/qemu-qmp-machines
> 
> I am unable to access this repo, maybe it's not public?
> 
> Alistair

Hi Damien,

On 9/22/21 18:13, Damien Hedde wrote:
> 
> The goal of this work is to bring dynamic machine creation to QEMU:
> we want to setup a machine without compiling a specific machine C
> code. It would ease supporting highly configurable platforms (for
> example resulting from an automated design flow). The requirements
> for such configuration include begin able to specify the number of
> cores, available peripherals, emmory mapping, IRQ mapping, etc.
> 
> This series focuses on the first step: populating a machine with
> devices during its creation. We propose patches to support this
> using QMP commands. This is a working set of patches and improves
> over the earlier rfc (posted in May):
> https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg03706.html

Do you have a roadmap for the following steps? Or are you done with
this series?

Yesterday I was thinking about this, and one thing I was wondering is
if it would be possible to have DeviceClass and MachineClass implement
a populate_fdt() handler, to automatically generate custom DTB for these
custom machines.

Maybe in your case you don't need that, as your framework generating the
QEMU machine also generates the DTB, or even parse a DTB to generate the
machine... :)

Regards,

Phil.

On 9/22/21 19:50, Philippe Mathieu-Daudé wrote:
> Hi Damien,
> 
> On 9/22/21 18:13, Damien Hedde wrote:
>>
>> The goal of this work is to bring dynamic machine creation to QEMU:
>> we want to setup a machine without compiling a specific machine C
>> code. It would ease supporting highly configurable platforms (for
>> example resulting from an automated design flow). The requirements
>> for such configuration include begin able to specify the number of
>> cores, available peripherals, emmory mapping, IRQ mapping, etc.
>>
>> This series focuses on the first step: populating a machine with
>> devices during its creation. We propose patches to support this
>> using QMP commands. This is a working set of patches and improves
>> over the earlier rfc (posted in May):
>> https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg03706.html
> 
> Do you have a roadmap for the following steps? Or are you done with
> this series?

Hi Philippe,

We would like to stick to this scope ("creating devices in a machine") 
for this particular series otherwise it will become very big and cover a 
huge scope.

We have some patches to "migrate" more devices to be early 
user-creatable (like the last 2 patches of this series) so that we can 
use more device when building a machine. But these are trivial and only 
depends on what is the condition to allow this. We plan to submit these 
when this series is done.

We plan to address other issues we have in others series of patches. We 
do not put a roadmap somewhere. But we can details this in a page in our 
github or in the qemu wiki if you think this is a good idea.
Here are the main remaining issues:
+ the base machine (we are using "none" here because it is "almost" 
empty and fit our needs but it has some limitations)
+ adding cpus
+ non-trivial memory mappings
+ solving backend (eg: net) connection issues

> 
> Yesterday I was thinking about this, and one thing I was wondering is
> if it would be possible to have DeviceClass and MachineClass implement
> a populate_fdt() handler, to automatically generate custom DTB for these
> custom machines.
> 
> Maybe in your case you don't need that, as your framework generating the
> QEMU machine also generates the DTB, or even parse a DTB to generate the
> machine... :)

You are right, we do not need this. We indeed use a device tree file to 
describe the platform but this is an "extended" device tree (it has more 
info than needed for linux). If it was not the case, I think it would 
still be easier to generate it from the source platform description than 
using qemu as an intermediate.

It is probably possible but it is tricky to generate the dtb: mapping in 
dtb are not standardized and really depends on the node types.

For example, to generate the interrupt-map property of a device node. 
You need to know the interrupt mapping (which interrupt line goes in 
which interrupt controller) and also have info about the interrupt 
controller's cells format (eg: how many bytes do we need to specify the 
interrupt). For example for some controller, you have specify the 
interrupt kind (rising or falling edge, high or low active level).

So you'll probably need some special "get_dtb_interrupt_cell" method in 
interrupt controllers to generate these entries for you so that a device 
can ask dtb data to its controller.

Bus mappings also depend on the bus type, but since qemu devices are 
already organized on a bus-type basis, this is probably easier to solve.

You'll have similar issues with every mapping. But bus and interrupt are 
the most important ones.

Thanks,
Damien

Hi,

This is both a ping and a small update. It would be great to have some 
feedback about patches 1 and 3.

Right now the device part of this series conflicts with Kevin 's work 
about replacing the QemuOpts by a QemuDict in device_add:
https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg06136.html

So I'll have at least to rebase on top of his series, remove and rework 
some of the patches.

Maybe this series is too big and we should split it anyway ? We could 
for example target 3 smaller series:
  1. -> about stopping during the machine 'initialized' phase
  2. -> about enabling using device_add QMP commands during this phase
  3. -> about the sysbus device case (some of the patches are even 
independent)

Thanks,
Damien

On 9/22/21 18:13, Damien Hedde wrote:
> Hi,
> 
> The goal of this work is to bring dynamic machine creation to QEMU:
> we want to setup a machine without compiling a specific machine C
> code. It would ease supporting highly configurable platforms (for
> example resulting from an automated design flow). The requirements
> for such configuration include begin able to specify the number of
> cores, available peripherals, emmory mapping, IRQ mapping, etc.
> 
> This series focuses on the first step: populating a machine with
> devices during its creation. We propose patches to support this
> using QMP commands. This is a working set of patches and improves
> over the earlier rfc (posted in May):
> https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg03706.html
> 
> Although it is working and could be merged, it is tag as an RFC:
> we probably need to discuss the conditions for allowing a device to
> be created at an early stage. Patches 6, 10 and 13, 15 and 16 depend
> on such conditions and are subject to change. Other patches are
> unrelated to this point.
> 
> We address several issues in this series. They are detailed below.
> 
> ## 1. Stoping QEMU to populate the machine with devices
> 
> QEMU goes through several steps (called _machine phases_) when
> creating the machine: 'no-machine', 'machine-created',
> 'accel-created', 'initialized', and finally 'ready'. At 'ready'
> phase, QEMU is ready to start (see Paolo's page
> https://wiki.qemu.org/User:Paolo_Bonzini/Machine_init_sequence for
> more details).
> 
> Using the -preconfig CLI option, QEMU can be stopped today during
> the 'accel-created' phase. Then the 'x-exit-preconfig' QMP command
> triggers QEMU moving forwards to the completion of the machine
> creation ('ready' phase).
> 
> The devices are created during the 'initialized' phase.
> In this phase the machine init() method has been executed and thus
> machine properties have been handled. Although the sysbus exists and
> the machine may have been populated by the init(),
> _machine_init_done_ notifiers have not been called yet. At this point
> we can add more devices to a machine.
> 
> We propose to add 2 QMP commands:
> + The 'query-machine-phase' command would return the current machine
>    phase.
> + The 'x-machine-init' command would advance the machine phase to
>    'initialized'. 'x-exit-preconfig' could then still be used to
>    advance to the last phase.
> 
> ## 2. Adding devices
> 
> Right now, the user can create devices in 2 ways: using '-device' CLI
> option or 'device_add' QMP command. Both are executed after the
> machine is ready: such devices are hot-plugged. We propose to allow
> 'device_add' QMP command to be used during the 'initialized' phase.
> 
> In this series, we keep the constraint that the device must be
> 'user-creatable' (this is a device class flag). We do not see any
> reason why a device the user can hot-plug could not be created at an
> earlier stage.
> 
> This part is still RFC because, as Peter mentioned it (in this thread
> https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg01933.html),
> we may want additional or distinct conditions for:
> + device we can hot-plug
> + device we can add in '-preconfig' (cold-plug)
> We are open to suggestions. We could for example add a
> 'preconfig-creatable' or 'init-creatable' flag to device class, which
> can identify a set of devices we can create this way.
> 
> The main addition is how we handle the case of sysbus devices. Sysbus
> devices are particular because unlike, for example, pci devices, you
> have to manually handle the memory mapping and interrupts wiring. So
> right now, a sysbus device is dynamically creatable (using -device
> CLI option or device_add QMP command) only if:
> + it is 'user_creatable' (like any other device),
> + and it is in the current machine sysbus device allow list.
> 
> In this series, we propose to relax the second constraint during the
> earlier phases of machine creation so that when using -preconfig we
> can create any 'user-creatable' sysbus device. When the machine
> progresses to the 'ready' phase, sysbus devices creation will come
> back to the legacy behavior: it will be possible only based on the
> per-machine authorization basis.
> 
> For sysbus devices, wiring interrupts is not a problem as we can use
> the 'qom-set' QMP command, but memory mapping is.
> 
> ## 3. Memory mapping
> 
> There is no point allowing the creation sysbus devices if we cannot
> map them onto the memory bus (the 'sysbus').
> 
> As far as we know, right now, there is no way to add memory mapping
> for sysbus device using QMP commands. We propose a 'x-sysbus-mmio-map'
> command to do this. This command would only be allowed during the
> 'initialized' phase when using -preconfig.
> 
> ## 4. Working example
> 
> The last patches of the series add and modify devices in order to
> build a working machine starting from the 'none' machine.
> 
> We add a new sysbus device modeling a simple memory (ram or rom). We
> also set 'user-creatable' flag of some sysbus devices. These are
> trivial patches, but they depends on the conditions we choose to allow
> creating devices with -preconfig. Therefore, there is really no need
> to review them until we settled on the device conditions first.
> 
> With these devices (memory, ibex_uart, ibex_plic) we can dynamically
> configure a part (we did not add the timer, but we could) the
> opentitan machine very easily and run firmwares which demonstrates
> interrupts and memory-mapping are working.
> 
> We use the existing qmp-shell script to issue machine devices
> from a qmp commands script file which contains qmp commands listed in
> a file.
> 
> The following qmp commands add some memories, an interrupt controller
> and an uart with an interrupt.
> 
> cat > opentitan.qmp <<EOF
> x-machine-init
> 
> # ROM 0x00008000
> device_add        driver=sysbus-memory id=rom size=0x4000 readonly=true
> x-sysbus-mmio-map device=rom addr=32768
> 
> # FLASH 0x20000000
> device_add        driver=sysbus-memory id=flash size=0x80000 readonly=true
> x-sysbus-mmio-map device=flash addr=536870912
> 
> # RAM 0x10000000
> device_add        driver=sysbus-memory id=ram size=0x10000
> x-sysbus-mmio-map device=ram addr=268435456
> 
> # PLIC 0x41010000
> device_add        driver=ibex-plic id=plic
> x-sysbus-mmio-map device=plic addr=1090584576
> 
> # UART 0x40000000
> device_add        driver=ibex-uart id=uart chardev=serial0
> x-sysbus-mmio-map device=uart addr=1073741824
> qom-set path=uart property=sysbus-irq[1] value=plic/unnamed-gpio-in[2]
> 
> x-exit-preconfig
> EOF
> 
> We've put the opentitan.qmp and a firmware opentitan-echo.elf here
> (among some other qmp machine files we are working on):
> https://github.com/GreenSocs/qemu-qmp-machines
> This firmware is just a small interrupt-based program echoing back
> whatever is sent in the uart.
> 
> QEMU should be run using the following command:
> qemu-system-riscv32 -preconfig -qmp unix:/tmp/qmp-socket,server \
>      -display none \
>      -M none -cpu lowrisc-ibex \
>      -serial mon:stdio \
>      -device loader,addr=0x8090,cpu-num=0 \
>      -device loader,file=opentitan-hello.elf \
> 
> and in other terminal to do the configuration (grep is just here to
> remove comments):
> grep -v -e '^#' opentitan.qmp | qmp-shell -v /tmp/qmp-socket
> 
> Alternatively we can load the firmware on the existing machine and
> observe the same behavior:
> qemu-system-riscv32 -display none \
>       -M opentitan \
>       -serial mon:stdio \
>       -kernel opentitan-echo.elf
> 
> We chose this example because it is very simple and does not need a
> lot of devices.
> 
> This work has still a lot of limitations. Cpus config is done the
> normal way (the C machine does that): in our example we used the
> 'none' machine. We have work to do for handling backend
> connection (for example net/nic are complicated) because the way it
> is done in machine C code does not translate easily in QMP commands.
> Firmware loading is also a bit tricky. We plan to work on this in
> follow-up series.
> 
> The series is organized as follows:
> - Patches 1 to 3 add qmp support to stop QEMU at an early phase
>    to populate the machine with devices.
> - Patches 4 to 6 prepare and allow issuing device_add during this phase.
> - Patches 7 to 10 prepare and allow creating sysbus device during this phase.
> - Patches 11 and 12 add the x-sysbus-mmio-map QMP command
> - Patch 13 add the memory sysbus device to model ram and rom
> - Patch 14 adds some documentation
> - Patches 15 and 16 set 'user_creatable' flag of ibex_uart and ibex_plic.
> 
> This work is supported by Greensocs, Sifive and Xilinx.
> 
> Thanks,
> --
> Damien
> 
> Damien Hedde (12):
>    softmmu/qdev-monitor: add error handling in qdev_set_id
>    qdev-monitor: prevent conflicts between qmp/device_add and cli/-device
>    hw/core/machine: add machine_class_is_dynamic_sysbus_dev_allowed
>    qdev-monitor: Check sysbus device type before creating it
>    hw/core/machine: Remove the dynamic sysbus devices type check
>    qdev-monitor: allow adding any sysbus device before machine is ready
>    softmmu/memory: add memory_region_try_add_subregion function
>    add x-sysbus-mmio-map qmp command
>    hw/mem/system-memory: add a memory sysbus device
>    docs/system: add doc about the initialized machine phase and an
>      example
>    hw/char/ibex_uart: set user_creatable
>    hw/intc/ibex_plic: set user_creatable
> 
> Mirela Grujic (4):
>    rename MachineInitPhase enum constants for QAPI compatibility
>    qapi: Implement query-machine-phase QMP command
>    qapi: Implement x-machine-init QMP command
>    qapi: Allow device_add to execute in machine initialized phase
> 
>   docs/system/managed-startup.rst | 77 +++++++++++++++++++++++++++++
>   qapi/machine.json               | 79 ++++++++++++++++++++++++++++++
>   qapi/qdev.json                  | 24 ++++++++-
>   include/exec/memory.h           | 22 +++++++++
>   include/hw/boards.h             | 18 ++++++-
>   include/hw/mem/sysbus-memory.h  | 32 ++++++++++++
>   include/hw/qdev-core.h          | 30 +-----------
>   include/monitor/qdev.h          | 25 +++++++++-
>   hw/char/ibex_uart.c             |  1 +
>   hw/core/machine-qmp-cmds.c      | 11 ++++-
>   hw/core/machine.c               | 48 ++++++------------
>   hw/core/qdev.c                  |  7 ++-
>   hw/core/sysbus.c                | 41 ++++++++++++++++
>   hw/intc/ibex_plic.c             |  1 +
>   hw/mem/sysbus-memory.c          | 83 +++++++++++++++++++++++++++++++
>   hw/pci/pci.c                    |  2 +-
>   hw/usb/core.c                   |  2 +-
>   hw/virtio/virtio-iommu.c        |  2 +-
>   hw/xen/xen-legacy-backend.c     |  3 +-
>   monitor/hmp.c                   |  2 +-
>   monitor/misc.c                  |  2 +-
>   softmmu/memory.c                | 22 ++++++---
>   softmmu/qdev-monitor.c          | 86 +++++++++++++++++++++++++++------
>   softmmu/vl.c                    | 23 ++++++---
>   ui/console.c                    |  3 +-
>   hw/mem/meson.build              |  2 +
>   26 files changed, 547 insertions(+), 101 deletions(-)
>   create mode 100644 include/hw/mem/sysbus-memory.h
>   create mode 100644 hw/mem/sysbus-memory.c
>