X86 IOMMUs cannot be created more than one on a system yet.  Make it a
singleton so it guards the system from accidentally create yet another
IOMMU object when one already presents.

Now if someone tries to create more than one, e.g., via:

  ./qemu -M q35 -device intel-iommu -device intel-iommu

The error will change from:

  qemu-system-x86_64: -device intel-iommu: QEMU does not support multiple vIOMMUs for x86 yet.

To:

  qemu-system-x86_64: -device intel-iommu: Class 'intel-iommu' only supports one instance

Unfortunately, yet we can't remove the singleton check in the machine
hook (pc_machine_device_pre_plug_cb), because there can also be
virtio-iommu involved, which doesn't share a common parent class yet.

But with this, it should be closer to reach that goal to check singleton by
QOM one day.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 hw/i386/x86-iommu.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/hw/i386/x86-iommu.c b/hw/i386/x86-iommu.c
index 60af896225..4bfeb08705 100644
--- a/hw/i386/x86-iommu.c
+++ b/hw/i386/x86-iommu.c
@@ -26,6 +26,7 @@
 #include "qemu/error-report.h"
 #include "trace.h"
 #include "sysemu/kvm.h"
+#include "qom/object_interfaces.h"
 
 void x86_iommu_iec_register_notifier(X86IOMMUState *iommu,
                                      iec_notify_fn fn, void *data)
@@ -133,10 +134,19 @@ static Property x86_iommu_properties[] = {
     DEFINE_PROP_END_OF_LIST(),
 };
 
+static Object *x86_iommu_get_instance(Error **errp)
+{
+    return OBJECT(x86_iommu_get_default());
+}
+
 static void x86_iommu_class_init(ObjectClass *klass, void *data)
 {
     DeviceClass *dc = DEVICE_CLASS(klass);
+    SingletonClass *singleton = SINGLETON_CLASS(klass);
+
     dc->realize = x86_iommu_realize;
+    singleton->get_instance = x86_iommu_get_instance;
+
     device_class_set_props(dc, x86_iommu_properties);
 }
 
@@ -152,6 +162,10 @@ static const TypeInfo x86_iommu_info = {
     .class_init    = x86_iommu_class_init,
     .class_size    = sizeof(X86IOMMUClass),
     .abstract      = true,
+    .interfaces = (InterfaceInfo[]) {
+        { TYPE_SINGLETON },
+        { }
+    }
 };
 
 static void x86_iommu_register_types(void)
-- 
2.45.0

On Thu, Oct 24, 2024 at 12:56:25PM -0400, Peter Xu wrote:
> X86 IOMMUs cannot be created more than one on a system yet.  Make it a
> singleton so it guards the system from accidentally create yet another
> IOMMU object when one already presents.
> 
> Now if someone tries to create more than one, e.g., via:
> 
>   ./qemu -M q35 -device intel-iommu -device intel-iommu
> 
> The error will change from:
> 
>   qemu-system-x86_64: -device intel-iommu: QEMU does not support multiple vIOMMUs for x86 yet.
> 
> To:
> 
>   qemu-system-x86_64: -device intel-iommu: Class 'intel-iommu' only supports one instance
> 
> Unfortunately, yet we can't remove the singleton check in the machine
> hook (pc_machine_device_pre_plug_cb), because there can also be
> virtio-iommu involved, which doesn't share a common parent class yet.

Presumably the 'class' reported is the one that the user requested,
but this would imply if we were to do

  qemu-system-x86_64 -device intel-iommu -device virtio-iommu

Then QEMU would report

   "Class 'virtio-iommu' only supports one instance"

at which point the user is wondering, huh, I only requested one virtio-iommu
instance ?

IOW, the current error message would be better as it is not referring to a
specific subclass, but rather to the more general fact that only a single
IOMMU is permitted, no matter what it's impl is.

> 
> But with this, it should be closer to reach that goal to check singleton by
> QOM one day.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  hw/i386/x86-iommu.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/hw/i386/x86-iommu.c b/hw/i386/x86-iommu.c
> index 60af896225..4bfeb08705 100644
> --- a/hw/i386/x86-iommu.c
> +++ b/hw/i386/x86-iommu.c
> @@ -26,6 +26,7 @@
>  #include "qemu/error-report.h"
>  #include "trace.h"
>  #include "sysemu/kvm.h"
> +#include "qom/object_interfaces.h"
>  
>  void x86_iommu_iec_register_notifier(X86IOMMUState *iommu,
>                                       iec_notify_fn fn, void *data)
> @@ -133,10 +134,19 @@ static Property x86_iommu_properties[] = {
>      DEFINE_PROP_END_OF_LIST(),
>  };
>  
> +static Object *x86_iommu_get_instance(Error **errp)
> +{
> +    return OBJECT(x86_iommu_get_default());
> +}
> +
>  static void x86_iommu_class_init(ObjectClass *klass, void *data)
>  {
>      DeviceClass *dc = DEVICE_CLASS(klass);
> +    SingletonClass *singleton = SINGLETON_CLASS(klass);
> +
>      dc->realize = x86_iommu_realize;
> +    singleton->get_instance = x86_iommu_get_instance;
> +
>      device_class_set_props(dc, x86_iommu_properties);
>  }
>  
> @@ -152,6 +162,10 @@ static const TypeInfo x86_iommu_info = {
>      .class_init    = x86_iommu_class_init,
>      .class_size    = sizeof(X86IOMMUClass),
>      .abstract      = true,
> +    .interfaces = (InterfaceInfo[]) {
> +        { TYPE_SINGLETON },
> +        { }
> +    }
>  };
>  
>  static void x86_iommu_register_types(void)
> -- 
> 2.45.0
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Tue, Oct 29, 2024 at 10:47:06AM +0000, Daniel P. Berrangé wrote:
> On Thu, Oct 24, 2024 at 12:56:25PM -0400, Peter Xu wrote:
> > X86 IOMMUs cannot be created more than one on a system yet.  Make it a
> > singleton so it guards the system from accidentally create yet another
> > IOMMU object when one already presents.
> > 
> > Now if someone tries to create more than one, e.g., via:
> > 
> >   ./qemu -M q35 -device intel-iommu -device intel-iommu
> > 
> > The error will change from:
> > 
> >   qemu-system-x86_64: -device intel-iommu: QEMU does not support multiple vIOMMUs for x86 yet.
> > 
> > To:
> > 
> >   qemu-system-x86_64: -device intel-iommu: Class 'intel-iommu' only supports one instance
> > 
> > Unfortunately, yet we can't remove the singleton check in the machine
> > hook (pc_machine_device_pre_plug_cb), because there can also be
> > virtio-iommu involved, which doesn't share a common parent class yet.
> 
> Presumably the 'class' reported is the one that the user requested,
> but this would imply if we were to do
> 
>   qemu-system-x86_64 -device intel-iommu -device virtio-iommu
> 
> Then QEMU would report
> 
>    "Class 'virtio-iommu' only supports one instance"
> 
> at which point the user is wondering, huh, I only requested one virtio-iommu
> instance ?
> 
> IOW, the current error message would be better as it is not referring to a
> specific subclass, but rather to the more general fact that only a single
> IOMMU is permitted, no matter what it's impl is.

True.. though IIUC this is more or less a cosmetic change only.  E.g., if
we want (assuming after we could have object_new_allowed(Error **errp),
checking both abstract + singleton classes) we could make the error points
to the base class (rather than the top class to be initiated) that declared
TYPE_SINGLETON when it failed due to the singleton check.

One step further, we can even provide a custom Error for any singleton
class to say whatever it wants if it hits a duplicate.

So to me it's a separate issue from whether we would like to have a generic
way to define a singleton class.  I am still ok if we want to avoid
introducing the singleton, but just to mention I believe it can report
something similar as before if we want.

Thanks,

-- 
Peter Xu

Peter Xu <peterx@redhat.com> writes:

> X86 IOMMUs cannot be created more than one on a system yet.  Make it a
> singleton so it guards the system from accidentally create yet another
> IOMMU object when one already presents.
>
> Now if someone tries to create more than one, e.g., via:
>
>   ./qemu -M q35 -device intel-iommu -device intel-iommu
>
> The error will change from:
>
>   qemu-system-x86_64: -device intel-iommu: QEMU does not support multiple vIOMMUs for x86 yet.
>
> To:
>
>   qemu-system-x86_64: -device intel-iommu: Class 'intel-iommu' only supports one instance
>
> Unfortunately, yet we can't remove the singleton check in the machine
> hook (pc_machine_device_pre_plug_cb), because there can also be
> virtio-iommu involved, which doesn't share a common parent class yet.
>
> But with this, it should be closer to reach that goal to check singleton by
> QOM one day.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>

$ qemu-system-x86_64 -device amd-iommu,help
/work/armbru/qemu/include/hw/boards.h:24:MACHINE: Object 0x56473906f960 is not an instance of type machine
Aborted (core dumped)

(gdb) bt
#0  0x00007ffff4e43834 in __pthread_kill_implementation () at /lib64/libc.so.6
#1  0x00007ffff4df18ee in raise () at /lib64/libc.so.6
#2  0x00007ffff4dd98ff in abort () at /lib64/libc.so.6
#3  0x0000555555f75ef3 in object_dynamic_cast_assert
    (obj=0x555557e03960, typename=0x5555563c403e "machine", file=0x5555563c4018 "/work/armbru/qemu/include/hw/boards.h", line=24, func=0x5555563c4290 <__func__.7> "MACHINE") at ../qom/object.c:936
#4  0x0000555555d5db0f in MACHINE (obj=0x555557e03960)
    at /work/armbru/qemu/include/hw/boards.h:24
#5  0x0000555555d5e030 in x86_iommu_get_default () at ../hw/i386/x86-iommu.c:83
#6  0x0000555555d5e262 in x86_iommu_get_instance
    (errp=0x5555573d4918 <error_abort>) at ../hw/i386/x86-iommu.c:139
#7  0x0000555555f7c27c in singleton_get_instance (class=0x555557e00320)
    at ../qom/object_interfaces.c:371
#8  0x000055555612a842 in qmp_device_list_properties
    (typename=0x555557e001d0 "amd-iommu", errp=0x7fffffffda38)
    at ../qom/qom-qmp-cmds.c:147
#9  0x0000555555bf20b2 in qdev_device_help (opts=0x555557e001f0)
    at ../system/qdev-monitor.c:314
#10 0x0000555555bfe06d in device_help_func
    (opaque=0x0, opts=0x555557e001f0, errp=0x0) at ../system/vl.c:1208
#11 0x0000555556217186 in qemu_opts_foreach
    (list=0x55555729e5c0 <qemu_device_opts>, func=0x555555bfe04d <device_help_func>, opaque=0x0, errp=0x0) at ../util/qemu-option.c:1135
#12 0x0000555555c01d56 in qemu_process_help_options () at ../system/vl.c:2555
#13 0x0000555555c04d81 in qemu_init (argc=3, argv=0x7fffffffde28)
    at ../system/vl.c:3654
#14 0x000055555612ffae in main (argc=3, argv=0x7fffffffde28)
    at ../system/main.c:47

On Fri, Oct 25, 2024 at 11:25:23AM +0200, Markus Armbruster wrote:
> Peter Xu <peterx@redhat.com> writes:
> 
> > X86 IOMMUs cannot be created more than one on a system yet.  Make it a
> > singleton so it guards the system from accidentally create yet another
> > IOMMU object when one already presents.
> >
> > Now if someone tries to create more than one, e.g., via:
> >
> >   ./qemu -M q35 -device intel-iommu -device intel-iommu
> >
> > The error will change from:
> >
> >   qemu-system-x86_64: -device intel-iommu: QEMU does not support multiple vIOMMUs for x86 yet.
> >
> > To:
> >
> >   qemu-system-x86_64: -device intel-iommu: Class 'intel-iommu' only supports one instance
> >
> > Unfortunately, yet we can't remove the singleton check in the machine
> > hook (pc_machine_device_pre_plug_cb), because there can also be
> > virtio-iommu involved, which doesn't share a common parent class yet.
> >
> > But with this, it should be closer to reach that goal to check singleton by
> > QOM one day.
> >
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> 
> $ qemu-system-x86_64 -device amd-iommu,help
> /work/armbru/qemu/include/hw/boards.h:24:MACHINE: Object 0x56473906f960 is not an instance of type machine
> Aborted (core dumped)
> 
> (gdb) bt
> #0  0x00007ffff4e43834 in __pthread_kill_implementation () at /lib64/libc.so.6
> #1  0x00007ffff4df18ee in raise () at /lib64/libc.so.6
> #2  0x00007ffff4dd98ff in abort () at /lib64/libc.so.6
> #3  0x0000555555f75ef3 in object_dynamic_cast_assert
>     (obj=0x555557e03960, typename=0x5555563c403e "machine", file=0x5555563c4018 "/work/armbru/qemu/include/hw/boards.h", line=24, func=0x5555563c4290 <__func__.7> "MACHINE") at ../qom/object.c:936
> #4  0x0000555555d5db0f in MACHINE (obj=0x555557e03960)
>     at /work/armbru/qemu/include/hw/boards.h:24
> #5  0x0000555555d5e030 in x86_iommu_get_default () at ../hw/i386/x86-iommu.c:83
> #6  0x0000555555d5e262 in x86_iommu_get_instance
>     (errp=0x5555573d4918 <error_abort>) at ../hw/i386/x86-iommu.c:139
> #7  0x0000555555f7c27c in singleton_get_instance (class=0x555557e00320)
>     at ../qom/object_interfaces.c:371
> #8  0x000055555612a842 in qmp_device_list_properties
>     (typename=0x555557e001d0 "amd-iommu", errp=0x7fffffffda38)
>     at ../qom/qom-qmp-cmds.c:147
> #9  0x0000555555bf20b2 in qdev_device_help (opts=0x555557e001f0)
>     at ../system/qdev-monitor.c:314
> #10 0x0000555555bfe06d in device_help_func
>     (opaque=0x0, opts=0x555557e001f0, errp=0x0) at ../system/vl.c:1208
> #11 0x0000555556217186 in qemu_opts_foreach
>     (list=0x55555729e5c0 <qemu_device_opts>, func=0x555555bfe04d <device_help_func>, opaque=0x0, errp=0x0) at ../util/qemu-option.c:1135
> #12 0x0000555555c01d56 in qemu_process_help_options () at ../system/vl.c:2555
> #13 0x0000555555c04d81 in qemu_init (argc=3, argv=0x7fffffffde28)
>     at ../system/vl.c:3654
> #14 0x000055555612ffae in main (argc=3, argv=0x7fffffffde28)
>     at ../system/main.c:47
> 

Thanks for the report!

It turns out that qdev_get_machine() cannot be invoked too early, and the
singleton code can make it earlier..

We may want a pre-requisite patch to allow qdev_get_machine() to be invoked
anytime, like:

===8<===
diff --git a/hw/core/qdev.c b/hw/core/qdev.c
index db36f54d91..7ceae47139 100644
--- a/hw/core/qdev.c
+++ b/hw/core/qdev.c
@@ -831,6 +831,16 @@ Object *qdev_get_machine(void)
 {
     static Object *dev;
 
+    if (!phase_check(PHASE_MACHINE_CREATED)) {
+        /*
+         * When the machine is not created, below can wrongly create
+         * /machine to be a container.. this enables qdev_get_machine() to
+         * be used at any time and return NULL properly when machine is not
+         * created.
+         */
+        return NULL;
+    }
+
     if (dev == NULL) {
         dev = container_get(object_get_root(), "/machine");
     }
===8<===

I hope it makes sense on its own. Then callers who can be invoked earlier
could then handle NULL properly, in this case..

===8<===
diff --git a/hw/i386/x86-iommu.c b/hw/i386/x86-iommu.c
index 4bfeb08705..fceb7adfe0 100644
--- a/hw/i386/x86-iommu.c
+++ b/hw/i386/x86-iommu.c
@@ -80,9 +80,15 @@ void x86_iommu_irq_to_msi_message(X86IOMMUIrq *irq, MSIMessage *msg_out)
 
 X86IOMMUState *x86_iommu_get_default(void)
 {
-    MachineState *ms = MACHINE(qdev_get_machine());
-    PCMachineState *pcms =
-        PC_MACHINE(object_dynamic_cast(OBJECT(ms), TYPE_PC_MACHINE));
+    Object *machine = qdev_get_machine();
+    PCMachineState *pcms;
+
+    /* If machine has not been created, so is the vIOMMU */
+    if (!machine) {
+        return NULL;
+    }
+
+    pcms = PC_MACHINE(object_dynamic_cast(machine, TYPE_PC_MACHINE));
 
     if (pcms &&
         object_dynamic_cast(OBJECT(pcms->iommu), TYPE_X86_IOMMU_DEVICE)) {
===8<===

I'll make sure this works if I'll repost.

Thanks,

-- 
Peter Xu

On Fri, Oct 25, 2024 at 05:55:59PM -0400, Peter Xu wrote:
> On Fri, Oct 25, 2024 at 11:25:23AM +0200, Markus Armbruster wrote:
> > Peter Xu <peterx@redhat.com> writes:
> > 
> > > X86 IOMMUs cannot be created more than one on a system yet.  Make it a
> > > singleton so it guards the system from accidentally create yet another
> > > IOMMU object when one already presents.
> > >
> > > Now if someone tries to create more than one, e.g., via:
> > >
> > >   ./qemu -M q35 -device intel-iommu -device intel-iommu
> > >
> > > The error will change from:
> > >
> > >   qemu-system-x86_64: -device intel-iommu: QEMU does not support multiple vIOMMUs for x86 yet.
> > >
> > > To:
> > >
> > >   qemu-system-x86_64: -device intel-iommu: Class 'intel-iommu' only supports one instance
> > >
> > > Unfortunately, yet we can't remove the singleton check in the machine
> > > hook (pc_machine_device_pre_plug_cb), because there can also be
> > > virtio-iommu involved, which doesn't share a common parent class yet.
> > >
> > > But with this, it should be closer to reach that goal to check singleton by
> > > QOM one day.
> > >
> > > Signed-off-by: Peter Xu <peterx@redhat.com>
> > 
> > $ qemu-system-x86_64 -device amd-iommu,help
> > /work/armbru/qemu/include/hw/boards.h:24:MACHINE: Object 0x56473906f960 is not an instance of type machine
> > Aborted (core dumped)
> > 
> > (gdb) bt
> > #0  0x00007ffff4e43834 in __pthread_kill_implementation () at /lib64/libc.so.6
> > #1  0x00007ffff4df18ee in raise () at /lib64/libc.so.6
> > #2  0x00007ffff4dd98ff in abort () at /lib64/libc.so.6
> > #3  0x0000555555f75ef3 in object_dynamic_cast_assert
> >     (obj=0x555557e03960, typename=0x5555563c403e "machine", file=0x5555563c4018 "/work/armbru/qemu/include/hw/boards.h", line=24, func=0x5555563c4290 <__func__.7> "MACHINE") at ../qom/object.c:936
> > #4  0x0000555555d5db0f in MACHINE (obj=0x555557e03960)
> >     at /work/armbru/qemu/include/hw/boards.h:24
> > #5  0x0000555555d5e030 in x86_iommu_get_default () at ../hw/i386/x86-iommu.c:83
> > #6  0x0000555555d5e262 in x86_iommu_get_instance
> >     (errp=0x5555573d4918 <error_abort>) at ../hw/i386/x86-iommu.c:139
> > #7  0x0000555555f7c27c in singleton_get_instance (class=0x555557e00320)
> >     at ../qom/object_interfaces.c:371
> > #8  0x000055555612a842 in qmp_device_list_properties
> >     (typename=0x555557e001d0 "amd-iommu", errp=0x7fffffffda38)
> >     at ../qom/qom-qmp-cmds.c:147
> > #9  0x0000555555bf20b2 in qdev_device_help (opts=0x555557e001f0)
> >     at ../system/qdev-monitor.c:314
> > #10 0x0000555555bfe06d in device_help_func
> >     (opaque=0x0, opts=0x555557e001f0, errp=0x0) at ../system/vl.c:1208
> > #11 0x0000555556217186 in qemu_opts_foreach
> >     (list=0x55555729e5c0 <qemu_device_opts>, func=0x555555bfe04d <device_help_func>, opaque=0x0, errp=0x0) at ../util/qemu-option.c:1135
> > #12 0x0000555555c01d56 in qemu_process_help_options () at ../system/vl.c:2555
> > #13 0x0000555555c04d81 in qemu_init (argc=3, argv=0x7fffffffde28)
> >     at ../system/vl.c:3654
> > #14 0x000055555612ffae in main (argc=3, argv=0x7fffffffde28)
> >     at ../system/main.c:47
> > 
> 
> Thanks for the report!
> 
> It turns out that qdev_get_machine() cannot be invoked too early, and the
> singleton code can make it earlier..
> 
> We may want a pre-requisite patch to allow qdev_get_machine() to be invoked
> anytime, like:
> 
> ===8<===
> diff --git a/hw/core/qdev.c b/hw/core/qdev.c
> index db36f54d91..7ceae47139 100644
> --- a/hw/core/qdev.c
> +++ b/hw/core/qdev.c
> @@ -831,6 +831,16 @@ Object *qdev_get_machine(void)
>  {
>      static Object *dev;
>  
> +    if (!phase_check(PHASE_MACHINE_CREATED)) {
> +        /*
> +         * When the machine is not created, below can wrongly create
> +         * /machine to be a container.. this enables qdev_get_machine() to
> +         * be used at any time and return NULL properly when machine is not
> +         * created.
> +         */
> +        return NULL;
> +    }
> +
>      if (dev == NULL) {
>          dev = container_get(object_get_root(), "/machine");
>      }
> ===8<===
> 
> I hope it makes sense on its own.

My apologies, spoke too soon here.  This helper is used too after machine
is created, but right before switching to PHASE_MACHINE_CREATE stage..

So we need another way, like:

===8<===
diff --git a/hw/core/qdev.c b/hw/core/qdev.c
index db36f54d91..36a9fdb428 100644
--- a/hw/core/qdev.c
+++ b/hw/core/qdev.c
@@ -832,7 +832,13 @@ Object *qdev_get_machine(void)
     static Object *dev;
 
     if (dev == NULL) {
-        dev = container_get(object_get_root(), "/machine");
+        /*
+         * NOTE: dev can keep being NULL if machine is not yet created!
+         * In which case the function will properly return NULL.
+         *
+         * Whenever machine object is created and found once, we cache it.
+         */
+        dev = object_resolve_path_component(object_get_root(), "machine");
     }
 
     return dev;
===8<===

The idea is still the same.  Meanwhile I'll test more to see whether it has
other issues.

Thanks,

> Then callers who can be invoked earlier
> could then handle NULL properly, in this case..
> 
> ===8<===
> diff --git a/hw/i386/x86-iommu.c b/hw/i386/x86-iommu.c
> index 4bfeb08705..fceb7adfe0 100644
> --- a/hw/i386/x86-iommu.c
> +++ b/hw/i386/x86-iommu.c
> @@ -80,9 +80,15 @@ void x86_iommu_irq_to_msi_message(X86IOMMUIrq *irq, MSIMessage *msg_out)
>  
>  X86IOMMUState *x86_iommu_get_default(void)
>  {
> -    MachineState *ms = MACHINE(qdev_get_machine());
> -    PCMachineState *pcms =
> -        PC_MACHINE(object_dynamic_cast(OBJECT(ms), TYPE_PC_MACHINE));
> +    Object *machine = qdev_get_machine();
> +    PCMachineState *pcms;
> +
> +    /* If machine has not been created, so is the vIOMMU */
> +    if (!machine) {
> +        return NULL;
> +    }
> +
> +    pcms = PC_MACHINE(object_dynamic_cast(machine, TYPE_PC_MACHINE));
>  
>      if (pcms &&
>          object_dynamic_cast(OBJECT(pcms->iommu), TYPE_X86_IOMMU_DEVICE)) {
> ===8<===
> 
> I'll make sure this works if I'll repost.
> 
> Thanks,
> 
> -- 
> Peter Xu

-- 
Peter Xu

Peter Xu <peterx@redhat.com> writes:

> On Fri, Oct 25, 2024 at 05:55:59PM -0400, Peter Xu wrote:
>> On Fri, Oct 25, 2024 at 11:25:23AM +0200, Markus Armbruster wrote:
>> > Peter Xu <peterx@redhat.com> writes:
>> > 
>> > > X86 IOMMUs cannot be created more than one on a system yet.  Make it a
>> > > singleton so it guards the system from accidentally create yet another
>> > > IOMMU object when one already presents.
>> > >
>> > > Now if someone tries to create more than one, e.g., via:
>> > >
>> > >   ./qemu -M q35 -device intel-iommu -device intel-iommu
>> > >
>> > > The error will change from:
>> > >
>> > >   qemu-system-x86_64: -device intel-iommu: QEMU does not support multiple vIOMMUs for x86 yet.
>> > >
>> > > To:
>> > >
>> > >   qemu-system-x86_64: -device intel-iommu: Class 'intel-iommu' only supports one instance
>> > >
>> > > Unfortunately, yet we can't remove the singleton check in the machine
>> > > hook (pc_machine_device_pre_plug_cb), because there can also be
>> > > virtio-iommu involved, which doesn't share a common parent class yet.
>> > >
>> > > But with this, it should be closer to reach that goal to check singleton by
>> > > QOM one day.
>> > >
>> > > Signed-off-by: Peter Xu <peterx@redhat.com>
>> > 
>> > $ qemu-system-x86_64 -device amd-iommu,help
>> > /work/armbru/qemu/include/hw/boards.h:24:MACHINE: Object 0x56473906f960 is not an instance of type machine
>> > Aborted (core dumped)

[...]

>> Thanks for the report!
>> 
>> It turns out that qdev_get_machine() cannot be invoked too early, and the
>> singleton code can make it earlier..
>> 
>> We may want a pre-requisite patch to allow qdev_get_machine() to be invoked
>> anytime, like:
>> 
>> ===8<===
>> diff --git a/hw/core/qdev.c b/hw/core/qdev.c
>> index db36f54d91..7ceae47139 100644
>> --- a/hw/core/qdev.c
>> +++ b/hw/core/qdev.c
>> @@ -831,6 +831,16 @@ Object *qdev_get_machine(void)
>>  {
>>      static Object *dev;
>>  
>> +    if (!phase_check(PHASE_MACHINE_CREATED)) {
>> +        /*
>> +         * When the machine is not created, below can wrongly create
>> +         * /machine to be a container.. this enables qdev_get_machine() to
>> +         * be used at any time and return NULL properly when machine is not
>> +         * created.
>> +         */
>> +        return NULL;
>> +    }
>> +
>>      if (dev == NULL) {
>>          dev = container_get(object_get_root(), "/machine");
>>      }
>> ===8<===
>> 
>> I hope it makes sense on its own.
>
> My apologies, spoke too soon here.  This helper is used too after machine
> is created, but right before switching to PHASE_MACHINE_CREATE stage..

container_get() is a trap.

When the object to be gotten is always "container", it merely
complicates container creation: it's implicitly created on first get.
Which of the calls creates may be less than obvious.

When the object to be gotten is something else, such as a machine,
container_get() before creation is *wrong*, and will lead to trouble
later.

In my opinion:

* Hiding creation in getters is a bad idea unless creation has no
  material side effects.

* Getting anything but a container with container_get() is in bad taste.


> So we need another way, like:
>
> ===8<===
>
> diff --git a/hw/core/qdev.c b/hw/core/qdev.c
> index db36f54d91..36a9fdb428 100644
> --- a/hw/core/qdev.c
> +++ b/hw/core/qdev.c
> @@ -832,7 +832,13 @@ Object *qdev_get_machine(void)
>      static Object *dev;
>  
>      if (dev == NULL) {
> -        dev = container_get(object_get_root(), "/machine");
> +        /*
> +         * NOTE: dev can keep being NULL if machine is not yet created!
> +         * In which case the function will properly return NULL.
> +         *
> +         * Whenever machine object is created and found once, we cache it.
> +         */
> +        dev = object_resolve_path_component(object_get_root(), "machine");
>      }
>  
>      return dev;

Now returns null instead of a bogus container when called before machine
creation.  Improvement of sorts.  But none of the callers expect null...
shouldn't we assert(dev) here?

Hmm, below you add a caller that checks for null.

Another nice mess.

> ===8<===
>
> The idea is still the same.  Meanwhile I'll test more to see whether it has
> other issues.
>
> Thanks,
>
>> Then callers who can be invoked earlier
>> could then handle NULL properly, in this case..
>> 
>> ===8<===
>> diff --git a/hw/i386/x86-iommu.c b/hw/i386/x86-iommu.c
>> index 4bfeb08705..fceb7adfe0 100644
>> --- a/hw/i386/x86-iommu.c
>> +++ b/hw/i386/x86-iommu.c
>> @@ -80,9 +80,15 @@ void x86_iommu_irq_to_msi_message(X86IOMMUIrq *irq, MSIMessage *msg_out)
>>  
>>  X86IOMMUState *x86_iommu_get_default(void)
>>  {
>> -    MachineState *ms = MACHINE(qdev_get_machine());
>> -    PCMachineState *pcms =
>> -        PC_MACHINE(object_dynamic_cast(OBJECT(ms), TYPE_PC_MACHINE));
>> +    Object *machine = qdev_get_machine();
>> +    PCMachineState *pcms;
>> +
>> +    /* If machine has not been created, so is the vIOMMU */
>> +    if (!machine) {
>> +        return NULL;
>> +    }
>> +
>> +    pcms = PC_MACHINE(object_dynamic_cast(machine, TYPE_PC_MACHINE));
>>  
>>      if (pcms &&
>>          object_dynamic_cast(OBJECT(pcms->iommu), TYPE_X86_IOMMU_DEVICE)) {
>> ===8<===
>> 
>> I'll make sure this works if I'll repost.
>> 
>> Thanks,
>> 
>> -- 
>> Peter Xu

On Thu, Nov 07, 2024 at 12:12:10PM +0100, Markus Armbruster wrote:
> Peter Xu <peterx@redhat.com> writes:
> 
> > On Fri, Oct 25, 2024 at 05:55:59PM -0400, Peter Xu wrote:
> >> On Fri, Oct 25, 2024 at 11:25:23AM +0200, Markus Armbruster wrote:
> >> > Peter Xu <peterx@redhat.com> writes:
> >> > 
> >> > > X86 IOMMUs cannot be created more than one on a system yet.  Make it a
> >> > > singleton so it guards the system from accidentally create yet another
> >> > > IOMMU object when one already presents.
> >> > >
> >> > > Now if someone tries to create more than one, e.g., via:
> >> > >
> >> > >   ./qemu -M q35 -device intel-iommu -device intel-iommu
> >> > >
> >> > > The error will change from:
> >> > >
> >> > >   qemu-system-x86_64: -device intel-iommu: QEMU does not support multiple vIOMMUs for x86 yet.
> >> > >
> >> > > To:
> >> > >
> >> > >   qemu-system-x86_64: -device intel-iommu: Class 'intel-iommu' only supports one instance
> >> > >
> >> > > Unfortunately, yet we can't remove the singleton check in the machine
> >> > > hook (pc_machine_device_pre_plug_cb), because there can also be
> >> > > virtio-iommu involved, which doesn't share a common parent class yet.
> >> > >
> >> > > But with this, it should be closer to reach that goal to check singleton by
> >> > > QOM one day.
> >> > >
> >> > > Signed-off-by: Peter Xu <peterx@redhat.com>
> >> > 
> >> > $ qemu-system-x86_64 -device amd-iommu,help
> >> > /work/armbru/qemu/include/hw/boards.h:24:MACHINE: Object 0x56473906f960 is not an instance of type machine
> >> > Aborted (core dumped)
> 
> [...]
> 
> >> Thanks for the report!
> >> 
> >> It turns out that qdev_get_machine() cannot be invoked too early, and the
> >> singleton code can make it earlier..
> >> 
> >> We may want a pre-requisite patch to allow qdev_get_machine() to be invoked
> >> anytime, like:
> >> 
> >> ===8<===
> >> diff --git a/hw/core/qdev.c b/hw/core/qdev.c
> >> index db36f54d91..7ceae47139 100644
> >> --- a/hw/core/qdev.c
> >> +++ b/hw/core/qdev.c
> >> @@ -831,6 +831,16 @@ Object *qdev_get_machine(void)
> >>  {
> >>      static Object *dev;
> >>  
> >> +    if (!phase_check(PHASE_MACHINE_CREATED)) {
> >> +        /*
> >> +         * When the machine is not created, below can wrongly create
> >> +         * /machine to be a container.. this enables qdev_get_machine() to
> >> +         * be used at any time and return NULL properly when machine is not
> >> +         * created.
> >> +         */
> >> +        return NULL;
> >> +    }
> >> +
> >>      if (dev == NULL) {
> >>          dev = container_get(object_get_root(), "/machine");
> >>      }
> >> ===8<===
> >> 
> >> I hope it makes sense on its own.
> >
> > My apologies, spoke too soon here.  This helper is used too after machine
> > is created, but right before switching to PHASE_MACHINE_CREATE stage..
> 
> container_get() is a trap.

I had the same feeling..  Though I'd confess I'm not familiar enough with
this part of code.

> 
> When the object to be gotten is always "container", it merely
> complicates container creation: it's implicitly created on first get.
> Which of the calls creates may be less than obvious.
> 
> When the object to be gotten is something else, such as a machine,
> container_get() before creation is *wrong*, and will lead to trouble
> later.
> 
> In my opinion:
> 
> * Hiding creation in getters is a bad idea unless creation has no
>   material side effects.
> 
> * Getting anything but a container with container_get() is in bad taste.

Agreed.

IMHO container_get() interface might still be ok to implicitly create
containers, but only if it will: (1) always make sure what it walks is a
container along the way, and (2) never return any non-container.

> 
> 
> > So we need another way, like:
> >
> > ===8<===
> >
> > diff --git a/hw/core/qdev.c b/hw/core/qdev.c
> > index db36f54d91..36a9fdb428 100644
> > --- a/hw/core/qdev.c
> > +++ b/hw/core/qdev.c
> > @@ -832,7 +832,13 @@ Object *qdev_get_machine(void)
> >      static Object *dev;
> >  
> >      if (dev == NULL) {
> > -        dev = container_get(object_get_root(), "/machine");
> > +        /*
> > +         * NOTE: dev can keep being NULL if machine is not yet created!
> > +         * In which case the function will properly return NULL.
> > +         *
> > +         * Whenever machine object is created and found once, we cache it.
> > +         */
> > +        dev = object_resolve_path_component(object_get_root(), "machine");
> >      }
> >  
> >      return dev;
> 
> Now returns null instead of a bogus container when called before machine
> creation.  Improvement of sorts.  But none of the callers expect null...
> shouldn't we assert(dev) here?
> 
> Hmm, below you add a caller that checks for null.
> 
> Another nice mess.

I plan to put aside the application of singletons to x86-iommu as of now,
due to the fact that qdev complexity may better be done separately.

IOW, before that, I wonder whether we should clean up the container_get()
as you discussed: it doesn't sound like a good interface to return
non-container objects.

I had a quick look, I only see two outliers of such, and besides the
"abuse" in qdev_get_machine(), the only other one is
e500_pcihost_bridge_realize():

*** hw/core/qdev.c:
qdev_get_machine[820]          dev = container_get(object_get_root(), "/machine");

*** hw/pci-host/ppce500.c:
e500_pcihost_bridge_realize[422] PPCE500CCSRState *ccsr = CCSR(container_get(qdev_get_machine(),

If any of us thinks this is the right way to go, I can try to clean it up
(for 10.0).  qdev_get_machine() may still need to be able to return NULL
when singleton applies to IOMMUs, but that can be for later.  Before that,
we can still assert(qdev), I think.

Just to mention I've posted rfcv2 for this series, again feel free to
ignore patch 3-5 as of now:

[PATCH RFC v2 0/7] QOM: Singleton interface
https://lore.kernel.org/r/20241029211607.2114845-1-peterx@redhat.com

I think the plan is Dan may keep collecting feedbacks on his other rfc:

[RFC 0/5] RFC: require error handling for dynamically created objects
https://lore.kernel.org/r/20241031155350.3240361-1-berrange@redhat.com

Then after Dan's lands, I'll rebase my rfcv2 on top of his, dropping
iommu/qdev changes.

Thanks,

-- 
Peter Xu

Peter Xu <peterx@redhat.com> writes:

> On Thu, Nov 07, 2024 at 12:12:10PM +0100, Markus Armbruster wrote:
>> Peter Xu <peterx@redhat.com> writes:
>> 
>> > On Fri, Oct 25, 2024 at 05:55:59PM -0400, Peter Xu wrote:
>> >> On Fri, Oct 25, 2024 at 11:25:23AM +0200, Markus Armbruster wrote:
>> >> > Peter Xu <peterx@redhat.com> writes:
>> >> > 
>> >> > > X86 IOMMUs cannot be created more than one on a system yet.  Make it a
>> >> > > singleton so it guards the system from accidentally create yet another
>> >> > > IOMMU object when one already presents.
>> >> > >
>> >> > > Now if someone tries to create more than one, e.g., via:
>> >> > >
>> >> > >   ./qemu -M q35 -device intel-iommu -device intel-iommu
>> >> > >
>> >> > > The error will change from:
>> >> > >
>> >> > >   qemu-system-x86_64: -device intel-iommu: QEMU does not support multiple vIOMMUs for x86 yet.
>> >> > >
>> >> > > To:
>> >> > >
>> >> > >   qemu-system-x86_64: -device intel-iommu: Class 'intel-iommu' only supports one instance
>> >> > >
>> >> > > Unfortunately, yet we can't remove the singleton check in the machine
>> >> > > hook (pc_machine_device_pre_plug_cb), because there can also be
>> >> > > virtio-iommu involved, which doesn't share a common parent class yet.
>> >> > >
>> >> > > But with this, it should be closer to reach that goal to check singleton by
>> >> > > QOM one day.
>> >> > >
>> >> > > Signed-off-by: Peter Xu <peterx@redhat.com>
>> >> > 
>> >> > $ qemu-system-x86_64 -device amd-iommu,help
>> >> > /work/armbru/qemu/include/hw/boards.h:24:MACHINE: Object 0x56473906f960 is not an instance of type machine
>> >> > Aborted (core dumped)
>> 
>> [...]
>> 
>> >> Thanks for the report!
>> >> 
>> >> It turns out that qdev_get_machine() cannot be invoked too early, and the
>> >> singleton code can make it earlier..
>> >> 
>> >> We may want a pre-requisite patch to allow qdev_get_machine() to be invoked
>> >> anytime, like:
>> >> 
>> >> ===8<===
>> >> diff --git a/hw/core/qdev.c b/hw/core/qdev.c
>> >> index db36f54d91..7ceae47139 100644
>> >> --- a/hw/core/qdev.c
>> >> +++ b/hw/core/qdev.c
>> >> @@ -831,6 +831,16 @@ Object *qdev_get_machine(void)
>> >>  {
>> >>      static Object *dev;
>> >>  
>> >> +    if (!phase_check(PHASE_MACHINE_CREATED)) {
>> >> +        /*
>> >> +         * When the machine is not created, below can wrongly create
>> >> +         * /machine to be a container.. this enables qdev_get_machine() to
>> >> +         * be used at any time and return NULL properly when machine is not
>> >> +         * created.
>> >> +         */
>> >> +        return NULL;
>> >> +    }
>> >> +
>> >>      if (dev == NULL) {
>> >>          dev = container_get(object_get_root(), "/machine");
>> >>      }
>> >> ===8<===
>> >> 
>> >> I hope it makes sense on its own.
>> >
>> > My apologies, spoke too soon here.  This helper is used too after machine
>> > is created, but right before switching to PHASE_MACHINE_CREATE stage..
>> 
>> container_get() is a trap.
>
> I had the same feeling..  Though I'd confess I'm not familiar enough with
> this part of code.
>
>> 
>> When the object to be gotten is always "container", it merely
>> complicates container creation: it's implicitly created on first get.
>> Which of the calls creates may be less than obvious.
>> 
>> When the object to be gotten is something else, such as a machine,
>> container_get() before creation is *wrong*, and will lead to trouble
>> later.
>> 
>> In my opinion:
>> 
>> * Hiding creation in getters is a bad idea unless creation has no
>>   material side effects.
>> 
>> * Getting anything but a container with container_get() is in bad taste.
>
> Agreed.
>
> IMHO container_get() interface might still be ok to implicitly create
> containers,

Creation on demand is fine when we want to create the thing only when
there is demand.

I guess it can also be okay when we want to create it always, but don't
want to decide when exactly (must be before first use), although I
suspect that's just lazy more often than not.

>             but only if it will: (1) always make sure what it walks is a
> container along the way, and (2) never return any non-container.

Yes.  Anything else invites abuse.

>> > So we need another way, like:
>> >
>> > ===8<===
>> >
>> > diff --git a/hw/core/qdev.c b/hw/core/qdev.c
>> > index db36f54d91..36a9fdb428 100644
>> > --- a/hw/core/qdev.c
>> > +++ b/hw/core/qdev.c
>> > @@ -832,7 +832,13 @@ Object *qdev_get_machine(void)
>> >      static Object *dev;
>> >  
>> >      if (dev == NULL) {
>> > -        dev = container_get(object_get_root(), "/machine");
>> > +        /*
>> > +         * NOTE: dev can keep being NULL if machine is not yet created!
>> > +         * In which case the function will properly return NULL.
>> > +         *
>> > +         * Whenever machine object is created and found once, we cache it.
>> > +         */
>> > +        dev = object_resolve_path_component(object_get_root(), "machine");
>> >      }
>> >  
>> >      return dev;
>> 
>> Now returns null instead of a bogus container when called before machine
>> creation.  Improvement of sorts.  But none of the callers expect null...
>> shouldn't we assert(dev) here?
>> 
>> Hmm, below you add a caller that checks for null.
>> 
>> Another nice mess.
>
> I plan to put aside the application of singletons to x86-iommu as of now,
> due to the fact that qdev complexity may better be done separately.
>
> IOW, before that, I wonder whether we should clean up the container_get()
> as you discussed: it doesn't sound like a good interface to return
> non-container objects.
>
> I had a quick look, I only see two outliers of such, and besides the
> "abuse" in qdev_get_machine(), the only other one is
> e500_pcihost_bridge_realize():
>
> *** hw/core/qdev.c:
> qdev_get_machine[820]          dev = container_get(object_get_root(), "/machine");
>
> *** hw/pci-host/ppce500.c:
> e500_pcihost_bridge_realize[422] PPCE500CCSRState *ccsr = CCSR(container_get(qdev_get_machine(),
                                                                "/e500-ccsr"));

Yes, this abuses container_get() to get an "e500-ccsr", which is a
device, not a container.

By the way, intentation is confusing here.

> If any of us thinks this is the right way to go, I can try to clean it up
> (for 10.0).  qdev_get_machine() may still need to be able to return NULL
> when singleton applies to IOMMUs, but that can be for later.  Before that,
> we can still assert(qdev), I think.

I think it's worthwhile.

> Just to mention I've posted rfcv2 for this series, again feel free to
> ignore patch 3-5 as of now:
>
> [PATCH RFC v2 0/7] QOM: Singleton interface
> https://lore.kernel.org/r/20241029211607.2114845-1-peterx@redhat.com
>
> I think the plan is Dan may keep collecting feedbacks on his other rfc:
>
> [RFC 0/5] RFC: require error handling for dynamically created objects
> https://lore.kernel.org/r/20241031155350.3240361-1-berrange@redhat.com
>
> Then after Dan's lands, I'll rebase my rfcv2 on top of his, dropping
> iommu/qdev changes.
>
> Thanks,

Makes sense.  Thanks!