From: Zhao Liu <zhao1.liu@intel.com>

Hi list,

This RFC series is to introduce hybrid topology in QEMU, and is based
on our previous patches about cluster support for x86 [1] ([PATCH RESEND
00/18] Support smp.clusters for x86).

Though our cluster support patches [1] has not yet accepted, we are
still posting the hybrid topology support patches in advance, hoping to
get discussion and feedbacks.

In this RFC series, we mainly have the following work:
* Introduce the new cpu topology under heterogeneous architecture, which
  we called "hybrid" cpu topology.
      - since the name "hmp" is already used in QEMU and "hybrid" is
        also a synonym for heterogeneous.
* Reorganized the relevant topology information in qemu to adapt to the
  hybrid topology.
* Introduce a new command "hybrid", which allows user to set a custom
  hybrid topology.
      - currently limited to custom cluster and custom core
* Support hybrid topology in i386 PC machine.
* Expose basic topology information (topology type and core type) in
  query_cpus_fast.

We hope that the hybrid topology can be general enough to be compatible
with the needs of hybrids with various architectures in the future.

Welcome your feedbacks!


# 1. Background

Heterogeneous computing refers to systems that use more than one kind of
processor or cores [2]. Now there are more and more heterogeneous
computing architectures, typically including arm's big.LITTLE [3] and
intel hybrid architecture [4].

The development of heterogeneous computing brings new challenges to
virtualization. The first problem we face is how to support the vCPU
topology under the heterogeneous architecture.

Here, to distinguish it from "smp" cpu topology, we call it "hybrid" cpu
topology.


# 2. Motivation

In a typical hybrid system, different types of core may have different
capabilities. This difference may be reflected in different capacities,
performance or power efficiencies, and even different instruction-per-
cycle (IPC) capabilities. The direct consequence of these differences is
to affect the performance of the workload.

For a virtual machines, vCPUs are just threads, although we can bind
vCPUs to different cores to benefit from the different advantages of
these physical cores, it's obviously not enough since CPU topology (and
cache topology) will still have a significant impact on scheduling
performance.

In this case (binding vCPU to pCPU) - actually it's a very common
application example, especially on the client side - it is very
necessary for vCPU to obtain the same (or similar) topology as pCPU.

Unfortunately, the current "smp" command cannot do this. For example,
Alder Lake P [5] has 6 Core cores (2 threads per Core) and 8 Atom cores
(1 threads per Atom), and each Core has a L2 cache as well as every 4
Atoms (in the same module) share a L2 cache. "smp" cannot set the
correct cluster topology for "Atom" vCPU or "Core" vCPU.

Therefore, we need a new way to set up hybrid cpu topology for virtual
machines.


# 3. Design overview

## 3.1. General topology structure

Considering that not only intel has implemented the hybrid architecture,
but also other architectures have related implementations, such as arm,
we hope to make the hybrid topology more general, so we changed the
topology structure in MachineState to make it compatible with smp or
hybrid, which also avoids the conflict between the new topology in an
arch-specific machine and the general MachineState.smp.

The new topology structure is as follows:

typedef struct CpuTopology {
    unsigned int cpus;
    unsigned int max_cpus;
    CpuTopoType topo_type;
    union {
        SmpCpuTopology smp;
        HybridCpuTopology hybrid;
    };
} CpuTopology;

Here "smp" and "hybrid" are wrapped in a union, and the new topology
flag "topo_type" identifies whether it is currently smp or hybrid.

Please note, the hybrid topology can be compatible with the smp topology
type, that is, different parts on the same level of the hybrid topology
can set to be the same, but the hybrid topology will introduce more
complexities:
* need to allocate more memory, organized with array or linked-list.
* if someone wants specific cluster/core/thread information, the cpu
  index is required.

These complexities are obviously very different from the usage habits
of "smp", so the original smp topology support is retained while
introducing the hybrid topology, and the hybrid topology is only built
when the hybrid is explicitly required.

Because of the complexity brought by the new topology type and union, we
also wrap the access to this structure into a series of helpers for other
modules to call:

- General interfaces, no need to worry about whether the underlying
  topology is smp or hybrid:

* machine_topo_get_cpus()
* machine_topo_get_max_cpus()
* machine_topo_is_smp()
* machine_topo_get_sockets()
* machine_topo_get_dies()
* machine_topo_get_clusters()
* machine_topo_get_cores()
* machine_topo_get_threads()
* machine_topo_get_threads_by_idx()
* machine_topo_get_cores_per_socket()
* machine_topo_get_threads_per_socket()

- smp-specific interfaces - provided for the cases that are clearly
known to be smp topology:

* machine_topo_get_smp_cores()
* machine_topo_get_smp_threads()

- hybrid-specific interface:

* machine_topo_get_hybrid_core_type()


## 3.2. hybrid topology structures

For most hybrid systems, the threads are the same. The difference is
mainly at the core level: different cores may have different numbers of
threads, or may belong to different core type. So we introduce the new
hybrid core topology structure:

typedef struct HybridCore {
    unsigned int threads;
    unsigned int core_type;
} HybridCore;

At the same time, it should be noted that since the cores are different,
the clusters containing different cores will also be different.
Correspondingly, we also introduce a new hybrid cluster topology:

typedef struct HybridCluster {
    unsigned int cores;
    HybridCore *core_list;
    QSLIST_HEAD(, HybridCorePack) core_pack_list;
} HybridCluster;

With the single linked list "core_pack_list" in cluster structure, the
different cores can be inserted into specific clusters. This provides
much flexibility for the "hybrid" command. After "hybrid" parsing, the
"core_list" array will be built from "core_pack_list", and this is the
topological form of the core level maintained by MachineState since
core-id can be used well on the array.

As of now, hybrid core and hybrid cluster are enough for us, so we limit
the dies and sockets in the current hybrid topology to be the same
respectively. In order to list clusters in one die, we define an array
of clusters in the general hybrid topology structure. The reason why the
linked list is not used here is that the array is simple enough and can
already meet the needs of the "hybrid" command.

typedef struct HybridCpuTopology {
    unsigned int sockets;
    unsigned int dies;
    unsigned int clusters;
    HybridCluster *cluster_list;
} HybridCpuTopology;


## 3.3. "-hybrid" command

For hybrid cpu topology configuration, the original "-smp" lack of
flexibility to expand, and unables to customize different cores.

So we introduce "-hybrid" command and implement it as the multi-
line command. The multi-line command format is more complex than the
single-line one, but it can bring stronger scalability and
intuitiveness. In the future, it can also be easily extended to more
heterogeneous topology levels.

"-hybrid" command is as follows:

-hybrid socket,sockets=n
-hybrid die,dies=n
-hybrid cluster,clusters=n
-hybrid core,cores=n,type=core_type[,threads=threads]
        [,clusterid=cluster]

Here, we first define the corresponding qapi options for these 4
topology levels: core, cluster, die and socket.

We doesn't need a thread level since thread doesn't have different
type.

For example:

-hybrid socket,sockets=1
-hybrid die,dies=1
-hybrid cluster,clusters=4
-hybrid core,cores=1,coretype="core",threads=2,clusterid=0-2
-hybrid core,cores=4,coretype="atom",threads=1

Here we can build a hybrid cpu topology, which has 1 socket, 1 die per
socket, 4 clusters per die. And in each die, every clusters has 4 "atom"
core with 1 threads, and cluster0, cluster1 and cluster2 have 1 "core"
cores with 2 threads.

Please note this example is not an actual machine topology, but it shows
the powerful flexibility of "hybrid" command.


## 3.4. Reorganized topology information in CPUState

CPUState maintains nr_cores and nr_threads for a long time, but
there're some issues:
* The naming of nr_cores and nr_threads is not clear enough to cause
  misunderstandings in topology-related modifications.
* smp has already supported more topology levels, but these added
  topology information is stored in the architecture-specific code,
  therefore, this results in the fragmentation of topology information,
  which is not conducive to maintenance.
    - For example, i386 store nr_dies and nr_modules in CPUX86State.

So, we reorganized the CPUState topology information and cleaned up the
topology information fragments scattered in the specific architecture
code (i386's CPUX86State).

The new topology structure in CPUState is as follows:

typedef struct TopologyState {
    int sockets;
    int cores_per_socket;
    int threads_per_socket;
    int dies_per_socket;
    int clusters_per_die;
    int cores_per_cluster;
    int threads_per_core;
} TopologyState;


Since then, the different topologies in qemu have the clear purposes and
scopes:
* MachineState.topo maintains the topology information of the
entire machine.
* CPUState.topo maintains the topology information required by the
current CPU.


## 3.5. topology information in query_cpus_fast

The topology type and core type are useful to users, so expose them
in query_cpus_fast.

Since the hybrid core is pre-set and does not support hotplug for
the entire core, the topology type and core type are not exposed in
query-hotpluggable-cpus.

For example:

{
    "cpu-index": 0,
    "props": {
        "cluster-id": 0,
        "core-id": 0,
        "socket-id": 0,
        "thread-id": 0
    },
    "qom-path": "/machine/unattached/device[0]",
    "target": "x86_64",
    "thread-id": 313464,
    "topo": {
        "core-type": "core",
        "topo-type": "hybrid"
    }
}


# 4. Patch summary

- patch 1-4 fix historical topology-related issues, which will affect
            subsequent hybrid topology support.
- patch 5-9 add the new general topology structure support which
            is compatible with hybrid and smp.
- patch 10-27 replace origial smp topology structure with the new
              general topology structure. For generic code, hybrid
              topology must be respected, so case by case is required
              for replacement (patch 10-18). As for arch-specific code,
              restricting the topology to smp is also enough until the
              arch supports hybrid (patch 19-27).
- pacth 28-29 add complete topology information in CPUState and remove
              the nr_dies and nr_modules in CPUX86State. This avoid
              fragmentation of topology information and facilitate
              subsequent maintenance. Since then, MachineState.topo
              keeps the system topology information, and CPUState.topo
              owns the topology information for current CPU.
- patch 30-34 cleanups X86CPUTopoInfo uses. X86CPUTopoInfo original was
              used to keep topology informations and used to generate
              APIC ID. For hybrid topology, these two functions cannot
              be preserved at the same time. So make it only used to do
              APIC ID related topology work, and specifc topology
              information should get from CPUState.topo or
              MachineState.topo.
- patch 35-39 add the hybrid topology related support for i386.
- patch 40-50 introduce the "hybrid" command.
- patch 51-52 expose topology type and core type in query_cpus_fast.


# 5. TODOs

- Add the unit test for "hybrid" command.
- Support complete online cpu custom configuration for "hybrid" command.
- Support coretype and core level can be omitted.
- Add a limit on the maximum apicid.


# 6. References

[1]: [PATCH RESEND 00/18] Support smp.clusters for x86:
     https://lists.gnu.org/archive/html/qemu-devel/2023-02/msg03184.html
[2]: https://en.wikipedia.org/wiki/Heterogeneous_computing
[3]: https://www.arm.com/technologies/big-little
[4]: https://www.intel.com/content/www/us/en/support/articles/000091896/processors.html
[5]: https://www.intel.com/content/www/us/en/products/platforms/details/alder-lake-p.html

---
Zhao Liu (52):
  hw/smbios: Fix smbios_smp_sockets caculation
  hw/smbios: Fix thread count in type4
  hw/smbios: Fix core count in type4
  i386/WHPX: Fix error message when fail to set ProcessorCount
  hw/core/machine: Rename machine-smp.c to machine-topo.c
  hw/cpu: Introduce hybrid CPU topology
  hw/core/machine: Add the new topology support in MachineState
  machine: Add helpers to get cpu topology info from MachineState.topo
  hw/machine: Introduce core type for hybrid topology
  machine: Replace MachineState.topo.smp access with topology helpers
  accel/kvm: Add hybrid info when check cpu num
  hw/acpi: Replace MachineState.smp access with topology helpers
  cpu/core: Use generic topology helper for "help" to set nr_threads
  hw/smbios: Use generic topology name and helper
  migration/postcopy-ram: Use generic topology name and helper
  plugins: Use generic topology name and helper
  softmmu/cpus: Use generic topology helper in vcpus initialization
  general: Replace MachineState.smp access with topology helpers
  i386: Replace MachineState.smp access with topology helpers
  s390x: Replace MachineState.smp access with topology helpers
  ppc: Replace MachineState.smp access with topology helpers
  riscv: Replace MachineState.smp access with topology helpers
  arm: Replace MachineState.smp access with topology helpers
  loongarch: Replace MachineState.smp access with topology helpers
  mips: Replace MachineState.smp access with topology helpers
  hw: Replace MachineState.smp access with topology helpers for all
    remaining archs
  test/test-smp-parse: Check fields of MachineState.topo.smp
  hw/core/machine: Remove support of MachineState.smp
  hw/core/cpu: Introduce TopologyState in CPUState
  i386: Drop nr_dies and nr_modules CPUX86State
  i386/cpu: Use CPUState.topo to replace X86CPUTopoInfo to get topology
    info
  i386: Rename X86CPUTopoInfo and its members to reflect relationship
    with APIC ID
  i386: Rename init_topo_info() to init_apic_topo_info()
  i386: Rename variable topo_info to apicid_topo
  i386: Support APIC ID topology for hybrid CPU topology
  i386: Use init_apicid_topo_info() to initialize APIC ID topology for
    system emulator
  i386: Update X86CPUTopoIDs generating rule for hybrid topology
  i386: Introduce hybrid_core_type to CPUX86State
  i386/cpu: Add Intel hybrid related CPUID support
  qapi: Introduce hybrid options
  machine: Introduce core_type() hook
  hw/machine: Add hybrid_supported in generic topo properties
  hw/machine: Rename MachineClass.smp_props to MachineClass.topo_props
  machine: Add "-hybrid" parsing rule
  hw/machine: Add hybrid cpu topology validation
  hw/machine: build core level hybrid topology form HybridCorePack
  hw/machine: Use opts_visitor to parse hybrid topo
  machine: Support "-hybrid" command
  i386/pc: Support hybrid cpu topology
  qemu-options: Add the document of hybrid command
  qapi: Expose CPU topology info in query_cpus_fast
  i386: Support cpu_index_to_core_type() for x86

 MAINTAINERS                          |   3 +-
 accel/kvm/kvm-all.c                  |   5 +-
 accel/tcg/tcg-accel-ops-mttcg.c      |   2 +-
 accel/tcg/tcg-all.c                  |   2 +-
 backends/hostmem.c                   |   2 +-
 contrib/plugins/hwprofile.c          |   2 +-
 gdbstub/gdbstub.c                    |   2 +-
 hw/acpi/aml-build.c                  |   4 +-
 hw/alpha/dp264.c                     |   2 +-
 hw/arm/fsl-imx6.c                    |   4 +-
 hw/arm/fsl-imx6ul.c                  |   4 +-
 hw/arm/fsl-imx7.c                    |   4 +-
 hw/arm/highbank.c                    |   2 +-
 hw/arm/realview.c                    |   2 +-
 hw/arm/sbsa-ref.c                    |   8 +-
 hw/arm/vexpress.c                    |   2 +-
 hw/arm/virt-acpi-build.c             |   4 +-
 hw/arm/virt.c                        |  56 +-
 hw/arm/xlnx-zynqmp.c                 |   6 +-
 hw/core/cpu-common.c                 |  14 +-
 hw/core/machine-qmp-cmds.c           |  11 +
 hw/core/machine-smp.c                | 198 -------
 hw/core/machine-topo.c               | 853 +++++++++++++++++++++++++++
 hw/core/machine.c                    |  33 +-
 hw/core/meson.build                  |   2 +-
 hw/core/numa.c                       |   2 +-
 hw/cpu/core.c                        |   6 +-
 hw/hppa/machine.c                    |   6 +-
 hw/i386/acpi-build.c                 |   2 +-
 hw/i386/kvmvapic.c                   |   4 +-
 hw/i386/microvm.c                    |   4 +-
 hw/i386/pc.c                         |   5 +-
 hw/i386/pc_piix.c                    |   2 +-
 hw/i386/pc_q35.c                     |   2 +-
 hw/i386/x86.c                        | 255 ++++++--
 hw/i386/xen/xen-hvm.c                |   4 +-
 hw/intc/spapr_xive_kvm.c             |   3 +-
 hw/intc/xics_kvm.c                   |   3 +-
 hw/loongarch/acpi-build.c            |   4 +-
 hw/loongarch/fw_cfg.c                |   4 +-
 hw/loongarch/virt.c                  |  30 +-
 hw/mips/boston.c                     |   8 +-
 hw/mips/loongson3_bootp.c            |   7 +-
 hw/mips/loongson3_virt.c             |   8 +-
 hw/mips/malta.c                      |  22 +-
 hw/openrisc/openrisc_sim.c           |   2 +-
 hw/openrisc/virt.c                   |   2 +-
 hw/ppc/e500.c                        |   6 +-
 hw/ppc/mac_newworld.c                |  16 +-
 hw/ppc/mac_oldworld.c                |  12 +-
 hw/ppc/pnv.c                         |  11 +-
 hw/ppc/prep.c                        |   3 +-
 hw/ppc/spapr.c                       |  39 +-
 hw/ppc/spapr_rtas.c                  |  17 +-
 hw/riscv/microchip_pfsoc.c           |  11 +-
 hw/riscv/numa.c                      |  21 +-
 hw/riscv/opentitan.c                 |   8 +-
 hw/riscv/shakti_c.c                  |   2 +-
 hw/riscv/sifive_e.c                  |  10 +-
 hw/riscv/sifive_u.c                  |  28 +-
 hw/riscv/virt.c                      |  24 +-
 hw/s390x/s390-virtio-ccw.c           |   8 +-
 hw/s390x/sclp.c                      |   3 +-
 hw/smbios/smbios.c                   |  22 +-
 hw/sparc/sun4m.c                     |   4 +-
 hw/sparc64/sun4u.c                   |   6 +-
 hw/virtio/virtio-pci.c               |   2 +-
 hw/xtensa/sim.c                      |   2 +-
 hw/xtensa/xtfpga.c                   |   2 +-
 include/hw/arm/virt.h                |   2 +-
 include/hw/boards.h                  |  93 ++-
 include/hw/core/cpu.h                |  26 +-
 include/hw/cpu/cpu-topology.h        | 137 +++++
 include/hw/i386/topology.h           | 128 ++--
 include/hw/i386/x86.h                |   3 +-
 include/qemu/qemu-plugin.h           |   4 +-
 migration/postcopy-ram.c             |  24 +-
 plugins/api.c                        |   4 +-
 plugins/loader.c                     |   4 +-
 qapi/machine.json                    | 119 +++-
 qemu-options.hx                      |  84 +++
 softmmu/cpus.c                       |  18 +-
 softmmu/dirtylimit.c                 |   8 +-
 softmmu/vl.c                         |  35 +-
 target/arm/cpu.c                     |   2 +-
 target/arm/cpu_tcg.c                 |   2 +-
 target/arm/kvm.c                     |   2 +-
 target/i386/cpu.c                    | 172 ++++--
 target/i386/cpu.h                    |  16 +-
 target/i386/hax/hax-all.c            |   2 +-
 target/i386/hvf/x86_emu.c            |   6 +-
 target/i386/kvm/kvm.c                |   8 +-
 target/i386/tcg/sysemu/misc_helper.c |   2 +-
 target/i386/whpx/whpx-all.c          |   6 +-
 target/mips/tcg/sysemu/cp0_helper.c  |   4 +-
 target/openrisc/sys_helper.c         |   2 +-
 target/ppc/compat.c                  |   2 +-
 target/s390x/cpu-sysemu.c            |   2 +-
 target/s390x/kvm/kvm.c               |  15 +-
 target/s390x/tcg/excp_helper.c       |   2 +-
 tests/unit/meson.build               |   2 +-
 tests/unit/test-smp-parse.c          |  74 ++-
 tests/unit/test-x86-apicid.c         | 211 ++++---
 103 files changed, 2250 insertions(+), 840 deletions(-)
 delete mode 100644 hw/core/machine-smp.c
 create mode 100644 hw/core/machine-topo.c
 create mode 100644 include/hw/cpu/cpu-topology.h

-- 
2.34.1

Zhao Liu <zhao1.liu@linux.intel.com> writes:

> From: Zhao Liu <zhao1.liu@intel.com>
>
> Hi list,
>
> This RFC series is to introduce hybrid topology in QEMU, and is based
> on our previous patches about cluster support for x86 [1] ([PATCH RESEND
> 00/18] Support smp.clusters for x86).
>
> Though our cluster support patches [1] has not yet accepted, we are
> still posting the hybrid topology support patches in advance, hoping to
> get discussion and feedbacks.
>
> In this RFC series, we mainly have the following work:
> * Introduce the new cpu topology under heterogeneous architecture, which
>   we called "hybrid" cpu topology.
>       - since the name "hmp" is already used in QEMU and "hybrid" is
>         also a synonym for heterogeneous.
> * Reorganized the relevant topology information in qemu to adapt to the
>   hybrid topology.
> * Introduce a new command "hybrid", which allows user to set a custom
>   hybrid topology.
>       - currently limited to custom cluster and custom core
> * Support hybrid topology in i386 PC machine.
> * Expose basic topology information (topology type and core type) in
>   query_cpus_fast.
>
> We hope that the hybrid topology can be general enough to be compatible
> with the needs of hybrids with various architectures in the future.
>
> Welcome your feedbacks!
>
>
> # 1. Background
>
> Heterogeneous computing refers to systems that use more than one kind of
> processor or cores [2]. Now there are more and more heterogeneous
> computing architectures, typically including arm's big.LITTLE [3] and
> intel hybrid architecture [4].
>
> The development of heterogeneous computing brings new challenges to
> virtualization. The first problem we face is how to support the vCPU
> topology under the heterogeneous architecture.
>
> Here, to distinguish it from "smp" cpu topology, we call it "hybrid" cpu
> topology.
>
>
> # 2. Motivation
>
> In a typical hybrid system, different types of core may have different
> capabilities. This difference may be reflected in different capacities,
> performance or power efficiencies, and even different instruction-per-
> cycle (IPC) capabilities. The direct consequence of these differences is
> to affect the performance of the workload.

This elides a use case for heterogeneous computing under emulation where
different cores can have totally different ISA's. We already support
emulation models where we have varying classes of the same ISA, for
example xlnx-zyncmp supports 4 A profile and 2 R profile Arm chips.

I've glanced over the patch set and I don't think there is any direct
clash with that as this is mainly about representing the topology for
x86 non-emulation accelerators but I just want ensure we don't end up
stepping on each others toes.

> For a virtual machines, vCPUs are just threads, although we can bind
> vCPUs to different cores to benefit from the different advantages of
> these physical cores, it's obviously not enough since CPU topology (and
> cache topology) will still have a significant impact on scheduling
> performance.
>
<snip>
> ## 3.3. "-hybrid" command
>
> For hybrid cpu topology configuration, the original "-smp" lack of
> flexibility to expand, and unables to customize different cores.
>
> So we introduce "-hybrid" command and implement it as the multi-
> line command. The multi-line command format is more complex than the
> single-line one, but it can bring stronger scalability and
> intuitiveness. In the future, it can also be easily extended to more
> heterogeneous topology levels.
>
> "-hybrid" command is as follows:
>
> -hybrid socket,sockets=n
> -hybrid die,dies=n
> -hybrid cluster,clusters=n
> -hybrid core,cores=n,type=core_type[,threads=threads]
>         [,clusterid=cluster]
>
> Here, we first define the corresponding qapi options for these 4
> topology levels: core, cluster, die and socket.
>
> We doesn't need a thread level since thread doesn't have different
> type.
>
> For example:
>
> -hybrid socket,sockets=1
> -hybrid die,dies=1
> -hybrid cluster,clusters=4
> -hybrid core,cores=1,coretype="core",threads=2,clusterid=0-2
> -hybrid core,cores=4,coretype="atom",threads=1
>
> Here we can build a hybrid cpu topology, which has 1 socket, 1 die per
> socket, 4 clusters per die. And in each die, every clusters has 4 "atom"
> core with 1 threads, and cluster0, cluster1 and cluster2 have 1 "core"
> cores with 2 threads.
>
> Please note this example is not an actual machine topology, but it shows
> the powerful flexibility of "hybrid" command.

Is there any intention to apply this command to emulation or will it
only be for hardware virtualisation? For example the x86 translator
should be capable of emulating two different classes of x86 with
different subsets of instructions. It requires ensuring each CPU is
assigned to the correct TYPE_CPU_CLUSTER so we don't re-use TBs from
the "wrong" CPU.

<snip>

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

On Mon, Feb 13, 2023 at 10:14:22AM +0000, Alex Bennée wrote:
> Date: Mon, 13 Feb 2023 10:14:22 +0000
> From: Alex Bennée <alex.bennee@linaro.org>
> Subject: Re: [RFC 00/52] Introduce hybrid CPU topology
> 
> 
> Zhao Liu <zhao1.liu@linux.intel.com> writes:
> 
> > From: Zhao Liu <zhao1.liu@intel.com>
> >
> > Hi list,
> >
> > This RFC series is to introduce hybrid topology in QEMU, and is based
> > on our previous patches about cluster support for x86 [1] ([PATCH RESEND
> > 00/18] Support smp.clusters for x86).
> >
> > Though our cluster support patches [1] has not yet accepted, we are
> > still posting the hybrid topology support patches in advance, hoping to
> > get discussion and feedbacks.
> >
> > In this RFC series, we mainly have the following work:
> > * Introduce the new cpu topology under heterogeneous architecture, which
> >   we called "hybrid" cpu topology.
> >       - since the name "hmp" is already used in QEMU and "hybrid" is
> >         also a synonym for heterogeneous.
> > * Reorganized the relevant topology information in qemu to adapt to the
> >   hybrid topology.
> > * Introduce a new command "hybrid", which allows user to set a custom
> >   hybrid topology.
> >       - currently limited to custom cluster and custom core
> > * Support hybrid topology in i386 PC machine.
> > * Expose basic topology information (topology type and core type) in
> >   query_cpus_fast.
> >
> > We hope that the hybrid topology can be general enough to be compatible
> > with the needs of hybrids with various architectures in the future.
> >
> > Welcome your feedbacks!
> >
> >
> > # 1. Background
> >
> > Heterogeneous computing refers to systems that use more than one kind of
> > processor or cores [2]. Now there are more and more heterogeneous
> > computing architectures, typically including arm's big.LITTLE [3] and
> > intel hybrid architecture [4].
> >
> > The development of heterogeneous computing brings new challenges to
> > virtualization. The first problem we face is how to support the vCPU
> > topology under the heterogeneous architecture.
> >
> > Here, to distinguish it from "smp" cpu topology, we call it "hybrid" cpu
> > topology.
> >
> >
> > # 2. Motivation
> >
> > In a typical hybrid system, different types of core may have different
> > capabilities. This difference may be reflected in different capacities,
> > performance or power efficiencies, and even different instruction-per-
> > cycle (IPC) capabilities. The direct consequence of these differences is
> > to affect the performance of the workload.
> 
> This elides a use case for heterogeneous computing under emulation where
> different cores can have totally different ISA's. We already support
> emulation models where we have varying classes of the same ISA, for
> example xlnx-zyncmp supports 4 A profile and 2 R profile Arm chips.

This is emulation mode, so you also need to define heterogeneous CPU
topology in this case, right?

> 
> I've glanced over the patch set and I don't think there is any direct
> clash with that as this is mainly about representing the topology for
> x86 non-emulation accelerators but I just want ensure we don't end up
> stepping on each others toes.

Yes, this is for non-emulation case.

> 
> > For a virtual machines, vCPUs are just threads, although we can bind
> > vCPUs to different cores to benefit from the different advantages of
> > these physical cores, it's obviously not enough since CPU topology (and
> > cache topology) will still have a significant impact on scheduling
> > performance.
> >
> <snip>
> > ## 3.3. "-hybrid" command
> >
> > For hybrid cpu topology configuration, the original "-smp" lack of
> > flexibility to expand, and unables to customize different cores.
> >
> > So we introduce "-hybrid" command and implement it as the multi-
> > line command. The multi-line command format is more complex than the
> > single-line one, but it can bring stronger scalability and
> > intuitiveness. In the future, it can also be easily extended to more
> > heterogeneous topology levels.
> >
> > "-hybrid" command is as follows:
> >
> > -hybrid socket,sockets=n
> > -hybrid die,dies=n
> > -hybrid cluster,clusters=n
> > -hybrid core,cores=n,type=core_type[,threads=threads]
> >         [,clusterid=cluster]
> >
> > Here, we first define the corresponding qapi options for these 4
> > topology levels: core, cluster, die and socket.
> >
> > We doesn't need a thread level since thread doesn't have different
> > type.
> >
> > For example:
> >
> > -hybrid socket,sockets=1
> > -hybrid die,dies=1
> > -hybrid cluster,clusters=4
> > -hybrid core,cores=1,coretype="core",threads=2,clusterid=0-2
> > -hybrid core,cores=4,coretype="atom",threads=1
> >
> > Here we can build a hybrid cpu topology, which has 1 socket, 1 die per
> > socket, 4 clusters per die. And in each die, every clusters has 4 "atom"
> > core with 1 threads, and cluster0, cluster1 and cluster2 have 1 "core"
> > cores with 2 threads.
> >
> > Please note this example is not an actual machine topology, but it shows
> > the powerful flexibility of "hybrid" command.
> 
> Is there any intention to apply this command to emulation or will it
> only be for hardware virtualisation? For example the x86 translator
> should be capable of emulating two different classes of x86 with
> different subsets of instructions.

We have not considered the hybrid case in emulation mode, and
the current hybrid CPUs of intel are same-ISA.

> It requires ensuring each CPU is
> assigned to the correct TYPE_CPU_CLUSTER so we don't re-use TBs from
> the "wrong" CPU.

From my understanding, this also seems like the topology of heterogeneous
core and heterogeneous cluster.

Zhao

> 
> <snip>
> 
> -- 
> Alex Bennée
> Virtualisation Tech Lead @ Linaro

On Mon, Feb 13, 2023 at 05:49:43PM +0800, Zhao Liu wrote:
> From: Zhao Liu <zhao1.liu@intel.com>
> ## 3.3. "-hybrid" command
> 
> For hybrid cpu topology configuration, the original "-smp" lack of
> flexibility to expand, and unables to customize different cores.
> 
> So we introduce "-hybrid" command and implement it as the multi-
> line command. The multi-line command format is more complex than the
> single-line one, but it can bring stronger scalability and
> intuitiveness. In the future, it can also be easily extended to more
> heterogeneous topology levels.
> 
> "-hybrid" command is as follows:
> 
> -hybrid socket,sockets=n
> -hybrid die,dies=n
> -hybrid cluster,clusters=n
> -hybrid core,cores=n,type=core_type[,threads=threads]
>         [,clusterid=cluster]
> 
> Here, we first define the corresponding qapi options for these 4
> topology levels: core, cluster, die and socket.
> 
> We doesn't need a thread level since thread doesn't have different
> type.
> 
> For example:
> 
> -hybrid socket,sockets=1
> -hybrid die,dies=1
> -hybrid cluster,clusters=4
> -hybrid core,cores=1,coretype="core",threads=2,clusterid=0-2
> -hybrid core,cores=4,coretype="atom",threads=1
> 
> Here we can build a hybrid cpu topology, which has 1 socket, 1 die per
> socket, 4 clusters per die. And in each die, every clusters has 4 "atom"
> core with 1 threads, and cluster0, cluster1 and cluster2 have 1 "core"
> cores with 2 threads.

How will this interact with the -cpu  parameter. Presumably we now
need two distinct -cpu parameters, one for the 'core' CPU model and
one for the 'atom' CPU model ?

> Please note this example is not an actual machine topology, but it shows
> the powerful flexibility of "hybrid" command.

IIUC the functionality offered by -hybrid should be a superset
of the -smp functionality. IOW, -smp ought to be possible to
re-implement -smp as an alias for -hybrid, such that internally
code only ever has to deal with the modern approach. Having to
keep support for both -smp and -hybrid throughout the code is
undesirable IMHO. Keeping the compat at the CLI parsing level
limits the burden.


As a more general thought, rather than introducing a new top level
command line argument -hybrid, I'm thinking we should possibly just
define this all using QOM and thus re-use the existing -object
argument. 

I'm also finding the above example command lines quite difficult
to understand, as there is alot of implicit linkage and expansion
between the different levels. With devices we're much more
explicitly with the parent/child relationships, and have to
express everything with no automatic expansion, linking it all
together via the id=/bus= properties.  This is quite a bit more
verbose, but it is also very effective at letting us express
arbitrarily complex relationships.

I think it would be worth exploring that approach for the CPU
topology expression too.

If we followed the more explicit device approach to modelling
then instead of:

 -cpu core,...
 -cpu atom,...
 -hybrid socket,sockets=1
 -hybrid die,dies=1
 -hybrid cluster,clusters=4
 -hybrid core,cores=1,coretype="core",threads=2,clusterid=0-2
 -hybrid core,cores=4,coretype="atom",threads=1

we would end up with something like

  -object cpu-socket,id=sock0
  -object cpu-die,id=die0,parent=sock0
  -object cpu-cluster,id=cluster0,parent=die0
  -object cpu-cluster,id=cluster1,parent=die0
  -object cpu-cluster,id=cluster2,parent=die0
  -object cpu-cluster,id=cluster3,parent=die0
  -object x86-cpu-model-atom,id=cpu0,parent=cluster0
  -object x86-cpu-model-atom,id=cpu1,parent=cluster0
  -object x86-cpu-model-atom,id=cpu2,parent=cluster0
  -object x86-cpu-model-atom,id=cpu3,parent=cluster0
  -object x86-cpu-model-core,id=cpu4,parent=cluster0,threads=2
  -object x86-cpu-model-atom,id=cpu5,parent=cluster1
  -object x86-cpu-model-atom,id=cpu6,parent=cluster1
  -object x86-cpu-model-atom,id=cpu7,parent=cluster1
  -object x86-cpu-model-atom,id=cpu8,parent=cluster1
  -object x86-cpu-model-core,id=cpu9,parent=cluster1,threads=2
  -object x86-cpu-model-atom,id=cpu10,parent=cluster2
  -object x86-cpu-model-atom,id=cpu11,parent=cluster2
  -object x86-cpu-model-atom,id=cpu12,parent=cluster2
  -object x86-cpu-model-atom,id=cpu13,parent=cluster2
  -object x86-cpu-model-core,id=cpu14,parent=cluster2,threads=2
  -object x86-cpu-model-atom,id=cpu15,parent=cluster3
  -object x86-cpu-model-atom,id=cpu16,parent=cluster3
  -object x86-cpu-model-atom,id=cpu17,parent=cluster3
  -object x86-cpu-model-atom,id=cpu18,parent=cluster3
  -object x86-cpu-model-core,id=cpu19,parent=cluster3,threads=2

The really obvious downside is that it is much more verbose.

This example only has 20 CPUs. For a VM with say 1000 CPUs
this will be very big, but that doesn't neccesarily make it
wrong.

On the flipside

 * It is really clear exactly how many CPUs I've added

 * The relationship between the topology levels is clear

 * Every CPU has a unique ID given that can be used in
   later QMP commands

 * Whether or not 'threads' are permitted is now a property
   of the specific CPU model implementation, not the global
   config. IOW we can express that some CPU models allowing
   for threads, and some don't.

 * The -cpu arg is also obsoleted, replaced by the
   -object x86-cpu-model-core. This might facilitate the
   modelling of machines with CPUs from different architectures.


We could potentially compress the leaf node level by expressing
how many instances of an object we want. it we want. ie, define
a more convenient shorthand syntax to creating many instances of
an object. so eg

  -object-set $TYPE,$PROPS,idbase=foo,count=4

would be functionally identical to

  -object $TYPE,$PROPS,id=foo.0
  -object $TYPE,$PROPS,id=foo.1
  -object $TYPE,$PROPS,id=foo.2
  -object $TYPE,$PROPS,id=foo.3

QEMU just expands it and creates all the objects internally.

So the huge example I have above for 20 cpus would become much
shorter: e.g.

  -object cpu-socket,id=sock0
  -object cpu-die,id=die0,parent=sock0
  -object cpu-cluster,id=cluster0,parent=die0
  -object cpu-cluster,id=cluster1,parent=die0
  -object cpu-cluster,id=cluster2,parent=die0
  -object cpu-cluster,id=cluster3,parent=die0
  -object-set x86-cpu-core-atom,idbase=cpu0,parent=cluster0,count=4
  -object-set x86-cpu-core-core,id=cpu1,parent=cluster0,threads=2,count=1
  -object-set x86-cpu-core-atom,idbase=cpu2,parent=cluster1,count=4
  -object-set x86-cpu-core-core,id=cpu3,parent=cluster1,threads=2,count=1
  -object-set x86-cpu-core-atom,idbase=cpu4,parent=cluster2,count=4
  -object-set x86-cpu-core-core,id=cpu5,parent=cluster2,threads=2,count=1
  -object-set x86-cpu-core-atom,idbase=cpu6,parent=cluster3,count=4
  -object-set x86-cpu-core-core,id=cpu7,parent=cluster3,threads=2,count=1

IOW, the size of the CLI config only depends on the number of elements
in the hierarchy, and is independant of the number of leaf CPU cores.

Obviously in describing all of the above, I've ignored any complexity
of dealing with our existing code implementation and pain of getting
it converted to the new model.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Hi Daniel and folks,

Let me pick up this thread again to discuss some updates from my work
on QOM CPU topology. I would like to know if I'm on the right track. :-)

On Mon, Feb 13, 2023 at 01:38:28PM +0000, Daniel P. Berrangé wrote:
>

[snip]

> 
> IIUC the functionality offered by -hybrid should be a superset
> of the -smp functionality. IOW, -smp ought to be possible to
> re-implement -smp as an alias for -hybrid, such that internally
> code only ever has to deal with the modern approach. Having to
> keep support for both -smp and -hybrid throughout the code is
> undesirable IMHO. Keeping the compat at the CLI parsing level
> limits the burden.
> 
> 
> As a more general thought, rather than introducing a new top level
> command line argument -hybrid, I'm thinking we should possibly just
> define this all using QOM and thus re-use the existing -object
> argument. 
> 
> I'm also finding the above example command lines quite difficult
> to understand, as there is alot of implicit linkage and expansion
> between the different levels. With devices we're much more
> explicitly with the parent/child relationships, and have to
> express everything with no automatic expansion, linking it all
> together via the id=/bus= properties.  This is quite a bit more
> verbose, but it is also very effective at letting us express
> arbitrarily complex relationships.
> 
> I think it would be worth exploring that approach for the CPU
> topology expression too.
> 
> If we followed the more explicit device approach to modelling
> then instead of:
> 
>  -cpu core,...
>  -cpu atom,...
>  -hybrid socket,sockets=1
>  -hybrid die,dies=1
>  -hybrid cluster,clusters=4
>  -hybrid core,cores=1,coretype="core",threads=2,clusterid=0-2
>  -hybrid core,cores=4,coretype="atom",threads=1
> 
> we would end up with something like
> 
>   -object cpu-socket,id=sock0
>   -object cpu-die,id=die0,parent=sock0
>   -object cpu-cluster,id=cluster0,parent=die0
>   -object cpu-cluster,id=cluster1,parent=die0
>   -object cpu-cluster,id=cluster2,parent=die0
>   -object cpu-cluster,id=cluster3,parent=die0
>   -object x86-cpu-model-atom,id=cpu0,parent=cluster0
>   -object x86-cpu-model-atom,id=cpu1,parent=cluster0
>   -object x86-cpu-model-atom,id=cpu2,parent=cluster0
>   -object x86-cpu-model-atom,id=cpu3,parent=cluster0
>   -object x86-cpu-model-core,id=cpu4,parent=cluster0,threads=2
>   -object x86-cpu-model-atom,id=cpu5,parent=cluster1
>   -object x86-cpu-model-atom,id=cpu6,parent=cluster1
>   -object x86-cpu-model-atom,id=cpu7,parent=cluster1
>   -object x86-cpu-model-atom,id=cpu8,parent=cluster1
>   -object x86-cpu-model-core,id=cpu9,parent=cluster1,threads=2
>   -object x86-cpu-model-atom,id=cpu10,parent=cluster2
>   -object x86-cpu-model-atom,id=cpu11,parent=cluster2
>   -object x86-cpu-model-atom,id=cpu12,parent=cluster2
>   -object x86-cpu-model-atom,id=cpu13,parent=cluster2
>   -object x86-cpu-model-core,id=cpu14,parent=cluster2,threads=2
>   -object x86-cpu-model-atom,id=cpu15,parent=cluster3
>   -object x86-cpu-model-atom,id=cpu16,parent=cluster3
>   -object x86-cpu-model-atom,id=cpu17,parent=cluster3
>   -object x86-cpu-model-atom,id=cpu18,parent=cluster3
>   -object x86-cpu-model-core,id=cpu19,parent=cluster3,threads=2
> 
> The really obvious downside is that it is much more verbose.

I find the "core" and "cluster" already have the abstraction as the
device, and Andreas also tried to abstract "socket" as the device.

So I find in this QOM way, all CPU topology levels should be
abstracted to devices and created via "-device". And I also need to
extend "cluster" device to support VM case as a general CPU topology
abstraction, not only just used in TCG emulation.

I've done a preliminary POC so far, where I can create the CPU topology
via "-device", and also added the ability to create the "child"
relationship in "-device" (since neither the "link" relationship nor
the bus looks like a good representation of the CPU hierarchy).

There's an example:

-device cpu-socket,id=sock0 \
-device cpu-die,id=die0,parent=sock0 \
-device cpu-die,id=die1,parent=sock0 \
-device cpu-cluster,id=cluster0,parent=die0 \
-device cpu-cluster,id=cluster1,parent=die0 \
-device cpu-cluster,id=cluster2,parent=die1 \
-device x86-intel-core,id=core0,parent=cluster0,threads=3 \
-device x86-intel-atom,id=core1,parent=cluster0,threads=2 \
-device x86-core,id=core2,parent=cluster1,threads=1 \
-device x86-intel-core,id=core3,parent=cluster2,threads=5 \

with the above format, I could build the the more accurate canonical
path for CPUs like this:

(QEMU) query-hotpluggable-cpus
{
    "arguments": {},
    "execute": "query-hotpluggable-cpus"
}
{
    "return": [
        {
            "props": {
                "cluster-id": 0,
                "core-id": 0,
                "die-id": 1,
                "socket-id": 0,
                "thread-id": 4
            },
            "qom-path": "/machine/peripheral/cpu-slot/sock0/die1/cluster2/core3/host-x86_64-cpu[10]",
            "type": "host-x86_64-cpu",
            "vcpus-count": 1
        },
        {
            "props": {
                "cluster-id": 0,
                "core-id": 0,
                "die-id": 1,
                "socket-id": 0,
                "thread-id": 3
            },
            "qom-path": "/machine/peripheral/cpu-slot/sock0/die1/cluster2/core3/host-x86_64-cpu[9]",
            "type": "host-x86_64-cpu",
            "vcpus-count": 1
        },
        {
            "props": {
                "cluster-id": 0,
                "core-id": 0,
                "die-id": 1,
                "socket-id": 0,
                "thread-id": 2
            },
            "qom-path": "/machine/peripheral/cpu-slot/sock0/die1/cluster2/core3/host-x86_64-cpu[8]",
            "type": "host-x86_64-cpu",
            "vcpus-count": 1
        },
        {
            "props": {
                "cluster-id": 0,
                "core-id": 0,
                "die-id": 1,
                "socket-id": 0,
                "thread-id": 1
            },
            "qom-path": "/machine/peripheral/cpu-slot/sock0/die1/cluster2/core3/host-x86_64-cpu[7]",
            "type": "host-x86_64-cpu",
            "vcpus-count": 1
        },
        {
            "props": {
                "cluster-id": 0,
                "core-id": 0,
                "die-id": 1,
                "socket-id": 0,
                "thread-id": 0
            },
            "qom-path": "/machine/peripheral/cpu-slot/sock0/die1/cluster2/core3/host-x86_64-cpu[6]",
            "type": "host-x86_64-cpu",
            "vcpus-count": 1
        },
        {
            "props": {
                "cluster-id": 1,
                "core-id": 0,
                "die-id": 0,
                "socket-id": 0,
                "thread-id": 0
            },
            "qom-path": "/machine/peripheral/cpu-slot/sock0/die0/cluster1/core2/host-x86_64-cpu[5]",
            "type": "host-x86_64-cpu",
            "vcpus-count": 1
        },
        {
            "props": {
                "cluster-id": 0,
                "core-id": 1,
                "die-id": 0,
                "socket-id": 0,
                "thread-id": 1
            },
            "qom-path": "/machine/peripheral/cpu-slot/sock0/die0/cluster0/core1/host-x86_64-cpu[4]",
            "type": "host-x86_64-cpu",
            "vcpus-count": 1
        },
        {
            "props": {
                "cluster-id": 0,
                "core-id": 1,
                "die-id": 0,
                "socket-id": 0,
                "thread-id": 0
            },
            "qom-path": "/machine/peripheral/cpu-slot/sock0/die0/cluster0/core1/host-x86_64-cpu[3]",
            "type": "host-x86_64-cpu",
            "vcpus-count": 1
        },
        {
            "props": {
                "cluster-id": 0,
                "core-id": 0,
                "die-id": 0,
                "socket-id": 0,
                "thread-id": 2
            },
            "qom-path": "/machine/peripheral/cpu-slot/sock0/die0/cluster0/core0/host-x86_64-cpu[2]",
            "type": "host-x86_64-cpu",
            "vcpus-count": 1
        },
        {
            "props": {
                "cluster-id": 0,
                "core-id": 0,
                "die-id": 0,
                "socket-id": 0,
                "thread-id": 1
            },
            "qom-path": "/machine/peripheral/cpu-slot/sock0/die0/cluster0/core0/host-x86_64-cpu[1]",
            "type": "host-x86_64-cpu",
            "vcpus-count": 1
        },
        {
            "props": {
                "cluster-id": 0,
                "core-id": 0,
                "die-id": 0,
                "socket-id": 0,
                "thread-id": 0
            },
            "qom-path": "/machine/peripheral/cpu-slot/sock0/die0/cluster0/core0/host-x86_64-cpu[0]",
            "type": "host-x86_64-cpu",
            "vcpus-count": 1
        }
    ]
}

Of course, I'm still a bit far from the full QOM topology support, but
would like to hear comments at the POC stage as well! :-)

Thanks and BR,
Zhao

> 
> This example only has 20 CPUs. For a VM with say 1000 CPUs
> this will be very big, but that doesn't neccesarily make it
> wrong.
> 
> On the flipside
> 
>  * It is really clear exactly how many CPUs I've added
> 
>  * The relationship between the topology levels is clear
> 
>  * Every CPU has a unique ID given that can be used in
>    later QMP commands
> 
>  * Whether or not 'threads' are permitted is now a property
>    of the specific CPU model implementation, not the global
>    config. IOW we can express that some CPU models allowing
>    for threads, and some don't.
> 
>  * The -cpu arg is also obsoleted, replaced by the
>    -object x86-cpu-model-core. This might facilitate the
>    modelling of machines with CPUs from different architectures.
> 
> 
> We could potentially compress the leaf node level by expressing
> how many instances of an object we want. it we want. ie, define
> a more convenient shorthand syntax to creating many instances of
> an object. so eg
> 
>   -object-set $TYPE,$PROPS,idbase=foo,count=4
> 
> would be functionally identical to
> 
>   -object $TYPE,$PROPS,id=foo.0
>   -object $TYPE,$PROPS,id=foo.1
>   -object $TYPE,$PROPS,id=foo.2
>   -object $TYPE,$PROPS,id=foo.3
> 
> QEMU just expands it and creates all the objects internally.
> 
> So the huge example I have above for 20 cpus would become much
> shorter: e.g.
> 
>   -object cpu-socket,id=sock0
>   -object cpu-die,id=die0,parent=sock0
>   -object cpu-cluster,id=cluster0,parent=die0
>   -object cpu-cluster,id=cluster1,parent=die0
>   -object cpu-cluster,id=cluster2,parent=die0
>   -object cpu-cluster,id=cluster3,parent=die0
>   -object-set x86-cpu-core-atom,idbase=cpu0,parent=cluster0,count=4
>   -object-set x86-cpu-core-core,id=cpu1,parent=cluster0,threads=2,count=1
>   -object-set x86-cpu-core-atom,idbase=cpu2,parent=cluster1,count=4
>   -object-set x86-cpu-core-core,id=cpu3,parent=cluster1,threads=2,count=1
>   -object-set x86-cpu-core-atom,idbase=cpu4,parent=cluster2,count=4
>   -object-set x86-cpu-core-core,id=cpu5,parent=cluster2,threads=2,count=1
>   -object-set x86-cpu-core-atom,idbase=cpu6,parent=cluster3,count=4
>   -object-set x86-cpu-core-core,id=cpu7,parent=cluster3,threads=2,count=1
> 
> IOW, the size of the CLI config only depends on the number of elements
> in the hierarchy, and is independant of the number of leaf CPU cores.
> 
> Obviously in describing all of the above, I've ignored any complexity
> of dealing with our existing code implementation and pain of getting
> it converted to the new model.
> 
> With regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 

On Mon, Feb 13, 2023 at 01:38:28PM +0000, Daniel P. Berrangé wrote:
> Date: Mon, 13 Feb 2023 13:38:28 +0000
> From: "Daniel P. Berrangé" <berrange@redhat.com>
> Subject: Re: [RFC 00/52] Introduce hybrid CPU topology
> 
> On Mon, Feb 13, 2023 at 05:49:43PM +0800, Zhao Liu wrote:
> > From: Zhao Liu <zhao1.liu@intel.com>
> > ## 3.3. "-hybrid" command
> > 
> > For hybrid cpu topology configuration, the original "-smp" lack of
> > flexibility to expand, and unables to customize different cores.
> > 
> > So we introduce "-hybrid" command and implement it as the multi-
> > line command. The multi-line command format is more complex than the
> > single-line one, but it can bring stronger scalability and
> > intuitiveness. In the future, it can also be easily extended to more
> > heterogeneous topology levels.
> > 
> > "-hybrid" command is as follows:
> > 
> > -hybrid socket,sockets=n
> > -hybrid die,dies=n
> > -hybrid cluster,clusters=n
> > -hybrid core,cores=n,type=core_type[,threads=threads]
> >         [,clusterid=cluster]
> > 
> > Here, we first define the corresponding qapi options for these 4
> > topology levels: core, cluster, die and socket.
> > 
> > We doesn't need a thread level since thread doesn't have different
> > type.
> > 
> > For example:
> > 
> > -hybrid socket,sockets=1
> > -hybrid die,dies=1
> > -hybrid cluster,clusters=4
> > -hybrid core,cores=1,coretype="core",threads=2,clusterid=0-2
> > -hybrid core,cores=4,coretype="atom",threads=1
> > 
> > Here we can build a hybrid cpu topology, which has 1 socket, 1 die per
> > socket, 4 clusters per die. And in each die, every clusters has 4 "atom"
> > core with 1 threads, and cluster0, cluster1 and cluster2 have 1 "core"
> > cores with 2 threads.
> 
> How will this interact with the -cpu  parameter. Presumably we now
> need two distinct -cpu parameters, one for the 'core' CPU model and
> one for the 'atom' CPU model ?

(About this explanation for core type, also CC Philippe again)

For intel's hybrid, the core type is not a specific cpu model, and it
refers CPUID.1AH, sounds like a flag to distinguish between "core" and
"atom". For example, Alder Lake (not defined in QEMU yet) is a CPU model,
the Alder Lake CPU can be "core" or "atom".

Although the real "core" and "atom" have different microarchitectures,
they belong to the same CPU model and have the same ISA, and the
difference that the software layer can recognize is whether the encode
in CPUID.1AH is "core" type or "atom" type.

I think this is also the difference between intel hybrid and other
heterogeneous architectures. IIUC the heterogeneity of arm contains
different CPU models, while intel hybrid only marks the cpu as "core"
or "atom" in the same CPU model.

I think this difference between intel hybrid and other heterogeneous can
be distinguished by different handling methods of mc->core_type() which
can be defined by other arch to check "coretype" field [1].

For x86, mc->core_type() asks "coretype" as "core" or "atom" and treat
it as a hybrid "flag". But for other arch, who contains different CPU
models in heterogeneous (maybe arm), its mc->coretype asks for the
correct cpu model.

In this sense, x86 does not need a different "-cpu" to correspond to
core/atom, and even "-cpu" can be set to "host" (ie, -cpu host). But
other arch, who contains different CPU models in heterogeneous, needs
to support distinct -cpu parameters.

Do you think it is appropriate to use arch-specific mc->coretype to
handle different requirements for CPU model/core type?

[1]: [RFC 41/52] machine: Introduce core_type() hook,
     https://lists.gnu.org/archive/html/qemu-devel/2023-02/msg03244.html

> 
> > Please note this example is not an actual machine topology, but it shows
> > the powerful flexibility of "hybrid" command.
> 
> IIUC the functionality offered by -hybrid should be a superset
> of the -smp functionality. IOW, -smp ought to be possible to
> re-implement -smp as an alias for -hybrid, such that internally
> code only ever has to deal with the modern approach.

Yes, I agree.

> Having to
> keep support for both -smp and -hybrid throughout the code is
> undesirable IMHO. Keeping the compat at the CLI parsing level
> limits the burden.
>


"-hybrid" is introduced as a completely different implementation from
"-smp", for these 2 reasons:

1. Single-line commands can be very complex, and it is more difficult
to identify the relationship between different levels.

2. smp is implemented as the property of machine [2] and I don't have a
good way to define "hybrid" also as a property since I need to use
opts_visitor for multi-line commands instead of input_visitor.

[2]: fe68090: machine: add smp compound property.

> 
> As a more general thought, rather than introducing a new top level
> command line argument -hybrid, I'm thinking we should possibly just
> define this all using QOM and thus re-use the existing -object
> argument. 
> 
> I'm also finding the above example command lines quite difficult
> to understand, as there is alot of implicit linkage and expansion
> between the different levels. With devices we're much more
> explicitly with the parent/child relationships, and have to
> express everything with no automatic expansion, linking it all
> together via the id=/bus= properties.  This is quite a bit more
> verbose, but it is also very effective at letting us express
> arbitrarily complex relationships.
> 
> I think it would be worth exploring that approach for the CPU
> topology expression too.
> 
> If we followed the more explicit device approach to modelling
> then instead of:
> 
>  -cpu core,...
>  -cpu atom,...
>  -hybrid socket,sockets=1
>  -hybrid die,dies=1
>  -hybrid cluster,clusters=4
>  -hybrid core,cores=1,coretype="core",threads=2,clusterid=0-2
>  -hybrid core,cores=4,coretype="atom",threads=1
> 
> we would end up with something like
> 
>   -object cpu-socket,id=sock0
>   -object cpu-die,id=die0,parent=sock0
>   -object cpu-cluster,id=cluster0,parent=die0
>   -object cpu-cluster,id=cluster1,parent=die0
>   -object cpu-cluster,id=cluster2,parent=die0
>   -object cpu-cluster,id=cluster3,parent=die0
>   -object x86-cpu-model-atom,id=cpu0,parent=cluster0
>   -object x86-cpu-model-atom,id=cpu1,parent=cluster0
>   -object x86-cpu-model-atom,id=cpu2,parent=cluster0
>   -object x86-cpu-model-atom,id=cpu3,parent=cluster0
>   -object x86-cpu-model-core,id=cpu4,parent=cluster0,threads=2
>   -object x86-cpu-model-atom,id=cpu5,parent=cluster1
>   -object x86-cpu-model-atom,id=cpu6,parent=cluster1
>   -object x86-cpu-model-atom,id=cpu7,parent=cluster1
>   -object x86-cpu-model-atom,id=cpu8,parent=cluster1
>   -object x86-cpu-model-core,id=cpu9,parent=cluster1,threads=2
>   -object x86-cpu-model-atom,id=cpu10,parent=cluster2
>   -object x86-cpu-model-atom,id=cpu11,parent=cluster2
>   -object x86-cpu-model-atom,id=cpu12,parent=cluster2
>   -object x86-cpu-model-atom,id=cpu13,parent=cluster2
>   -object x86-cpu-model-core,id=cpu14,parent=cluster2,threads=2
>   -object x86-cpu-model-atom,id=cpu15,parent=cluster3
>   -object x86-cpu-model-atom,id=cpu16,parent=cluster3
>   -object x86-cpu-model-atom,id=cpu17,parent=cluster3
>   -object x86-cpu-model-atom,id=cpu18,parent=cluster3
>   -object x86-cpu-model-core,id=cpu19,parent=cluster3,threads=2


Given that x86 hybrid currently has the same cpu model and "core"/"atom"
is not CPU model, so the -object still needs a coretype fields to specify
"core" or "atom", right?

Given an example of "-cpu host":

-cpu host,...
-hybrid socket,sockets=1
-hybrid die,dies=1
-hybrid cluster,clusters=4
-hybrid core,cores=1,coretype="core",threads=2,clusterid=0-2
-hybrid core,cores=4,coretype="atom",threads=1


It occurred to me that there may be two conversion formats for qom.

The one with "coretype" field like:

-object x86-cpu-model-host,coretype="atom",id=cpu0,parent=cluster0
-object x86-cpu-model-host,coretype="core",id=cpu1,parent=cluster0

or another format like "x86-cpu-model-[real model name]-[core type]":

-object x86-cpu-model-host-atom,id=cpu0,parent=cluster0
-object x86-cpu-model-host-core,id=cpu1,parent=cluster0

Can qom support the parsing of the second naming format?

At the same time, I understand that for other architectures, there is
no need to consider the issue of core type, just specify different
heterogeneous CPU models.

> 
> The really obvious downside is that it is much more verbose.
> 
> This example only has 20 CPUs. For a VM with say 1000 CPUs
> this will be very big, but that doesn't neccesarily make it
> wrong.
> 
> On the flipside
> 
>  * It is really clear exactly how many CPUs I've added
> 
>  * The relationship between the topology levels is clear
> 
>  * Every CPU has a unique ID given that can be used in
>    later QMP commands
> 
>  * Whether or not 'threads' are permitted is now a property
>    of the specific CPU model implementation, not the global
>    config. IOW we can express that some CPU models allowing
>    for threads, and some don't.

Will this way break the smp property of machine (the above mentioned [2])?

If so, IIUC we still need keep smp code to support this smp format:

"-machine smp.{cores,threads,cpu,...}",

but we can mark this smp format as deprecated and at some point completely
remove the smp code and replace it with the qom implementation, right?

> 
>  * The -cpu arg is also obsoleted, replaced by the
>    -object x86-cpu-model-core. This might facilitate the
>    modelling of machines with CPUs from different architectures.
> 
> 
> We could potentially compress the leaf node level by expressing
> how many instances of an object we want. it we want. ie, define
> a more convenient shorthand syntax to creating many instances of
> an object. so eg
> 
>   -object-set $TYPE,$PROPS,idbase=foo,count=4
> 
> would be functionally identical to
> 
>   -object $TYPE,$PROPS,id=foo.0
>   -object $TYPE,$PROPS,id=foo.1
>   -object $TYPE,$PROPS,id=foo.2
>   -object $TYPE,$PROPS,id=foo.3
> 
> QEMU just expands it and creates all the objects internally.
> 
> So the huge example I have above for 20 cpus would become much
> shorter: e.g.
> 
>   -object cpu-socket,id=sock0
>   -object cpu-die,id=die0,parent=sock0
>   -object cpu-cluster,id=cluster0,parent=die0
>   -object cpu-cluster,id=cluster1,parent=die0
>   -object cpu-cluster,id=cluster2,parent=die0
>   -object cpu-cluster,id=cluster3,parent=die0
>   -object-set x86-cpu-core-atom,idbase=cpu0,parent=cluster0,count=4
>   -object-set x86-cpu-core-core,id=cpu1,parent=cluster0,threads=2,count=1
>   -object-set x86-cpu-core-atom,idbase=cpu2,parent=cluster1,count=4
>   -object-set x86-cpu-core-core,id=cpu3,parent=cluster1,threads=2,count=1
>   -object-set x86-cpu-core-atom,idbase=cpu4,parent=cluster2,count=4
>   -object-set x86-cpu-core-core,id=cpu5,parent=cluster2,threads=2,count=1
>   -object-set x86-cpu-core-atom,idbase=cpu6,parent=cluster3,count=4
>   -object-set x86-cpu-core-core,id=cpu7,parent=cluster3,threads=2,count=1

What's difference between "id" and "idbase"?

> 
> IOW, the size of the CLI config only depends on the number of elements
> in the hierarchy, and is independant of the number of leaf CPU cores.
> 
> Obviously in describing all of the above, I've ignored any complexity
> of dealing with our existing code implementation and pain of getting
> it converted to the new model.

Thanks and sounds cool! I think I need some time to study this way.


Zhao

> 
> With regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>