From nobody Mon Feb 9 19:30:24 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) client-ip=170.10.129.124; envelope-from=philmd@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=philmd@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1640954940; cv=none; d=zohomail.com; s=zohoarc; b=i2EqppNqw3RPkJY8bW6eKtFugoao9Mtqtpxbl929h0UhEfep5U7WZxoIonbs6Zc0GJ0taVy6bivaXkmvmegJtiOwn5VqlsLz9M2+vmeL7IDzeK1MkGk40fPr1Zwdho+Px1S+aakBmmBS2Trm/Bp3I4y2gJjCXRVFhyZO8Z1LF8U= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1640954940; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=wRGnTZ/XLILKmWFtYsMNltRv2S3GsR7Kd42iwaxNPRc=; b=SRG/SnTzgVcIriIr+yqQwXMWUecEyOkdas43oXEDDS1YwhZvvDQbnZ3vNT+1rxNap3WAFnpkmkBMZg6W06fptHNo+u2t2+8FMa9NThblLe2kVoSbxn+1dCQebWYEDSPxdTCipUWsDnv0xZYhsC72qY/87z9RzqQYYC1lciFwy40= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=philmd@redhat.com; dmarc=pass header.from= (p=none dis=none) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mx.zohomail.com with SMTPS id 1640954940168975.4577933930763; Fri, 31 Dec 2021 04:49:00 -0800 (PST) Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-527-abNR2s1XP5SX-xv8P7OM2A-1; Fri, 31 Dec 2021 07:48:58 -0500 Received: by mail-wr1-f72.google.com with SMTP id t30-20020adfa2de000000b001a24004e1fbso7515265wra.10 for ; Fri, 31 Dec 2021 04:48:57 -0800 (PST) Return-Path: Return-Path: Received: from x1w.redhat.com (26.red-83-50-86.dynamicip.rima-tde.net. [83.50.86.26]) by smtp.gmail.com with ESMTPSA id ay29sm28438051wmb.13.2021.12.31.04.48.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 31 Dec 2021 04:48:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1640954939; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wRGnTZ/XLILKmWFtYsMNltRv2S3GsR7Kd42iwaxNPRc=; b=c3TQtHBXu6D0QCs+Zu+cNknhg1E8kDeO71BS5pm5tI1gdNYlwCMA10Vht37wpN/v9734aI OgHb5YyAvv7ps5J4+nkfUoPOj1KcNRDSKm6OlOHmuzrRZQh/MyvNoLaprvD+4nIrgdMsMO sVjQQLHa2YTg0LPjeDjuS0IIkRI2qcs= X-MC-Unique: abNR2s1XP5SX-xv8P7OM2A-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wRGnTZ/XLILKmWFtYsMNltRv2S3GsR7Kd42iwaxNPRc=; b=xw6sbuvIWaALCzzAoVobwyBXCku0dA82Cjou6rvbwGRSblpczZZLT7w+/njCZCmDvX PpxZ520DnF+q0xQwFm5wAtI3H++1Lrb51+clZwZimQ+cFDYSxQ+aP2Mt4dCCMzjyXa6n 2jNts4iDVG6jeirY2pBa5a+HiGcYaE4y+rKJdCKtz5SJ2pT4svtLyVYR9jFOOs0XCtqB MCqBA0aSzHWPO2IXJXn1eVggBKRjPuFEsZ6LgOYrnBa4m9ArxqYn1qbt6C9YLxXEozQ9 sJs0PBJJbjGSFaiA2xlwDuUe2lTKLzUSTJZo8Kofwt1wfvXQFM1h76u8U1h+gssP8N8b rQWQ== X-Gm-Message-State: AOAM530BTV8HtKF0Uk4Z+1xf1Jjw0fN5xe1nXNj/vyS8HKwrJ+TAHypB BMIpE3n/x689KfmcKDwl+NNEMyxw28blz7W7k39b2GSQFevTTNcjA1+pd2n0Fnv/TtXs4X+gH9Y aQZrmBQwEs1BHlA== X-Received: by 2002:a05:6000:2c9:: with SMTP id o9mr29313038wry.377.1640954936851; Fri, 31 Dec 2021 04:48:56 -0800 (PST) X-Google-Smtp-Source: ABdhPJzOX7ujFYtjAJ/WubxHtWh4q+rNs2hLoB/xDUv76yYDBP3mJiM8jtRG3xb7jDx5xYvvh1ziHg== X-Received: by 2002:a05:6000:2c9:: with SMTP id o9mr29313027wry.377.1640954936582; Fri, 31 Dec 2021 04:48:56 -0800 (PST) From: =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= To: qemu-devel@nongnu.org Cc: Yanan Wang , Eduardo Habkost , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= Subject: [PULL 14/20] hw/core/machine: Introduce CPU cluster topology support Date: Fri, 31 Dec 2021 13:47:48 +0100 Message-Id: <20211231124754.1005747-15-philmd@redhat.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211231124754.1005747-1-philmd@redhat.com> References: <20211231124754.1005747-1-philmd@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=philmd@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1640954941764000001 From: Yanan Wang The new Cluster-Aware Scheduling support has landed in Linux 5.16, which has been proved to benefit the scheduling performance (e.g. load balance and wake_affine strategy) on both x86_64 and AArch64. So now in Linux 5.16 we have four-level arch-neutral CPU topology definition like below and a new scheduler level for clusters. struct cpu_topology { int thread_id; int core_id; int cluster_id; int package_id; int llc_id; cpumask_t thread_sibling; cpumask_t core_sibling; cpumask_t cluster_sibling; cpumask_t llc_sibling; } A cluster generally means a group of CPU cores which share L2 cache or other mid-level resources, and it is the shared resources that is used to improve scheduler's behavior. From the point of view of the size range, it's between CPU die and CPU core. For example, on some ARM64 Kunpeng servers, we have 6 clusters in each NUMA node, and 4 CPU cores in each cluster. The 4 CPU cores share a separate L2 cache and a L3 cache tag, which brings cache affinity advantage. In virtualization, on the Hosts which have pClusters (physical clusters), if we can design a vCPU topology with cluster level for guest kernel and have a dedicated vCPU pinning. A Cluster-Aware Guest kernel can also make use of the cache affinity of CPU clusters to gain similar scheduling performance. This patch adds infrastructure for CPU cluster level topology configuration and parsing, so that the user can specify cluster parameter if their machines support it. Signed-off-by: Yanan Wang Message-Id: <20211228092221.21068-3-wangyanan55@huawei.com> Reviewed-by: Philippe Mathieu-Daud=C3=A9 [PMD: Added '(since 7.0)' to @clusters in qapi/machine.json] Signed-off-by: Philippe Mathieu-Daud=C3=A9 --- qapi/machine.json | 5 ++++- include/hw/boards.h | 6 +++++- hw/core/machine-smp.c | 26 +++++++++++++++++++------- hw/core/machine.c | 3 +++ softmmu/vl.c | 3 +++ qemu-options.hx | 7 ++++--- 6 files changed, 38 insertions(+), 12 deletions(-) diff --git a/qapi/machine.json b/qapi/machine.json index edeab6084b7..372535b3482 100644 --- a/qapi/machine.json +++ b/qapi/machine.json @@ -1404,7 +1404,9 @@ # # @dies: number of dies per socket in the CPU topology # -# @cores: number of cores per die in the CPU topology +# @clusters: number of clusters per die in the CPU topology (since 7.0) +# +# @cores: number of cores per cluster in the CPU topology # # @threads: number of threads per core in the CPU topology # @@ -1416,6 +1418,7 @@ '*cpus': 'int', '*sockets': 'int', '*dies': 'int', + '*clusters': 'int', '*cores': 'int', '*threads': 'int', '*maxcpus': 'int' } } diff --git a/include/hw/boards.h b/include/hw/boards.h index 7597cec4400..f49a2578ead 100644 --- a/include/hw/boards.h +++ b/include/hw/boards.h @@ -129,10 +129,12 @@ typedef struct { * SMPCompatProps: * @prefer_sockets - whether sockets are preferred over cores in smp parsi= ng * @dies_supported - whether dies are supported by the machine + * @clusters_supported - whether clusters are supported by the machine */ typedef struct { bool prefer_sockets; bool dies_supported; + bool clusters_supported; } SMPCompatProps; =20 /** @@ -299,7 +301,8 @@ typedef struct DeviceMemoryState { * @cpus: the number of present logical processors on the machine * @sockets: the number of sockets on the machine * @dies: the number of dies in one socket - * @cores: the number of cores in one die + * @clusters: the number of clusters in one die + * @cores: the number of cores in one cluster * @threads: the number of threads in one core * @max_cpus: the maximum number of logical processors on the machine */ @@ -307,6 +310,7 @@ typedef struct CpuTopology { unsigned int cpus; unsigned int sockets; unsigned int dies; + unsigned int clusters; unsigned int cores; unsigned int threads; unsigned int max_cpus; diff --git a/hw/core/machine-smp.c b/hw/core/machine-smp.c index 2cbfd574293..b39ed21e654 100644 --- a/hw/core/machine-smp.c +++ b/hw/core/machine-smp.c @@ -37,6 +37,10 @@ static char *cpu_hierarchy_to_string(MachineState *ms) g_string_append_printf(s, " * dies (%u)", ms->smp.dies); } =20 + if (mc->smp_props.clusters_supported) { + g_string_append_printf(s, " * clusters (%u)", ms->smp.clusters); + } + g_string_append_printf(s, " * cores (%u)", ms->smp.cores); g_string_append_printf(s, " * threads (%u)", ms->smp.threads); =20 @@ -71,6 +75,7 @@ void machine_parse_smp_config(MachineState *ms, unsigned cpus =3D config->has_cpus ? config->cpus : 0; unsigned sockets =3D config->has_sockets ? config->sockets : 0; unsigned dies =3D config->has_dies ? config->dies : 0; + unsigned clusters =3D config->has_clusters ? config->clusters : 0; unsigned cores =3D config->has_cores ? config->cores : 0; unsigned threads =3D config->has_threads ? config->threads : 0; unsigned maxcpus =3D config->has_maxcpus ? config->maxcpus : 0; @@ -82,6 +87,7 @@ void machine_parse_smp_config(MachineState *ms, if ((config->has_cpus && config->cpus =3D=3D 0) || (config->has_sockets && config->sockets =3D=3D 0) || (config->has_dies && config->dies =3D=3D 0) || + (config->has_clusters && config->clusters =3D=3D 0) || (config->has_cores && config->cores =3D=3D 0) || (config->has_threads && config->threads =3D=3D 0) || (config->has_maxcpus && config->maxcpus =3D=3D 0)) { @@ -97,8 +103,13 @@ void machine_parse_smp_config(MachineState *ms, error_setg(errp, "dies not supported by this machine's CPU topolog= y"); return; } + if (!mc->smp_props.clusters_supported && clusters > 1) { + error_setg(errp, "clusters not supported by this machine's CPU top= ology"); + return; + } =20 dies =3D dies > 0 ? dies : 1; + clusters =3D clusters > 0 ? clusters : 1; =20 /* compute missing values based on the provided ones */ if (cpus =3D=3D 0 && maxcpus =3D=3D 0) { @@ -113,41 +124,42 @@ void machine_parse_smp_config(MachineState *ms, if (sockets =3D=3D 0) { cores =3D cores > 0 ? cores : 1; threads =3D threads > 0 ? threads : 1; - sockets =3D maxcpus / (dies * cores * threads); + sockets =3D maxcpus / (dies * clusters * cores * threads); } else if (cores =3D=3D 0) { threads =3D threads > 0 ? threads : 1; - cores =3D maxcpus / (sockets * dies * threads); + cores =3D maxcpus / (sockets * dies * clusters * threads); } } else { /* prefer cores over sockets since 6.2 */ if (cores =3D=3D 0) { sockets =3D sockets > 0 ? sockets : 1; threads =3D threads > 0 ? threads : 1; - cores =3D maxcpus / (sockets * dies * threads); + cores =3D maxcpus / (sockets * dies * clusters * threads); } else if (sockets =3D=3D 0) { threads =3D threads > 0 ? threads : 1; - sockets =3D maxcpus / (dies * cores * threads); + sockets =3D maxcpus / (dies * clusters * cores * threads); } } =20 /* try to calculate omitted threads at last */ if (threads =3D=3D 0) { - threads =3D maxcpus / (sockets * dies * cores); + threads =3D maxcpus / (sockets * dies * clusters * cores); } } =20 - maxcpus =3D maxcpus > 0 ? maxcpus : sockets * dies * cores * threads; + maxcpus =3D maxcpus > 0 ? maxcpus : sockets * dies * clusters * cores = * threads; cpus =3D cpus > 0 ? cpus : maxcpus; =20 ms->smp.cpus =3D cpus; ms->smp.sockets =3D sockets; ms->smp.dies =3D dies; + ms->smp.clusters =3D clusters; ms->smp.cores =3D cores; ms->smp.threads =3D threads; ms->smp.max_cpus =3D maxcpus; =20 /* sanity-check of the computed topology */ - if (sockets * dies * cores * threads !=3D maxcpus) { + if (sockets * dies * clusters * cores * threads !=3D maxcpus) { g_autofree char *topo_msg =3D cpu_hierarchy_to_string(ms); error_setg(errp, "Invalid CPU topology: " "product of the hierarchy must match maxcpus: " diff --git a/hw/core/machine.c b/hw/core/machine.c index 3993c534b90..a4a2df405f3 100644 --- a/hw/core/machine.c +++ b/hw/core/machine.c @@ -742,10 +742,12 @@ static void machine_get_smp(Object *obj, Visitor *v, = const char *name, .has_cpus =3D true, .cpus =3D ms->smp.cpus, .has_sockets =3D true, .sockets =3D ms->smp.sockets, .has_dies =3D true, .dies =3D ms->smp.dies, + .has_clusters =3D true, .clusters =3D ms->smp.clusters, .has_cores =3D true, .cores =3D ms->smp.cores, .has_threads =3D true, .threads =3D ms->smp.threads, .has_maxcpus =3D true, .maxcpus =3D ms->smp.max_cpus, }; + if (!visit_type_SMPConfiguration(v, name, &config, &error_abort)) { return; } @@ -932,6 +934,7 @@ static void machine_initfn(Object *obj) ms->smp.max_cpus =3D mc->default_cpus; ms->smp.sockets =3D 1; ms->smp.dies =3D 1; + ms->smp.clusters =3D 1; ms->smp.cores =3D 1; ms->smp.threads =3D 1; } diff --git a/softmmu/vl.c b/softmmu/vl.c index 620a1f1367e..d9e4c619d3e 100644 --- a/softmmu/vl.c +++ b/softmmu/vl.c @@ -726,6 +726,9 @@ static QemuOptsList qemu_smp_opts =3D { }, { .name =3D "dies", .type =3D QEMU_OPT_NUMBER, + }, { + .name =3D "clusters", + .type =3D QEMU_OPT_NUMBER, }, { .name =3D "cores", .type =3D QEMU_OPT_NUMBER, diff --git a/qemu-options.hx b/qemu-options.hx index b39377de3fc..fd1f8135fb5 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -206,13 +206,14 @@ SRST ERST =20 DEF("smp", HAS_ARG, QEMU_OPTION_smp, - "-smp [[cpus=3D]n][,maxcpus=3Dmaxcpus][,sockets=3Dsockets][,dies=3Ddie= s][,cores=3Dcores][,threads=3Dthreads]\n" + "-smp [[cpus=3D]n][,maxcpus=3Dmaxcpus][,sockets=3Dsockets][,dies=3Ddie= s][,clusters=3Dclusters][,cores=3Dcores][,threads=3Dthreads]\n" " set the number of initial CPUs to 'n' [default=3D1]\n" " maxcpus=3D maximum number of total CPUs, including\n" " offline CPUs for hotplug, etc\n" " sockets=3D number of sockets on the machine board\n" " dies=3D number of dies in one socket\n" - " cores=3D number of cores in one die\n" + " clusters=3D number of clusters in one die\n" + " cores=3D number of cores in one cluster\n" " threads=3D number of threads in one core\n" "Note: Different machines may have different subsets of the CPU topolo= gy\n" " parameters supported, so the actual meaning of the supported pa= rameters\n" @@ -228,7 +229,7 @@ DEF("smp", HAS_ARG, QEMU_OPTION_smp, " must be set as 1 in the purpose of correct parsing.\n", QEMU_ARCH_ALL) SRST -``-smp [[cpus=3D]n][,maxcpus=3Dmaxcpus][,sockets=3Dsockets][,dies=3Ddies][= ,cores=3Dcores][,threads=3Dthreads]`` +``-smp [[cpus=3D]n][,maxcpus=3Dmaxcpus][,sockets=3Dsockets][,dies=3Ddies][= ,clusters=3Dclusters][,cores=3Dcores][,threads=3Dthreads]`` Simulate a SMP system with '\ ``n``\ ' CPUs initially present on the machine type board. On boards supporting CPU hotplug, the optional '\ ``maxcpus``\ ' parameter can be set to enable further CPUs to be --=20 2.33.1