From nobody Sun Feb 8 14:07:28 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 207.211.31.120 as permitted sender) client-ip=207.211.31.120; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-1.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 207.211.31.120 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1593006594; cv=none; d=zohomail.com; s=zohoarc; b=X3TNkjJcnr1z5bvV7J926MMaNOFxTH0S76HA9xP8FrLylrYvOePiEgfeusYOjmWW1vZfc3AHH/2Kae+j+hJKLz/ssb6ztta0FQwVBNBqT9SwSaLOwGlNqO1J/NOxjIFyMcqHbYy/TWGMWkZ93fhNOZ4/l+Gl9VYysMQDEOn5y68= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1593006594; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=qmgXAnts+qY55XYCEOtPO4GR+MWWpQCzU635YFmW4zw=; b=T8u8UPqUo1TFIpSCxdJjlFL/weqFtELFLEPdQca0m8hPR6OCm6hgMUszLit8oT6WJ1EFGqqe9YaoDfCM2nPXrtUONST/J5b7z0NyJ2/Ft/NL5fAU6lhJ5WITEimsorg4aKtJLmyX3KbP+8VKYJcV64b6HvNcHha/FRfpeAvcIAQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 207.211.31.120 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) by mx.zohomail.com with SMTPS id 1593006594848685.1752508953771; Wed, 24 Jun 2020 06:49:54 -0700 (PDT) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-77-1jugvUbuOfmceudOxFZwlA-1; Wed, 24 Jun 2020 09:49:49 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C67DC107ACF5; Wed, 24 Jun 2020 13:49:40 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A423F2B47E; Wed, 24 Jun 2020 13:49:40 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 6C57D1806B0D; Wed, 24 Jun 2020 13:49:40 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 05ODnRdG015734 for ; Wed, 24 Jun 2020 09:49:27 -0400 Received: by smtp.corp.redhat.com (Postfix) id 384115DA27; Wed, 24 Jun 2020 13:49:27 +0000 (UTC) Received: from localhost.localdomain (unknown [10.40.195.7]) by smtp.corp.redhat.com (Postfix) with ESMTP id AD1045D9DC for ; Wed, 24 Jun 2020 13:49:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1593006592; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=qmgXAnts+qY55XYCEOtPO4GR+MWWpQCzU635YFmW4zw=; b=Ark0DtW50ClGbaSIzcV4s3ZwEXUHV9QbWR0gHJNJ89eg698Akwzz7XLbKjfGivFYCMZkp0 6nGRfzUI3DvXeiK1450CP5JTuKThjj0/aXLaIcVvyWiIV2Rq2dyIZOWPsMNjGwpDcA4G+Z DGaOBNby7zKPXzvdmXFncITY6LszIdU= X-MC-Unique: 1jugvUbuOfmceudOxFZwlA-1 From: Michal Privoznik To: libvir-list@redhat.com Subject: [PATCH 12/13] qemu: Build HMAT command line Date: Wed, 24 Jun 2020 15:49:10 +0200 Message-Id: <22d7cec7fb17765bba75cbd787af03528e8460fe.1593006346.git.mprivozn@redhat.com> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=3D1786303 Signed-off-by: Michal Privoznik Reviewed-by: Daniel Henrique Barboza --- src/conf/numa_conf.c | 7 + src/qemu/qemu_command.c | 183 ++++++++++++++++++ .../numatune-hmat.x86_64-latest.args | 53 +++++ tests/qemuxml2argvtest.c | 1 + 4 files changed, 244 insertions(+) create mode 100644 tests/qemuxml2argvdata/numatune-hmat.x86_64-latest.args diff --git a/src/conf/numa_conf.c b/src/conf/numa_conf.c index b35e5040fc..7fbbe0f52e 100644 --- a/src/conf/numa_conf.c +++ b/src/conf/numa_conf.c @@ -1904,6 +1904,13 @@ virDomainNumaGetNodeInitiator(const virDomainNuma *n= uma, if (!numa || node >=3D numa->nmem_nodes) return -1; =20 + /* A NUMA node which has at least one vCPU is initiator to itself by + * definition. */ + if (numa->mem_nodes[node].cpumask) + return node; + + /* For the rest, "NUMA node that has best performance (the lowest + * latency or largest bandwidth) to this NUMA node." */ for (i =3D 0; i < numa->nlatencies; i++) { const virDomainNumaLatency *l =3D &numa->latencies[i]; =20 diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index 628ff970bd..b28c43f110 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -6932,6 +6932,16 @@ qemuBuildMachineCommandLine(virCommandPtr cmd, virBufferAsprintf(&buf, ",pflash1=3D%s", priv->pflash1->nodefo= rmat); } =20 + if (virDomainNumaHasHMAT(def->numa)) { + if (!virQEMUCapsGet(qemuCaps, QEMU_CAPS_NUMA_HMAT)) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("HMAT is not supported with this QEMU")); + return -1; + } + + virBufferAddLit(&buf, ",hmat=3Don"); + } + virCommandAddArgBuffer(cmd, &buf); =20 return 0; @@ -7114,6 +7124,134 @@ qemuBuildIOThreadCommandLine(virCommandPtr cmd, } =20 =20 +static int +qemuBuilNumaCellCache(virCommandPtr cmd, + const virDomainDef *def, + size_t cell) +{ + size_t ncaches =3D virDomainNumaGetNodeCacheCount(def->numa, cell); + size_t i; + + if (ncaches =3D=3D 0) + return 0; + + for (i =3D 0; i < ncaches; i++) { + g_auto(virBuffer) buf =3D VIR_BUFFER_INITIALIZER; + unsigned int level; + unsigned int size; + unsigned int line; + virDomainCacheAssociativity associativity; + virDomainCachePolicy policy; + + if (virDomainNumaGetNodeCache(def->numa, cell, i, + &level, &size, &line, + &associativity, &policy) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("Unable to format NUMA node cache")); + return -1; + } + + virBufferAsprintf(&buf, + "hmat-cache,node-id=3D%zu,size=3D%uK,level=3D%u", + cell, size, level); + + switch (associativity) { + case VIR_DOMAIN_CACHE_ASSOCIATIVITY_NONE: + virBufferAddLit(&buf, ",associativity=3Dnone"); + break; + case VIR_DOMAIN_CACHE_ASSOCIATIVITY_DIRECT: + virBufferAddLit(&buf, ",associativity=3Ddirect"); + break; + case VIR_DOMAIN_CACHE_ASSOCIATIVITY_FULL: + virBufferAddLit(&buf, ",associativity=3Dcomplex"); + break; + case VIR_DOMAIN_CACHE_ASSOCIATIVITY_LAST: + break; + } + + switch (policy) { + case VIR_DOMAIN_CACHE_POLICY_NONE: + virBufferAddLit(&buf, ",policy=3Dnone"); + break; + case VIR_DOMAIN_CACHE_POLICY_WRITEBACK: + virBufferAddLit(&buf, ",policy=3Dwrite-back"); + break; + case VIR_DOMAIN_CACHE_POLICY_WRITETHROUGH: + virBufferAddLit(&buf, ",policy=3Dwrite-through"); + break; + case VIR_DOMAIN_CACHE_POLICY_LAST: + break; + } + + if (line > 0) + virBufferAsprintf(&buf, ",line=3D%u", line); + + virCommandAddArg(cmd, "-numa"); + virCommandAddArgBuffer(cmd, &buf); + } + + return 0; +} + + +VIR_ENUM_DECL(qemuDomainMemoryHierarchy); +VIR_ENUM_IMPL(qemuDomainMemoryHierarchy, + 4, /* Maximum level of cache */ + "memory", /* Special case, whole memory not specific cache */ + "first-level", + "second-level", + "third-level"); + +static int +qemuBuildNumaHMATCommandLine(virCommandPtr cmd, + const virDomainDef *def) +{ + size_t nlatencies; + size_t i; + + if (!def->numa) + return 0; + + nlatencies =3D virDomainNumaGetLatenciesCount(def->numa); + for (i =3D 0; i < nlatencies; i++) { + g_auto(virBuffer) buf =3D VIR_BUFFER_INITIALIZER; + virDomainNumaLatencyType type; + unsigned int initiator; + unsigned int target; + unsigned int cache; + virDomainMemoryLatency accessType; + unsigned long value; + const char *hierarchyStr; + const char *accessStr; + + if (virDomainNumaGetLatency(def->numa, i, + &type, &initiator, &target, + &cache, &accessType, &value) < 0) + return -1; + + hierarchyStr =3D qemuDomainMemoryHierarchyTypeToString(cache); + accessStr =3D virDomainMemoryLatencyTypeToString(accessType); + virBufferAsprintf(&buf, + "hmat-lb,initiator=3D%u,target=3D%u,hierarchy=3D= %s,data-type=3D%s-", + initiator, target, hierarchyStr, accessStr); + + switch (type) { + case VIR_DOMAIN_NUMA_LATENCY_TYPE_LATENCY: + virBufferAsprintf(&buf, "latency,latency=3D%lu", value); + break; + case VIR_DOMAIN_NUMA_LATENCY_TYPE_BANDWIDTH: + virBufferAsprintf(&buf, "bandwidth,bandwidth=3D%luK", value); + break; + } + + virCommandAddArg(cmd, "-numa"); + virCommandAddArgBuffer(cmd, &buf); + } + + return 0; +} + + static int qemuBuildNumaCommandLine(virQEMUDriverConfigPtr cfg, virDomainDefPtr def, @@ -7126,9 +7264,11 @@ qemuBuildNumaCommandLine(virQEMUDriverConfigPtr cfg, char *next =3D NULL; virBufferPtr nodeBackends =3D NULL; bool needBackend =3D false; + bool hmat =3D false; int rc; int ret =3D -1; size_t ncells =3D virDomainNumaGetNodeCount(def->numa); + ssize_t masterInitiator =3D -1; =20 if (!virDomainNumatuneNodesetIsAvailable(def->numa, priv->autoNodeset)) goto cleanup; @@ -7138,6 +7278,16 @@ qemuBuildNumaCommandLine(virQEMUDriverConfigPtr cfg, def->os.machine)) needBackend =3D true; =20 + if (virDomainNumaHasHMAT(def->numa)) { + if (!virQEMUCapsGet(qemuCaps, QEMU_CAPS_NUMA_HMAT)) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("HMAT is not supported with this QEMU")); + goto cleanup; + } + needBackend =3D true; + hmat =3D true; + } + if (VIR_ALLOC_N(nodeBackends, ncells) < 0) goto cleanup; =20 @@ -7166,8 +7316,22 @@ qemuBuildNumaCommandLine(virQEMUDriverConfigPtr cfg, qemuBuildMemPathStr(def, cmd, priv) < 0) goto cleanup; =20 + for (i =3D 0; i < ncells; i++) { + if (virDomainNumaGetNodeCpumask(def->numa, i)) { + masterInitiator =3D i; + break; + } + } + + if (masterInitiator) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("At least one NUMA node has to have CPUs")); + goto cleanup; + } + for (i =3D 0; i < ncells; i++) { virBitmapPtr cpumask =3D virDomainNumaGetNodeCpumask(def->numa, i); + ssize_t initiator =3D virDomainNumaGetNodeInitiator(def->numa, i); =20 if (needBackend) { virCommandAddArg(cmd, "-object"); @@ -7192,6 +7356,13 @@ qemuBuildNumaCommandLine(virQEMUDriverConfigPtr cfg, } } =20 + if (hmat) { + if (initiator < 0) + initiator =3D masterInitiator; + + virBufferAsprintf(&buf, ",initiator=3D%zd", initiator); + } + if (needBackend) virBufferAsprintf(&buf, ",memdev=3Dram-node%zu", i); else @@ -7217,6 +7388,18 @@ qemuBuildNumaCommandLine(virQEMUDriverConfigPtr cfg, } } =20 + if (hmat) { + if (qemuBuildNumaHMATCommandLine(cmd, def) < 0) + goto cleanup; + + /* This can't be moved into any of the loops above, + * because hmat-cache can be specified only after hmat-lb. */ + for (i =3D 0; i < ncells; i++) { + if (qemuBuilNumaCellCache(cmd, def, i) < 0) + goto cleanup; + } + } + ret =3D 0; =20 cleanup: diff --git a/tests/qemuxml2argvdata/numatune-hmat.x86_64-latest.args b/test= s/qemuxml2argvdata/numatune-hmat.x86_64-latest.args new file mode 100644 index 0000000000..c52015caa8 --- /dev/null +++ b/tests/qemuxml2argvdata/numatune-hmat.x86_64-latest.args @@ -0,0 +1,53 @@ +LC_ALL=3DC \ +PATH=3D/bin \ +HOME=3D/tmp/lib/domain--1-QEMUGuest \ +USER=3Dtest \ +LOGNAME=3Dtest \ +XDG_DATA_HOME=3D/tmp/lib/domain--1-QEMUGuest/.local/share \ +XDG_CACHE_HOME=3D/tmp/lib/domain--1-QEMUGuest/.cache \ +XDG_CONFIG_HOME=3D/tmp/lib/domain--1-QEMUGuest/.config \ +QEMU_AUDIO_DRV=3Dnone \ +/usr/bin/qemu-system-x86_64 \ +-name guest=3DQEMUGuest,debug-threads=3Don \ +-S \ +-object secret,id=3DmasterKey0,format=3Draw,\ +file=3D/tmp/lib/domain--1-QEMUGuest/master-key.aes \ +-machine pc,accel=3Dtcg,usb=3Doff,dump-guest-core=3Doff,hmat=3Don \ +-cpu qemu64 \ +-m 12288 \ +-overcommit mem-lock=3Doff \ +-smp 12,sockets=3D12,cores=3D1,threads=3D1 \ +-object memory-backend-ram,id=3Dram-node0,size=3D2147483648 \ +-numa node,nodeid=3D0,cpus=3D0-3,initiator=3D0,memdev=3Dram-node0 \ +-object memory-backend-ram,id=3Dram-node1,size=3D2147483648 \ +-numa node,nodeid=3D1,cpus=3D4-7,initiator=3D1,memdev=3Dram-node1 \ +-object memory-backend-ram,id=3Dram-node2,size=3D2147483648 \ +-numa node,nodeid=3D2,cpus=3D8-11,initiator=3D2,memdev=3Dram-node2 \ +-object memory-backend-ram,id=3Dram-node3,size=3D2147483648 \ +-numa node,nodeid=3D3,initiator=3D0,memdev=3Dram-node3 \ +-object memory-backend-ram,id=3Dram-node4,size=3D2147483648 \ +-numa node,nodeid=3D4,initiator=3D0,memdev=3Dram-node4 \ +-object memory-backend-ram,id=3Dram-node5,size=3D2147483648 \ +-numa node,nodeid=3D5,initiator=3D0,memdev=3Dram-node5 \ +-numa hmat-lb,initiator=3D0,target=3D0,hierarchy=3Dmemory,data-type=3Dacce= ss-latency,\ +latency=3D5 \ +-numa hmat-lb,initiator=3D0,target=3D0,hierarchy=3Dfirst-level,\ +data-type=3Daccess-latency,latency=3D10 \ +-numa hmat-lb,initiator=3D0,target=3D0,hierarchy=3Dmemory,data-type=3Dacce= ss-bandwidth,\ +bandwidth=3D204800K \ +-numa hmat-cache,node-id=3D0,size=3D10K,level=3D1,associativity=3Ddirect,\ +policy=3Dwrite-back,line=3D8 \ +-uuid c7a5fdb2-cdaf-9455-926a-d65c16db1809 \ +-display none \ +-no-user-config \ +-nodefaults \ +-chardev socket,id=3Dcharmonitor,fd=3D1729,server,nowait \ +-mon chardev=3Dcharmonitor,id=3Dmonitor,mode=3Dcontrol \ +-rtc base=3Dutc \ +-no-shutdown \ +-boot strict=3Don \ +-device piix3-usb-uhci,id=3Dusb,bus=3Dpci.0,addr=3D0x1.0x2 \ +-device virtio-balloon-pci,id=3Dballoon0,bus=3Dpci.0,addr=3D0x3 \ +-sandbox on,obsolete=3Ddeny,elevateprivileges=3Ddeny,spawn=3Ddeny,\ +resourcecontrol=3Ddeny \ +-msg timestamp=3Don diff --git a/tests/qemuxml2argvtest.c b/tests/qemuxml2argvtest.c index 98d9a7b5b2..3fdcd41317 100644 --- a/tests/qemuxml2argvtest.c +++ b/tests/qemuxml2argvtest.c @@ -1913,6 +1913,7 @@ mymain(void) =20 DO_TEST("numatune-distances", QEMU_CAPS_NUMA, QEMU_CAPS_NUMA_DIST); DO_TEST("numatune-no-vcpu", NONE); + DO_TEST_CAPS_LATEST("numatune-hmat"); =20 DO_TEST("numatune-auto-nodeset-invalid", NONE); DO_TEST("numatune-auto-prefer", QEMU_CAPS_OBJECT_MEMORY_RAM, --=20 2.26.2