From nobody Sun May 5 05:58:19 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) client-ip=63.128.21.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail(p=quarantine dis=quarantine) header.from=selectel.ru ARC-Seal: i=1; a=rsa-sha256; t=1613765376; cv=none; d=zohomail.com; s=zohoarc; b=RTrPUA0XzRtrtFATe6QLqnN5lGbO33/5Z+S45kUzMYynWZnU+xEzVRn8tLbCwsn3iZu02A7oSA0BfDqZpY3cvNK5i0jDOfRR1tbbK8dFkh1Xd+wkSuW+6H4gp0FXJ9/8eyD6SNrOl+pVbPUJoEljINJFqTX7iYovL5+d6Wjd2xU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1613765376; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=aqMEYD9g0UPHgezsGPtyqT1EwM/zY/5cCHGyM+ORC6Y=; b=P6uC21gcCC/YQSc+OP7yjG9zJOBmgDuJNHhnRskR1LJC1B1VkxqFsqSCrAZYCjlNBoigZUNBjiwK1UUTE01MrqlaYD9/ios9eh63M1+p6XDtj2YYZ6D5zJIHUhqKS8JPr+wXEpKVV5JZjp7fCd4ypgrpLAUPkfngsbGE6IIIyEM= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail header.from= (p=quarantine dis=quarantine) header.from= Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by mx.zohomail.com with SMTPS id 1613765375806112.69424954967042; Fri, 19 Feb 2021 12:09:35 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-584-iftZ4aYxOt-w8HCCSzOBDg-1; Fri, 19 Feb 2021 15:09:33 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 0043C56AA9; Fri, 19 Feb 2021 20:09:28 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E84D45C1BB; Fri, 19 Feb 2021 20:09:26 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id C62774EE4D; Fri, 19 Feb 2021 20:09:25 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11JK9Nc6009008 for ; Fri, 19 Feb 2021 15:09:24 -0500 Received: by smtp.corp.redhat.com (Postfix) id D440C14E780; Fri, 19 Feb 2021 20:09:23 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast01.extmail.prod.ext.rdu2.redhat.com [10.11.55.17]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7EF1F14E79F for ; Fri, 19 Feb 2021 20:09:20 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B2E2B85A5A6 for ; Fri, 19 Feb 2021 20:09:20 +0000 (UTC) Received: from mail-lj1-f177.google.com (mail-lj1-f177.google.com [209.85.208.177]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-57-fSuZOGf9PDKg6EcUEOkXcw-1; Fri, 19 Feb 2021 15:09:18 -0500 Received: by mail-lj1-f177.google.com with SMTP id k22so27133355ljg.3 for ; Fri, 19 Feb 2021 12:09:17 -0800 (PST) Received: from some-laptop.eq.selectel.org (89-110-38-38.static.avangarddsl.ru. [89.110.38.38]) by smtp.gmail.com with ESMTPSA id a12sm223310lfi.53.2021.02.19.12.09.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 19 Feb 2021 12:09:16 -0800 (PST) X-MC-Unique: iftZ4aYxOt-w8HCCSzOBDg-1 X-MC-Unique: fSuZOGf9PDKg6EcUEOkXcw-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=aqMEYD9g0UPHgezsGPtyqT1EwM/zY/5cCHGyM+ORC6Y=; b=rXtFnNB/V8HQKelJGmQh5wD0Jd4Nn/deBNn6bpOE3uRRycn2YzA3HLUcR4W3y2gkRO hAHVl8/GtstaKK8dZCO+zKAwWg4609AdVGw0gMhE4alMgrXi3nCFzcFB81aYnv22LPm9 gbcBktcMhZ9xFMTOs/iLmGqm8/kcJkwlI76ckg3PMKGzs/D8TK8pT0hFZu31qzuAtgOZ V5SVuN5PeLvjFMYCWkxNT7ykprkBqcPD3hxnK5bGFKSiI5i5tu767+1cC9G2XbYv1MI2 pb1f+Szh9Ur9iZzs60n0ZVQhq/fn4Nx888Se9wmHT7IhUsIhm7xRTOYGhp8sgH3Aw2xA b//A== X-Gm-Message-State: AOAM5319yEdOt1WTIClKzMcaHzS7ElimVGAXUldHlc/FjAHfAg5g1NCM y+V7kP2IPoQdpE/wwU0lObyQ/Lw/mAtXug== X-Google-Smtp-Source: ABdhPJyrj9B6g0rqj2b23dXWvhHYo9ReVZvpiQS7c/9vryZ9d89cxwxKtE/LMif766ZnYflnDVmovQ== X-Received: by 2002:a2e:a376:: with SMTP id i22mr6648528ljn.299.1613765356393; Fri, 19 Feb 2021 12:09:16 -0800 (PST) From: Aleksei Zakharov To: libvir-list@redhat.com Subject: [PATCH v4 1/1] qemu: add per-vcpu delay stats Date: Fri, 19 Feb 2021 23:08:34 +0300 Message-Id: <20210219200834.11913-2-zaharov@selectel.ru> In-Reply-To: <20210219200834.11913-1-zaharov@selectel.ru> References: <20210219200834.11913-1-zaharov@selectel.ru> X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-loop: libvir-list@redhat.com Cc: zakharov.a.g@yandex.ru, Aleksei Zakharov X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 2 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch adds delay time (steal time inside guest) to libvirt domain per-vcpu stats. Delay time is an important performance metric. It is a consequence of the overloaded CPU. Knowledge of the delay time of a virtual machine helps to understand if it is affected and estimate the impact. As a result, it is possible to react exactly when needed and rebalance the load between hosts. This is used by cloud providers to provide quality of service, especially when the CPU is oversubscribed. It's more convenient to work with this metric in a context of a libvirt domain. Any monitoring software may use this information. Signed-off-by: Aleksei Zakharov Reviewed-by: Daniel Henrique Barboza Reviewed-by: Michal Privoznik --- docs/manpages/virsh.rst | 4 ++++ src/libvirt-domain.c | 4 ++++ src/qemu/qemu_driver.c | 44 +++++++++++++++++++++++++++++++++++++++-- 3 files changed, 50 insertions(+), 2 deletions(-) diff --git a/docs/manpages/virsh.rst b/docs/manpages/virsh.rst index e3afa48f7b..a2b83ddfaa 100644 --- a/docs/manpages/virsh.rst +++ b/docs/manpages/virsh.rst @@ -2295,6 +2295,10 @@ When selecting the *--state* group the following fie= lds are returned: * ``vcpu..halted`` - virtual CPU is halted: yes or no (may indicate the processor is idle or even disabled, depending on the architecture) +* ``vcpu..delay`` - time the vCPU thread was enqueued by the + host scheduler, but was waiting in the queue instead of running. + Exposed to the VM as a steal time. + =20 =20 *--interface* returns: diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index 4af0166872..e54d11e513 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -11693,6 +11693,10 @@ virConnectGetDomainCapabilities(virConnectPtr conn, * "vcpu..halted" - virtual CPU is halted, may indicate the * processor is idle or even disabled, depending * on the architecture) + * "vcpu..delay" - time the vCPU thread was enqueued by the + * host scheduler, but was waiting in the queue + * instead of running. Exposed to the VM as a ste= al + * time. * * VIR_DOMAIN_STATS_INTERFACE: * Return network interface statistics (from domain point of view). diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index b9bbdf8d48..9ec3c8fce7 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -1332,6 +1332,34 @@ static char *qemuConnectGetCapabilities(virConnectPt= r conn) { return virCapabilitiesFormatXML(caps); } =20 +static int +qemuGetSchedstatDelay(unsigned long long *cpudelay, + pid_t pid, pid_t tid) +{ + g_autofree char *proc =3D NULL; + unsigned long long oncpu =3D 0; + g_autofree FILE *schedstat =3D NULL; + + if (tid) + proc =3D g_strdup_printf("/proc/%d/task/%d/schedstat", (int)pid, (= int)tid); + else + proc =3D g_strdup_printf("/proc/%d/schedstat", (int)pid); + + schedstat =3D fopen(proc, "r"); + if (!schedstat) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("Unable to open schedstat file at '%s'"), + proc); + } + if (fscanf(schedstat, "%llu %llu", &oncpu, cpudelay) < 2) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("Unable to parse schedstat info at '%s'"), + proc); + } + + VIR_FORCE_FCLOSE(schedstat); + return 0; +} =20 static int qemuGetSchedInfo(unsigned long long *cpuWait, @@ -1470,6 +1498,7 @@ static int qemuDomainHelperGetVcpus(virDomainObjPtr vm, virVcpuInfoPtr info, unsigned long long *cpuwait, + unsigned long long *cpudelay, int maxinfo, unsigned char *cpumaps, int maplen) @@ -1529,6 +1558,11 @@ qemuDomainHelperGetVcpus(virDomainObjPtr vm, return -1; } =20 + if (cpudelay) { + if (qemuGetSchedstatDelay(&(cpudelay[ncpuinfo]), vm->pid, vcpu= pid) < 0) + return -1; + } + ncpuinfo++; } =20 @@ -4873,7 +4907,7 @@ qemuDomainGetVcpus(virDomainPtr dom, goto cleanup; } =20 - ret =3D qemuDomainHelperGetVcpus(vm, info, NULL, maxinfo, cpumaps, map= len); + ret =3D qemuDomainHelperGetVcpus(vm, info, NULL, NULL, maxinfo, cpumap= s, maplen); =20 cleanup: virDomainObjEndAPI(&vm); @@ -17934,6 +17968,7 @@ qemuDomainGetStatsVcpu(virQEMUDriverPtr driver, int ret =3D -1; virVcpuInfoPtr cpuinfo =3D NULL; g_autofree unsigned long long *cpuwait =3D NULL; + g_autofree unsigned long long *cpudelay =3D NULL; =20 if (virTypedParamListAddUInt(params, virDomainDefGetVcpus(dom->def), "vcpu.current") < 0) @@ -17945,6 +17980,7 @@ qemuDomainGetStatsVcpu(virQEMUDriverPtr driver, =20 cpuinfo =3D g_new0(virVcpuInfo, virDomainDefGetVcpus(dom->def)); cpuwait =3D g_new0(unsigned long long, virDomainDefGetVcpus(dom->def)); + cpudelay =3D g_new0(unsigned long long, virDomainDefGetVcpus(dom->def)= ); =20 if (HAVE_JOB(privflags) && virDomainObjIsActive(dom) && qemuDomainRefreshVcpuHalted(driver, dom, QEMU_ASYNC_JOB_NONE) < 0)= { @@ -17953,7 +17989,7 @@ qemuDomainGetStatsVcpu(virQEMUDriverPtr driver, virResetLastError(); } =20 - if (qemuDomainHelperGetVcpus(dom, cpuinfo, cpuwait, + if (qemuDomainHelperGetVcpus(dom, cpuinfo, cpuwait, cpudelay, virDomainDefGetVcpus(dom->def), NULL, 0) < 0) { virResetLastError(); @@ -17978,6 +18014,10 @@ qemuDomainGetStatsVcpu(virQEMUDriverPtr driver, "vcpu.%u.wait", cpuinfo[i].number) = < 0) goto cleanup; =20 + if (virTypedParamListAddULLong(params, cpudelay[i], + "vcpu.%u.delay", cpuinfo[i].number= ) < 0) + goto cleanup; + /* state below is extracted from the individual vcpu structs */ if (!(vcpu =3D virDomainDefGetVcpu(dom->def, cpuinfo[i].number))) continue; --=20 2.17.1