From nobody Fri May 3 18:51:56 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 207.211.31.81 as permitted sender) client-ip=207.211.31.81; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-1.mimecast.com; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of redhat.com designates 207.211.31.81 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail(p=none dis=none) header.from=bytedance.com ARC-Seal: i=1; a=rsa-sha256; t=1600691189; cv=none; d=zohomail.com; s=zohoarc; b=aMOA06HQJuQSRHZNvf5SCn89Dmtv+mCaA2Q1nGx9BIaLHYOz7p2Eqt3l+b1nS+bDlflKSoTWSX2sKlp1NodANC5yADDAVXQlH5qKmfXnWjKXe+IOxf1jAqaQ8KgCKUfP8vzbP56ySVFMi1l5PEu1cZhdFxyvrGjAnOjCeyV9siA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600691189; h=Content-Type:Cc:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:Sender:Subject:To; bh=LMpXOU2cluVS/GUoesMhyT5yi6g6/r2g2zqw4MM2uLc=; b=Xzb3HbWRKpmPT42q9cYVq7RkEDlN+JkQRf5sdVbHdaAuTuGvFIEnLY/nBGSVt1tH8AaAYlJHs9FmF1RsUEOnGUG36Yo/uqQIY9LHUFIWhFSdpfEqdYILjF0coC443/wvUQU49j7O5ObZlbN3IupPx7gfjyyU4CfhcRT+wwdyNRE= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of redhat.com designates 207.211.31.81 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-1.mimecast.com (us-smtp-2.mimecast.com [207.211.31.81]) by mx.zohomail.com with SMTPS id 1600691189053522.4528766234636; Mon, 21 Sep 2020 05:26:29 -0700 (PDT) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-19-_7aXACodOiKW05iv45c4Mw-1; Mon, 21 Sep 2020 08:26:22 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id BE7CB802B69; Mon, 21 Sep 2020 12:26:14 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8D53B10013DB; Mon, 21 Sep 2020 12:26:13 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id DA19718561B3; Mon, 21 Sep 2020 12:26:09 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 08LCQ8qH015175 for ; Mon, 21 Sep 2020 08:26:08 -0400 Received: by smtp.corp.redhat.com (Postfix) id 2B8202166B28; Mon, 21 Sep 2020 12:26:08 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast02.extmail.prod.ext.rdu2.redhat.com [10.11.55.18]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 270D92166BA3 for ; Mon, 21 Sep 2020 12:26:05 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-2.mimecast.com [205.139.110.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1D56F802D3D for ; Mon, 21 Sep 2020 12:26:05 +0000 (UTC) Received: from mail-pg1-f196.google.com (mail-pg1-f196.google.com [209.85.215.196]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-376-H6gcK-DaOtG1vIcaOK-y2w-1; Mon, 21 Sep 2020 08:26:02 -0400 Received: by mail-pg1-f196.google.com with SMTP id l71so8896106pge.4 for ; Mon, 21 Sep 2020 05:26:02 -0700 (PDT) Received: from libai.bytedance.net ([61.120.150.71]) by smtp.gmail.com with ESMTPSA id q20sm6734775pgm.24.2020.09.21.05.25.59 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Sep 2020 05:26:01 -0700 (PDT) X-MC-Unique: _7aXACodOiKW05iv45c4Mw-1 X-MC-Unique: H6gcK-DaOtG1vIcaOK-y2w-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=LMpXOU2cluVS/GUoesMhyT5yi6g6/r2g2zqw4MM2uLc=; b=Co1pbm7VhxXB2yVbAtgy9XS4/FJ4/hLfX0mQB7JD+YPwVbbUQoVz7rTvoLk1Ba4p6Y E9lvbc8VfB0aM/BtpZtuvpkYZ/1kJIpH7nlFmw2NmEq9V1aGFlmWxQ7IbAtKCu945m9k MnVjCFWQjU15/d+yY/NMg6KgIiCKZ0TPAbmmM0XFQU6/hsTiZux2d3QPdGhJtjGTnBTm jC1Vba/Df6N/iTKn4274mtL4LHdwpwYpgcZiuqtRr3Jw2Lg2zjVl8N/8omSbohatkIlA nxpd5f214adImbs+1vtHDJ6hnOHEVo0GRvTUqeE6yugw80Y6/Gac/gBpqKmV43/WQ8+I VFPQ== X-Gm-Message-State: AOAM532KMiGZpy3Iq7MjhRb9v87U1+v+AUOVOjASy62LHL3OzvERdML8 vjBiJsbfKm77z7ncTGBotR1e+g== X-Google-Smtp-Source: ABdhPJx0SOcP/EqzOw1DTYnhoEOsl6AYPfwDclj2BgWha5wovjOGnZrsUvh2YMwxVgd+J1p9MXejcA== X-Received: by 2002:a63:6881:: with SMTP id d123mr37279150pgc.370.1600691161732; Mon, 21 Sep 2020 05:26:01 -0700 (PDT) From: zhenwei pi To: berrange@redhat.com Subject: [PATCH] util: support PCI passthrough net device stats collection Date: Mon, 21 Sep 2020 20:25:55 +0800 Message-Id: <20200921122555.42826-1-pizhenwei@bytedance.com> X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false; X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-loop: libvir-list@redhat.com Cc: libvir-list@redhat.com, mprivozn@redhat.com, pizhenwei@bytedance.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 2 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Collect PCI passthrough net device stats from kernel by netlink API. Currently, libvirt can not get PCI passthrough net device stats, run command: #virsh domifstat instance --interface=3D52:54:00:2d:b2:35 error: Failed to get interface stats instance 52:54:00:2d:b2:35 error: internal error: Interface name not provided The PCI device(usually SR-IOV virtual function device) is detached while it's used in PCI passthrough mode. And we can not parse this device from /proc/net/dev any more. In this patch, libvirt check net device is VF of not firstly, then query virNetDevVFInterfaceStats(new API). virNetDevVFInterfaceStats parses VFs info of all PFs, compares MAC address until the two MAC addresses match. '#ip -s link show' can get the same result. Instead of parsing the output result, implement this feature by libnl API. Notice that this feature deponds on driver of PF. Test on Mellanox ConnectX-4 Lx, it works well. Also test on Intel Corporation 82599ES, it works, but only get 0. (ip-link command get the same result). Signed-off-by: zhenwei pi --- meson.build | 4 ++ src/libvirt_private.syms | 1 + src/qemu/qemu_driver.c | 3 + src/util/virnetdev.c | 158 +++++++++++++++++++++++++++++++++++++++++++= ++++ src/util/virnetdev.h | 5 ++ 5 files changed, 171 insertions(+) diff --git a/meson.build b/meson.build index 24535a403c..e17da9e2b9 100644 --- a/meson.build +++ b/meson.build @@ -1392,6 +1392,10 @@ if not get_option('virtualport').disabled() endif endif =20 +if cc.has_header_symbol('linux/if_link.h', 'IFLA_VF_STATS') + conf.set('WITH_VF_STATS', 1) +endif + if host_machine.system() =3D=3D 'windows' ole32_dep =3D cc.find_library('ole32') oleaut32_dep =3D cc.find_library('oleaut32') diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms index bdbe3431b8..bcc40b8d69 100644 --- a/src/libvirt_private.syms +++ b/src/libvirt_private.syms @@ -2585,6 +2585,7 @@ virNetDevSetRcvMulti; virNetDevSetupControl; virNetDevSysfsFile; virNetDevValidateConfig; +virNetDevVFInterfaceStats; =20 =20 # util/virnetdevbandwidth.h diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index ae715c01d7..f554010c40 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -10196,6 +10196,9 @@ qemuDomainInterfaceStats(virDomainPtr dom, if (virDomainNetGetActualType(net) =3D=3D VIR_DOMAIN_NET_TYPE_VHOSTUSE= R) { if (virNetDevOpenvswitchInterfaceStats(net->ifname, stats) < 0) goto cleanup; + } else if (virDomainNetGetActualType(net) =3D=3D VIR_DOMAIN_NET_TYPE_H= OSTDEV) { + if (virNetDevVFInterfaceStats(&net->mac, stats) < 0) + goto cleanup; } else { if (virNetDevTapInterfaceStats(net->ifname, stats, !virDomainNetTypeSharesHostView(net= )) < 0) diff --git a/src/util/virnetdev.c b/src/util/virnetdev.c index e1a4cc2bef..be9b8ce4a9 100644 --- a/src/util/virnetdev.c +++ b/src/util/virnetdev.c @@ -1489,6 +1489,9 @@ static struct nla_policy ifla_vf_policy[IFLA_VF_MAX+1= ] =3D { .maxlen =3D sizeof(struct ifla_vf_mac) }, [IFLA_VF_VLAN] =3D { .type =3D NLA_UNSPEC, .maxlen =3D sizeof(struct ifla_vf_vlan) }, +#if defined(WITH_VF_STATS) + [IFLA_VF_STATS] =3D { .type =3D NLA_NESTED }, +#endif }; =20 =20 @@ -2265,6 +2268,151 @@ virNetDevSetNetConfig(const char *linkdev, int vf, return 0; } =20 +#if defined(WITH_VF_STATS) +static struct nla_policy ifla_vfstats_policy[IFLA_VF_STATS_MAX+1] =3D { + [IFLA_VF_STATS_RX_PACKETS] =3D { .type =3D NLA_U64 }, + [IFLA_VF_STATS_TX_PACKETS] =3D { .type =3D NLA_U64 }, + [IFLA_VF_STATS_RX_BYTES] =3D { .type =3D NLA_U64 }, + [IFLA_VF_STATS_TX_BYTES] =3D { .type =3D NLA_U64 }, + [IFLA_VF_STATS_BROADCAST] =3D { .type =3D NLA_U64 }, + [IFLA_VF_STATS_MULTICAST] =3D { .type =3D NLA_U64 }, +}; + +static int +virNetDevParseVfStats(struct nlattr **tb, virMacAddrPtr mac, + virDomainInterfaceStatsPtr stats) +{ + int ret =3D -1, len; + struct ifla_vf_mac *vf_lladdr; + struct nlattr *nla, *t[IFLA_VF_MAX+1]; + struct nlattr *stb[IFLA_VF_STATS_MAX+1]; + + if (tb =3D=3D NULL || mac =3D=3D NULL || stats =3D=3D NULL) { + return -1; + } + + if (!tb[IFLA_VFINFO_LIST]) + return -1; + + len =3D nla_len(tb[IFLA_VFINFO_LIST]); + + for (nla =3D nla_data(tb[IFLA_VFINFO_LIST]); nla_ok(nla, len); + nla =3D nla_next(nla, &len)) { + ret =3D nla_parse(t, IFLA_VF_MAX, nla_data(nla), nla_len(nla), + ifla_vf_policy); + if (ret < 0) + return -1; + + if (t[IFLA_VF_MAC] =3D=3D NULL) { + continue; + } + + vf_lladdr =3D nla_data(t[IFLA_VF_MAC]); + if(virMacAddrCmpRaw(mac, vf_lladdr->mac)) { + continue; + } + + if (t[IFLA_VF_STATS]) { + ret =3D nla_parse_nested(stb, IFLA_VF_STATS_MAX, + t[IFLA_VF_STATS], + ifla_vfstats_policy); + if (ret < 0) + return -1; + + stats->rx_bytes =3D nla_get_u64(stb[IFLA_VF_STATS_RX_BYTES]); + stats->tx_bytes =3D nla_get_u64(stb[IFLA_VF_STATS_TX_BYTES]); + stats->rx_packets =3D nla_get_u64(stb[IFLA_VF_STATS_RX_PACKETS= ]); + stats->tx_packets =3D nla_get_u64(stb[IFLA_VF_STATS_TX_PACKETS= ]); + } + return 0; + } + + return ret; +} + +/** + * virNetDevVFInterfaceStats: + * @mac: MAC address of the VF interface + * @stats: returns stats of the VF interface + * + * Get the VF interface from kernel by netlink. + * Returns 0 on success, -1 on failure. + */ +int +virNetDevVFInterfaceStats(virMacAddrPtr mac, + virDomainInterfaceStatsPtr stats) +{ + FILE *fp; + char line[256], *colon, *ifname; + int rc =3D -1; + void *nlData =3D NULL; + struct nlattr *tb[IFLA_MAX + 1] =3D {NULL, }; + char *sysfsDevicePath =3D NULL; + + fp =3D fopen("/proc/net/dev", "r"); + if (!fp) { + virReportSystemError(errno, "%s", + _("Could not open /proc/net/dev")); + return -1; + } + + /* get all PCI net devices, and parse VFs list from netlink API. + * compare MAC address, collect device stats if matching. + */ + while (fgets(line, sizeof(line), fp)) { + /* The line looks like: + * " eth0:..." + * Split it at the colon. and strip blank from head. + */ + colon =3D strchr(line, ':'); + if (!colon) + continue; + *colon =3D '\0'; + ifname =3D line; + while((*ifname =3D=3D ' ') && (ifname < colon)) + ifname++; + + if (virNetDevSysfsFile(&sysfsDevicePath, ifname, "device") < 0) + break; + + if (virNetDevIsPCIDevice(sysfsDevicePath)) { + rc =3D virNetlinkDumpLink(ifname, -1, &nlData, tb, 0, 0); + if (rc < 0) { + rc =3D -1; + goto cleanup; + } + + rc =3D virNetDevParseVfStats(tb, mac, stats); + VIR_FREE(nlData); + if (rc =3D=3D 0) + goto cleanup; + } + VIR_FREE(sysfsDevicePath); + } + +cleanup: + VIR_FREE(sysfsDevicePath); + VIR_FORCE_FCLOSE(fp); + return rc; +} + +#else /* #if defined(WITH_VF_STATS) */ + +int +virNetDevVFInterfaceStats(virMacAddrPtr mac, + virDomainInterfaceStatsPtr stats) +{ + virReportSystemError(ENOSYS, "%s", + _("Unable to get VF net device stats on this kern= el, please upgrade kernel at lease linux-4.2")); + + /* no need to do anything, just fix compling error here */ + if (mac =3D=3D NULL || stats =3D=3D NULL) { + return -1; + } + + return -1; +} +#endif /* #if defined(WITH_VF_STATS) */ =20 #else /* defined(__linux__) && defined(WITH_LIBNL) && defined(IFLA_VF_MAX= ) */ =20 @@ -2309,6 +2457,16 @@ virNetDevSetNetConfig(const char *linkdev G_GNUC_UNU= SED, } =20 =20 +int +virNetDevVFInterfaceStats(virMacAddrPtr mac, + virDomainInterfaceStatsPtr stats) +{ + virReportSystemError(ENOSYS, "%s", + _("Unable to get VF net device stats on this plat= form")); + return -1; +} + + #endif /* defined(__linux__) && defined(WITH_LIBNL) && defined(IFLA_VF_MAX= ) */ =20 VIR_ENUM_IMPL(virNetDevIfState, diff --git a/src/util/virnetdev.h b/src/util/virnetdev.h index 5f581323ed..ff59d9d341 100644 --- a/src/util/virnetdev.h +++ b/src/util/virnetdev.h @@ -312,4 +312,9 @@ int virNetDevSysfsFile(char **pf_sysfs_device_link, int virNetDevRunEthernetScript(const char *ifname, const char *script) G_GNUC_NO_INLINE; =20 +int virNetDevVFInterfaceStats(virMacAddrPtr mac, + virDomainInterfaceStatsPtr stats) +ATTRIBUTE_NONNULL(1) ATTRIBUTE_NONNULL(2); + + G_DEFINE_AUTOPTR_CLEANUP_FUNC(virNetDevRxFilter, virNetDevRxFilterFree); --=20 2.11.0