From nobody Sat May 4 08:32:39 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) client-ip=170.10.133.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail(p=none dis=none) header.from=canonical.com Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.zohomail.com with SMTPS id 1634907635318119.08943572401245; Fri, 22 Oct 2021 06:00:35 -0700 (PDT) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-458-L0cw93YzNduhxP3tGOBKaQ-1; Fri, 22 Oct 2021 09:00:32 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7BCC580668E; Fri, 22 Oct 2021 13:00:26 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4A36C70F6A; Fri, 22 Oct 2021 13:00:26 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 10B4C1800B9E; Fri, 22 Oct 2021 13:00:26 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 19MD0OS0012666 for ; Fri, 22 Oct 2021 09:00:24 -0400 Received: by smtp.corp.redhat.com (Postfix) id 4C0072166B2D; Fri, 22 Oct 2021 13:00:24 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast03.extmail.prod.ext.rdu2.redhat.com [10.11.55.19]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 45F3B2166B25 for ; Fri, 22 Oct 2021 13:00:19 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-2.mimecast.com [207.211.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 281FA811E7A for ; Fri, 22 Oct 2021 13:00:19 +0000 (UTC) Received: from smtp-relay-internal-1.canonical.com (smtp-relay-internal-1.canonical.com [185.125.188.123]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-357-NulLmrycOTSBVZJibpxHdg-1; Fri, 22 Oct 2021 09:00:17 -0400 Received: from mail-lf1-f70.google.com (mail-lf1-f70.google.com [209.85.167.70]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id 3BE133FFF2 for ; Fri, 22 Oct 2021 13:00:16 +0000 (UTC) Received: by mail-lf1-f70.google.com with SMTP id y40-20020a0565123f2800b003fded085638so1731985lfa.0 for ; Fri, 22 Oct 2021 06:00:16 -0700 (PDT) Received: from ws.lan.d-node.is ([95.165.29.203]) by smtp.gmail.com with ESMTPSA id h18sm315324lfu.85.2021.10.22.06.00.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Oct 2021 06:00:15 -0700 (PDT) X-MC-Unique: L0cw93YzNduhxP3tGOBKaQ-1 X-MC-Unique: NulLmrycOTSBVZJibpxHdg-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2gAQPcGhM7HkXDS5m5tlWLURoLVTcQgbiW1eIQAa14s=; b=G5jGT+6z15qzJCliIfuVPMnaRJgzKM4k5Sc+oLoF1RqvobU7h963BkAA7vDc/Lml29 VJ3qdD9jghURarIm9Ogg9bCoNm68IP0cjwaCVz+srBiP09F5Bs6C3DnDCOehBD1eNyiM ue/RebB9Akg/bI5oZ74YtZdl6auKct9o2CUDeD8e2yh517fOvZuzEwoL9stofOAOsP05 XzaqMWaCtJru+QyxKVaVA2Us8wcdfHQ3fjAJOwpi5pazCFuW64TF4Hm1sJKayrxvaoYQ 2XranxiA/dV3Q71MgW+IPGgXc/sIVafU3xf0DxMHQJqJHBHKnHgzTaj6mAeZ4nqB7FK3 UOpg== X-Gm-Message-State: AOAM532oQ7UthDgB5AG0zQ+GTX4vhvlyzOEtHSlV42Eq5ud8Mu/L+gFw FsePje5ZNf6PcRtn6IhsOICgTwWDvUI2YpdhOEAZG+X8hYXM3jX570M5NSLCN5RxRoeqa3SlKd2 e8HIjnouj/+9XZgMxYJFQWc20L5d6jWchsA== X-Received: by 2002:ac2:4c14:: with SMTP id t20mr10466780lfq.343.1634907615560; Fri, 22 Oct 2021 06:00:15 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzewlNvdO0tn0/CqhWqoo1iDbcEsN4obb+p8y1BCx02DaW9TA7b6v+a+QDc9OkONp8TqxHETg== X-Received: by 2002:ac2:4c14:: with SMTP id t20mr10466760lfq.343.1634907615279; Fri, 22 Oct 2021 06:00:15 -0700 (PDT) From: Dmitrii Shcherbakov To: libvir-list@redhat.com, dmitrii.shcherbakov@canonical.com Subject: [libvirt PATCH 1/1] Ignore EPERM on attempts to clear VF VLAN ID Date: Fri, 22 Oct 2021 16:00:14 +0300 Message-Id: <20211022130014.416398-2-dmitrii.shcherbakov@canonical.com> In-Reply-To: <20211022130014.416398-1-dmitrii.shcherbakov@canonical.com> References: <20211022130014.416398-1-dmitrii.shcherbakov@canonical.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-MIME-Autoconverted: from quoted-printable to 8bit by lists01.pubmisc.prod.ext.phx2.redhat.com id 19MD0OS0012666 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1634907649547100003 Content-Type: text/plain; charset="utf-8" SmartNIC DPUs may not expose some privileged eswitch operations to the hypervisor hosts. For example, this happens with Bluefield devices running in the ECPF (default) mode for security reasons. While VF MAC address programming is possible via an RTM_SETLINK operation, trying to set a VLAN ID in the same operation will fail with EPERM. The equivalent ip link commands below provide an illustration: 1. This works: sudo ip link set enp130s0f0 vf 2 mac de:ad:be:ef:ca:fe 2. Setting (or clearing) a VLAN fails with EPERM: sudo ip link set enp130s0f0 vf 2 vlan 0 RTNETLINK answers: Operation not permitted 3. This is what Libvirt attempts to do today (when trying to clear a VF VLAN at the same time as programming a VF MAC). sudo ip link set enp130s0f0 vf 2 vlan 0 mac de:ad:be:ef:ca:fe RTNETLINK answers: Operation not permitted If setting an explicit VLAN ID results in an EPERM, clearing a VLAN (setting a VLAN ID to 0) can be handled gracefully by ignoring the EPERM error with the rationale being that if we cannot set this state in the first place, we cannot clear it either. Thus, virNetDevSetVfConfig is split into two distinct functions. If clearing a VLAN ID fails with EPERM, the error is simply ignored. An alternative to this could be providing a higher level control plane mechanism that would provide metadata about a device being remotely managed in which case Libvirt would avoid trying to set or clear a VLAN ID. This would be more complicated since other software (like Nova in the OpenStack case) would have to annotate every guest device with an attribute indicating whether a device is remotely managed or not based on operator provided configuration so that Libvirt can act on this and avoid VLAN programming. Signed-off-by: Dmitrii Shcherbakov --- src/util/virnetdev.c | 144 +++++++++++++++++++++++++++++++++++++------ 1 file changed, 124 insertions(+), 20 deletions(-) diff --git a/src/util/virnetdev.c b/src/util/virnetdev.c index 5b4c585716..7b0904e699 100644 --- a/src/util/virnetdev.c +++ b/src/util/virnetdev.c @@ -1527,11 +1527,114 @@ static struct nla_policy ifla_vfstats_policy[IFLA_= VF_STATS_MAX+1] =3D { [IFLA_VF_STATS_MULTICAST] =3D { .type =3D NLA_U64 }, }; =20 +static int +virNetDevSetVfVlan(const char *ifname, int vf, int vlanid) +{ + int rc =3D -1; + g_autofree struct nlmsghdr *resp =3D NULL; + struct nlmsgerr *err; + unsigned int recvbuflen =3D 0; + struct nl_msg *nl_msg; + struct nlattr *vfinfolist, *vfinfo; + struct ifinfomsg ifinfo =3D { + .ifi_family =3D AF_UNSPEC, + .ifi_index =3D -1, + }; + + if (vlanid < 0) + return -1; + + nl_msg =3D virNetlinkMsgNew(RTM_SETLINK, NLM_F_REQUEST); + + if (nlmsg_append(nl_msg, &ifinfo, sizeof(ifinfo), NLMSG_ALIGNTO) < 0) + goto buffer_too_small; + + if (ifname && + nla_put(nl_msg, IFLA_IFNAME, strlen(ifname)+1, ifname) < 0) + goto buffer_too_small; + + + if (!(vfinfolist =3D nla_nest_start(nl_msg, IFLA_VFINFO_LIST))) + goto buffer_too_small; + + if (!(vfinfo =3D nla_nest_start(nl_msg, IFLA_VF_INFO))) + goto buffer_too_small; + + if (vlanid >=3D 0) { + struct ifla_vf_vlan ifla_vf_vlan =3D { + .vf =3D vf, + .vlan =3D vlanid, + .qos =3D 0, + }; + + if (nla_put(nl_msg, IFLA_VF_VLAN, sizeof(ifla_vf_vlan), + &ifla_vf_vlan) < 0) + goto buffer_too_small; + } + + nla_nest_end(nl_msg, vfinfo); + nla_nest_end(nl_msg, vfinfolist); + + if (virNetlinkCommand(nl_msg, &resp, &recvbuflen, 0, 0, + NETLINK_ROUTE, 0) < 0) + goto cleanup; + + if (recvbuflen < NLMSG_LENGTH(0) || resp =3D=3D NULL) + goto malformed_resp; + + switch (resp->nlmsg_type) { + case NLMSG_ERROR: + err =3D (struct nlmsgerr *)NLMSG_DATA(resp); + if (resp->nlmsg_len < NLMSG_LENGTH(sizeof(*err))) + goto malformed_resp; + + /* If vlanid is 0 - we are attempting to clear an existing VLAN id. + * An EPERM received at this stage is an indicator that the embedd= ed + * switch is not exposed to this host and the network driver is not + * able to set a VLAN for a VF. */ + if (err->error =3D=3D -EPERM && vlanid =3D=3D 0) { + rc =3D 0; + goto cleanup; + } else if (err->error) { + virReportSystemError(-err->error, + _("Cannot set interface vlanid to %d " + "for ifname %s vf %d"), + vlanid, + ifname ? ifname : "(unspecified)", + vf); + goto cleanup; + } + break; + case NLMSG_DONE: + break; + default: + goto malformed_resp; + } + + rc =3D 0; + cleanup: + VIR_DEBUG("RTM_SETLINK %s vf %d vlanid=3D%d - %s", + ifname, vf, + vlanid, rc < 0 ? "Fail" : "Success"); + + nlmsg_free(nl_msg); + return rc; + + malformed_resp: + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("malformed netlink response message")); + goto cleanup; + + buffer_too_small: + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("allocated netlink buffer is too small")); + goto cleanup; +} =20 static int -virNetDevSetVfConfig(const char *ifname, int vf, - const virMacAddr *macaddr, int vlanid, - bool *allowRetry) +virNetDevSetVfMac(const char *ifname, int vf, + const virMacAddr *macaddr, + bool *allowRetry) { int rc =3D -1; char macstr[VIR_MAC_STRING_BUFLEN]; @@ -1545,7 +1648,7 @@ virNetDevSetVfConfig(const char *ifname, int vf, .ifi_index =3D -1, }; =20 - if (!macaddr && vlanid < 0) + if (!macaddr) return -1; =20 nl_msg =3D virNetlinkMsgNew(RTM_SETLINK, NLM_F_REQUEST); @@ -1577,18 +1680,6 @@ virNetDevSetVfConfig(const char *ifname, int vf, goto buffer_too_small; } =20 - if (vlanid >=3D 0) { - struct ifla_vf_vlan ifla_vf_vlan =3D { - .vf =3D vf, - .vlan =3D vlanid, - .qos =3D 0, - }; - - if (nla_put(nl_msg, IFLA_VF_VLAN, sizeof(ifla_vf_vlan), - &ifla_vf_vlan) < 0) - goto buffer_too_small; - } - nla_nest_end(nl_msg, vfinfo); nla_nest_end(nl_msg, vfinfolist); =20 @@ -1615,12 +1706,11 @@ virNetDevSetVfConfig(const char *ifname, int vf, } else if (err->error) { /* other errors are permanent */ virReportSystemError(-err->error, - _("Cannot set interface MAC/vlanid to %s/= %d " + _("Cannot set interface MAC to %s " "for ifname %s vf %d"), (macaddr ? virMacAddrFormat(macaddr, macstr) : "(unchanged)"), - vlanid, ifname ? ifname : "(unspecified)", vf); *allowRetry =3D false; /* no use retrying */ @@ -1637,10 +1727,10 @@ virNetDevSetVfConfig(const char *ifname, int vf, =20 rc =3D 0; cleanup: - VIR_DEBUG("RTM_SETLINK %s vf %d MAC=3D%s vlanid=3D%d - %s", + VIR_DEBUG("RTM_SETLINK %s vf %d MAC=3D%s - %s", ifname, vf, macaddr ? virMacAddrFormat(macaddr, macstr) : "(unchanged)", - vlanid, rc < 0 ? "Fail" : "Success"); + rc < 0 ? "Fail" : "Success"); =20 nlmsg_free(nl_msg); return rc; @@ -1656,6 +1746,20 @@ virNetDevSetVfConfig(const char *ifname, int vf, goto cleanup; } =20 +static int +virNetDevSetVfConfig(const char *ifname, int vf, + const virMacAddr *macaddr, int vlanid, + bool *allowRetry) +{ + int rc =3D 0; + if ((rc =3D virNetDevSetVfMac(ifname, vf, macaddr, allowRetry)) < 0) { + return rc; + } else if ((rc =3D virNetDevSetVfVlan(ifname, vf, vlanid)) < 0) { + return rc; + } + return rc; +} + /** * virNetDevParseVfInfo: * Get the VF interface information from kernel by netlink, To make netlink --=20 2.32.0