From nobody Sat Feb 7 09:37:03 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 205.139.110.120 as permitted sender) client-ip=205.139.110.120; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-1.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 205.139.110.120 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1574775363; cv=none; d=zohomail.com; s=zohoarc; b=XgRkQFQhwnc8nDQrGUyjUvkPnyiSCiksFW9zCvd9QLei2hQB7X1mXXVe74O42wTm38YXbqE+BQztaj+lEWv87OMMVdBAWpPjF0zqEq6DM5iLsYJSrxSuUfcO7DvxqJb+0DdS79lbfJ9Y68UZ0oywvB0GzoXKsBAJuAgcDZW3Y1c= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1574775363; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=bUjPFdWQrrwxueoVHK5TSKazSywhuE9d6JMAKwbQljY=; b=C4I3LnBKqdXe8eY45fl/wU5g2t7afFta/V8KcCh1bJWXzB0QlS3RJgIKpXgCvBpOW+2mSDglZiXDiVc2qcZYPBg42hdGkSLgCQKJEVTKnDl5ToroxdOgoH2zgxAw1ufZmI2U4wMB5OhKbhMTKq8nv6aYr0TqDU0rmulR4jzYYww= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 205.139.110.120 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.120]) by mx.zohomail.com with SMTPS id 1574775363398706.96332307349; Tue, 26 Nov 2019 05:36:03 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-142-A_wRvinBNCSB5Q4UW3j9pA-1; Tue, 26 Nov 2019 08:36:00 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7A06B8024E3; Tue, 26 Nov 2019 13:35:55 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 51265600CA; Tue, 26 Nov 2019 13:35:55 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 05D924E56D; Tue, 26 Nov 2019 13:35:55 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id xAQDYlBx004742 for ; Tue, 26 Nov 2019 08:34:47 -0500 Received: by smtp.corp.redhat.com (Postfix) id 178EB1C669; Tue, 26 Nov 2019 13:34:47 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast04.extmail.prod.ext.rdu2.redhat.com [10.11.55.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 12A3EED16E for ; Tue, 26 Nov 2019 13:34:44 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 92ED61022E93 for ; Tue, 26 Nov 2019 13:34:44 +0000 (UTC) Received: from mail-qv1-f66.google.com (mail-qv1-f66.google.com [209.85.219.66]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-175-KBZJglHZPeaT0qG1ge5ZIQ-1; Tue, 26 Nov 2019 08:34:40 -0500 Received: by mail-qv1-f66.google.com with SMTP id n4so7248953qvq.9; Tue, 26 Nov 2019 05:34:40 -0800 (PST) Received: from rekt.ibmuc.com ([2804:431:c7c6:afb:97a3:bc66:755b:2f57]) by smtp.gmail.com with ESMTPSA id i41sm5931229qti.42.2019.11.26.05.34.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Nov 2019 05:34:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1574775362; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=bUjPFdWQrrwxueoVHK5TSKazSywhuE9d6JMAKwbQljY=; b=dJYuzIX+at6OE5oPm58cWVS87ER1vr3LvHivyawVdqwnu2Y6V15dhaYIgwwVvQL5rGFHQd H9DUGEAoEB6xXAkohdB9qCPqsOojl+SmR5wM5z62H6Z+CrWm04PoltWHO2GGcdp/FcRorX /nLUfkAfr/Eh3nAELrCyczZ5FUMS56Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tigpZ7TRbDUY1H9PSts/IrB7bb+O+VLwwyKhCEzkx2U=; b=FuRTr18mujmyJNbGo8UkCm1wwHEz6byROrRWuRLv6aoSmITzEstzeJurkL9zRv9Rap JMt910UoGdkQEVZwfzghYkO1oer99bM6hlvhnCE/OW66WxTOEtYZdRKqZo1A2R6nNWmr knSrL4G/sOgaegsu9V6P1M25WpXZQqQ+HjQArhRUwR9KhKfpdFhGOin9U//Yr1FKGMkW HPuTWUZERWmQFXS8bm39z0XlTJyul+SaItgZ+pWsT5FhYJjot9CYZ8wre2v4TRYg1Mwg CBp8GH1RUOZyDCYDTiYDkNU/BaK0w175neowulvnPsYr41Edl3mnr6/v7RvSn3ZAVwCg /riw== X-Gm-Message-State: APjAAAVljFOwGISmRo2Af0hhXrCxBP7GoT86vXduV1aa/8O/i+begVEl PvEYCs6qpAbT1x3fL1mGLCDs5gvG X-Google-Smtp-Source: APXvYqyvT8Tzu/2CQw5fWG+OR1RFs31DwBpM9MS/vpBz3UQnSaTk2Q9JfPuiuQTi6FX1IPbytsxTDQ== X-Received: by 2002:a05:6214:852:: with SMTP id dg18mr33432800qvb.8.1574775279980; Tue, 26 Nov 2019 05:34:39 -0800 (PST) From: Daniel Henrique Barboza To: libvir-list@redhat.com Date: Tue, 26 Nov 2019 10:33:58 -0300 Message-Id: <20191126133401.1242133-4-danielhb413@gmail.com> In-Reply-To: <20191126133401.1242133-1-danielhb413@gmail.com> References: <20191126133401.1242133-1-danielhb413@gmail.com> MIME-Version: 1.0 X-MC-Unique: KBZJglHZPeaT0qG1ge5ZIQ-1 X-MC-Unique: A_wRvinBNCSB5Q4UW3j9pA-1 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-MIME-Autoconverted: from quoted-printable to 8bit by lists01.pubmisc.prod.ext.phx2.redhat.com id xAQDYlBx004742 X-loop: libvir-list@redhat.com Cc: Daniel Henrique Barboza , darkstego@gmail.com, laine@laine.org Subject: [libvirt] [PATCH v3 3/6] virhostdev.c: check all IOMMU devs in virHostdevPreparePCIDevices X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Mimecast-Spam-Score: 0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" virHostdevPreparePCIDevices verifies if a PCI hostdev "A" is in use by another domain by checking the 'activePCIHostdevs' list. It also verifies if other domains are using any other hostdev that belongs to the same IOMMU of "A". This is not enough to cover all the cases. Suppose a PCI hostdev "B" that shares the IOMMU with "A". We are currently able to abort guest launch if "B" is being used by another domain, but if "B" is not being used by any domain, we'll proceed guest execution. What happens then is: if "A" is being used in managed mode, the guest will fail to launch because Libvirt didn't detach "B" from the host. If "A" is being used in un-managed mode, the guest might fail or succeed to launch depending on whether both "A" and "B" were detached manually prior to guest start. Libvirt is now able to deal with these scenarios in a homogeneous fashion, providing the same behavior for both managed and un-managed mode while allowing parcial assignment for both cases as well. This patch changes virHostdevPreparePCIDevices to check all the PCI hostdevs of the domain against all the PCI hostdevs of the IOMMU, failing to launch if the domain does not contain all the PCI hostdevs declared, in both managed and un-managed cases. After this patch, any existing domains that were using PCI hostdevs with un-managed mode, and were getting away with partial assignment without declaring all the IOMMU PCI hostdevs in the domain XML, will be forced to do so. The missing PCI hostdevs can be declared with "
" to keep them invisible to the guest, making the guest oblivious of this change. This is an annoyance for such domains, but the payoff is more consistency of behavior between managed and un-managed mode and more clarity on the domain XML, forcing the admin to declare all PCI hostdevs of the IOMMU and knowing exactly what will be detached from the host. This is an example of this newly added error condition when the domain does not declare all the PCI hostdevs of the same IOMMU: $ sudo ./run tools/virsh start multipci-coldplug-partial-fail error: Failed to start domain multipci-coldplug-partial-fail error: Requested operation is not valid: All devices of the same IOMMU grou= p 1 of the PCI device 0001:09:00.0 must belong to domain multipci-coldplug-partial= -fail $ Signed-off-by: Daniel Henrique Barboza --- src/util/virhostdev.c | 64 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 57 insertions(+), 7 deletions(-) diff --git a/src/util/virhostdev.c b/src/util/virhostdev.c index 39e6b8f49f..2e7885112f 100644 --- a/src/util/virhostdev.c +++ b/src/util/virhostdev.c @@ -56,6 +56,13 @@ struct virHostdevIsPCINodeDeviceUsedData { bool usesVFIO; }; =20 +struct virHostdevIsAllIOMMUGroupUsedData { + virPCIDeviceListPtr pcidevs; + const char *domainName; + const char *deviceName; + int iommuGroup; +}; + /* This module makes heavy use of bookkeeping lists contained inside a * virHostdevManager instance to keep track of the devices' status. To make * it easy to spot potential ownership errors when moving devices from one @@ -111,6 +118,27 @@ static int virHostdevIsPCINodeDeviceUsed(virPCIDeviceA= ddressPtr devAddr, void *o return 0; } =20 +static int virHostdevIsAllIOMMUGroupUsed(virPCIDeviceAddressPtr devAddr, v= oid *opaque) +{ + struct virHostdevIsAllIOMMUGroupUsedData *helperData =3D opaque; + virPCIDevicePtr actual; + + actual =3D virPCIDeviceListFindByIDs(helperData->pcidevs, + devAddr->domain, devAddr->bus, + devAddr->slot, devAddr->function); + if (actual) { + return 0; + } else { + virReportError(VIR_ERR_OPERATION_INVALID, + _("All devices of the same IOMMU group %d of " + "the PCI device %s must belong to domain %s"), + helperData->iommuGroup, + helperData->deviceName, + helperData->domainName); + return -1; + } +} + static int virHostdevManagerOnceInit(void) { if (!VIR_CLASS_NEW(virHostdevManager, virClassForObject())) @@ -724,9 +752,10 @@ virHostdevPreparePCIDevices(virHostdevManagerPtr mgr, unsigned int flags) { g_autoptr(virPCIDeviceList) pcidevs =3D NULL; + g_autofree unsigned int *searchedIOMMUs =3D NULL; int last_processed_hostdev_vf =3D -1; - size_t i; - int ret =3D -1; + size_t i, j; + int ret =3D -1, nSearchedIOMMUs =3D 0; virPCIDeviceAddressPtr devAddr =3D NULL; =20 if (!nhostdevs) @@ -735,6 +764,8 @@ virHostdevPreparePCIDevices(virHostdevManagerPtr mgr, if (!(pcidevs =3D virHostdevGetPCIHostDeviceList(hostdevs, nhostdevs))) return -1; =20 + searchedIOMMUs =3D g_new0(unsigned int, virPCIDeviceListCount(pcidevs)= ); + virObjectLock(mgr->activePCIHostdevs); virObjectLock(mgr->inactivePCIHostdevs); =20 @@ -778,13 +809,32 @@ virHostdevPreparePCIDevices(virHostdevManagerPtr mgr, goto cleanup; =20 /* VFIO devices belonging to same IOMMU group can't be - * shared across guests. Check if that's the case. */ + * shared across guests, and all of the must belong to the + * same domain. If a device isn't going to be assigned + * (flag unassigned is true) the guest will not use it, but + * the actual device must be detached together from the host + * anyway. */ if (usesVFIO) { - data.usesVFIO =3D true; - if (virPCIDeviceAddressIOMMUGroupIterate(devAddr, - virHostdevIsPCINodeDe= viceUsed, - &data) < 0) + int devIOMMUGroup =3D virPCIDeviceAddressGetIOMMUGroupNum(devA= ddr); + struct virHostdevIsAllIOMMUGroupUsedData helper =3D { + pcidevs, dom_name, virPCIDeviceGetName(pci), devIOMMUGroup= }; + bool alreadySearched =3D false; + + for (j =3D 0; j < nSearchedIOMMUs; j++) { + if (devIOMMUGroup =3D=3D searchedIOMMUs[j]) { + alreadySearched =3D true; + break; + } + } + + if (alreadySearched) + continue; + + if (virPCIDeviceAddressIOMMUGroupIterate( + devAddr, virHostdevIsAllIOMMUGroupUsed, &helper) < 0) goto cleanup; + + searchedIOMMUs[nSearchedIOMMUs++] =3D devIOMMUGroup; } } =20 --=20 2.23.0 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list