From nobody Sun Dec 22 09:01:36 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.libvirt.org designates 8.43.85.245 as permitted sender) client-ip=8.43.85.245; envelope-from=devel-bounces@lists.libvirt.org; helo=lists.libvirt.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of lists.libvirt.org designates 8.43.85.245 as permitted sender) smtp.mailfrom=devel-bounces@lists.libvirt.org; arc=fail (BodyHash is different from the expected one); dmarc=pass(p=reject dis=none) header.from=lists.libvirt.org Return-Path: Received: from lists.libvirt.org (lists.libvirt.org [8.43.85.245]) by mx.zohomail.com with SMTPS id 1733963280086229.56661111045423; Wed, 11 Dec 2024 16:28:00 -0800 (PST) Received: by lists.libvirt.org (Postfix, from userid 996) id E4D5311B0; Wed, 11 Dec 2024 19:27:58 -0500 (EST) Received: from lists.libvirt.org (localhost [IPv6:::1]) by lists.libvirt.org (Postfix) with ESMTP id 469291225; Wed, 11 Dec 2024 19:25:20 -0500 (EST) Received: by lists.libvirt.org (Postfix, from userid 996) id 235E611E4; Wed, 11 Dec 2024 19:25:14 -0500 (EST) Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2085.outbound.protection.outlook.com [40.107.212.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.libvirt.org (Postfix) with ESMTPS id 772C011C2 for ; Wed, 11 Dec 2024 19:24:56 -0500 (EST) Received: from SN7PR12MB6838.namprd12.prod.outlook.com (2603:10b6:806:266::18) by CY5PR12MB6322.namprd12.prod.outlook.com (2603:10b6:930:21::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8251.14; Thu, 12 Dec 2024 00:24:49 +0000 Received: from SN7PR12MB6838.namprd12.prod.outlook.com ([fe80::529d:478:bc5d:b400]) by SN7PR12MB6838.namprd12.prod.outlook.com ([fe80::529d:478:bc5d:b400%6]) with mapi id 15.20.8251.008; Thu, 12 Dec 2024 00:24:49 +0000 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on lists.libvirt.org X-Spam-Level: X-Spam-Status: No, score=0.2 required=5.0 tests=DKIM_INVALID,DKIM_SIGNED, FORGED_SPF_HELO,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2, RCVD_IN_VALIDITY_RPBL_BLOCKED,RCVD_IN_VALIDITY_SAFE_BLOCKED, SPF_HELO_PASS autolearn=no autolearn_force=no version=3.4.4 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=A2Mab4CAvmPwtCDg9n4iOjiPt881av6hRyzN79FRFGgLRHYe29wUxRQ/DyVcM6REm6QUy53XNIZY3xiqa4rx6dL6Nlx34V9vD6UESsHC3txDDXGuNhgcAZsNIPBbL5v+8EljSiF94ZjCzt/+xqzona1RQbLunHkes4BgQUAft5TSbkTSszGZXPtY/52bztZ9aTarVX+9iv4ipAwi4WFc5wlIbG7QahZXj7qtJhiHnuUR4pMGV0E4KNnBl9yaYaQTG7TthRnyYPxv4SlVCbCTsZ5lJD8WNUAzqCRWCwMR5oPpIOsK+ZmSEFLFXiG34oK1LMFLxMORjBJCZEyh2mYxyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9PWA0WMEpt+mmdqddHfXlhRVqpXdQTjq0fZ7kER9KNI=; b=p0iMXbiJSFcQxqvZR0GMQSuOIRyCRxPkOQHT93ZVqJN7/yJp/SDw+SvwFjcBLeQ+rC3I5dXCI2YHYAzB3WZLtSrGrnqCkhQwTFiGRo6rG/gZhFzw0+Pw/N5AUMPu0LOc71ynSZ4qw9GgeLasS6+QtQopmtCrgAwaodM7rBoqZ88CjhUlPduX5q/l+vkn0qfgNXVgStGuOp41QhipiCsS21Sz9DuYfZ8zEmBK9I3nKyTVOL4FEljiTaHrqMjwRTuZxLK9luNjZCuvqKIy34tyERsFj2VfS/Iy/tQikJ9CTk0nwqImCerx427fKsgwKIT2nY8Zp2+LjcyETSsqpMdiIA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9PWA0WMEpt+mmdqddHfXlhRVqpXdQTjq0fZ7kER9KNI=; b=L8lvkiFn/TkWBbQPoGw9MbnDSgZe3QvhCSYBVkYVgfrCV7OUdq6qQvwhxcjKBBeSDy1UqRtqtwK3kbnQ01pusOVBDLD3TJ3DtInSydupDCw3KIywoxZVu73hupVyP46Hc7jVjiJBeL318lExPJJaAFUGh2h7GuB0j1pTEc5mrPSEjE4gDMtoQHVhTeIa5HHK8uWYTyH/xBn1SBHJwAPBVD2/Nv2faXKU2oHQC5SX2vQMgZmBpqzvVjuHsSW1aNsxsrqDhJIG+mgPhK8K2tvYaT7X06bojgFXUL10b+l9X6C6CIec/vBGLjDR4p7DLTu4uNoJzqI/9n4ipf1IdONQ3w== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; To: devel@lists.libvirt.org Subject: [RFC PATCH 3/5] qemu: Create PXBs and auto-assign VFIO devs and nested SMMUs Date: Wed, 11 Dec 2024 16:24:21 -0800 Message-Id: <20241212002423.1050892-4-nathanc@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241212002423.1050892-1-nathanc@nvidia.com> References: <20241212002423.1050892-1-nathanc@nvidia.com> X-ClientProxiedBy: BYAPR05CA0028.namprd05.prod.outlook.com (2603:10b6:a03:c0::41) To SN7PR12MB6838.namprd12.prod.outlook.com (2603:10b6:806:266::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN7PR12MB6838:EE_|CY5PR12MB6322:EE_ X-MS-Office365-Filtering-Correlation-Id: 6b14225a-d56c-43be-4a0c-08dd1a4363d9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|366016; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?tZpa+yx+iEHB029YVpXnnlcdBRBUnDmRCXCsaSpq/IVk5NGAZGddVtjRjZX4?= =?us-ascii?Q?/indzZ/6bjGw6Pb+RjVT+NihKCdsmJJzOkm5GvGiEl4kgUPtHzRtIMB9SK1y?= =?us-ascii?Q?HKppMRA4iRLSBspXL2jOxyh/4LyUqLCGc2UET4kUS98n+UV+s5nv5HfjVGfB?= =?us-ascii?Q?bACIaIV1Yl710DWQALzmJDY8rTRdrRpdd7gJOnx8ow1SOYv2gYK/EqQ3SUqQ?= =?us-ascii?Q?gILaktqEnOrcgqkxTtPPqrYEESIX99uiL1hUal1sKaKwg/hobDTFS9kGv7Lm?= =?us-ascii?Q?fiPlG4tvJrdpBPEAZiLV/ZEL8aOiANiWYYLtcaBlDQK9zLlRflfmtLMtlc+j?= =?us-ascii?Q?R/j+jvjSqMmPpX+/V4g4IkNYGu81V3yxKdnkJKWle08nhE3Ds7JOThnSU7wi?= =?us-ascii?Q?S1FEvk5eWzma4BAA+5kRn3bmjsEzQN7PA6e6jb7ve11F8a81iTo3JXaDIx1B?= =?us-ascii?Q?dMAJ7ZeXNeXi3aijDRaH9a6tWm1Jxv1j782SKnREn8g/1F8E+Yjb4y81YGjx?= =?us-ascii?Q?muDNtilONeZquNRViXauwOWqYir1zVBrUB6ChdPEXF6VMu9kv/HQ9gxrO52P?= =?us-ascii?Q?kRzib9UmBpMx7yAqSiPNV5Q1RhbIHibA5XoIXHTx3VJ6GUBFjHgUM5dD65I7?= =?us-ascii?Q?L/eAHUW6ERgUsRHglGGnrFw9A2FsXu3C5tUMNJTe+bOJ5cwSbuQPNWKQJ6m8?= =?us-ascii?Q?krjFC2QmBs9iLyh1UpF58zGAGSoRaGUB5q3y21F1FKdmHqHCjFfj1lZruc+Z?= =?us-ascii?Q?5kBNcUYFjdowsDU8OQligNptXkOUVI6Mq16s0FOBcdUgNec8qMAAgI7MO8/L?= =?us-ascii?Q?iLWNhjwOd5UuwHL7XcJedgj1ymueL4L83uiQ2nEJ9eRCRvXvIGcIzbyrombV?= =?us-ascii?Q?BGlMtBJGQVYnmlfFgqgWyCMoAMoCpG+7o3fxw6X8k82dL95gSPsjCIF0dHMK?= =?us-ascii?Q?r8suNVVrh7Vk1lbxQT+gbpkpVYXqXYsEzcPG+/C4NMfaTOMJDAyK3TU6es6S?= =?us-ascii?Q?fRShdiQX6TLD75HYLJPEqFfHTrxyhH9HC9ElqKaYnz68Rbq5mRgq2jMXVWAp?= =?us-ascii?Q?zFelgyc0biGxm3ZSEnzKKJv8FRURk6B3r9cug7eRjPikvSgYoT8gjVjySnno?= =?us-ascii?Q?vUXyGMAcK89fAbp0DRXR46W7Ol8dPeVPjW46uuVIkqFtlgOSdc3IliqX4HsV?= =?us-ascii?Q?1EQ8ShXSRqYZq6ufgoym7+R8QYQ3oVKkPDO1Jzamt0XrFfTDS7r72XGGV0VS?= =?us-ascii?Q?MLEWYs4C0H4E/aDlZQiEZ7T5E+U/ytYRzpJ853k2x3xwH6ofgnq5JmOuHzHL?= =?us-ascii?Q?u5qZErh+/fwb9b1e7dP53gfQ3X9qAYaVZ0CjlDIpi15tSQ=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SN7PR12MB6838.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?bxHhiug7MfKYwDz6luKoVhyfhVdW8NLGIdGdvMw28x8AllZfhZqruJhBZj6a?= =?us-ascii?Q?YdGk0ilUvCMzi3wDIqV0fi07DnL91vonW6oFAaDIHIQKUVcksr1xJtPDiIYf?= =?us-ascii?Q?0Tn/r0fCfkoAuQUdz1PVn9IxneUw5YK5a2saUWUnfGf3Il71a/nrVsGoRRi6?= =?us-ascii?Q?oobvTffXweRgXG2ZvmRZwXLot2jr4G/hJ9t57xKWiCM46Me1e1a2Ykyyr3pY?= =?us-ascii?Q?BLdqaASI4jvEcBPATGwXuImTV47KFbuSrpxKO0JigpMv2n3ejRyyTql1xDfm?= =?us-ascii?Q?7REK00ilCrhoYET6r5eWNVIG9hd/+o6zfnkQ18rKTF5WMOQXjgkuSmm5ej0F?= =?us-ascii?Q?86Np6iQCPSF7NenMwu1hS/O/j2pKy4DwkpkPfyjTWJRhIpBX1ayTKGvjycdr?= =?us-ascii?Q?HV99tncHL6Yof3hvRb6aLj4yTAwdrjQGQ1n11AB8lzWHsXCwWdjKaCURC84u?= =?us-ascii?Q?h4YIJyVXxHquxqSyuGsgf1xDmO6NpV4RZtz3GFOVNSOA2NAe24AidRmKBVz0?= =?us-ascii?Q?EBgyWl7B/LvAe1IQcXWT0aWHV1t4tyya/v8uhH48OB6G0OhWyM6BvQcNbMxu?= =?us-ascii?Q?waS7ym1xwd2tz64nxpBhZJgCU4OG1PXu6m2U9YmjKxCmm8EgURoSG6TcIAFd?= =?us-ascii?Q?S/lhEJfLfzO/nqaA+rreI/jUV3yjg0TdCokrqjyLzHfVggauQZwDYnuK5IA2?= =?us-ascii?Q?5iVSw9c0/NqjUOngBjpC5/fjjsVTCUxxzZt1cUXvh13oBkKRvSkHD+k9o3aW?= =?us-ascii?Q?2/LfOwzK60zdybeprDdCeJe15OJxDRFDSzBWA2LnK+MQ2K4bPkOSZQFP83xa?= =?us-ascii?Q?HU4GUTTBPwB9DYeMuNZ1R6bjXdDu+paLiwYd0qrx5AfUD4eoDVFkE1MUhUkV?= =?us-ascii?Q?m1cXhsYSQJaSVLpy79vyXICBUqZKxo7d8Po8+scJSC6o2zz9S+j9AalxBT9D?= =?us-ascii?Q?Loitia/5+JACE9h6n76b6WhovN2+aCymH0/45UdG67bNHp31TwG/4QuGT/m/?= =?us-ascii?Q?xIoe6jFJEaVa8SzkIRiy0AU+u/bi0fpbjieJknP3scCI+S7OSqT8BgvdizJR?= =?us-ascii?Q?vSdIsyWDpTruKLiulhtZt4v61cD8brNYTKhH1BDr3s2engWmV7F87BBL4cNS?= =?us-ascii?Q?Jt7mW+ZJFvvurIXCbGPBbocEu041+xOnaBU8pZaW1GWE63LYbfQs8NY7/Ip4?= =?us-ascii?Q?vRBeL5scs5A4Ko+fHpgdK6QkpnXRJHM920C3r17B+MwQ+Hk2firmuN/SPe6j?= =?us-ascii?Q?0IuXhzWL8gFLCGGf5SrKSaMtRKRDNK/B44+2GqrmvcCNsb+btRgZeTI9LHh8?= =?us-ascii?Q?YGZ1Ir4iyqvRbd6zdjFxblguj82//19HJOVDnrbUOGAC3E3vfCaKZIJaJ1kS?= =?us-ascii?Q?F2ID39c4AaywZR4Ky15IqkhULtRArZvrV9LRgKd887luNlXfWOJW6Iqg8sbh?= =?us-ascii?Q?CUBWAWByMmfZG3KID/X1iMFVX2WJvrKNFSNpOc/UPhdSLZUyZ6UuoeRHmAzY?= =?us-ascii?Q?JkH9xPnlmN6/EAQhLen/IFZA5zpy0R0Y60k80oaS2yNj9laDM9dHdGM73F6O?= =?us-ascii?Q?1WIYD6Ui0VbyKkNKBQIxJINb0BVf+jyU58Z7PaMr?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6b14225a-d56c-43be-4a0c-08dd1a4363d9 X-MS-Exchange-CrossTenant-AuthSource: SN7PR12MB6838.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Dec 2024 00:24:49.8359 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 33H8c17GUvmk+T4cqGhxO+ORQh3KPvrkFrMeSU3WYiSvBtnYFHLrG8v4CHsEtiieBuyfDG9IgqoTqhNzMLj/3w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6322 Content-Transfer-Encoding: quoted-printable Message-ID-Hash: LDEOWD77CVBQWF2QPLN6BPYMWKEBCU7X X-Message-ID-Hash: LDEOWD77CVBQWF2QPLN6BPYMWKEBCU7X X-MailFrom: nathanc@nvidia.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-config-1; header-match-config-2; header-match-config-3; header-match-devel.lists.libvirt.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header CC: Nicolin Chen , Shameerali Kolothum Thodi , Nathan Chen X-Mailman-Version: 3.2.2 Precedence: list List-Id: Development discussions about the libvirt library & tools Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: From: Nathan Chen via Devel Reply-To: Nathan Chen X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1733963282502116600 Content-Type: text/plain; charset="utf-8" Add a pcie-expander-bus controller to the VM definition for each "nestedSmm= uv3" device that is generated when the "nestedSmmuv3" IOMMU model is parsed from= the VM definition. Assign each "nestedSmmuv3" device to one PXB controller, and route any unmanaged "hostdev" VFIO devices with associated host SMMU nodes = to their corresponding PXB controller based on the "name" attributes of "nestedSmmuv3" devices attached to these PXB controllers. Signed-off-by: Nathan Chen --- src/conf/domain_addr.c | 26 ++++++- src/conf/domain_addr.h | 3 +- src/conf/domain_conf.c | 1 + src/qemu/qemu_domain_address.c | 134 +++++++++++++++++++++++++++++++++ 4 files changed, 162 insertions(+), 2 deletions(-) diff --git a/src/conf/domain_addr.c b/src/conf/domain_addr.c index a53ff6df6c..6d8ca89025 100644 --- a/src/conf/domain_addr.c +++ b/src/conf/domain_addr.c @@ -386,6 +386,8 @@ virDomainPCIAddressFlagsCompatible(virPCIDeviceAddress = *addr, connectStr =3D "pcie-expander-bus"; } else if (devFlags & VIR_PCI_CONNECT_TYPE_PCI_BRIDGE) { connectStr =3D "pci-bridge"; + } else if (devFlags & VIR_PCI_CONNECT_TYPE_NESTED_SMMUV3) { + connectStr =3D "nestedSmmuv3 device"; } else { /* this should never happen. If it does, there is a * bug in the code that sets the flag bits for devices. @@ -565,7 +567,8 @@ virDomainPCIAddressBusSetModel(virDomainPCIAddressBus *= bus, * dmi-to-pci-bridge */ bus->flags =3D (VIR_PCI_CONNECT_TYPE_PCIE_ROOT_PORT | - VIR_PCI_CONNECT_TYPE_DMI_TO_PCI_BRIDGE); + VIR_PCI_CONNECT_TYPE_DMI_TO_PCI_BRIDGE | + VIR_PCI_CONNECT_TYPE_NESTED_SMMUV3); bus->minSlot =3D 0; bus->maxSlot =3D VIR_PCI_ADDRESS_SLOT_LAST; break; @@ -690,6 +693,8 @@ virDomainPCIAddressSetGrow(virDomainPCIAddressSet *addr= s, } else if (flags & (VIR_PCI_CONNECT_TYPE_PCIE_DEVICE | VIR_PCI_CONNECT_TYPE_PCIE_SWITCH_UPSTREAM_PORT)) { model =3D VIR_DOMAIN_CONTROLLER_MODEL_PCIE_ROOT_PORT; + } else if (flags & VIR_PCI_CONNECT_TYPE_NESTED_SMMUV3) { + model =3D VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS; } else { /* The types of devices that we can't auto-add a controller for: * @@ -1030,6 +1035,11 @@ virDomainPCIAddressFindUnusedFunctionOnBus(virDomain= PCIAddressBus *bus, break; } =20 + if (flags =3D=3D VIR_PCI_CONNECT_TYPE_NESTED_SMMUV3) { + *found =3D false; + break; + } + if (flags & VIR_PCI_CONNECT_AGGREGATE_SLOT && bus->slot[searchAddr->slot].aggregate) { /* slot and device are okay with aggregating devices */ @@ -1087,6 +1097,20 @@ virDomainPCIAddressGetNextAddr(virDomainPCIAddressSe= t *addrs, else a.function =3D function; =20 + if (flags =3D=3D VIR_PCI_CONNECT_TYPE_NESTED_SMMUV3) { + if (addrs->dryRun) { + virDomainPCIAddressBus *bus =3D &addrs->buses[addrs->nbuses - = 1]; + /* a is already set to the first new bus */ + a.bus =3D addrs->nbuses; + a.slot =3D bus->minSlot; + if (virDomainPCIAddressSetGrow(addrs, &a, flags) < 0) + return -1; + /* this device will use the first slot of the new bus */ + a.slot =3D addrs->buses[a.bus].minSlot; + goto success; + } + } + /* When looking for a suitable bus for the device, start by being * very strict and ignoring all those where the isolation groups * don't match. This ensures all devices sharing the same isolation diff --git a/src/conf/domain_addr.h b/src/conf/domain_addr.h index 9781685903..2881f2dadb 100644 --- a/src/conf/domain_addr.h +++ b/src/conf/domain_addr.h @@ -72,7 +72,8 @@ typedef enum { VIR_PCI_CONNECT_TYPE_PCI_EXPANDER_BUS | \ VIR_PCI_CONNECT_TYPE_PCIE_EXPANDER_BUS | \ VIR_PCI_CONNECT_TYPE_PCI_BRIDGE | \ - VIR_PCI_CONNECT_TYPE_PCIE_TO_PCI_BRIDGE) + VIR_PCI_CONNECT_TYPE_PCIE_TO_PCI_BRIDGE | \ + VIR_PCI_CONNECT_TYPE_NESTED_SMMUV3) =20 /* combination of all bits that could be used to connect a normal * endpoint device (i.e. excluding the connection possible between an diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c index 24aff1cfbe..46f9b9b0cf 100644 --- a/src/conf/domain_conf.c +++ b/src/conf/domain_conf.c @@ -25,6 +25,7 @@ #include #include #include +#include =20 #include "configmake.h" #include "internal.h" diff --git a/src/qemu/qemu_domain_address.c b/src/qemu/qemu_domain_address.c index 31004bfc7e..dee198a7d2 100644 --- a/src/qemu/qemu_domain_address.c +++ b/src/qemu/qemu_domain_address.c @@ -1627,6 +1627,19 @@ qemuDomainPCIAddressSetCreate(virDomainDef *def, =20 addrs->dryRun =3D dryRun; =20 + /* PXB indices must come before pcie-root-port indices in qemu, + * so add PXB buses to addrs before the pcie-root-ports. */ + + if (addrs->dryRun) { + for (i =3D 0; i < def->nnestedsmmus; i++) { + if (!virDeviceInfoPCIAddressIsWanted(def->nestedsmmus[i]->info= )) + continue; + if (qemuDomainPCIAddressReserveNextAddr(addrs, + def->nestedsmmus[i]->i= nfo) < 0) + return NULL; + } + } + /* pSeries domains support multiple pci-root controllers */ if (qemuDomainIsPSeries(def)) addrs->areMultipleRootsSupported =3D true; @@ -2030,6 +2043,109 @@ qemuDomainValidateDevicePCISlotsChipsets(virDomainD= ef *def, } =20 =20 +static char* +retrieveSysfsDevPath(virPCIDeviceAddress* addr, const char* path) +{ + return g_strdup_printf("/sys/bus/pci/devices/%04x:%02x:%02x.%01x%s", + addr->domain, + addr->bus, + addr->slot, + addr->function, + path ? path : ""); +} + + +static char * +nestedSmmuVfioHostdevFound(virDomainHostdevDef *hostdev, bool dryRun) +{ + char* devPath =3D NULL; + char* devSmmuPath =3D NULL; + char* devVFIOPath =3D NULL; + g_autoptr(DIR) dir =3D NULL; + g_autoptr(DIR) smmuDir =3D NULL; + g_autoptr(DIR) VFIODir =3D NULL; + char* dir_iommu =3D NULL; + char* smmu_node =3D NULL; + devPath =3D retrieveSysfsDevPath(&hostdev->source.subsys.u.pci.addr, "= "); + if (virDirOpenIfExists(&dir, devPath) < 1) + return NULL; + devSmmuPath =3D retrieveSysfsDevPath(&hostdev->source.subsys.u.pci.add= r, "/iommu"); + if (virDirOpenIfExists(&smmuDir, devSmmuPath) < 1) + return NULL; + devVFIOPath =3D retrieveSysfsDevPath(&hostdev->source.subsys.u.pci.add= r, "/vfio-dev"); + if (hostdev->source.subsys.type =3D=3D VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_= PCI && + (hostdev->info->addr.pci.bus !=3D 0 || dryRun)) { + // We only want to route vfio hostdevs + if (hostdev->managed || + (virDirOpenIfExists(&VFIODir, devVFIOPath) =3D=3D 1)) { + // Get the hostdev's associated SMMU node name + dir_iommu =3D realpath(devSmmuPath, NULL); + if (!dir_iommu) + return NULL; + smmu_node =3D g_path_get_basename(dir_iommu); + if (!smmu_node) + return NULL; + } + } + return smmu_node; +} + + +static virDomainControllerDef * +qemuDomainGetUpstreamCont(virDomainDef *def, + virDomainDeviceInfo *downstreamInfo, + int model) +{ + size_t i; + for (i =3D 0; i < def->ncontrollers; i++) { + if (def->controllers[i]->idx =3D=3D downstreamInfo->addr.pci.bus && + def->controllers[i]->model =3D=3D model) + return def->controllers[i]; + } + return NULL; +} + + +static int +qemuDomainAssignNestedSmmuv3HostdevSlots(virDomainDef *def, + virDomainPCIAddressSet *addrs) +{ + size_t i, j; + char* smmu_node =3D NULL; + virDomainControllerDef *rootPort; + virDomainPCIAddressSet *set =3D NULL; + if (def->iommu !=3D NULL && def->iommu->model =3D=3D VIR_DOMAIN_IOMMU_= MODEL_NESTED_SMMUV3 && + def->nnestedsmmus > 0) { + for (i =3D 0; i < def->nhostdevs; i++) { + if (!(smmu_node =3D nestedSmmuVfioHostdevFound(def->hostdevs[i= ], addrs->dryRun))) + continue; + /* Find a hostdev and nested SMMU pair */ + for (j =3D 0; j < def->nnestedsmmus; j++) { + unsigned int nestedSmmuBus =3D def->nestedsmmus[j]->info->= addr.pci.bus; + virDomainControllerDef *pxb; + if (!STREQLEN(def->nestedsmmus[j]->name, smmu_node, strlen= (smmu_node))) + continue; + /* Get the hostdev's pcie-root-port controller */ + rootPort =3D qemuDomainGetUpstreamCont(def, def->hostdevs[= i]->info, + VIR_DOMAIN_CONTROLLER= _MODEL_PCIE_ROOT_PORT); + /* Skip if already assigned */ + pxb =3D qemuDomainGetUpstreamCont(def, &rootPort->info, + VIR_DOMAIN_CONTROLLER_MODE= L_PCIE_EXPANDER_BUS); + if (pxb) + break; + /* Assign the controller to the next available slot/func on + * the corresponding PXB */ + set =3D virDomainPCIAddressSetAlloc(nestedSmmuBus + 1, + VIR_PCI_ADDRESS_EXTENSIO= N_NONE); + set->buses[nestedSmmuBus] =3D addrs->buses[nestedSmmuBus]; + qemuDomainPCIAddressReserveNextAddr(set, &rootPort->info); + break; + } + } + } + return 0; +} + /* * This assigns static PCI slots to all configured devices. * The ordering here is chosen to match the ordering used @@ -2262,6 +2378,18 @@ qemuDomainAssignDevicePCISlots(virDomainDef *def, return -1; } =20 + /* Nested SMMUs */ + if (!addrs->dryRun) { + for (i =3D 0; i < def->nnestedsmmus; i++) { + if (!virDeviceInfoPCIAddressIsWanted(def->nestedsmmus[i]->info= )) + continue; + + if (qemuDomainPCIAddressReserveNextAddr(addrs, + def->nestedsmmus[i]->i= nfo) < 0) + return -1; + } + } + /* Host PCI devices */ for (i =3D 0; i < def->nhostdevs; i++) { virDomainHostdevSubsys *subsys =3D &def->hostdevs[i]->source.subsy= s; @@ -2286,6 +2414,12 @@ qemuDomainAssignDevicePCISlots(virDomainDef *def, return -1; } =20 + // Route hostdevs to nested SMMUs + if (!addrs->dryRun) { + if (qemuDomainAssignNestedSmmuv3HostdevSlots(def, addrs) < 0) + return -1; + } + /* memballoon. the qemu driver only accepts virtio memballoon devices = */ if (virDomainDefHasMemballoon(def) && virDeviceInfoPCIAddressIsWanted(&def->memballoon->info)) { --=20 2.34.1