From nobody Thu Nov 21 21:47:41 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass(p=quarantine dis=none) header.from=amd.com ARC-Seal: i=2; a=rsa-sha256; t=1728660491; cv=pass; d=zohomail.com; s=zohoarc; b=m+/+FsgV5S176t5TaSlhuqhHA8LFw55yQpoj0Tpl/i5jy6VzKIuug+kQ14TCFvUY28+AdyvLt+s/lCpblppNHnVt/fYTm9b7MGkIJ0igBxVu0XSvx2t3Eeerwvx4KDuUsZbasCArUl6WXBIShtftEZz6K/qnIGJ3JyE03yGqSzo= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1728660491; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Bae18Nq7AOaW1aZ8YIQKtu/Xs/0xVUuY6JnFn3meqtw=; b=Ail5jiTLCaWYx6UBcigKgMWo3HBBbP2gPDnXt1To/iCr4jqMckuBIwV4yfY4P06TkLtmWN3GdqhAe1pDhgwtSEHvga3pw1H04Iv2p6yUgU3n6qqXGrkLTXRqgsG7S3moLwWEm/T/97dMBFn6f/rBxZOItrOBGhzPa1D2lLW7ijQ= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1728660491689555.9273585024991; Fri, 11 Oct 2024 08:28:11 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.817224.1231307 (Exim 4.92) (envelope-from ) id 1szHYY-0007Tj-9s; Fri, 11 Oct 2024 15:27:54 +0000 Received: by outflank-mailman (output) from mailman id 817224.1231307; Fri, 11 Oct 2024 15:27:54 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1szHYY-0007TS-6R; Fri, 11 Oct 2024 15:27:54 +0000 Received: by outflank-mailman (input) for mailman id 817224; Fri, 11 Oct 2024 15:27:52 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1szHYW-0007SS-Er for xen-devel@lists.xenproject.org; Fri, 11 Oct 2024 15:27:52 +0000 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on20622.outbound.protection.outlook.com [2a01:111:f403:2418::622]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 5e9855e1-87e5-11ef-a0bd-8be0dac302b0; Fri, 11 Oct 2024 17:27:51 +0200 (CEST) Received: from BN7PR02CA0008.namprd02.prod.outlook.com (2603:10b6:408:20::21) by IA1PR12MB7639.namprd12.prod.outlook.com (2603:10b6:208:425::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.16; Fri, 11 Oct 2024 15:27:42 +0000 Received: from BN2PEPF00004FBA.namprd04.prod.outlook.com (2603:10b6:408:20:cafe::3c) by BN7PR02CA0008.outlook.office365.com (2603:10b6:408:20::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.20 via Frontend Transport; Fri, 11 Oct 2024 15:27:42 +0000 Received: from SATLEXMB04.amd.com (165.204.84.17) by BN2PEPF00004FBA.mail.protection.outlook.com (10.167.243.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8048.13 via Frontend Transport; Fri, 11 Oct 2024 15:27:41 +0000 Received: from SATLEXMB04.amd.com (10.181.40.145) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 11 Oct 2024 10:27:41 -0500 Received: from ubuntu.mshome.net (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Fri, 11 Oct 2024 10:27:40 -0500 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 5e9855e1-87e5-11ef-a0bd-8be0dac302b0 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=PVuecqEyK9QUPIM/OucNgwHJLHQqk9B2HELaqoNmi/PrRqgkHLFflWRpNamcapp+o7YKCasjXPEn1FPGlO4ymN5ZPuz8zRkA8/5cdzuDWzzNKMBBZxfFE+rXAZ5QEIbsRMmD7bugkOuV3MJuMQ60/MvyzeZ3WWLwZMux4L9KiTg1uE3i9b0Q7KHus5VslEdfDlOa561d8t73iXq1zkkfFvgM23+ERobLWRwBQbVrFhGAw5trMacoZMQq1gn+d2kBpb3be8fJ9vb1UrT1uyCUvT796VDfusUIkIgnQLpSl64Ij8qF59GrYFhxiBqF2gec2DzYUCW7r2qb8Fo05tbaew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Bae18Nq7AOaW1aZ8YIQKtu/Xs/0xVUuY6JnFn3meqtw=; b=PTP82t8gf/ZrirNSf48Hfa8MFV17jamp7YngmCIlwzNbcIQ4L5FavPPRIyhSNYkGIyL5mlnzEvBg/ClAM2toHAmLBR6tI4UwUDpHcQ2m4tckU31hcpOUwjJrVLP+4ESyDS0Gz0jnyC5ANZbzKirMRlWeUXo08t9513mjW/+ZhOBlQ/MkimcPLmeBH05LT59+lb0N5NbUqGJ8U3BAvaHv6q8WM/k7O3iIKzMZvTEX0VE0/PN81YyHDmSxANRjFVTJFPN8kubavth4a9lM21RjRHE/gOEcvre0oaDDa4PmypliyA7HICJvq1cb151dE40XjZcirTl0Ri/602CbIuFNcA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Bae18Nq7AOaW1aZ8YIQKtu/Xs/0xVUuY6JnFn3meqtw=; b=XqoI5GEhYbqPzhOtXcYTs401jY++u3veKjDiOnUGKXUB3o6rVB7O4jVcd0u3ZVrYCi3ev9UELyQJHFbqjJjcBKzWRwsyjjJT0yISjF41YfPvAblrt15DG4HwPnSU4kv+tSZxjOA4hLf2nSyA0MX9zcaGCvl4lBZf64xOM6Qid+U= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C From: Stewart Hildebrand To: CC: Stewart Hildebrand , Jan Beulich , Andrew Cooper , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= Subject: [PATCH v5 1/3] x86/msi: harden stale pdev handling Date: Fri, 11 Oct 2024 11:27:06 -0400 Message-ID: <20241011152727.366770-2-stewart.hildebrand@amd.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241011152727.366770-1-stewart.hildebrand@amd.com> References: <20241011152727.366770-1-stewart.hildebrand@amd.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: None (SATLEXMB04.amd.com: stewart.hildebrand@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF00004FBA:EE_|IA1PR12MB7639:EE_ X-MS-Office365-Filtering-Correlation-Id: 55b1eea3-583b-4af9-a953-08dcea093f56 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?ScdZELdIUTo+L1yeaV1r4m9phOlErMBPrkEA3RznZclUiCyJY6ukI4UIXovs?= =?us-ascii?Q?6QvCMCMayECi6La05VwbD5QdFtSl5NSaDWqTkQrq6IpObSosFLHzgKW6dX7e?= =?us-ascii?Q?6AX7wrrNxKeLcVkO7B2HddhOB2PKkXAC9izAeNlZdiUprEQJuGnbyBLjDCo3?= =?us-ascii?Q?c/venKIyRs/xbxffLMa8q3PDnSSNDoea8zkmUxYY6CBPzK1/+HTnyccDGZ/e?= =?us-ascii?Q?Kq3RPEncr9NCuVrIAoYyYqB9TDcI1rTirMHCNy9lxnUL4oi3MCzdVh7ar2fb?= =?us-ascii?Q?vOO8H9R1R0JnZKZj+dJo0y3TLc2Kpqu5LDIDpl8NqPPprJctJQ0eoUs3vmeW?= =?us-ascii?Q?tA9z1J1USeHtA4OwWtL5cn3bPpSR+M0bg32rBZ3zyVRGHCUlSzWPZD9vg7Fz?= =?us-ascii?Q?jzEaSo7vHWddZuRPS9cXYXjxS9F0jA86bhPKWKQw4AK4Mz0FGEwQi+a4SwWJ?= =?us-ascii?Q?ZLsf5g/sxE782WggHjv2B3xV+5gyNFIR7/Huz4KBBX5ARhevRjor5nMebtWr?= =?us-ascii?Q?grnJvCPjh92xFtYXnIJRDuZJal0peMIq833Q4TMXR6rMqxa9AWG+9ZWSQ4tA?= =?us-ascii?Q?cDh1n/GuqANiDaXa7PU3iDNvI45jTbxDKxc1XZSKsJrIOeNBOSSu2q1arrAo?= =?us-ascii?Q?+UQWeQO/RoaCUOrv3ZYvgitVwr104YRcOps9gL4ggLyC/88klss3t7uD1xPv?= =?us-ascii?Q?sKBYRQopfCIpMUb46Xrk9QWW4Y7sjcnn80HqKd8ponDXKOn+o7dG1i8BFA4X?= =?us-ascii?Q?HZAs0MRLqk7XLaIut6PMYXb5cly+zbU0tazAAvTQsDJbpEamytAhqbtXkJr0?= =?us-ascii?Q?P5Be3BFmZaZCUNS3asmrUIZi1QVhPMyJ0X/G38XDeadKgvHBC6mQoukAhoNn?= =?us-ascii?Q?8rwmrPIrPZ91od/jjwwE6MEy4jUoSl88UwdPWVL46deEobwrh88rhOjP1wG4?= =?us-ascii?Q?KDeIyc0JUFv8dsFMY4XANZBsr6viwUvG6zHuQITM6EihXEoaclxWwtFOvayF?= =?us-ascii?Q?b7VZYCw9C26KqZDfaqXE7IEWzhHYuDfx+5IUuYG786UoYWwIuaQbr5dysT+K?= =?us-ascii?Q?q/K3uJfdthaG9R+XVtRrkH8gng+cbOPTU7JhjK5Y8gh5MnHOdeuiDGWtQymd?= =?us-ascii?Q?6J1iD084uJuqoN70fPrmSbX9hWgeXBGsZs2e11A4Aq/J6e4o6tLIr/mD8dij?= =?us-ascii?Q?xlfAy7WQAYzsaOBxGTj9TVjqhUreK41UzW9xlNyfKrFZHbLmXueAvCG99tsF?= =?us-ascii?Q?7e/NyD7kz3+CCLDfSbVr7sgbW3hjAef9zhAJynaKP1+XU3AxOcPy4oNkHFr+?= =?us-ascii?Q?ayxwz8fA8DvgOQ+nHXGnpvcJ436Rsm/5cMA7OTWHXAAxbDZPapzUZDe1466j?= =?us-ascii?Q?Q0wstyJkvQyl5CDCBCSJRxx30bSc?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Oct 2024 15:27:41.7392 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 55b1eea3-583b-4af9-a953-08dcea093f56 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF00004FBA.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB7639 X-ZohoMail-DKIM: pass (identity @amd.com) X-ZM-MESSAGEID: 1728660493437116600 Content-Type: text/plain; charset="utf-8" Dom0 normally informs Xen of PCI device removal via PHYSDEVOP_pci_device_remove, e.g. in response to SR-IOV disable or hot-unplug. We might find ourselves with stale pdevs if a buggy dom0 fails to report removal via PHYSDEVOP_pci_device_remove. In this case, attempts to access the config space of the stale pdevs would be invalid and return all 1s. Some possible conditions leading to this are: 1. Dom0 disables SR-IOV without reporting VF removal to Xen. The Linux SR-IOV subsystem normally reports VF removal when a PF driver disables SR-IOV. In case of a buggy dom0 SR-IOV subsystem, SR-IOV could become disabled with stale dangling VF pdevs in both dom0 Linux and Xen. 2. Dom0 reporting PF removal without reporting VF removal. During SR-IOV PF removal (hot-unplug), a buggy PF driver may fail to disable SR-IOV, thus failing to remove the VFs, leaving stale dangling VFs behind in both Xen and Linux. At least Linux warns in this case: [ 100.000000] 0000:01:00.0: driver left SR-IOV enabled after remove In either case, Xen is left with stale VF pdevs, risking invalid PCI config space accesses. When Xen is built with CONFIG_DEBUG=3Dy, the following Xen crashes were observed when dom0 attempted to access the config space of a stale VF: (XEN) Assertion 'pos' failed at arch/x86/msi.c:1274 (XEN) ----[ Xen-4.20-unstable x86_64 debug=3Dy Tainted: C ]---- ... (XEN) Xen call trace: (XEN) [] R pci_msi_conf_write_intercept+0xa2/0x1de (XEN) [] F pci_conf_write_intercept+0x68/0x78 (XEN) [] F arch/x86/pv/emul-priv-op.c#pci_cfg_ok+0xa0/= 0x114 (XEN) [] F arch/x86/pv/emul-priv-op.c#guest_io_write+0= xb5/0x1c8 (XEN) [] F arch/x86/pv/emul-priv-op.c#write_io+0x9a/0x= e0 (XEN) [] F x86_emulate+0x100e5/0x25f1e (XEN) [] F x86_emulate_wrapper+0x29/0x64 (XEN) [] F pv_emulate_privileged_op+0x12e/0x217 (XEN) [] F do_general_protection+0xc2/0x1b8 (XEN) [] F x86_64/entry.S#handle_exception_saved+0x2b/= 0x8c (XEN) Assertion 'pos' failed at arch/x86/msi.c:1246 (XEN) ----[ Xen-4.20-unstable x86_64 debug=3Dy Tainted: C ]---- ... (XEN) Xen call trace: (XEN) [] R pci_reset_msix_state+0x47/0x50 (XEN) [] F pdev_msix_assign+0x19/0x35 (XEN) [] F drivers/passthrough/pci.c#assign_device+0x1= 81/0x471 (XEN) [] F iommu_do_pci_domctl+0x248/0x2ec (XEN) [] F iommu_do_domctl+0x26/0x44 (XEN) [] F do_domctl+0x8c1/0x1660 (XEN) [] F pv_hypercall+0x5ce/0x6af (XEN) [] F lstar_enter+0x143/0x150 Replace the ASSERT(s) with an error, and mark the device broken to disallow passthrough to domUs. Fixes: 484d7c852e4f ("x86/MSI-X: track host and guest mask-all requests sep= arately") Fixes: 575e18d54d19 ("pci: clear {host/guest}_maskall field on assign") Signed-off-by: Stewart Hildebrand --- v4->v5: * new patch, independent of the rest of the series * new approach to fixing the issue: don't rely on dom0 to report any sort of device removal; rather, fix the condition directly --- Instructions to reproduce Requires Xen with CONFIG_DEBUG=3Dy Tested with Linux 6.11 1. Dom0 disables SR-IOV without reporting VF removal to Xen. * Hack the Linux SR-IOV subsystem to remove the call to pci_stop_and_remove_bus_device() in drivers/pci/iov.c:pci_iov_remove_virtfn(). * Enable SR-IOV, then disable SR-IOV echo 1 > /sys/bus/pci/devices/0000\:01\:00.0/sriov_numvfs echo 0 > /sys/bus/pci/devices/0000\:01\:00.0/sriov_numvfs * Now we have a stale VF. We can trigger the ASSERT either by unbinding the VF driver and issuing a reset... echo 0000\:01\:10.0 > /sys/bus/pci/devices/0000\:01\:10.0/driver/unbind echo 1 > /sys/bus/pci/devices/0000\:01\:10.0/reset ... or by doing xl pci-assignable-add xl pci-assignable-add 01:10.0 2. Dom0 reporting PF removal without reporting VF removal. * Hack your PF driver to leave SR-IOV enabled when removing the device * Enable SR-IOV echo 1 > /sys/bus/pci/devices/0000\:01\:00.0/sriov_numvfs * Unplug the PCI device (qemu) device_del mydev * Now we have a stale VF. We can trigger the ASSERT either by re-adding the PF device with SR-IOV disabled... echo 0000\:01\:10.0 > /sys/bus/pci/devices/0000\:01\:10.0/driver/unbind (qemu) device_add igb,id=3Dmydev,bus=3Dpcie.1,netdev=3Dnet1 ... or by reset / xl pci-assignable-add as above. --- xen/arch/x86/msi.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/xen/arch/x86/msi.c b/xen/arch/x86/msi.c index ff2e3d86878d..fbb07fe821b5 100644 --- a/xen/arch/x86/msi.c +++ b/xen/arch/x86/msi.c @@ -1243,7 +1243,12 @@ int pci_reset_msix_state(struct pci_dev *pdev) { unsigned int pos =3D pci_find_cap_offset(pdev->sbdf, PCI_CAP_ID_MSIX); =20 - ASSERT(pos); + if ( !pos ) + { + pdev->broken =3D true; + return -EFAULT; + } + /* * Xen expects the device state to be the after reset one, and hence * host_maskall =3D guest_maskall =3D false and all entries should hav= e the @@ -1271,7 +1276,12 @@ int pci_msi_conf_write_intercept(struct pci_dev *pde= v, unsigned int reg, entry =3D find_msi_entry(pdev, -1, PCI_CAP_ID_MSIX); pos =3D entry ? entry->msi_attrib.pos : pci_find_cap_offset(pdev->sbdf, PCI_CAP_ID_MSIX); - ASSERT(pos); + + if ( !pos ) + { + pdev->broken =3D true; + return -EFAULT; + } =20 if ( reg >=3D pos && reg < msix_pba_offset_reg(pos) + 4 ) { --=20 2.47.0 From nobody Thu Nov 21 21:47:41 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass(p=quarantine dis=none) header.from=amd.com ARC-Seal: i=2; a=rsa-sha256; t=1728660494; cv=pass; d=zohomail.com; s=zohoarc; b=aKAt09featFy8PeVMLQYCCVQ2kHmbKMawU2s8J9TZVikEWybx6OORkxW3FbHg8JnO/LmBBEC5Xih8TMQ2bmQ6KEyb7dvUgpbhGnWXD/YeN+0akZSWzriXTQ5hbVYPAqLx6WR3I+DSKwQQYiBKmi0hLjXwXlhfSMqoXx/Nv6my10= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1728660494; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=0hdJcd67VFCUntI2TnhTshHdpS7eovY/+F2haWAsl40=; b=CR38JJUFBA6ga/XmAyEPWT9szTN6EGNHc9QAjNO1qDKirg7guEAjyxHV+m3U99WKAebQqtqmqnpQz8kY9FNV8rpj7iPj7n0BjI3QD56B4IQvIibmtL2aTwJkAtgS9s6u/sFThrB49Xoap1+1Egpb3i94p/hglBAej/WqOYpbQ+U= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1728660494971329.7276342028215; Fri, 11 Oct 2024 08:28:14 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.817225.1231316 (Exim 4.92) (envelope-from ) id 1szHYb-0007kE-Iz; Fri, 11 Oct 2024 15:27:57 +0000 Received: by outflank-mailman (output) from mailman id 817225.1231316; Fri, 11 Oct 2024 15:27:57 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1szHYb-0007k3-Fa; Fri, 11 Oct 2024 15:27:57 +0000 Received: by outflank-mailman (input) for mailman id 817225; Fri, 11 Oct 2024 15:27:56 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1szHYa-0007SS-Eg for xen-devel@lists.xenproject.org; Fri, 11 Oct 2024 15:27:56 +0000 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2061a.outbound.protection.outlook.com [2a01:111:f403:2415::61a]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 626127fd-87e5-11ef-a0bd-8be0dac302b0; Fri, 11 Oct 2024 17:27:55 +0200 (CEST) Received: from BN7PR02CA0002.namprd02.prod.outlook.com (2603:10b6:408:20::15) by PH0PR12MB8125.namprd12.prod.outlook.com (2603:10b6:510:293::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.18; Fri, 11 Oct 2024 15:27:50 +0000 Received: from BN2PEPF00004FBA.namprd04.prod.outlook.com (2603:10b6:408:20:cafe::e6) by BN7PR02CA0002.outlook.office365.com (2603:10b6:408:20::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.20 via Frontend Transport; Fri, 11 Oct 2024 15:27:50 +0000 Received: from SATLEXMB04.amd.com (165.204.84.17) by BN2PEPF00004FBA.mail.protection.outlook.com (10.167.243.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8048.13 via Frontend Transport; Fri, 11 Oct 2024 15:27:50 +0000 Received: from SATLEXMB04.amd.com (10.181.40.145) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 11 Oct 2024 10:27:48 -0500 Received: from ubuntu.mshome.net (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Fri, 11 Oct 2024 10:27:47 -0500 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 626127fd-87e5-11ef-a0bd-8be0dac302b0 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=RBGo0ONO5nhwEO3UszMfRS/bOX2NDDgS7dfJWbD3JORg6nY6wQ4V9vZUShfYU2NAT0HXdFI+TKn86q0qicctR2Bi7056i+fVb14REjkxJ9HHlO3RACLGYb0MWGxQQi4AcPaQs+gMJYi2cGPgnbKmjrMMpNHas1hL0JcLMWe9luamStVR50HaWKl4o8EQhAkz/uSU9saiaWC9hl5P/J+gUOaPLecgCu1S05smi4HIPc1nkRHjMvnyyaHNq92EtPgqch1xLckES6dplXzinATOzs/iKTm4L9jGtJpBcFYBgsNIAA+umBsWknr72WtjjL/E9dDMKXlv0IfeBrcBgOZ0ug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0hdJcd67VFCUntI2TnhTshHdpS7eovY/+F2haWAsl40=; b=hpRh12Ja7KktDPBVh2Izavb3yg5R4N07JJNkry/C+tjs0Q5hGIbg6XMaAPw8XgzfL9aU0ToM7vRCbmMAKBVeiEPAVBV3draEpyrUm447/2rLfc2SAGdpr/owDlqnCUVftFPYUnk4Hv15EaTdA6QitTTnRko9GsN+sHsAymvpBragCSJse2cjv4AAJb3fYgptWKhqylSU1corIpOq/NUlbIfVELRFv68nljIeWSNuzElVHIq7Fx9fHe6oM8E7oHvGgKlHAepl3I88TSSxlWMyI0Wx5JnaSnL1ftUJzZoo/igAsOqbGfFypUwsfM9dwJE7J7aGHNePA9Xbhn6NMqHwrg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0hdJcd67VFCUntI2TnhTshHdpS7eovY/+F2haWAsl40=; b=R7pJYlvgCd4wxirE+pp4Ad2pWj/Z5W0fcaT42X9cbXFUGVSvbWe/xcHJwct1bu5QCelRAVOQHUd52+mJwIUVRx1ilCAcIrH0tVpVsvs3Cc/EwWqiXa6qSyHFSFoiulhRgc6YD3J0kV6euJjIGgDmdJCOJBgPfgCZezMtuZaSNwM= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C From: Stewart Hildebrand To: CC: Stewart Hildebrand , Jan Beulich , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Andrew Cooper , "Julien Grall" , Stefano Stabellini Subject: [PATCH v5 2/3] xen/pci: introduce PF<->VF links Date: Fri, 11 Oct 2024 11:27:07 -0400 Message-ID: <20241011152727.366770-3-stewart.hildebrand@amd.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241011152727.366770-1-stewart.hildebrand@amd.com> References: <20241011152727.366770-1-stewart.hildebrand@amd.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: None (SATLEXMB04.amd.com: stewart.hildebrand@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF00004FBA:EE_|PH0PR12MB8125:EE_ X-MS-Office365-Filtering-Correlation-Id: f847a9b3-49d2-4bd9-f711-08dcea094475 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|376014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?HjVEaIXfOLZtSrKpIg90qlVpiUInB6M0q0RQ9XN+vaWyi8tHamvSB60/X9fj?= =?us-ascii?Q?28674g9ccg2VLI8VZJ1Q+tdi9zeZGWGcO31Q5k25z66FCSRJtOfjwLj1PgND?= =?us-ascii?Q?DbqbnLZ4f5SmGNzQRrqaN1jrTIDTaDo8JcoWow5E/h46N30KFp9LLVwUx41o?= =?us-ascii?Q?9Txfd2cee2oR0d61Tu18FBPeIrv4pJSUSwUW/FIO5ktiSmR2J1YD6cQGqOKX?= =?us-ascii?Q?X++JXspoU/h7Wscw/BLPaYHGsL+LJSgSzHeNh3KLrujOftQhQhB74dEw4MqN?= =?us-ascii?Q?WMSH3WynGHzlx3MBsU7k8dEuDfVHah3zF/t1M/NQ7Phk43IXDIuGNuc/zjud?= =?us-ascii?Q?njYglJU2PAdY++qpv/IMtI9EopNIkJQU/78AwGcdqdOEueQPvLKY0YnLftAf?= =?us-ascii?Q?22vcE2cfoFiCZMHFK7aZlmiGEhfTmIrYdlgg2cnBWG38YDHeO3bjRF20d5Uz?= =?us-ascii?Q?aURM58Y3dhqt/N4iVc/vozqwJ+NdEMwylzuR42PKbs2znLxXIjTRCdvuXyDg?= =?us-ascii?Q?2En8hWSFvmSORZcl1QBM+StC4DKi3Y8p9tVCkbnvJQeoOz2HzYOKxU6XETUi?= =?us-ascii?Q?kObw52ZtFR52KCWEsOYvkyQaI04FPEhbRYz6q8o0dNtUXWSLhrZq4z7k7PT2?= =?us-ascii?Q?q6/BG2pt92O/hw5ECzZoHRCOiDSO7UJoxeriKh5L5z37zeNlPOYpZWU6sQ0o?= =?us-ascii?Q?ysIlCbPSLhliU8DSOsTkUu1kFt1kyfILsQlvjqr4yAcPVeckU35rOGh5ywho?= =?us-ascii?Q?AYnauPhmoyMEPShQLcu9jH2FFq3HPS0cMQp4HG7syjgiCrcXIfr5QM8XciLd?= =?us-ascii?Q?NL2dZyClKs4nD6Qu3n+YAFKNVkHc8aAkpNaqJt8FNoFdoGqQdXm2I/f02aqi?= =?us-ascii?Q?N730DkfnAFBHfPVLSOayIrEu24nhYi3sbfVe5zS0iUgKulOC24KnbuUMOb0k?= =?us-ascii?Q?zxaicWLpP9PF5AgsmZcfOTzjPKaOmyDBoGMu1wRCsY69pE9Asr0fDXLj8Abz?= =?us-ascii?Q?ZTqE17EvTc+bPjH28/AHebGsIz0OmR0DwBixItgd4pFXvtqjEi1XxDR+3I4h?= =?us-ascii?Q?9tFE1ESUFQFT2G88NvCldIVRZMUYHrLIY223kRixzHoYv6J6cXe7lOSrcpPh?= =?us-ascii?Q?UMVcY4e8wUR+iAc/IUb077J7etbJ0DOYfVwTZOh4ih1IPKg8OQweshT/AHxn?= =?us-ascii?Q?2qTLYKLEaK/kW0OfKAIVkCwkwV9VIX6aBewwdCrnTfiYeh2b6iKmsr1pfE/L?= =?us-ascii?Q?o9c4dnvCnyNEqDen6LTHNRzPhkI/srXPAXUemhkZmSLxNadt03j8PtnGXyjI?= =?us-ascii?Q?xAMyy4as9lfpgm0yIpuouxvbZNIzRQxW/A3A0qreEdIeVVMHRd40SVRObfOL?= =?us-ascii?Q?4RgEghk=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Oct 2024 15:27:50.3329 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f847a9b3-49d2-4bd9-f711-08dcea094475 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF00004FBA.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB8125 X-ZohoMail-DKIM: pass (identity @amd.com) X-ZM-MESSAGEID: 1728660495298116600 Content-Type: text/plain; charset="utf-8" Add links between a VF's struct pci_dev and its associated PF struct pci_dev. Move the calls to pci_get_pdev()/pci_add_device() down to avoid dropping and re-acquiring the pcidevs_lock(). During PF removal, unlink VF from PF and mark the VF broken. As before, VFs may exist without a corresponding PF, although now only with pdev->broken =3D true. If the PF is removed and re-added, dom0 is expected to also remove and re-add the VFs. Signed-off-by: Stewart Hildebrand --- Candidate for backport to 4.19 (the next patch depends on this one) v4->v5: * new patch, split from ("x86/msi: fix locking for SR-IOV devices") * move INIT_LIST_HEAD(&pdev->vf_list); earlier * collapse struct list_head instances * retain error code from pci_add_device() * unlink (and mark broken) VFs instead of removing them * const-ify VF->PF link --- xen/drivers/passthrough/pci.c | 70 ++++++++++++++++++++++++++++------- xen/include/xen/pci.h | 10 +++++ 2 files changed, 66 insertions(+), 14 deletions(-) diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 74d3895e1ef6..95a8ed850efd 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -333,6 +333,8 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg,= u8 bus, u8 devfn) *((u8*) &pdev->devfn) =3D devfn; pdev->domain =3D NULL; =20 + INIT_LIST_HEAD(&pdev->vf_list); + arch_pci_init_pdev(pdev); =20 rc =3D pdev_msi_init(pdev); @@ -449,6 +451,10 @@ static void free_pdev(struct pci_seg *pseg, struct pci= _dev *pdev) =20 list_del(&pdev->alldevs_list); pdev_msi_deinit(pdev); + + if ( pdev->info.is_virtfn && pdev->virtfn.pf_pdev ) + list_del(&pdev->vf_list); + xfree(pdev); } =20 @@ -656,24 +662,11 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, unsigned int slot =3D PCI_SLOT(devfn), func =3D PCI_FUNC(devfn); const char *type; int ret; - bool pf_is_extfn =3D false; =20 if ( !info ) type =3D "device"; else if ( info->is_virtfn ) - { - pcidevs_lock(); - pdev =3D pci_get_pdev(NULL, - PCI_SBDF(seg, info->physfn.bus, - info->physfn.devfn)); - if ( pdev ) - pf_is_extfn =3D pdev->info.is_extfn; - pcidevs_unlock(); - if ( !pdev ) - pci_add_device(seg, info->physfn.bus, info->physfn.devfn, - NULL, node); type =3D "virtual function"; - } else if ( info->is_extfn ) type =3D "extended function"; else @@ -703,7 +696,44 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, * extended function. */ if ( pdev->info.is_virtfn ) - pdev->info.is_extfn =3D pf_is_extfn; + { + struct pci_dev *pf_pdev; + + pf_pdev =3D pci_get_pdev(NULL, + PCI_SBDF(seg, info->physfn.bus, + info->physfn.devfn)); + + if ( !pf_pdev ) + { + ret =3D pci_add_device(seg, info->physfn.bus, info->physfn= .devfn, + NULL, node); + if ( ret ) + { + printk(XENLOG_WARNING "Failed to add SR-IOV device PF = %pp for VF %pp\n", + &PCI_SBDF(seg, info->physfn.bus, info->physfn.d= evfn), + &pdev->sbdf); + free_pdev(pseg, pdev); + goto out; + } + pf_pdev =3D pci_get_pdev(NULL, + PCI_SBDF(seg, info->physfn.bus, + info->physfn.devfn)); + if ( !pf_pdev ) + { + ASSERT_UNREACHABLE(); + printk(XENLOG_ERR "Failed to find SR-IOV device PF %pp= for VF %pp\n", + &PCI_SBDF(seg, info->physfn.bus, info->physfn.d= evfn), + &pdev->sbdf); + free_pdev(pseg, pdev); + ret =3D -EILSEQ; + goto out; + } + } + + pdev->info.is_extfn =3D pf_pdev->info.is_extfn; + pdev->virtfn.pf_pdev =3D pf_pdev; + list_add(&pdev->vf_list, &pf_pdev->vf_list); + } } =20 if ( !pdev->info.is_virtfn && !pdev->vf_rlen[0] ) @@ -821,6 +851,18 @@ int pci_remove_device(u16 seg, u8 bus, u8 devfn) list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list ) if ( pdev->bus =3D=3D bus && pdev->devfn =3D=3D devfn ) { + if ( !pdev->info.is_virtfn ) + { + struct pci_dev *vf_pdev, *tmp; + + list_for_each_entry_safe(vf_pdev, tmp, &pdev->vf_list, vf_= list) + { + list_del(&vf_pdev->vf_list); + vf_pdev->virtfn.pf_pdev =3D NULL; + vf_pdev->broken =3D true; + } + } + if ( pdev->domain ) { write_lock(&pdev->domain->pci_lock); diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index 63e49f0117e9..f9435b7f4eb9 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -150,7 +150,17 @@ struct pci_dev { unsigned int count; #define PT_FAULT_THRESHOLD 10 } fault; + + /* + * List head if info.is_virtfn =3D=3D false + * List entry if info.is_virtfn =3D=3D true + */ + struct list_head vf_list; u64 vf_rlen[6]; + struct { + /* Only populated for VFs (info.is_virtfn =3D=3D true) */ + const struct pci_dev *pf_pdev; /* Link from VF to PF */ + } virtfn; =20 /* Data for vPCI. */ struct vpci *vpci; --=20 2.47.0 From nobody Thu Nov 21 21:47:41 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass(p=quarantine dis=none) header.from=amd.com ARC-Seal: i=2; a=rsa-sha256; t=1728660502; cv=pass; d=zohomail.com; s=zohoarc; b=kjMipuqEgktUAtQ5e6SX6UaDlhiCn+BUdWm616wpEoKwu09T4wqUNhjm3v6PHRoj48P9svPvbZ9Y8zIfbbXo3oDqQMx89Bd8Ghe/LmD35N71D3Ikvhd/3F39SZWr2oqc08fa2o9ZIZWfjxKR6PfLfAhMMqX3DzytY9fMcthLv78= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1728660502; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=PoaEQUmZyPk/7lddPfXAooR8DtejrJ8pV0FXtQo30ts=; b=gToL+A2nbjvyeRklxsZ9kK5OZRmBStLxDmIur6zxXJSRewNE98vMs8F691DM9uzbxk1TwYNa0pF5UKKPOAF9rOjnalhHnn8jJVROZwAfD0Ejhg2eDSnVHqwkfdrB6lOoXhRwPCMcNrftE4GYCnx3lqzPexi81j/jYFwhLEyH7Ks= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1728660501971877.7646464849868; Fri, 11 Oct 2024 08:28:21 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.817227.1231327 (Exim 4.92) (envelope-from ) id 1szHYh-00086W-Sh; Fri, 11 Oct 2024 15:28:03 +0000 Received: by outflank-mailman (output) from mailman id 817227.1231327; Fri, 11 Oct 2024 15:28:03 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1szHYh-00086N-P7; Fri, 11 Oct 2024 15:28:03 +0000 Received: by outflank-mailman (input) for mailman id 817227; Fri, 11 Oct 2024 15:28:02 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1szHYg-0007SS-BX for xen-devel@lists.xenproject.org; Fri, 11 Oct 2024 15:28:02 +0000 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on20617.outbound.protection.outlook.com [2a01:111:f403:2416::617]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 660d22e1-87e5-11ef-a0bd-8be0dac302b0; Fri, 11 Oct 2024 17:28:01 +0200 (CEST) Received: from BL1PR13CA0263.namprd13.prod.outlook.com (2603:10b6:208:2ba::28) by PH7PR12MB6955.namprd12.prod.outlook.com (2603:10b6:510:1b8::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.18; Fri, 11 Oct 2024 15:27:57 +0000 Received: from BN2PEPF00004FC1.namprd04.prod.outlook.com (2603:10b6:208:2ba:cafe::54) by BL1PR13CA0263.outlook.office365.com (2603:10b6:208:2ba::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.12 via Frontend Transport; Fri, 11 Oct 2024 15:27:56 +0000 Received: from SATLEXMB04.amd.com (165.204.84.17) by BN2PEPF00004FC1.mail.protection.outlook.com (10.167.243.187) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8048.13 via Frontend Transport; Fri, 11 Oct 2024 15:27:55 +0000 Received: from SATLEXMB04.amd.com (10.181.40.145) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 11 Oct 2024 10:27:55 -0500 Received: from ubuntu.mshome.net (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Fri, 11 Oct 2024 10:27:54 -0500 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 660d22e1-87e5-11ef-a0bd-8be0dac302b0 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=PwyIC8+3adm599ynvdRaYap1uzZK6fVFCY/ENpRkwuJJkUSBtdwhadjrxhLJCF/IGEOxuLuA0CdKvQ1+iqM1A4EpxX4SHCG/fVPifyTPv7qAY3lOxBPLrjOSJGrvyjyDMLotefYOCJFJIx6oZeIvW/w/FFkhBpJ4ZOfSaCklTRoiuUjIBorJIINVPrYOx9qr5Hn73xqQll27hB9QIJAMqtvgPrI2LP7C1eQrhr2i8SdXymfucZhHFyqgzWdNCf0epTogenSGmBArEvoLhzdfz8AKzvqHH2/Xd42vYybmFkDzvL4hlwU9fSg5YdGw8ad8jJFJ7oC5r0qS5iTptCp8oQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PoaEQUmZyPk/7lddPfXAooR8DtejrJ8pV0FXtQo30ts=; b=TTeE0mfofZhndQAj3DuR4GcsgcBHjkeL0QXIQ/gWZNj5p7kheOj4tszI4tnfxeJy0tvMCJdm/K77RtvqP0on81WLH7bUxvZb3qNXximTZVkr4rhoNGi/CvH+9UR1cHALxvjCyNRDmj3XaOOxSPEjG+aiNoMsbI5ZnJ13kpDn5yBlXKgDkFXEpw/Nesd9OYhxtE3nXGN5wSHISIJYcAC32+BdJR3J86/soeVhjLvkifOdeLlKHACaJNTfNvCm/q6+14bZuSQxQEyhU8MnfvqXc8ceCgS7yjguNsJaV6t+uac+14/I33zUDBCadnqWIYzIu8RuGEuwdcPUUh1470YI3g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PoaEQUmZyPk/7lddPfXAooR8DtejrJ8pV0FXtQo30ts=; b=ASzh3HtiwofGi0mVNp3ZEFarWgicRpayCdmO7IojO4tgzOa42n5HfFGKNWCcd/1RhfnG67goX0tfvlvU5DAFSAC303H6CMHl+UoG2LE6XEX7FBVQ7vSSP0fRlvTMpm6NNZQbfNxV0Pmp+31EhzjeKdo7EOcX9CbslSNdSxemPi8= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C From: Stewart Hildebrand To: CC: Stewart Hildebrand , Jan Beulich , Andrew Cooper , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Julien Grall , Stefano Stabellini , Teddy Astie Subject: [PATCH v5 3/3] x86/msi: fix locking for SR-IOV devices Date: Fri, 11 Oct 2024 11:27:08 -0400 Message-ID: <20241011152727.366770-4-stewart.hildebrand@amd.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241011152727.366770-1-stewart.hildebrand@amd.com> References: <20241011152727.366770-1-stewart.hildebrand@amd.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: None (SATLEXMB04.amd.com: stewart.hildebrand@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF00004FC1:EE_|PH7PR12MB6955:EE_ X-MS-Office365-Filtering-Correlation-Id: 087b2fa2-2eaf-485e-02a3-08dcea0947b3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|376014|1800799024|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?bGIMa8YoU+u/7eiEe75cVEqeB/NEZQ83Uh30EzkNI0vyjDBzVbwbRLlyIPj+?= =?us-ascii?Q?cn109ayVPkOc4NV0FzGf0qYXw2Ivp8Vo0LEF4ywuyzddcn9g7BhfyFHTyaEX?= =?us-ascii?Q?gNPWcpmNuKZeYxiRugOpzSNshiEfAOkZuGzEiS0hRAadPqeayd77JztTIq8E?= =?us-ascii?Q?Pz8+QvZqTsAj7adIqbGUhrv4aksKG+LP/i6Zv3wLxQtGVhBq4qhX1MFy2uUl?= =?us-ascii?Q?Cz1EOG2RddJBZl6k067O0IR9PJ2NvJVuW+E271rRtfYKR9mkmPn/hCEYn0YM?= =?us-ascii?Q?DPwSCvHyhEOAXq40dhCdhnaQHw6ymFGH84hrvQS0HNswmRcD2Hm2IudGsSw3?= =?us-ascii?Q?TDBZsTBxBX3Bccw6Pe6w4hPRblprh5zSAlOHTeyTjbhP4XNhNXtDN2+hGS8M?= =?us-ascii?Q?dgPKhr3jhWkUHmKe2ntf2c4Bm394pYY8bzuhBc6s2t7Z/r4Yp1S/lkDqrqQj?= =?us-ascii?Q?cGoQw0Q4i96MROw7aK9Na3Vi441XVcGX0Ue10p2FYggPOx3VNyLszZd90rkJ?= =?us-ascii?Q?SPMW1Mw+LxPxYHK3RVZ8loElFUR/8AjrUTzH420I5hvN4LK3Ib3R5jTZwAWw?= =?us-ascii?Q?E9zosvJfQnVSOe6JtNw9QEQ5jFuXstQ5z0Ovm83FmA8aqz4DQc14OullUs5s?= =?us-ascii?Q?q8r12JSwzbP28aY7BYC7kFzprHzk5SxMBzCBkp4llVW338ctSQMUdSHM07qm?= =?us-ascii?Q?h3xlt2RuRgnIihuU5J4FG4WUPGfbeAZp2/s5eDYz/11anO0eYKXMI0JK+MK/?= =?us-ascii?Q?3T0IJlIPl66WnxdtG3yUl67s/IqflsiowNxu5aPhR00VI2wy7czjcP8oXckC?= =?us-ascii?Q?LGolulpaTN+s+QJbzk9bWeYf3m9gV3kzO8E/2tklBPB3tZ2fc0WaxnaNgcY3?= =?us-ascii?Q?cvC9EgktFWujJuiZNlj//X0lkhl99fyxPqPVu+0e7vZ0GcFJygvcfFKLV6CD?= =?us-ascii?Q?ntV8KRkeQ1LUc3zCwVjgSTW/r6Z9LzBW/ULltYXSwWreKQjKV+8fWLZldvfz?= =?us-ascii?Q?v/pbgVJ1+Fwj6DI42NsTYhHbPeehLwYdH04laxwbt/+3BH0bCXYe8sy5qmMe?= =?us-ascii?Q?GpiIaELoZWVfDfoAtXtg3AtmM3UKrOLofxny5cvgrVLNjyNzinpXjtq1BQjP?= =?us-ascii?Q?Te/A9L3umY5EdBHZAIfKmNK/jNFb0Nh9UgPXTWJX3OZFeRzRM+1yN5yNy9A7?= =?us-ascii?Q?ey0lZIWHhVkykfhdfydTU2/gQbydEZDi0IVYJIlCSpH22e5yglACOF3lNadj?= =?us-ascii?Q?XcE4Bj6AfMyh2BPnW8y+aqP6XOS3H0wkw+EbCwELOVXNnpt41OrSZ2nJm9yl?= =?us-ascii?Q?Unw5B+Eq0kdZJj1aSFGj8j/7sDFAZsJqMOyCGpzmF2tSVHI+7qVjqdv4ZHAk?= =?us-ascii?Q?OSzEVWceHAGeyNZrfG6DNX0aE9mf?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(376014)(1800799024)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Oct 2024 15:27:55.7662 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 087b2fa2-2eaf-485e-02a3-08dcea0947b3 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF00004FC1.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6955 X-ZohoMail-DKIM: pass (identity @amd.com) X-ZM-MESSAGEID: 1728660503277116600 Content-Type: text/plain; charset="utf-8" In commit 4f78438b45e2 ("vpci: use per-domain PCI lock to protect vpci structure") a lock was moved from allocate_and_map_msi_pirq() to the caller and changed from pcidevs_lock() to read_lock(&d->pci_lock). However, one call path wasn't updated to reflect the change, leading to a failed assertion observed under the following conditions: * PV dom0 * Debug build (CONFIG_DEBUG=3Dy) of Xen * There is an SR-IOV device in the system with one or more VFs enabled * Dom0 has loaded the driver for the VF and enabled MSI-X (XEN) Assertion 'd || pcidevs_locked()' failed at drivers/passthrough/pci.c= :535 (XEN) ----[ Xen-4.20-unstable x86_64 debug=3Dy Not tainted ]---- ... (XEN) Xen call trace: (XEN) [] R pci_get_pdev+0x4c/0xab (XEN) [] F arch/x86/msi.c#read_pci_mem_bar+0x58/0x272 (XEN) [] F arch/x86/msi.c#msix_capability_init+0x198/0= x755 (XEN) [] F arch/x86/msi.c#__pci_enable_msix+0x82/0xe8 (XEN) [] F pci_enable_msi+0x3f/0x78 (XEN) [] F map_domain_pirq+0x2a4/0x6dc (XEN) [] F allocate_and_map_msi_pirq+0x103/0x262 (XEN) [] F physdev_map_pirq+0x210/0x259 (XEN) [] F do_physdev_op+0x9c3/0x1454 (XEN) [] F pv_hypercall+0x5ac/0x6af (XEN) [] F lstar_enter+0x143/0x150 In read_pci_mem_bar(), the VF obtains the struct pci_dev pointer for its associated PF to access the vf_rlen array. This array is initialized in pci_add_device() and is only populated in the associated PF's struct pci_dev. Access the vf_rlen array via the link to the PF, and remove the troublesome call to pci_get_pdev(). Fixes: 4f78438b45e2 ("vpci: use per-domain PCI lock to protect vpci structu= re") Reported-by: Teddy Astie Signed-off-by: Stewart Hildebrand Reviewed-by: Jan Beulich --- Candidate for backport to 4.19 Patch #2 ("xen/pci: introduce PF<->VF links") is pre-requisite v4->v5: * split the PF<->VF links to a pre-requisite patch * pass pci_sbdf_t to read_pci_mem_bar() * use stdint.h types on changed lines * re-add NULL check for pf_info in read_pci_mem_bar(), as pf_info could be NULL v3->v4: * handle case when PF is removed with VFs enabled, then re-added with VFs disabled v2->v3: * link from VF to PF's struct pci_dev * v1->v2: * remove call to pci_get_pdev() --- xen/arch/x86/msi.c | 37 ++++++++++++++++++++++------------- xen/drivers/passthrough/pci.c | 8 +++++--- xen/include/xen/pci.h | 15 +++++++++----- 3 files changed, 38 insertions(+), 22 deletions(-) diff --git a/xen/arch/x86/msi.c b/xen/arch/x86/msi.c index fbb07fe821b5..254a403dfb51 100644 --- a/xen/arch/x86/msi.c +++ b/xen/arch/x86/msi.c @@ -662,34 +662,35 @@ static int msi_capability_init(struct pci_dev *dev, return 0; } =20 -static u64 read_pci_mem_bar(u16 seg, u8 bus, u8 slot, u8 func, u8 bir, int= vf) +static uint64_t read_pci_mem_bar(pci_sbdf_t sbdf, uint8_t bir, int vf, + const struct pf_info *pf_info) { + uint16_t seg =3D sbdf.seg; + uint8_t bus =3D sbdf.bus, slot =3D sbdf.dev, func =3D sbdf.fn; u8 limit; u32 addr, base =3D PCI_BASE_ADDRESS_0; u64 disp =3D 0; =20 if ( vf >=3D 0 ) { - struct pci_dev *pdev =3D pci_get_pdev(NULL, - PCI_SBDF(seg, bus, slot, func)= ); unsigned int pos; uint16_t ctrl, num_vf, offset, stride; =20 - if ( !pdev ) + if ( !pf_info ) return 0; =20 - pos =3D pci_find_ext_capability(pdev->sbdf, PCI_EXT_CAP_ID_SRIOV); - ctrl =3D pci_conf_read16(pdev->sbdf, pos + PCI_SRIOV_CTRL); - num_vf =3D pci_conf_read16(pdev->sbdf, pos + PCI_SRIOV_NUM_VF); - offset =3D pci_conf_read16(pdev->sbdf, pos + PCI_SRIOV_VF_OFFSET); - stride =3D pci_conf_read16(pdev->sbdf, pos + PCI_SRIOV_VF_STRIDE); + pos =3D pci_find_ext_capability(sbdf, PCI_EXT_CAP_ID_SRIOV); + ctrl =3D pci_conf_read16(sbdf, pos + PCI_SRIOV_CTRL); + num_vf =3D pci_conf_read16(sbdf, pos + PCI_SRIOV_NUM_VF); + offset =3D pci_conf_read16(sbdf, pos + PCI_SRIOV_VF_OFFSET); + stride =3D pci_conf_read16(sbdf, pos + PCI_SRIOV_VF_STRIDE); =20 if ( !pos || !(ctrl & PCI_SRIOV_CTRL_VFE) || !(ctrl & PCI_SRIOV_CTRL_MSE) || !num_vf || !offset || (num_vf > 1 && !stride) || bir >=3D PCI_SRIOV_NUM_BARS || - !pdev->vf_rlen[bir] ) + !pf_info->vf_rlen[bir] ) return 0; base =3D pos + PCI_SRIOV_BAR; vf -=3D PCI_BDF(bus, slot, func) + offset; @@ -703,8 +704,8 @@ static u64 read_pci_mem_bar(u16 seg, u8 bus, u8 slot, u= 8 func, u8 bir, int vf) } if ( vf >=3D num_vf ) return 0; - BUILD_BUG_ON(ARRAY_SIZE(pdev->vf_rlen) !=3D PCI_SRIOV_NUM_BARS); - disp =3D vf * pdev->vf_rlen[bir]; + BUILD_BUG_ON(ARRAY_SIZE(pf_info->vf_rlen) !=3D PCI_SRIOV_NUM_BARS); + disp =3D vf * pf_info->vf_rlen[bir]; limit =3D PCI_SRIOV_NUM_BARS; } else switch ( pci_conf_read8(PCI_SBDF(seg, bus, slot, func), @@ -813,6 +814,7 @@ static int msix_capability_init(struct pci_dev *dev, int vf; paddr_t pba_paddr; unsigned int pba_offset; + const struct pf_info *pf_info; =20 if ( !dev->info.is_virtfn ) { @@ -820,6 +822,7 @@ static int msix_capability_init(struct pci_dev *dev, pslot =3D slot; pfunc =3D func; vf =3D -1; + pf_info =3D NULL; } else { @@ -827,9 +830,14 @@ static int msix_capability_init(struct pci_dev *dev, pslot =3D PCI_SLOT(dev->info.physfn.devfn); pfunc =3D PCI_FUNC(dev->info.physfn.devfn); vf =3D dev->sbdf.bdf; + if ( dev->virtfn.pf_pdev ) + pf_info =3D &dev->virtfn.pf_pdev->physfn; + else + pf_info =3D NULL; } =20 - table_paddr =3D read_pci_mem_bar(seg, pbus, pslot, pfunc, bir, vf); + table_paddr =3D read_pci_mem_bar(PCI_SBDF(seg, pbus, pslot, pfunc)= , bir, + vf, pf_info); WARN_ON(msi && msi->table_base !=3D table_paddr); if ( !table_paddr ) { @@ -852,7 +860,8 @@ static int msix_capability_init(struct pci_dev *dev, =20 pba_offset =3D pci_conf_read32(dev->sbdf, msix_pba_offset_reg(pos)= ); bir =3D (u8)(pba_offset & PCI_MSIX_BIRMASK); - pba_paddr =3D read_pci_mem_bar(seg, pbus, pslot, pfunc, bir, vf); + pba_paddr =3D read_pci_mem_bar(PCI_SBDF(seg, pbus, pslot, pfunc), = bir, vf, + pf_info); WARN_ON(!pba_paddr); pba_paddr +=3D pba_offset & ~PCI_MSIX_BIRMASK; =20 diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 95a8ed850efd..8209efdebecf 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -736,7 +736,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, } } =20 - if ( !pdev->info.is_virtfn && !pdev->vf_rlen[0] ) + if ( !pdev->info.is_virtfn && !pdev->physfn.vf_rlen[0] ) { unsigned int pos =3D pci_find_ext_capability(pdev->sbdf, PCI_EXT_CAP_ID_SRIOV); @@ -748,7 +748,9 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, { unsigned int i; =20 - BUILD_BUG_ON(ARRAY_SIZE(pdev->vf_rlen) !=3D PCI_SRIOV_NUM_BARS= ); + BUILD_BUG_ON(ARRAY_SIZE(pdev->physfn.vf_rlen) !=3D + PCI_SRIOV_NUM_BARS); + for ( i =3D 0; i < PCI_SRIOV_NUM_BARS; ) { unsigned int idx =3D pos + PCI_SRIOV_BAR + i * 4; @@ -763,7 +765,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, continue; } ret =3D pci_size_mem_bar(pdev->sbdf, idx, NULL, - &pdev->vf_rlen[i], + &pdev->physfn.vf_rlen[i], PCI_BAR_VF | ((i =3D=3D PCI_SRIOV_NUM_BARS - 1) ? PCI_BAR_LAST : 0)); diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index f9435b7f4eb9..12f1c599e467 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -156,11 +156,16 @@ struct pci_dev { * List entry if info.is_virtfn =3D=3D true */ struct list_head vf_list; - u64 vf_rlen[6]; - struct { - /* Only populated for VFs (info.is_virtfn =3D=3D true) */ - const struct pci_dev *pf_pdev; /* Link from VF to PF */ - } virtfn; + union { + struct pf_info { + /* Only populated for PFs (info.is_virtfn =3D=3D false) */ + uint64_t vf_rlen[PCI_SRIOV_NUM_BARS]; + } physfn; + struct { + /* Only populated for VFs (info.is_virtfn =3D=3D true) */ + const struct pci_dev *pf_pdev; /* Link from VF to PF */ + } virtfn; + }; =20 /* Data for vPCI. */ struct vpci *vpci; --=20 2.47.0