From nobody Sun Nov 24 01:40:31 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass(p=quarantine dis=none) header.from=amd.com ARC-Seal: i=2; a=rsa-sha256; t=1729284008; cv=pass; d=zohomail.com; s=zohoarc; b=SXDuEu0rMpMaHRbrey1GOKZvIUP0xjDhVcsHXN+r2+eG/6PmQGkiNvDFWopbfmNy4xG1zi5GVTZVfiG37fkdcVWx/PVgFLdkwNyabMLFGcQl3/yQDjgxfo6bTbwuJb1NQSC/xExYkpo9P+v8RyIh1bnZ5uMwyWQT/7JH/kIkvbE= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1729284008; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=uhSgenMyww7CPsK0AMuHmuq3dKJ97wqxyXgBiGyLZRY=; b=L7fOIQr32Xht+4W6mrvQJHdEv30UxidcZwPB3ntpv9CmwJnDsdVBoDeUkmhHxs/77PGH8PL0SHVxJKrKn2dp2D0jbzNWEw2WwBKtGoyq6+Jwv9UmRzKZO9jhL/8bumgi+Q0rVRx+JYR0boryYWhfPDxgOmiXU4a+MYPIewJYOjc= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1729284008294828.6276982293981; Fri, 18 Oct 2024 13:40:08 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.822149.1236119 (Exim 4.92) (envelope-from ) id 1t1tl2-0001DL-0T; Fri, 18 Oct 2024 20:39:36 +0000 Received: by outflank-mailman (output) from mailman id 822149.1236119; Fri, 18 Oct 2024 20:39:35 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1t1tl1-0001DE-TR; Fri, 18 Oct 2024 20:39:35 +0000 Received: by outflank-mailman (input) for mailman id 822149; Fri, 18 Oct 2024 20:39:34 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1t1tl0-0001Cl-9l for xen-devel@lists.xenproject.org; Fri, 18 Oct 2024 20:39:34 +0000 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on20627.outbound.protection.outlook.com [2a01:111:f403:2409::627]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 12d601b8-8d91-11ef-99a3-01e77a169b0f; Fri, 18 Oct 2024 22:39:31 +0200 (CEST) Received: from CH2PR18CA0060.namprd18.prod.outlook.com (2603:10b6:610:55::40) by SA3PR12MB8046.namprd12.prod.outlook.com (2603:10b6:806:304::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.20; Fri, 18 Oct 2024 20:39:25 +0000 Received: from CH2PEPF00000149.namprd02.prod.outlook.com (2603:10b6:610:55:cafe::77) by CH2PR18CA0060.outlook.office365.com (2603:10b6:610:55::40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.21 via Frontend Transport; Fri, 18 Oct 2024 20:39:25 +0000 Received: from SATLEXMB03.amd.com (165.204.84.17) by CH2PEPF00000149.mail.protection.outlook.com (10.167.244.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8069.17 via Frontend Transport; Fri, 18 Oct 2024 20:39:25 +0000 Received: from SATLEXMB05.amd.com (10.181.40.146) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 18 Oct 2024 15:39:24 -0500 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB05.amd.com (10.181.40.146) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 18 Oct 2024 15:39:24 -0500 Received: from ubuntu.mshome.net (10.180.168.240) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Fri, 18 Oct 2024 15:39:23 -0500 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 12d601b8-8d91-11ef-99a3-01e77a169b0f ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=OapSGjE5nWZCxDGN+2OXRd7fRBmd08+D3NDufWu31YYgq2tEpbi1pzJ8Lujwa3o8H+RBv8G93bm0K9fpJig1RSwL2RQ3lcdKYqQKtxH6z1Annh6icdySah7wKukabCpCqDZF5jAlH5QsWof2AI25WHSLaUO5i2v4sd2LHy/PsP/FXcrJMOETwuUcF/6oZSeQ+SMgbZelJn+ZoobkV45UltO8vA+mi3/B108ygt9yLHYKFC6rypbotVbPOLOniGbZpiIWUbO66F2j4LzIE2uRQZO3TWKBBn224ittJ+vEebAodRy6eViEcf7LVpIsVEZVWJ2PnnIQreluaYNL+Dzpuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=uhSgenMyww7CPsK0AMuHmuq3dKJ97wqxyXgBiGyLZRY=; b=mCKeI1ODwPqiaGGahftuMIrTlamxcX5HeyBRGSQ7hZhtehq1bgHyUsHJrqRZFPDimOW43tlfOoplVzK/d4XoxtPOMQ4MmGEQoFBPdNg7oEpybvLGJ1Yo2kPHUs1z/LV0xkrj9vmzISb79ZXonfVtDcYhk/isz0QhLq6yHr1DFsH8pGS2bHNnc/9KTEHWl/2oMd25Z57qoCnykbvfEg71Gsl/pHJKOh1GlB8G2+7i1Ogve/BymLCOoyX6OfH0Pz2w7zBf60bLuezwdomGR1mSoFQxfZnGy2NnqORb4uFvbUASjvpBmryv44C8jic8LkzBMMZHqWU7x0Lr3OojbTDE0g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uhSgenMyww7CPsK0AMuHmuq3dKJ97wqxyXgBiGyLZRY=; b=Uh5v40aSuaDBvDWnZod0WmLMSbUPj+U1s+lZWlY7y2s7NTyZAyQMT2oNe4u+UlNGXLPU7z3w9NGqR0t2B9IJE1HK3YAPp+vhPTeDJ+nqJxubPo1TjnjYo7eSwEWUjPJA978G5b7sF1C1GpslDEZztOr4bcoRWB5fjAwYOYPbC9o= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB03.amd.com; pr=C From: Stewart Hildebrand To: CC: Stewart Hildebrand , Jan Beulich , Andrew Cooper , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Julien Grall , Stefano Stabellini Subject: [PATCH v6 1/3] x86/msi: harden stale pdev handling Date: Fri, 18 Oct 2024 16:39:08 -0400 Message-ID: <20241018203913.1162962-2-stewart.hildebrand@amd.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241018203913.1162962-1-stewart.hildebrand@amd.com> References: <20241018203913.1162962-1-stewart.hildebrand@amd.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: None (SATLEXMB05.amd.com: stewart.hildebrand@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000149:EE_|SA3PR12MB8046:EE_ X-MS-Office365-Filtering-Correlation-Id: 576cb1c1-6113-4765-1066-08dcefb4f450 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|376014|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?fN0Ewg4HcNoQ8V4Gyx1u1x5lqzJT3wbUmrBR3CG5Z2enm17SiI+hlVnoLsHf?= =?us-ascii?Q?l1om8OGtV6eP0LV0m7BcWAmcTLBcxQmQ1sQDtIWQ+Ob07i3dIv/5LW9yMUHp?= =?us-ascii?Q?B1Fhcw5hKYkHA9UklbbQO2Qlz6bMUkj9RkQmTOxHJCbT8gyFQVEL3+skLi6u?= =?us-ascii?Q?7nhTcfIrzwRviMVXvsepmVRXxq8wTQKwfC1cxGOIUpG4h3EGXG1y0yAGvhtR?= =?us-ascii?Q?HE8NsMh0jvwhGEkCElWbLLgLNZzEmnaERimI+qPc5KBNBGtiEZ/wXdixWLYa?= =?us-ascii?Q?e675mNoSOq71u/Kou5nxFHhO3m+CGHOPjoJkWVY+bCwvzLW7HmbaS9hIKxkj?= =?us-ascii?Q?aGVVqkLp2B1+/9KG7AYtuPEOMReEW06I5R1r9WC1yKDnWOh1ddKAuf2ytZmT?= =?us-ascii?Q?MMditvO9Lp224bRAABm0/rY6v/5XlLLayry+5LgC/8AxuWRYOftxrHESs5KS?= =?us-ascii?Q?4SMLDLsZ96yklGMXYfMXDDI6x5r/0vQ86/EQ5rKB/X7gs9r5RZAjOwLN0hzm?= =?us-ascii?Q?GoLHjyFW9x8TBjGegszvy/SOxPQrOo7E/n653UTJlIR6mlEHvxmj3XSR3qmS?= =?us-ascii?Q?NO4Qvcz0PgKdwI8F+LhHuyiE98lxbdKH3r4gi/k4eAxW9Nro0eTdBNPL6JD+?= =?us-ascii?Q?CjzUhzBYdMs+D6KD/X+MbMNgQKfHS0FmkXDksubwPCxDmIqQWSFVb8u3Rhr7?= =?us-ascii?Q?KOAlHwkzYuQajxPO/o0OPjNIbx5TzclwRo0y60FFNoaqmf+TwDX0BReePS0s?= =?us-ascii?Q?tyogP2Gcw0cbayjIgWaqSmsn+9Jsa/WYmk2me2/F+kh2SgCmUFQZRMcKM0lQ?= =?us-ascii?Q?zcrRoityj8rbiEa4xGBvd2VqhSxwyR+zm25vWOshHDZ3xgUFmd+KWOLa57Mm?= =?us-ascii?Q?Q0sM3EfB0wvrKL5SzYGikym8sV1RPz1bHIKEGljL1+wUKkaut4FS7iGWhNm+?= =?us-ascii?Q?HiQXkmaOt8n2yA6olEI2lAxIcY2CttggCb+07zef0z9j+wTQHS/vkKAQqMzH?= =?us-ascii?Q?mM8YwG6mZnLCVby0iJPfsaGeNkAR8iZBOc3IehxJX4OTu19kMg6if1cfLFpd?= =?us-ascii?Q?6B66HW5ouM8HE+ae9d7Be6zxioPOGvl6UxqhS/OJLip/gCnl3WWwQ55P9u30?= =?us-ascii?Q?afg95Q+/x8n/bKnwEOI+hhGV7/ApGKXarnL0RQgRzmbTvtjJgiwxPEjH5HaH?= =?us-ascii?Q?u65w2lZx04vQpDwIGhgmY1YJ+7Il1sXdaF8XEHILVnMXDMBRyLDHEwwWNB1P?= =?us-ascii?Q?Di24mDIZC1WNc6hxd6h5JBkA3+T+NSSoxJjAiG9008iqSrg1md7JEb0ui8/B?= =?us-ascii?Q?+1a/liwrNLwVJ3nzRckjhkOnSBJ66ikeeb4uRyQqNS/nrVNUENpmlPhGyU06?= =?us-ascii?Q?zWgB4ePZFyRuYMnEQqfVFKNaezYSm2XoDuZ+kzrE8Q4hPSGPog=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB03.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(376014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2024 20:39:25.0869 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 576cb1c1-6113-4765-1066-08dcefb4f450 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB03.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000149.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB8046 X-ZohoMail-DKIM: pass (identity @amd.com) X-ZM-MESSAGEID: 1729284008926116600 Content-Type: text/plain; charset="utf-8" Dom0 normally informs Xen of PCI device removal via PHYSDEVOP_pci_device_remove, e.g. in response to SR-IOV disable or hot-unplug. We might find ourselves with stale pdevs if a buggy dom0 fails to report removal via PHYSDEVOP_pci_device_remove. In this case, attempts to access the config space of the stale pdevs would be invalid and return all 1s. Some possible conditions leading to this are: 1. Dom0 disables SR-IOV without reporting VF removal to Xen. The Linux SR-IOV subsystem normally reports VF removal when a PF driver disables SR-IOV. In case of a buggy dom0 SR-IOV subsystem, SR-IOV could become disabled with stale dangling VF pdevs in both dom0 Linux and Xen. 2. Dom0 reporting PF removal without reporting VF removal. During SR-IOV PF removal (hot-unplug), a buggy PF driver may fail to disable SR-IOV, thus failing to remove the VFs, leaving stale dangling VFs behind in both Xen and Linux. At least Linux warns in this case: [ 100.000000] 0000:01:00.0: driver left SR-IOV enabled after remove In either case, Xen is left with stale VF pdevs, risking invalid PCI config space accesses. When Xen is built with CONFIG_DEBUG=3Dy, the following Xen crashes were observed when dom0 attempted to access the config space of a stale VF: (XEN) Assertion 'pos' failed at arch/x86/msi.c:1274 (XEN) ----[ Xen-4.20-unstable x86_64 debug=3Dy Tainted: C ]---- ... (XEN) Xen call trace: (XEN) [] R pci_msi_conf_write_intercept+0xa2/0x1de (XEN) [] F pci_conf_write_intercept+0x68/0x78 (XEN) [] F arch/x86/pv/emul-priv-op.c#pci_cfg_ok+0xa0/= 0x114 (XEN) [] F arch/x86/pv/emul-priv-op.c#guest_io_write+0= xb5/0x1c8 (XEN) [] F arch/x86/pv/emul-priv-op.c#write_io+0x9a/0x= e0 (XEN) [] F x86_emulate+0x100e5/0x25f1e (XEN) [] F x86_emulate_wrapper+0x29/0x64 (XEN) [] F pv_emulate_privileged_op+0x12e/0x217 (XEN) [] F do_general_protection+0xc2/0x1b8 (XEN) [] F x86_64/entry.S#handle_exception_saved+0x2b/= 0x8c (XEN) Assertion 'pos' failed at arch/x86/msi.c:1246 (XEN) ----[ Xen-4.20-unstable x86_64 debug=3Dy Tainted: C ]---- ... (XEN) Xen call trace: (XEN) [] R pci_reset_msix_state+0x47/0x50 (XEN) [] F pdev_msix_assign+0x19/0x35 (XEN) [] F drivers/passthrough/pci.c#assign_device+0x1= 81/0x471 (XEN) [] F iommu_do_pci_domctl+0x248/0x2ec (XEN) [] F iommu_do_domctl+0x26/0x44 (XEN) [] F do_domctl+0x8c1/0x1660 (XEN) [] F pv_hypercall+0x5ce/0x6af (XEN) [] F lstar_enter+0x143/0x150 These ASSERTs triggered because the MSI-X capability position can't be found for a stale pdev. Latch the capability positions of MSI and MSI-X during device init, and replace instances of pci_find_cap_offset(..., PCI_CAP_ID_MSI{,X}) with the stored value. Introduce one additional ASSERT, while the two existing ASSERTs in question continue to work as intended, even with a stale pdev. Fixes: 484d7c852e4f ("x86/MSI-X: track host and guest mask-all requests sep= arately") Fixes: 575e18d54d19 ("pci: clear {host/guest}_maskall field on assign") Signed-off-by: Stewart Hildebrand Reviewed-by: Jan Beulich Reviewed-by: Roger Pau Monn=C3=A9 --- v5->v6; * latch MSI/MSI-X capability position during device init v4->v5: * new patch, independent of the rest of the series * new approach to fixing the issue: don't rely on dom0 to report any sort of device removal; rather, fix the condition directly --- Instructions to reproduce Requires Xen with CONFIG_DEBUG=3Dy Tested with Linux 6.11 1. Dom0 disables SR-IOV without reporting VF removal to Xen. * Hack the Linux SR-IOV subsystem to remove the call to pci_stop_and_remove_bus_device() in drivers/pci/iov.c:pci_iov_remove_virtfn(). * Enable SR-IOV, then disable SR-IOV echo 1 > /sys/bus/pci/devices/0000\:01\:00.0/sriov_numvfs echo 0 > /sys/bus/pci/devices/0000\:01\:00.0/sriov_numvfs * Now we have a stale VF. We can trigger the ASSERT either by unbinding the VF driver and issuing a reset... echo 0000\:01\:10.0 > /sys/bus/pci/devices/0000\:01\:10.0/driver/unbind echo 1 > /sys/bus/pci/devices/0000\:01\:10.0/reset ... or by doing xl pci-assignable-add xl pci-assignable-add 01:10.0 2. Dom0 reporting PF removal without reporting VF removal. * Hack your PF driver to leave SR-IOV enabled when removing the device * Enable SR-IOV echo 1 > /sys/bus/pci/devices/0000\:01\:00.0/sriov_numvfs * Unplug the PCI device (qemu) device_del mydev * Now we have a stale VF. We can trigger the ASSERT either by re-adding the PF device with SR-IOV disabled... echo 0000\:01\:10.0 > /sys/bus/pci/devices/0000\:01\:10.0/driver/unbind (qemu) device_add igb,id=3Dmydev,bus=3Dpcie.1,netdev=3Dnet1 ... or by reset / xl pci-assignable-add as above. --- xen/arch/x86/msi.c | 19 +++++++++---------- xen/drivers/passthrough/msi.c | 3 +++ xen/drivers/vpci/msi.c | 2 +- xen/drivers/vpci/msix.c | 2 +- xen/include/xen/pci.h | 3 +++ 5 files changed, 17 insertions(+), 12 deletions(-) diff --git a/xen/arch/x86/msi.c b/xen/arch/x86/msi.c index ff2e3d86878d..5e24df7be0c0 100644 --- a/xen/arch/x86/msi.c +++ b/xen/arch/x86/msi.c @@ -278,23 +278,21 @@ void __msi_set_enable(u16 seg, u8 bus, u8 slot, u8 fu= nc, int pos, int enable) =20 static void msi_set_enable(struct pci_dev *dev, int enable) { - int pos; + int pos =3D dev->msi_pos; u16 seg =3D dev->seg; u8 bus =3D dev->bus; u8 slot =3D PCI_SLOT(dev->devfn); u8 func =3D PCI_FUNC(dev->devfn); =20 - pos =3D pci_find_cap_offset(dev->sbdf, PCI_CAP_ID_MSI); if ( pos ) __msi_set_enable(seg, bus, slot, func, pos, enable); } =20 static void msix_set_enable(struct pci_dev *dev, int enable) { - int pos; + int pos =3D dev->msix_pos; uint16_t control; =20 - pos =3D pci_find_cap_offset(dev->sbdf, PCI_CAP_ID_MSIX); if ( pos ) { control =3D pci_conf_read16(dev->sbdf, msix_control_reg(pos)); @@ -601,7 +599,7 @@ static int msi_capability_init(struct pci_dev *dev, uint16_t control; =20 ASSERT_PDEV_LIST_IS_READ_LOCKED(dev->domain); - pos =3D pci_find_cap_offset(dev->sbdf, PCI_CAP_ID_MSI); + pos =3D dev->msi_pos; if ( !pos ) return -ENODEV; control =3D pci_conf_read16(dev->sbdf, msi_control_reg(pos)); @@ -764,7 +762,7 @@ static int msix_capability_init(struct pci_dev *dev, u8 slot =3D PCI_SLOT(dev->devfn); u8 func =3D PCI_FUNC(dev->devfn); bool maskall =3D msix->host_maskall, zap_on_error =3D false; - unsigned int pos =3D pci_find_cap_offset(dev->sbdf, PCI_CAP_ID_MSIX); + unsigned int pos =3D dev->msix_pos; =20 if ( !pos ) return -ENODEV; @@ -1133,11 +1131,13 @@ static void _pci_cleanup_msix(struct arch_msix *msi= x) static void __pci_disable_msix(struct msi_desc *entry) { struct pci_dev *dev =3D entry->dev; - unsigned int pos =3D pci_find_cap_offset(dev->sbdf, PCI_CAP_ID_MSIX); + unsigned int pos =3D dev->msix_pos; u16 control =3D pci_conf_read16(dev->sbdf, msix_control_reg(entry->msi_attrib.pos)); bool maskall =3D dev->msix->host_maskall; =20 + ASSERT(pos); + if ( unlikely(!(control & PCI_MSIX_FLAGS_ENABLE)) ) { dev->msix->host_maskall =3D 1; @@ -1241,7 +1241,7 @@ void pci_cleanup_msi(struct pci_dev *pdev) =20 int pci_reset_msix_state(struct pci_dev *pdev) { - unsigned int pos =3D pci_find_cap_offset(pdev->sbdf, PCI_CAP_ID_MSIX); + unsigned int pos =3D pdev->msix_pos; =20 ASSERT(pos); /* @@ -1269,8 +1269,7 @@ int pci_msi_conf_write_intercept(struct pci_dev *pdev= , unsigned int reg, if ( pdev->msix ) { entry =3D find_msi_entry(pdev, -1, PCI_CAP_ID_MSIX); - pos =3D entry ? entry->msi_attrib.pos - : pci_find_cap_offset(pdev->sbdf, PCI_CAP_ID_MSIX); + pos =3D entry ? entry->msi_attrib.pos : pdev->msix_pos; ASSERT(pos); =20 if ( reg >=3D pos && reg < msix_pba_offset_reg(pos) + 4 ) diff --git a/xen/drivers/passthrough/msi.c b/xen/drivers/passthrough/msi.c index 13d904692ef8..ed2bc7ebe635 100644 --- a/xen/drivers/passthrough/msi.c +++ b/xen/drivers/passthrough/msi.c @@ -29,6 +29,7 @@ int pdev_msi_init(struct pci_dev *pdev) { uint16_t ctrl =3D pci_conf_read16(pdev->sbdf, msi_control_reg(pos)= ); =20 + pdev->msi_pos =3D pos; pdev->msi_maxvec =3D multi_msi_capable(ctrl); } =20 @@ -41,6 +42,8 @@ int pdev_msi_init(struct pci_dev *pdev) if ( !msix ) return -ENOMEM; =20 + pdev->msix_pos =3D pos; + spin_lock_init(&msix->table_lock); =20 ctrl =3D pci_conf_read16(pdev->sbdf, msix_control_reg(pos)); diff --git a/xen/drivers/vpci/msi.c b/xen/drivers/vpci/msi.c index dd6620ec5674..66e5a8a116be 100644 --- a/xen/drivers/vpci/msi.c +++ b/xen/drivers/vpci/msi.c @@ -195,7 +195,7 @@ static void cf_check mask_write( =20 static int cf_check init_msi(struct pci_dev *pdev) { - unsigned int pos =3D pci_find_cap_offset(pdev->sbdf, PCI_CAP_ID_MSI); + unsigned int pos =3D pdev->msi_pos; uint16_t control; int ret; =20 diff --git a/xen/drivers/vpci/msix.c b/xen/drivers/vpci/msix.c index 5bb4444ce21f..6bd8c55bb48e 100644 --- a/xen/drivers/vpci/msix.c +++ b/xen/drivers/vpci/msix.c @@ -711,7 +711,7 @@ static int cf_check init_msix(struct pci_dev *pdev) struct vpci_msix *msix; int rc; =20 - msix_offset =3D pci_find_cap_offset(pdev->sbdf, PCI_CAP_ID_MSIX); + msix_offset =3D pdev->msix_pos; if ( !msix_offset ) return 0; =20 diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index 63e49f0117e9..ef56e80651d6 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -113,6 +113,9 @@ struct pci_dev { pci_sbdf_t sbdf; }; =20 + unsigned int msi_pos; + unsigned int msix_pos; + uint8_t msi_maxvec; uint8_t phantom_stride; =20 --=20 2.47.0 From nobody Sun Nov 24 01:40:31 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass(p=quarantine dis=none) header.from=amd.com ARC-Seal: i=2; a=rsa-sha256; t=1729284006; cv=pass; d=zohomail.com; s=zohoarc; b=Abr6f9y+A3jakDr+7ymmr5ODs7RmGctkJ4Ya5xli1O55abYnkFbzfeyt9gimW7jTPaK4ysbeNPILckOlGQOG51BAPBC2f+FhuSAr2PRFKh7borSIsrj83rUFz2giJN7WlHTO0AAl37g4FcFRAaiB1y9Padw8rAkhmevCu5Pc9oY= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1729284006; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=5fq6KP4wVu47kB55ME6GPz2MtcQHWslf85p2ZaJ9WoQ=; b=UKU31zQjy0O/V8lGI8fhvnSAVMjLA58AQCVLxJGEsv1NBbMqGab7zc02EmsIZIn1xdKVfS1FjIRg3gFs7boObEssCrk6l2fJnx9yeYznGngqHz5RIY1SNgnaC5ce6+D8rQwOPGdRMhOu3bI2GKqY+grwUh6jEInSf9t8I6My0/k= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 17292840067141010.323750681781; Fri, 18 Oct 2024 13:40:06 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.822150.1236129 (Exim 4.92) (envelope-from ) id 1t1tl8-0001V4-A9; Fri, 18 Oct 2024 20:39:42 +0000 Received: by outflank-mailman (output) from mailman id 822150.1236129; Fri, 18 Oct 2024 20:39:42 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1t1tl8-0001Ux-5m; Fri, 18 Oct 2024 20:39:42 +0000 Received: by outflank-mailman (input) for mailman id 822150; Fri, 18 Oct 2024 20:39:41 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1t1tl7-0000xC-G4 for xen-devel@lists.xenproject.org; Fri, 18 Oct 2024 20:39:41 +0000 Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on20629.outbound.protection.outlook.com [2a01:111:f403:240a::629]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 189e824f-8d91-11ef-a0be-8be0dac302b0; Fri, 18 Oct 2024 22:39:40 +0200 (CEST) Received: from BY3PR05CA0023.namprd05.prod.outlook.com (2603:10b6:a03:254::28) by PH7PR12MB5853.namprd12.prod.outlook.com (2603:10b6:510:1d4::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.27; Fri, 18 Oct 2024 20:39:34 +0000 Received: from SJ1PEPF00002310.namprd03.prod.outlook.com (2603:10b6:a03:254:cafe::23) by BY3PR05CA0023.outlook.office365.com (2603:10b6:a03:254::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8093.10 via Frontend Transport; Fri, 18 Oct 2024 20:39:34 +0000 Received: from SATLEXMB04.amd.com (165.204.84.17) by SJ1PEPF00002310.mail.protection.outlook.com (10.167.242.164) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8069.17 via Frontend Transport; Fri, 18 Oct 2024 20:39:32 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 18 Oct 2024 15:39:32 -0500 Received: from ubuntu.mshome.net (10.180.168.240) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Fri, 18 Oct 2024 15:39:30 -0500 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 189e824f-8d91-11ef-a0be-8be0dac302b0 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Cv2J0MgqpazzhLjeyG63sHGxWuLt7ZVUSdd0KdofanA7yDEH8bw3QXS9Y7OPy6LOIf5DbpIvmxjTWP0hFx/h98k7kXEh/JPRBSwHl0Ilgh4ffpg9CCIFOk0j+7/zIbirnxjRZ4idG6Qv8usIJThdSjgqIyXP4lJgKhb5z5DseJgbjwzTI+NgkWT7dqsGA++NzvyzNrbAh6PBVcC0DU5cXZEjmmvmP3UOwxiMoftF6RD+0+xpJNRCS+6P+csUbSiFkDt7YdQfqizToKy9pWsMbxr5BF+oSjjKJDUKgTfJGtJlbnkA69AtcGveOrXnN72yOOcBQNXqiU6ku1qhWEzgBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5fq6KP4wVu47kB55ME6GPz2MtcQHWslf85p2ZaJ9WoQ=; b=ykw8fcwvufKrJhuX22BIw/6P8KamRUbPK/Xlzsi16DpADEeyc/S9M3ct9+f34VT/jbSar4H6j+pf8z52VM/6aqBWj9x3T70cy23l5OvdBYaBSjnF/n6LHzdEjCOdiVpGzvOlWIvSEKZvBJw8+qBm6Uu2knciLgUEiTRi9rU91pFp31aAjxtVmaF3+keyxDmlO+tOCiWnzHqoMAXabiFdHSeDyTVLmqVva/AOgoFoIAfRU5yS6ttjpHCQBEfgTp0OsAKUhWju+x5ZBsi3Cbhdzqe2pGvKtaQ7LQCsT5aTlVD1xag/yHI+5IWr6W1KOqy/GssqjuCNl/HH24BqAn1Kug== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5fq6KP4wVu47kB55ME6GPz2MtcQHWslf85p2ZaJ9WoQ=; b=nO3TspkTNOZMz6/ExV5qiHKosYq9LZxlGZc9pp03SIryrAyucVvxi0bUyt/aqdq7XoLJELL1oLGxekbtDTNO8/cxps2ygs/SR5SY2nnXhdfh7iz4HrxFKIAlh8SuJk8R6uaQkHbepSYn/HBl5Z/XOGe4embPuUeKBpmUyAUlyjA= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C From: Stewart Hildebrand To: CC: Stewart Hildebrand , Jan Beulich , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Andrew Cooper , "Julien Grall" , Stefano Stabellini Subject: [PATCH v6 2/3] xen/pci: introduce PF<->VF links Date: Fri, 18 Oct 2024 16:39:09 -0400 Message-ID: <20241018203913.1162962-3-stewart.hildebrand@amd.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241018203913.1162962-1-stewart.hildebrand@amd.com> References: <20241018203913.1162962-1-stewart.hildebrand@amd.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: None (SATLEXMB04.amd.com: stewart.hildebrand@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF00002310:EE_|PH7PR12MB5853:EE_ X-MS-Office365-Filtering-Correlation-Id: 3aa28764-1c2a-4471-cffc-08dcefb4f90b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|82310400026|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?WQGirubvFsMHFc/rghVPZsvKPdBCWJT49f86mJmn8n4o/wJIu7Xw7RffwylS?= =?us-ascii?Q?vLEFkqOCpzecJ0p2mhLdSy0J2NpKsvgZ50k+5PNn59TLz1Qkhq3Rph5XYVr3?= =?us-ascii?Q?bflyCLx/3pjVgU42Jf3FWiWOhCb2Kokbm2Kvga7WemRzHQ6Ijh6dzo3G3toE?= =?us-ascii?Q?mZ0FdzjF3ijxsZSLlMndIVE+NwF99enshGFmCIzwbrINizPVeOPwrWsyskdg?= =?us-ascii?Q?0b5/yi538y/lLhcDZ1JZe1fWtHIOa6rBIYvFX5LsctSbA7jlGUzjsUSub+ZW?= =?us-ascii?Q?8B8yV3+DpxamhJVHGzqsVnfddHWQEXwP31AYylde6CBfDhEqQ3tjwCrzJ0Uf?= =?us-ascii?Q?BtwBJ707xPTC/0M8v94Z2BApRPv+zGgtYptFlSylnDWIzvnEGVgj0bHE1lWT?= =?us-ascii?Q?337RPQ3MiUi5ZH0yH93PLWn9Bb1lxrpLNCCm2iNUtapOZ38sxi8riSRlQC3w?= =?us-ascii?Q?hLR0hAj/7yg9QMqeohxoKEGnD23CxPG4Nil/YgNnN4EHTGOvbyFLy9ppbtLm?= =?us-ascii?Q?XtPpFkJydTBX5SYw9vg4Er03ICZTJqK3E59RJ12Badf3phILJz5FlRxnuZO/?= =?us-ascii?Q?58seha9sNPL6uSSU3An9RHSicHjAdNSQvBNMCHnfI2FkmNY24fS6LMMCKvNf?= =?us-ascii?Q?zxLNrMCD3vjEm1SbNf741cAmeYQ9x3Xv1T/4M3H3SwVUUYQty/UjeoppjSeM?= =?us-ascii?Q?oNdnuz3AGCRXJNX845y1BCpd6kXyOFa992Iz6MSvHlFM1wilQz6q+FqSU70G?= =?us-ascii?Q?Dg3TOa7gNzU1ZSPguhSTEpBOe7BeUphj7+eD4hxkjAlqVeqjHFL9NZzj2kTC?= =?us-ascii?Q?2BhHeCyUnBLfp5hacWKNqzHJHoCIB77beFPtNcSe5l8dOTe+Ovy7Fr2GPAra?= =?us-ascii?Q?OWQUslkhW2pOKZe0uXbweogEWI9x8DTj31qhLnC7IOkRn746zyPGfF4meDfz?= =?us-ascii?Q?OACEJPzZGlUXycid+0Bts0y4uA3eH4rmh9ACq/kTCqVmRmHDHthEY2wAl9nJ?= =?us-ascii?Q?liecRitO5Ee4Dm4YUbKYv3/mDri/OxiudcEn1U6wZvvVRrTv7aEU1DsvvJ4O?= =?us-ascii?Q?jASTYHCFDZ1yNlCOFASRkCrJ4o/GfBnp0h2rzWj2L8pfd52zRUlP2TseREbK?= =?us-ascii?Q?P7WGNvXuqhuPCxrHJWffukHq6W2yFmTN/uGofdJ5NHWz5KaXZ7mlh1HudT+j?= =?us-ascii?Q?80Mpu/QzGQ47oyY5kV1/RGzcv1F5SwUKzdyaalcr9KxjEeFrGHXxy6ZuzREC?= =?us-ascii?Q?SZnTt98XHFUtV0avjoDIq8dQ2wwWr8X+Mn1XbVX0mC5dbdGhdOct1eXiSWgL?= =?us-ascii?Q?bqCcYqjbMj5y1zBnyHK0UOQAZ793ZrDnCF2ituJ0U5861U7Vvk0TGQmIarxI?= =?us-ascii?Q?bmMEOeBLtLGn/plfVtzuT8XzJUHUp4hQcvs0J3KCmu6JABxNJw=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(82310400026)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2024 20:39:32.9657 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3aa28764-1c2a-4471-cffc-08dcefb4f90b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF00002310.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5853 X-ZohoMail-DKIM: pass (identity @amd.com) X-ZM-MESSAGEID: 1729284008672116600 Content-Type: text/plain; charset="utf-8" Add links between a VF's struct pci_dev and its associated PF struct pci_dev. Move the calls to pci_get_pdev()/pci_add_device() down to avoid dropping and re-acquiring the pcidevs_lock(). During PF removal, unlink VF from PF and mark the VF broken. As before, VFs may exist without a corresponding PF, although now only with pdev->broken =3D true. The hardware domain is expected to remove the associated VFs before removing the PF. Print a warning in case a PF is removed with associated VFs still present. Signed-off-by: Stewart Hildebrand --- Candidate for backport to 4.19 (the next patch depends on this one) v5->v6: * move printk() before ASSERT_UNREACHABLE() * warn about PF removal with VFs still present * clarify commit message v4->v5: * new patch, split from ("x86/msi: fix locking for SR-IOV devices") * move INIT_LIST_HEAD(&pdev->vf_list); earlier * collapse struct list_head instances * retain error code from pci_add_device() * unlink (and mark broken) VFs instead of removing them * const-ify VF->PF link --- xen/drivers/passthrough/pci.c | 76 ++++++++++++++++++++++++++++------- xen/include/xen/pci.h | 10 +++++ 2 files changed, 72 insertions(+), 14 deletions(-) diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 74d3895e1ef6..fe31255b1207 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -333,6 +333,8 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg,= u8 bus, u8 devfn) *((u8*) &pdev->devfn) =3D devfn; pdev->domain =3D NULL; =20 + INIT_LIST_HEAD(&pdev->vf_list); + arch_pci_init_pdev(pdev); =20 rc =3D pdev_msi_init(pdev); @@ -449,6 +451,10 @@ static void free_pdev(struct pci_seg *pseg, struct pci= _dev *pdev) =20 list_del(&pdev->alldevs_list); pdev_msi_deinit(pdev); + + if ( pdev->info.is_virtfn && pdev->virtfn.pf_pdev ) + list_del(&pdev->vf_list); + xfree(pdev); } =20 @@ -656,24 +662,11 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, unsigned int slot =3D PCI_SLOT(devfn), func =3D PCI_FUNC(devfn); const char *type; int ret; - bool pf_is_extfn =3D false; =20 if ( !info ) type =3D "device"; else if ( info->is_virtfn ) - { - pcidevs_lock(); - pdev =3D pci_get_pdev(NULL, - PCI_SBDF(seg, info->physfn.bus, - info->physfn.devfn)); - if ( pdev ) - pf_is_extfn =3D pdev->info.is_extfn; - pcidevs_unlock(); - if ( !pdev ) - pci_add_device(seg, info->physfn.bus, info->physfn.devfn, - NULL, node); type =3D "virtual function"; - } else if ( info->is_extfn ) type =3D "extended function"; else @@ -703,7 +696,44 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, * extended function. */ if ( pdev->info.is_virtfn ) - pdev->info.is_extfn =3D pf_is_extfn; + { + struct pci_dev *pf_pdev; + + pf_pdev =3D pci_get_pdev(NULL, + PCI_SBDF(seg, info->physfn.bus, + info->physfn.devfn)); + + if ( !pf_pdev ) + { + ret =3D pci_add_device(seg, info->physfn.bus, info->physfn= .devfn, + NULL, node); + if ( ret ) + { + printk(XENLOG_WARNING "Failed to add SR-IOV device PF = %pp for VF %pp\n", + &PCI_SBDF(seg, info->physfn.bus, info->physfn.d= evfn), + &pdev->sbdf); + free_pdev(pseg, pdev); + goto out; + } + pf_pdev =3D pci_get_pdev(NULL, + PCI_SBDF(seg, info->physfn.bus, + info->physfn.devfn)); + if ( !pf_pdev ) + { + printk(XENLOG_ERR "Failed to find SR-IOV device PF %pp= for VF %pp\n", + &PCI_SBDF(seg, info->physfn.bus, info->physfn.d= evfn), + &pdev->sbdf); + ASSERT_UNREACHABLE(); + free_pdev(pseg, pdev); + ret =3D -EILSEQ; + goto out; + } + } + + pdev->info.is_extfn =3D pf_pdev->info.is_extfn; + pdev->virtfn.pf_pdev =3D pf_pdev; + list_add(&pdev->vf_list, &pf_pdev->vf_list); + } } =20 if ( !pdev->info.is_virtfn && !pdev->vf_rlen[0] ) @@ -821,6 +851,24 @@ int pci_remove_device(u16 seg, u8 bus, u8 devfn) list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list ) if ( pdev->bus =3D=3D bus && pdev->devfn =3D=3D devfn ) { + if ( !pdev->info.is_virtfn ) + { + struct pci_dev *vf_pdev, *tmp; + bool warn_stale_vfs =3D false; + + list_for_each_entry_safe(vf_pdev, tmp, &pdev->vf_list, vf_= list) + { + list_del(&vf_pdev->vf_list); + vf_pdev->virtfn.pf_pdev =3D NULL; + vf_pdev->broken =3D true; + warn_stale_vfs =3D true; + } + + if ( warn_stale_vfs ) + printk(XENLOG_WARNING "PCI SR-IOV PF %pp removed with = VFs still present\n", + &pdev->sbdf); + } + if ( pdev->domain ) { write_lock(&pdev->domain->pci_lock); diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index ef56e80651d6..2ea168d5f914 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -153,7 +153,17 @@ struct pci_dev { unsigned int count; #define PT_FAULT_THRESHOLD 10 } fault; + + /* + * List head if info.is_virtfn =3D=3D false + * List entry if info.is_virtfn =3D=3D true + */ + struct list_head vf_list; u64 vf_rlen[6]; + struct { + /* Only populated for VFs (info.is_virtfn =3D=3D true) */ + const struct pci_dev *pf_pdev; /* Link from VF to PF */ + } virtfn; =20 /* Data for vPCI. */ struct vpci *vpci; --=20 2.47.0 From nobody Sun Nov 24 01:40:31 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass(p=quarantine dis=none) header.from=amd.com ARC-Seal: i=2; a=rsa-sha256; t=1729284008; cv=pass; d=zohomail.com; s=zohoarc; b=KsAH4p2m8nNJZuSSIBPapDhMR/O60wRt+PcwATp8iabRP0wtCdco3BkqFbdgfq25Leu3afxIfJ8PRU3rum/GY3YVVPf88HwL2hGIsHlW6cn8pexChMb509FcB9ZaO2rdzWAo/Bwsv576Lj+C92+IV5+quxCbNXQYcNW4gpdk+Fk= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1729284008; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=PYAsN6CZPwERab7Ma9P37546OYVBrhDtCRKV6N+OMD0=; b=eV6H2cTJ8Ls2Oq8qX5DQm2f2U31pSq+JrGV2bxIl6w6kZ8oV3i324SuQ3bo8AOY0HGXHPJ2TL6wzLZrt57ITEN4ZynO4/c02hg78qdpfqJ54a405ADa9/Q2uCTqD8eYihPeOkwBa94enMOGUoWxwRmXtqp6EpsQ9wFbsbd+gEio= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1729284008619610.4114000587504; Fri, 18 Oct 2024 13:40:08 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.822152.1236139 (Exim 4.92) (envelope-from ) id 1t1tlF-0001uG-NT; Fri, 18 Oct 2024 20:39:49 +0000 Received: by outflank-mailman (output) from mailman id 822152.1236139; Fri, 18 Oct 2024 20:39:49 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1t1tlF-0001u9-IP; Fri, 18 Oct 2024 20:39:49 +0000 Received: by outflank-mailman (input) for mailman id 822152; Fri, 18 Oct 2024 20:39:48 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1t1tlE-0001Cl-CZ for xen-devel@lists.xenproject.org; Fri, 18 Oct 2024 20:39:48 +0000 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on20622.outbound.protection.outlook.com [2a01:111:f403:2415::622]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 1c11bab8-8d91-11ef-99a3-01e77a169b0f; Fri, 18 Oct 2024 22:39:46 +0200 (CEST) Received: from SJ0PR13CA0217.namprd13.prod.outlook.com (2603:10b6:a03:2c1::12) by MW4PR12MB5667.namprd12.prod.outlook.com (2603:10b6:303:18a::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.26; Fri, 18 Oct 2024 20:39:40 +0000 Received: from SJ1PEPF00002316.namprd03.prod.outlook.com (2603:10b6:a03:2c1:cafe::fc) by SJ0PR13CA0217.outlook.office365.com (2603:10b6:a03:2c1::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8093.8 via Frontend Transport; Fri, 18 Oct 2024 20:39:40 +0000 Received: from SATLEXMB04.amd.com (165.204.84.17) by SJ1PEPF00002316.mail.protection.outlook.com (10.167.242.170) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8069.17 via Frontend Transport; Fri, 18 Oct 2024 20:39:40 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 18 Oct 2024 15:39:39 -0500 Received: from ubuntu.mshome.net (10.180.168.240) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Fri, 18 Oct 2024 15:39:38 -0500 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 1c11bab8-8d91-11ef-99a3-01e77a169b0f ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=GHxmXdiw5IU7oEfNP0HaZXGgUZId99mv4mG3SeZCS2GgBPUynfPwoUGS7BVFpFgSvV6no9aVFdQxlTSTWN/WhmI2kuJXf2E0NzvaDO5AY05hLNRuysAgyukOv/s0rdPsC8N1H9OTHSgt2VpYBZvjRTmOJXFkGIRdJdeGPpMQld8AyqQYpeO3sYaO/V/8OaD3XmxtD7XqnoAH0qLZ7Phah+N1PKeTgJ552rdASgOMo4oGE+plqNDVvim39xGLMLmxmqro6l/qhw7EnlmhttfKRPWfXj082Y3iKb37ygh26W2nZpdnx6ntiN+I2cEfaR273obH7Btv7GNzelf88wnHrQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PYAsN6CZPwERab7Ma9P37546OYVBrhDtCRKV6N+OMD0=; b=wpZcka6lRPbwqkN1O7JFMWLQvhXi0u2dukNDCTzFcufw3LOh0jivqV4KQSNLzScDgtKrRisKHSWFc2Xa18yjblrONvd6ZkPTzZy5EhvG4HRSMIDp067VW208/o1a6Z4aOnU7KQudQ1eq2rR3g1bvQIYLHuS18RuzeA9RZiYAGEodTOdF2Ixd9HeXqt4vIvy21+yu83Gkz1yOnBVqyxAR4eb0OreCGpuA9hYg9trU0NKbPQCzC4MO0ioQ4S8BjKlaWpydmqFPUIjbfAjEZYR2aGbVRFzVrK6eXZoW0cEtZy+sZS2wqiF5bFtxE1zN9gFqSk3lPCxFNpORHpFj0Vn19w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PYAsN6CZPwERab7Ma9P37546OYVBrhDtCRKV6N+OMD0=; b=qJrLA6fbYJXB5DF5u5hmqnMjzO7t7L5UqsP7FbZGVktZbEeAK8TyvOx51r7Eno3D0PvbMD1HF9dat1MMT/Gv/zy/w0yBKU/uBPePMNfsY+20T5f8brxa/hSzbGvfduom2OJo4ljUWF0//mZwwAcQa7ldK10NW4yShlUrsJMiVsE= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C From: Stewart Hildebrand To: CC: Stewart Hildebrand , Jan Beulich , Andrew Cooper , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Julien Grall , Stefano Stabellini , Teddy Astie Subject: [PATCH v6 3/3] x86/msi: fix locking for SR-IOV devices Date: Fri, 18 Oct 2024 16:39:10 -0400 Message-ID: <20241018203913.1162962-4-stewart.hildebrand@amd.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241018203913.1162962-1-stewart.hildebrand@amd.com> References: <20241018203913.1162962-1-stewart.hildebrand@amd.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: None (SATLEXMB04.amd.com: stewart.hildebrand@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF00002316:EE_|MW4PR12MB5667:EE_ X-MS-Office365-Filtering-Correlation-Id: 2694bcef-708d-4ff1-2a94-08dcefb4fd85 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|376014|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?oe5+MRPL6/ZqUvsviAbciql0oRQEzSLi3j3jH7CLofKauSjF+Ah6WujdgLst?= =?us-ascii?Q?V3xKwshuT0KFfBx3t2BVFT03aTimJr5n9rQDDciv+sjQhFYlpv4XqChLV5yb?= =?us-ascii?Q?Lg3hFO2SU1oEtCjIWDjPKfYsxWZRY82tOjaxQ6eIWBirURE4j/XF/RTYUUY7?= =?us-ascii?Q?LfGqorhgKIKkXIzmoBhaQM05nrmrCfjRq4FbebNb4brZD7gXsvyFNq1+wjHJ?= =?us-ascii?Q?D2VvHxMAS1ZPI9o0uDdJ9TsjoCESDnPBI8kQV/IQ+p3zYSPqvW6dQH2HUJjK?= =?us-ascii?Q?gupL7xTwPLVqcZlMeVJZFS3HOuIB+DRQQkuedhu89t8w0Y3bf7nMoABub79G?= =?us-ascii?Q?yNdaFN11lmvfoKz9YxicrHNQvKvf1EDjP9fpHsEwJPql9s6eQVRGva8NEASC?= =?us-ascii?Q?2rhw0Y9Ci0C4Fu6kECh7HOz1IV7Zntiwa8S6VxXPbdNIzBMhHWdGg4oUSPGn?= =?us-ascii?Q?vyBkNioH+d94O6Dge4aoQqa8Vtl6HZt5wBbQobs9a76bQziSTHo5vt0l3Ji3?= =?us-ascii?Q?Z9p0kiRcLsnkHM3TyIMMEGp3UwXCuT4vcT4aGNs3B7Mm14HsYVfBenZav6Vg?= =?us-ascii?Q?9EZd7FnxzDYgS/JTjOP5L1cSRNdOHLpgt2y04ZjPPlAM5/60jUQKtxv30IQf?= =?us-ascii?Q?CVvIrffLi13YWEZBne5dvWuupHQgHH691vXmTp2uV6KBZfC9jj8veJ7m9kVZ?= =?us-ascii?Q?xbJOox1vziSNRggzEikYvv8FHufR0l/rh1XkdtmdV4OFBmSI3Dtcr40XG5CX?= =?us-ascii?Q?9wVtm8M8eC+VMojfhavFfVmQ//XHjXZjaaaqOGj+Uk7OivwD6/82xqz6ottu?= =?us-ascii?Q?rEcLDHdJBOcHz4BBzURoDc/VsNxNLbGbYuerjCZCbk5ujqQkfDa0xecNqZr2?= =?us-ascii?Q?566Ox8B3MGL6ClBV0Abq9W3dmAnF5ePMeEPqceYLLL6Rv6Op3d2sljr5R2r+?= =?us-ascii?Q?egoebmWZt6WJTc34ITj3OqLhNehHlSTdORv/0ZlcUTV4uCFdFsjEXdnbFtfR?= =?us-ascii?Q?RCf+449sgkonIyXrHWR7coMjuOoKQm5leyetGlETM6bKBOaaYBJYNsCPBdVv?= =?us-ascii?Q?VYYP5bl0ybvppkzNIHAtUe2bJfKzc8XHzDZwgts9kElGd+nqf4GroFje4ST2?= =?us-ascii?Q?d0PT7pi3AEVZ/yahhDOzeVTr+KEBLQh6zlYYLBYuKyYKPEGYSovHj2w74igT?= =?us-ascii?Q?gf+pj6B7nC7XTq2deYuHWX3WoGCNxuZ0VgzcZjm63m03yHPvJICCKUOmjjdI?= =?us-ascii?Q?NiSG9cGa3V2s+tXqtxHIbfqeDquVymKJPUmG1yl9GHjnCwdgkUiQONlGWUnm?= =?us-ascii?Q?805th9Hxl0v/ZGgMb3pCyRZ20gW0xyuWPAUGMoA4iiu0LGkGh46FqDLJ8Vec?= =?us-ascii?Q?Tai2v1huY0JK6hwVByTOEQywKhvoaTta1Z3BpMkSI4eLLe/eNw=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(376014)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2024 20:39:40.4876 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2694bcef-708d-4ff1-2a94-08dcefb4fd85 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF00002316.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB5667 X-ZohoMail-DKIM: pass (identity @amd.com) X-ZM-MESSAGEID: 1729284010701116600 Content-Type: text/plain; charset="utf-8" In commit 4f78438b45e2 ("vpci: use per-domain PCI lock to protect vpci structure") a lock was moved from allocate_and_map_msi_pirq() to the caller and changed from pcidevs_lock() to read_lock(&d->pci_lock). However, one call path wasn't updated to reflect the change, leading to a failed assertion observed under the following conditions: * PV dom0 * Debug build (CONFIG_DEBUG=3Dy) of Xen * There is an SR-IOV device in the system with one or more VFs enabled * Dom0 has loaded the driver for the VF and enabled MSI-X (XEN) Assertion 'd || pcidevs_locked()' failed at drivers/passthrough/pci.c= :535 (XEN) ----[ Xen-4.20-unstable x86_64 debug=3Dy Not tainted ]---- ... (XEN) Xen call trace: (XEN) [] R pci_get_pdev+0x4c/0xab (XEN) [] F arch/x86/msi.c#read_pci_mem_bar+0x58/0x272 (XEN) [] F arch/x86/msi.c#msix_capability_init+0x198/0= x755 (XEN) [] F arch/x86/msi.c#__pci_enable_msix+0x82/0xe8 (XEN) [] F pci_enable_msi+0x3f/0x78 (XEN) [] F map_domain_pirq+0x2a4/0x6dc (XEN) [] F allocate_and_map_msi_pirq+0x103/0x262 (XEN) [] F physdev_map_pirq+0x210/0x259 (XEN) [] F do_physdev_op+0x9c3/0x1454 (XEN) [] F pv_hypercall+0x5ac/0x6af (XEN) [] F lstar_enter+0x143/0x150 In read_pci_mem_bar(), the VF obtains the struct pci_dev pointer for its associated PF to access the vf_rlen array. This array is initialized in pci_add_device() and is only populated in the associated PF's struct pci_dev. Access the vf_rlen array via the link to the PF, and remove the troublesome call to pci_get_pdev(). Fixes: 4f78438b45e2 ("vpci: use per-domain PCI lock to protect vpci structu= re") Reported-by: Teddy Astie Signed-off-by: Stewart Hildebrand Reviewed-by: Jan Beulich --- Candidate for backport to 4.19 Patch #2 ("xen/pci: introduce PF<->VF links") is pre-requisite v5->v6: * add Jan's R-b v4->v5: * split the PF<->VF links to a pre-requisite patch * pass pci_sbdf_t to read_pci_mem_bar() * use stdint.h types on changed lines * re-add NULL check for pf_info in read_pci_mem_bar(), as pf_info could be NULL v3->v4: * handle case when PF is removed with VFs enabled, then re-added with VFs disabled v2->v3: * link from VF to PF's struct pci_dev * v1->v2: * remove call to pci_get_pdev() --- xen/arch/x86/msi.c | 37 ++++++++++++++++++++++------------- xen/drivers/passthrough/pci.c | 8 +++++--- xen/include/xen/pci.h | 15 +++++++++----- 3 files changed, 38 insertions(+), 22 deletions(-) diff --git a/xen/arch/x86/msi.c b/xen/arch/x86/msi.c index 5e24df7be0c0..79fff9de4305 100644 --- a/xen/arch/x86/msi.c +++ b/xen/arch/x86/msi.c @@ -660,34 +660,35 @@ static int msi_capability_init(struct pci_dev *dev, return 0; } =20 -static u64 read_pci_mem_bar(u16 seg, u8 bus, u8 slot, u8 func, u8 bir, int= vf) +static uint64_t read_pci_mem_bar(pci_sbdf_t sbdf, uint8_t bir, int vf, + const struct pf_info *pf_info) { + uint16_t seg =3D sbdf.seg; + uint8_t bus =3D sbdf.bus, slot =3D sbdf.dev, func =3D sbdf.fn; u8 limit; u32 addr, base =3D PCI_BASE_ADDRESS_0; u64 disp =3D 0; =20 if ( vf >=3D 0 ) { - struct pci_dev *pdev =3D pci_get_pdev(NULL, - PCI_SBDF(seg, bus, slot, func)= ); unsigned int pos; uint16_t ctrl, num_vf, offset, stride; =20 - if ( !pdev ) + if ( !pf_info ) return 0; =20 - pos =3D pci_find_ext_capability(pdev->sbdf, PCI_EXT_CAP_ID_SRIOV); - ctrl =3D pci_conf_read16(pdev->sbdf, pos + PCI_SRIOV_CTRL); - num_vf =3D pci_conf_read16(pdev->sbdf, pos + PCI_SRIOV_NUM_VF); - offset =3D pci_conf_read16(pdev->sbdf, pos + PCI_SRIOV_VF_OFFSET); - stride =3D pci_conf_read16(pdev->sbdf, pos + PCI_SRIOV_VF_STRIDE); + pos =3D pci_find_ext_capability(sbdf, PCI_EXT_CAP_ID_SRIOV); + ctrl =3D pci_conf_read16(sbdf, pos + PCI_SRIOV_CTRL); + num_vf =3D pci_conf_read16(sbdf, pos + PCI_SRIOV_NUM_VF); + offset =3D pci_conf_read16(sbdf, pos + PCI_SRIOV_VF_OFFSET); + stride =3D pci_conf_read16(sbdf, pos + PCI_SRIOV_VF_STRIDE); =20 if ( !pos || !(ctrl & PCI_SRIOV_CTRL_VFE) || !(ctrl & PCI_SRIOV_CTRL_MSE) || !num_vf || !offset || (num_vf > 1 && !stride) || bir >=3D PCI_SRIOV_NUM_BARS || - !pdev->vf_rlen[bir] ) + !pf_info->vf_rlen[bir] ) return 0; base =3D pos + PCI_SRIOV_BAR; vf -=3D PCI_BDF(bus, slot, func) + offset; @@ -701,8 +702,8 @@ static u64 read_pci_mem_bar(u16 seg, u8 bus, u8 slot, u= 8 func, u8 bir, int vf) } if ( vf >=3D num_vf ) return 0; - BUILD_BUG_ON(ARRAY_SIZE(pdev->vf_rlen) !=3D PCI_SRIOV_NUM_BARS); - disp =3D vf * pdev->vf_rlen[bir]; + BUILD_BUG_ON(ARRAY_SIZE(pf_info->vf_rlen) !=3D PCI_SRIOV_NUM_BARS); + disp =3D vf * pf_info->vf_rlen[bir]; limit =3D PCI_SRIOV_NUM_BARS; } else switch ( pci_conf_read8(PCI_SBDF(seg, bus, slot, func), @@ -811,6 +812,7 @@ static int msix_capability_init(struct pci_dev *dev, int vf; paddr_t pba_paddr; unsigned int pba_offset; + const struct pf_info *pf_info; =20 if ( !dev->info.is_virtfn ) { @@ -818,6 +820,7 @@ static int msix_capability_init(struct pci_dev *dev, pslot =3D slot; pfunc =3D func; vf =3D -1; + pf_info =3D NULL; } else { @@ -825,9 +828,14 @@ static int msix_capability_init(struct pci_dev *dev, pslot =3D PCI_SLOT(dev->info.physfn.devfn); pfunc =3D PCI_FUNC(dev->info.physfn.devfn); vf =3D dev->sbdf.bdf; + if ( dev->virtfn.pf_pdev ) + pf_info =3D &dev->virtfn.pf_pdev->physfn; + else + pf_info =3D NULL; } =20 - table_paddr =3D read_pci_mem_bar(seg, pbus, pslot, pfunc, bir, vf); + table_paddr =3D read_pci_mem_bar(PCI_SBDF(seg, pbus, pslot, pfunc)= , bir, + vf, pf_info); WARN_ON(msi && msi->table_base !=3D table_paddr); if ( !table_paddr ) { @@ -850,7 +858,8 @@ static int msix_capability_init(struct pci_dev *dev, =20 pba_offset =3D pci_conf_read32(dev->sbdf, msix_pba_offset_reg(pos)= ); bir =3D (u8)(pba_offset & PCI_MSIX_BIRMASK); - pba_paddr =3D read_pci_mem_bar(seg, pbus, pslot, pfunc, bir, vf); + pba_paddr =3D read_pci_mem_bar(PCI_SBDF(seg, pbus, pslot, pfunc), = bir, vf, + pf_info); WARN_ON(!pba_paddr); pba_paddr +=3D pba_offset & ~PCI_MSIX_BIRMASK; =20 diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index fe31255b1207..9182723ece1b 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -736,7 +736,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, } } =20 - if ( !pdev->info.is_virtfn && !pdev->vf_rlen[0] ) + if ( !pdev->info.is_virtfn && !pdev->physfn.vf_rlen[0] ) { unsigned int pos =3D pci_find_ext_capability(pdev->sbdf, PCI_EXT_CAP_ID_SRIOV); @@ -748,7 +748,9 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, { unsigned int i; =20 - BUILD_BUG_ON(ARRAY_SIZE(pdev->vf_rlen) !=3D PCI_SRIOV_NUM_BARS= ); + BUILD_BUG_ON(ARRAY_SIZE(pdev->physfn.vf_rlen) !=3D + PCI_SRIOV_NUM_BARS); + for ( i =3D 0; i < PCI_SRIOV_NUM_BARS; ) { unsigned int idx =3D pos + PCI_SRIOV_BAR + i * 4; @@ -763,7 +765,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, continue; } ret =3D pci_size_mem_bar(pdev->sbdf, idx, NULL, - &pdev->vf_rlen[i], + &pdev->physfn.vf_rlen[i], PCI_BAR_VF | ((i =3D=3D PCI_SRIOV_NUM_BARS - 1) ? PCI_BAR_LAST : 0)); diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index 2ea168d5f914..e961a3c66799 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -159,11 +159,16 @@ struct pci_dev { * List entry if info.is_virtfn =3D=3D true */ struct list_head vf_list; - u64 vf_rlen[6]; - struct { - /* Only populated for VFs (info.is_virtfn =3D=3D true) */ - const struct pci_dev *pf_pdev; /* Link from VF to PF */ - } virtfn; + union { + struct pf_info { + /* Only populated for PFs (info.is_virtfn =3D=3D false) */ + uint64_t vf_rlen[PCI_SRIOV_NUM_BARS]; + } physfn; + struct { + /* Only populated for VFs (info.is_virtfn =3D=3D true) */ + const struct pci_dev *pf_pdev; /* Link from VF to PF */ + } virtfn; + }; =20 /* Data for vPCI. */ struct vpci *vpci; --=20 2.47.0