From nobody Mon Nov 3 06:56:10 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass(p=quarantine dis=none) header.from=amd.com ARC-Seal: i=2; a=rsa-sha256; t=1731444850; cv=pass; d=zohomail.com; s=zohoarc; b=BD+y5DV3IW/MGhJVI3aViLLJHk4KUVrjMRVX37nsGW1HKgVaQUxztiaD9dNlj47zaDXPSKpCSiJ/K1y0+5yY4RNIv78wd3I3aKA4+gRjc7umzpvtFsK99aylsM3Z5JjrPWZq/+BAr0TjouVxZfGiMrKfehKTN4FkDEBZZNpDBwY= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1731444850; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=ky8HxzXjH0qC2Vc4vqFd2aklwcMHHpdbS4ww5q29aEk=; b=M3yXzkbq5HpUKAD1BpBrLNeqI0Y51DuRdtM3NHzdHf4z/4xrq2S3UL7hkCMBNbD9rTvXqPLcreT5X1Btu2f1zYLXH9F+9dwjnc+aafVhm8xAPBANNkEbhpEULRPeBT2NhOzRHJ4cdMaQi/F3jtZ5GI5s9/5Cal/Cm1wQZk1w7ck= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1731444850536362.02226105670104; Tue, 12 Nov 2024 12:54:10 -0800 (PST) Received: from list by lists.xenproject.org with outflank-mailman.834911.1250716 (Exim 4.92) (envelope-from ) id 1tAxtN-0006mg-T6; Tue, 12 Nov 2024 20:53:41 +0000 Received: by outflank-mailman (output) from mailman id 834911.1250716; Tue, 12 Nov 2024 20:53:41 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tAxtN-0006mZ-QU; Tue, 12 Nov 2024 20:53:41 +0000 Received: by outflank-mailman (input) for mailman id 834911; Tue, 12 Nov 2024 20:53:40 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tAxtM-0006YG-42 for xen-devel@lists.xenproject.org; Tue, 12 Nov 2024 20:53:40 +0000 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on20625.outbound.protection.outlook.com [2a01:111:f403:200a::625]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 2f938939-a138-11ef-a0c7-8be0dac302b0; Tue, 12 Nov 2024 21:53:37 +0100 (CET) Received: from BLAPR05CA0001.namprd05.prod.outlook.com (2603:10b6:208:36e::6) by SJ2PR12MB8783.namprd12.prod.outlook.com (2603:10b6:a03:4d0::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8137.28; Tue, 12 Nov 2024 20:53:32 +0000 Received: from MN1PEPF0000ECDA.namprd02.prod.outlook.com (2603:10b6:208:36e:cafe::77) by BLAPR05CA0001.outlook.office365.com (2603:10b6:208:36e::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8158.15 via Frontend Transport; Tue, 12 Nov 2024 20:53:31 +0000 Received: from SATLEXMB03.amd.com (165.204.84.17) by MN1PEPF0000ECDA.mail.protection.outlook.com (10.167.242.134) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8158.14 via Frontend Transport; Tue, 12 Nov 2024 20:53:31 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 12 Nov 2024 14:53:30 -0600 Received: from ubuntu.mshome.net (10.180.168.240) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Tue, 12 Nov 2024 14:53:29 -0600 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 2f938939-a138-11ef-a0c7-8be0dac302b0 X-Custom-Connection: eyJyZW1vdGVpcCI6IjJhMDE6MTExOmY0MDM6MjAwYTo6NjI1IiwiaGVsbyI6Ik5BTTEyLU1XMi1vYmUub3V0Ym91bmQucHJvdGVjdGlvbi5vdXRsb29rLmNvbSJ9 X-Custom-Transaction: eyJpZCI6IjJmOTM4OTM5LWExMzgtMTFlZi1hMGM3LThiZTBkYWMzMDJiMCIsInRzIjoxNzMxNDQ0ODE3LjU0NDkzMywic2VuZGVyIjoic3Rld2FydC5oaWxkZWJyYW5kQGFtZC5jb20iLCJyZWNpcGllbnQiOiJ4ZW4tZGV2ZWxAbGlzdHMueGVucHJvamVjdC5vcmcifQ== ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=iS2jnoy4aJW7ve7HO5rr2EixV40FBpLlH1/uuGV3BgnvP74lHO100pKqCQSjaUukLax2lTF1RT1M+DbPsmoLh7m6aJO4x9GKmJVWqHkyVVdBt/Vyq69s94DCd2sRdULvNpOOX900UriWbGesT7VCcBoCm8dQ+TcODmzaweoM41HuxtxaRDRxaoQ9UagLLXfP28xJnIYbu9iPz25Ll7/4Yc0EYTctD6sIpLu2tNG8ssuyhYs2H5ScxkyuQ9cvCe7I9Or6yekEoDV4WO/pPCl1ZzCPrj0Aj2nL3JDPezinTHpoeS0ptO17I/cGPgWh8X3/g/IjcMsTxNwxlg65tsychg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ky8HxzXjH0qC2Vc4vqFd2aklwcMHHpdbS4ww5q29aEk=; b=Nh1KXwM8WXWSyTgNML782YduUNDSLrngq2NnZy+YdEJAXf9x6xXMxtD/P7SJ3Xe0lHZTE7xsi6YguyPf8g8nhkHCIM0WotH2D3mRQrqFwMHMoU1cI0JiE4AyNAaBCN80IM7Bl/YfSLsHWErDjXa0xs5Xu0Fvc9S+qNlE/hLXp9d7ZQ/dbQJiHbt20kiuIsUsx3R+e91W3U+ARWduLJY22ZFvT5jvXv7GiFMjLC6ulPkdBHKKFs40499DhQqv6Xu1bYyjtKTZI+oHJJyt3KhuLwFO4cz43XYdenYdK9m+ToM/4tE+VTTst7eV2PBjePGogYaz8uskrL+5IrDFoKbXBg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ky8HxzXjH0qC2Vc4vqFd2aklwcMHHpdbS4ww5q29aEk=; b=QOV+7rQ5CPwiln+/q2LJzxoFX9bCo6vSq72G64A43iADQXuHM/FGKDWzxSgfP2QNKOzhCVNC1VNx2AAxixtWpOSUzgnFaqI/uuyKahnDJRlnFXmL0zdaoeGRWuKQbnjEqLBYL0TPLmYuF2lR1uchyhL9i13l0qwEVOeWROj6RSE= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB03.amd.com; pr=C From: Stewart Hildebrand To: CC: Stewart Hildebrand , Jan Beulich , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Andrew Cooper , "Julien Grall" , Stefano Stabellini Subject: [PATCH v7 1/2] xen/pci: introduce PF<->VF links Date: Tue, 12 Nov 2024 15:53:19 -0500 Message-ID: <20241112205321.186622-2-stewart.hildebrand@amd.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241112205321.186622-1-stewart.hildebrand@amd.com> References: <20241112205321.186622-1-stewart.hildebrand@amd.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: None (SATLEXMB03.amd.com: stewart.hildebrand@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000ECDA:EE_|SJ2PR12MB8783:EE_ X-MS-Office365-Filtering-Correlation-Id: 40ff51b7-ef80-4318-5713-08dd035c1118 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|376014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?By4rK9PtHmnwASVBJRBz5fplmo85mXM61Ss8N9qZGbRvGlAtqi0o1hg5e4ix?= =?us-ascii?Q?DlWQXzzpReO4qeMmof33TNW/i+cMygboxO3YDeYNet5l4jA1/3V8I5YzbPO2?= =?us-ascii?Q?xxp/iRuNC9ggpSpE662GT+9W5QBUPChJoafTy35PZo1ARq6DuBMmcaLOGxyr?= =?us-ascii?Q?zhfm7Mh3cJSMIkYjik5znSMzWn0f6G/CwmCwyE1W/4J4jI7pF+XDbhFWKCNw?= =?us-ascii?Q?5tdg0g7L4Q5c/5z9Pl9yUAwPzkYJ33kxDZyVXcB8U2vT7GTD2y52sf5TL90K?= =?us-ascii?Q?iBqEJBqzZHZtFqjeBsCm0earU48jl2wwqcN5y6i7jdJ3ld2V4py74jbSs0Xf?= =?us-ascii?Q?AybdydyfdxZJyZ4ziWStSIwMOUTRHil4NWsPt1xMJ/c013j9yyHJ68eNSorH?= =?us-ascii?Q?HToKwHLSD8O+WcU09X+0/1KdpUHtyxga4QCzfc6M4rNuQU+ZUGhrXQ4SDDCE?= =?us-ascii?Q?xByhhODwlgjr+OXleSGZMypyLJ7lLXzK+VP6ZmB+kSIcGkzzKV+PwG5wTgS+?= =?us-ascii?Q?KEsFlP+/Es8KNT+SBEqS5uAzwEfn9E4kIP2nR0yU1yvCBMBoBDHO23qFv9wm?= =?us-ascii?Q?2E4q1mCGSVa+Nrjdt/cDzhTGjFHgILXk5ST15LAxG0Hqc21alLqpQOQQq5nH?= =?us-ascii?Q?1RSbiEtCtaWi51t/ADg7y91bI9QMAluoU2hL0S46MdiwMRU8+TfYYEHrWkWf?= =?us-ascii?Q?EP96w3WJH4+yO9Rro7iWtVe/IP80nSrI2tiQ75d/n9grjeIM+KBCuySwjz/b?= =?us-ascii?Q?DDf0eUSIQ6bI1XsDREaa9v3On/t77ZBga6B0ztFZGsVNzAybJdX4zrcxcx/K?= =?us-ascii?Q?lbwHoANQqWVLmUtnMeRTIRIpRe983p2kY/lrLpZB4cBblLX9+IXUEJMr2cVN?= =?us-ascii?Q?+1dWHc44gNeXOIVfWRK8tr7KHjq4QglzxMzUUTforbYaLVybS8nWtbYVzoQj?= =?us-ascii?Q?L2czr/Hbx61NdFobrAYmKk0jEjE0MFyfYKguO7u+zXKM/7RYfIWGYiqUyOXh?= =?us-ascii?Q?lcCAav7ISaw7TDctY7PfdAsW/8z4yYj2XsDlVf3ByVSm2JKAe70fqipX/fz8?= =?us-ascii?Q?eMLKYvR1Dp2duiCa4kAYnSWnaDuMK0U3ArD3jE71hlMF8gG/gz4eOH9iLK+T?= =?us-ascii?Q?GcQ00Eb+YkPoPKijTQrHKRg1fYDYgJ8b6uJHQ4Ss5+X54CGEu97wOpBOKm6Z?= =?us-ascii?Q?PIBERTct+gs7KmkAQXku87mpZMTXYM12dNaqq19uKvFA1xSk9kZgJfr43+7L?= =?us-ascii?Q?R+v1g4xDz7A4HmDuuRPWNf+QeWERo5eUkrPcup+Eq4wkXiDT1SHCsuC6pr69?= =?us-ascii?Q?g04H1T0HnD6VPE2XoSl1vgN2NvWlO0szSzv/6UGfFOHRBcG8GPhWT9S/jfI3?= =?us-ascii?Q?qF4ZKFcgvWVrEsawsn9K8XkAYKdLyyJs44u0o9lVfhrV5fwh8g=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB03.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(376014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Nov 2024 20:53:31.4379 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 40ff51b7-ef80-4318-5713-08dd035c1118 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB03.amd.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000ECDA.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8783 X-ZohoMail-DKIM: pass (identity @amd.com) X-ZM-MESSAGEID: 1731444851464116600 Content-Type: text/plain; charset="utf-8" Add links between a VF's struct pci_dev and its associated PF struct pci_dev. The hardware domain is expected to add a PF first before adding associated VFs. Similarly, the hardware domain is expected to remove the associated VFs before removing the PF. If adding/removing happens out of order, print a warning and return an error. This means that VFs can only exist with an associated PF. Additionally, if the hardware domain attempts to remove a PF with VFs still present, mark the PF and VFs broken, because Linux Dom0 has been observed to not respect the error returned. Move the call to pci_get_pdev() down to avoid dropping and re-acquiring the pcidevs_lock(). Drop the call to pci_add_device() as it is invalid to add a VF without an existing PF. Signed-off-by: Stewart Hildebrand Reviewed-by: Jan Beulich --- Candidate for backport to 4.19 (the next patch depends on this one) v6->v7: * cope with multiple invocations of pci_add_device for VFs * get rid of enclosing struct for single member * during PF removal attempt with VFs still present: * keep PF * mark broken * don't unlink * return error * during VF add: * initialize pf_pdev in declaration * remove logic catering to adding VFs without PF v5->v6: * move printk() before ASSERT_UNREACHABLE() * warn about PF removal with VFs still present * clarify commit message v4->v5: * new patch, split from ("x86/msi: fix locking for SR-IOV devices") * move INIT_LIST_HEAD(&pdev->vf_list); earlier * collapse struct list_head instances * retain error code from pci_add_device() * unlink (and mark broken) VFs instead of removing them * const-ify VF->PF link --- xen/drivers/passthrough/pci.c | 74 ++++++++++++++++++++++++++++------- xen/include/xen/pci.h | 8 ++++ 2 files changed, 68 insertions(+), 14 deletions(-) diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 74d3895e1ef6..d4167cea09c0 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -333,6 +333,8 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg,= u8 bus, u8 devfn) *((u8*) &pdev->devfn) =3D devfn; pdev->domain =3D NULL; =20 + INIT_LIST_HEAD(&pdev->vf_list); + arch_pci_init_pdev(pdev); =20 rc =3D pdev_msi_init(pdev); @@ -449,6 +451,10 @@ static void free_pdev(struct pci_seg *pseg, struct pci= _dev *pdev) =20 list_del(&pdev->alldevs_list); pdev_msi_deinit(pdev); + + if ( pdev->info.is_virtfn ) + list_del(&pdev->vf_list); + xfree(pdev); } =20 @@ -656,24 +662,11 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, unsigned int slot =3D PCI_SLOT(devfn), func =3D PCI_FUNC(devfn); const char *type; int ret; - bool pf_is_extfn =3D false; =20 if ( !info ) type =3D "device"; else if ( info->is_virtfn ) - { - pcidevs_lock(); - pdev =3D pci_get_pdev(NULL, - PCI_SBDF(seg, info->physfn.bus, - info->physfn.devfn)); - if ( pdev ) - pf_is_extfn =3D pdev->info.is_extfn; - pcidevs_unlock(); - if ( !pdev ) - pci_add_device(seg, info->physfn.bus, info->physfn.devfn, - NULL, node); type =3D "virtual function"; - } else if ( info->is_extfn ) type =3D "extended function"; else @@ -703,7 +696,38 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, * extended function. */ if ( pdev->info.is_virtfn ) - pdev->info.is_extfn =3D pf_is_extfn; + { + struct pci_dev *pf_pdev =3D pci_get_pdev(NULL, + PCI_SBDF(seg, + info->physfn.bu= s, + info->physfn.de= vfn)); + struct pci_dev *vf_pdev; + bool already_added =3D false; + + if ( !pf_pdev ) + { + printk(XENLOG_WARNING + "Attempted to add SR-IOV device VF %pp without PF %= pp\n", + &pdev->sbdf, + &PCI_SBDF(seg, info->physfn.bus, info->physfn.devfn= )); + free_pdev(pseg, pdev); + ret =3D -ENODEV; + goto out; + } + + pdev->info.is_extfn =3D pf_pdev->info.is_extfn; + pdev->pf_pdev =3D pf_pdev; + list_for_each_entry(vf_pdev, &pf_pdev->vf_list, vf_list) + { + if ( vf_pdev =3D=3D pdev ) + { + already_added =3D true; + break; + } + } + if ( !already_added ) + list_add(&pdev->vf_list, &pf_pdev->vf_list); + } } =20 if ( !pdev->info.is_virtfn && !pdev->vf_rlen[0] ) @@ -821,6 +845,28 @@ int pci_remove_device(u16 seg, u8 bus, u8 devfn) list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list ) if ( pdev->bus =3D=3D bus && pdev->devfn =3D=3D devfn ) { + if ( !pdev->info.is_virtfn && !list_empty(&pdev->vf_list) ) + { + struct pci_dev *vf_pdev; + + /* + * Linux Dom0 has been observed to not respect an error co= de + * returned from PHYSDEVOP_pci_device_remove. Mark VFs and= PF + * broken. + */ + list_for_each_entry(vf_pdev, &pdev->vf_list, vf_list) + vf_pdev->broken =3D true; + + pdev->broken =3D true; + + printk(XENLOG_WARNING + "Attempted to remove PCI SR-IOV PF %pp with VFs sti= ll present\n", + &pdev->sbdf); + + ret =3D -EBUSY; + break; + } + if ( pdev->domain ) { write_lock(&pdev->domain->pci_lock); diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index 1e4fe68c60fb..977c0d08f78a 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -153,7 +153,15 @@ struct pci_dev { unsigned int count; #define PT_FAULT_THRESHOLD 10 } fault; + + /* + * List head if PF. + * List entry if VF. + */ + struct list_head vf_list; u64 vf_rlen[6]; + /* Link from VF to PF. Only populated for VFs. */ + const struct pci_dev *pf_pdev; =20 /* Data for vPCI. */ struct vpci *vpci; --=20 2.47.0 From nobody Mon Nov 3 06:56:10 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass(p=quarantine dis=none) header.from=amd.com ARC-Seal: i=2; a=rsa-sha256; t=1731444850; cv=pass; d=zohomail.com; s=zohoarc; b=a4vJIJQr8BzlFn7nB4r6qXjcpTtgg6E30rvCwl3fma/2+rb7mDZNt/e4E7krpWpZFYOSGv4Hx9PWER7mf4q+hA6ePXC3Y9Q3a9WEgcALpsWzQYWJFguva9E+lbI5cSTBkJFJU+AYAICWrrQzGG8ZTsMrODJKmj9buOmW7fr5m6w= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1731444850; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=fvfdk7zwQONqaq/3Edn44Ev2SS3iqmtHkUeBX4CZLoU=; b=Gd3/g57QNUttVLCAZL9X8G1hKNTrkYclKRJbqAGEe+jzCSGEr6KKIYPQVZsrYLNHmfwyrAv4n8YpXX/fBObmJwdoiPwMh67BHQBjo1RNcyVQSqyPvsrRBIi8b61jefa7Vfr5zoXQtCHZt8K1RjQ26SoIUai/Tedh+c7PUeEIehQ= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1731444850275911.9341909080467; Tue, 12 Nov 2024 12:54:10 -0800 (PST) Received: from list by lists.xenproject.org with outflank-mailman.834913.1250727 (Exim 4.92) (envelope-from ) id 1tAxtW-00076O-9A; Tue, 12 Nov 2024 20:53:50 +0000 Received: by outflank-mailman (output) from mailman id 834913.1250727; Tue, 12 Nov 2024 20:53:50 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tAxtW-00076F-4U; Tue, 12 Nov 2024 20:53:50 +0000 Received: by outflank-mailman (input) for mailman id 834913; Tue, 12 Nov 2024 20:53:48 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tAxtU-0006YG-AQ for xen-devel@lists.xenproject.org; Tue, 12 Nov 2024 20:53:48 +0000 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on20619.outbound.protection.outlook.com [2a01:111:f403:2415::619]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 347bcc0d-a138-11ef-a0c7-8be0dac302b0; Tue, 12 Nov 2024 21:53:45 +0100 (CET) Received: from BL1PR13CA0019.namprd13.prod.outlook.com (2603:10b6:208:256::24) by DS7PR12MB6191.namprd12.prod.outlook.com (2603:10b6:8:98::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8158.17; Tue, 12 Nov 2024 20:53:40 +0000 Received: from BN1PEPF00005FFE.namprd05.prod.outlook.com (2603:10b6:208:256:cafe::7f) by BL1PR13CA0019.outlook.office365.com (2603:10b6:208:256::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8158.16 via Frontend Transport; Tue, 12 Nov 2024 20:53:39 +0000 Received: from SATLEXMB04.amd.com (165.204.84.17) by BN1PEPF00005FFE.mail.protection.outlook.com (10.167.243.230) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8158.14 via Frontend Transport; Tue, 12 Nov 2024 20:53:39 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 12 Nov 2024 14:53:37 -0600 Received: from ubuntu.mshome.net (10.180.168.240) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Tue, 12 Nov 2024 14:53:36 -0600 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 347bcc0d-a138-11ef-a0c7-8be0dac302b0 X-Custom-Connection: eyJyZW1vdGVpcCI6IjJhMDE6MTExOmY0MDM6MjQxNTo6NjE5IiwiaGVsbyI6Ik5BTTExLURNNi1vYmUub3V0Ym91bmQucHJvdGVjdGlvbi5vdXRsb29rLmNvbSJ9 X-Custom-Transaction: eyJpZCI6IjM0N2JjYzBkLWExMzgtMTFlZi1hMGM3LThiZTBkYWMzMDJiMCIsInRzIjoxNzMxNDQ0ODI1LjU0MTI2OCwic2VuZGVyIjoic3Rld2FydC5oaWxkZWJyYW5kQGFtZC5jb20iLCJyZWNpcGllbnQiOiJ4ZW4tZGV2ZWxAbGlzdHMueGVucHJvamVjdC5vcmcifQ== ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=FaQ/MgtYQNjpSwmecoOVcUT+Irr3S3GThy5V10vqXsdDKDqPlIDd8zG9sKWVeyNRnz8D4Pdko+iIfImvt5uYz++0a04+QW+Q/tE2+litX6fd2lO+/pFyOMjh1qjNyhVbYPwaH2elhsoEHayYCPAER8ySAIzafZA3wCOR9RlpYdAh0bE3T6IZ2RxfYqBsRL8QEDAMZfB4aCrEFxkNp/i/JqnDvoeY4ifVBlcMUN3RnzeyvU56+Rt1+qKS3xniMdy9qB1mVyWRDaueHARpRvaE9p9VEgbQwqV22RmCdll4p7o6R87q2cKBxr1F3/BhjzKVuZXUWb6Y9L5EpgZOEzfCvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fvfdk7zwQONqaq/3Edn44Ev2SS3iqmtHkUeBX4CZLoU=; b=jA95A5ClNtbAYlYXDGGiD2eB8JgDoRPl1/Akzgrw/WisPXXtfon2tQjJZZ2Z3hbhcARfwlnVgBrCRqXrTjTKUczQj2stcQ0atWoYyBOwdWehMmBIPr7V7rFGbB5Q4LH+n2K1DcI0za6MTQGMTZ0YVTMEjehkQD696Y4rTeVlTxIo2nj4l71tuNIwP4j5sM4LBlxg08L03nZDMYOT0/dLq9ELL1tijY+7HMlLsiM+XtyhjGA0L35+AXCyJeG5Di5xE3bZTJ4/3A8a3Zg+IYtt77+BPSt/Te6U3po/wisQEILdHN5ZVbnGBKhHi9q5/VBaHyfGBOxkZY8xMzJJklkpgQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fvfdk7zwQONqaq/3Edn44Ev2SS3iqmtHkUeBX4CZLoU=; b=0H72bbDtiEOuyWNx7PkGVuf46lNjhAeOXnKYoVFtRz27R0VIIHrLr/B4p0hzfCIiVRZ7t/cnMg422gKVLCE+Bv9TztTWIPag9FNQN9JHEW88Nr1Mzq7drEWJyTnVnrF4db7jJu+gr+tehBaCxRvS/XwdxplJnQJwlyVUqPiSAsY= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C From: Stewart Hildebrand To: CC: Stewart Hildebrand , Jan Beulich , Andrew Cooper , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Julien Grall , Stefano Stabellini , Teddy Astie Subject: [PATCH v7 2/2] x86/msi: fix locking for SR-IOV devices Date: Tue, 12 Nov 2024 15:53:20 -0500 Message-ID: <20241112205321.186622-3-stewart.hildebrand@amd.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241112205321.186622-1-stewart.hildebrand@amd.com> References: <20241112205321.186622-1-stewart.hildebrand@amd.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: None (SATLEXMB04.amd.com: stewart.hildebrand@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN1PEPF00005FFE:EE_|DS7PR12MB6191:EE_ X-MS-Office365-Filtering-Correlation-Id: e3ed892f-1733-41b5-0301-08dd035c15d8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?oe9XZV7Yp7pHVumzi7PUVFspyoHwi23OLO7xe10KXzKUvGLgj4sj1Toegeaq?= =?us-ascii?Q?jyo+22GkMqK+ZmaQRpejrlbX1enAu/RlSxQhW1ScUqk1YZ1uMIr8Ra1Wu6MI?= =?us-ascii?Q?SUZ+jkYXCHK2d01O2mOHvLY3ZNrFEfwDL9fEAy0zDiWnIYqSyOE6JRiLo556?= =?us-ascii?Q?D0E9R7yc0ex7/BKHmXbUgqLRTkqNI66lNkBPyD3B8dugHb6w1kN2Bc9HXU8Q?= =?us-ascii?Q?B/0DsIPzb0sOXJb421BlujFXpkkLDSYM0rR8CxRjKxSbs8Kh85KAQmeczadC?= =?us-ascii?Q?FMoFVWXDD1ArP7rxCc9FmuurOAGe8nfPa5+MgcqPgKaiziPnKbOuxJuV2JXo?= =?us-ascii?Q?vGZWXhexqSW6+Kgl9DoqIRLU0ckWzjNJRcXwc9qGdv0/cZOsZPrNncQW3XML?= =?us-ascii?Q?wy7XA9/b2+N78Wf555HwLzebcCXqKeD5G5OWqh7BEAGFAj/desd3p5tdIDNf?= =?us-ascii?Q?dW0WnGI7deW9TAkhAnwZ81fVfnY9IejLP0qYMbhRURTzJgKzuO1u4ejd/JVi?= =?us-ascii?Q?H5yHYgJiccsqiIw+Ykgi4uK6q4iK9x9Wx0X92C0P5+StlBBwAPf4ynLc7V9v?= =?us-ascii?Q?/UW2FH5bXwjvT72ms5Te/AD74G1YyxPCHVEyvDkUDEdXSCoEJK4goJrSFQ5h?= =?us-ascii?Q?bRPo1wtBJNbfKGPEJl9L2OY8yMmhLjB93B9EOH62EL7itDJKmQQ1JkRMZNBK?= =?us-ascii?Q?aFj2QoUOd6oWv2IwAB/GTPOXUyK2kBMNOX36BxE043MYPFKt72IEEkXE70Gj?= =?us-ascii?Q?jUoZclLs+9LZJHkVswvBmCaUiPtAWqUMBm6/XcXQ3VP9rJmQn1LhcXbET0aJ?= =?us-ascii?Q?7wxTtX7LvAXLYEB8sqxaA6quZPYmWOlmWcuk7PpLF/4eFn+1UJHCIJtjpufF?= =?us-ascii?Q?ab3vC4EDvLvb3XiFjWkEz56NIIhuEFqUbXxSXytdMYVQpZTzO31b4h0dph95?= =?us-ascii?Q?f9Z+qvGtRJ0W9Wjjt1pjXGsq6mQteuOWp7VK8e5ojA6thkWHAl2hwpelQMN9?= =?us-ascii?Q?zo0C4MgDkW8lwdFUo8HdhYvg/oI/UQ+mIIL/lXjJJtQW3wOBM/qb1t4EHd76?= =?us-ascii?Q?9jFTtMlGlyGDXxQSN1lmQEkyZboqqchC3IZcNbIgD6okSEUqRmg6nNOWq4ZU?= =?us-ascii?Q?DPZKdQZKafY8+JSs8EQmkCqm9APsvMePl2PFIhoRCBqJ7Qj3zyg+YWgkBj39?= =?us-ascii?Q?F6VC92PwibzK8yNnv5bTc1QE5Au11fHs/ztJMvMIA+RwVHL2GkJhAttb+6Oa?= =?us-ascii?Q?CrW/BugXGCXCmK6eLWh8XTjCMhsxGdSUQEitYkff5axopnKh6FFCE7Itsm3q?= =?us-ascii?Q?18J6W2u4YKOuh+hXZ84p/fkvjl3GKt4svGnnmIRtMdHJnvuj477H2FlXJudZ?= =?us-ascii?Q?DUVPpsv4ts/cUH+LHCb6muqZraYoadqYHc1xbcPrUbCDW9Rd1w=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(376014)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Nov 2024 20:53:39.4206 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e3ed892f-1733-41b5-0301-08dd035c15d8 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN1PEPF00005FFE.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB6191 X-ZohoMail-DKIM: pass (identity @amd.com) X-ZM-MESSAGEID: 1731444851545116600 Content-Type: text/plain; charset="utf-8" In commit 4f78438b45e2 ("vpci: use per-domain PCI lock to protect vpci structure") a lock was moved from allocate_and_map_msi_pirq() to the caller and changed from pcidevs_lock() to read_lock(&d->pci_lock). However, one call path wasn't updated to reflect the change, leading to a failed assertion observed under the following conditions: * PV dom0 * Debug build (CONFIG_DEBUG=3Dy) of Xen * There is an SR-IOV device in the system with one or more VFs enabled * Dom0 has loaded the driver for the VF and enabled MSI-X (XEN) Assertion 'd || pcidevs_locked()' failed at drivers/passthrough/pci.c= :535 (XEN) ----[ Xen-4.20-unstable x86_64 debug=3Dy Not tainted ]---- ... (XEN) Xen call trace: (XEN) [] R pci_get_pdev+0x4c/0xab (XEN) [] F arch/x86/msi.c#read_pci_mem_bar+0x58/0x272 (XEN) [] F arch/x86/msi.c#msix_capability_init+0x198/0= x755 (XEN) [] F arch/x86/msi.c#__pci_enable_msix+0x82/0xe8 (XEN) [] F pci_enable_msi+0x3f/0x78 (XEN) [] F map_domain_pirq+0x2a4/0x6dc (XEN) [] F allocate_and_map_msi_pirq+0x103/0x262 (XEN) [] F physdev_map_pirq+0x210/0x259 (XEN) [] F do_physdev_op+0x9c3/0x1454 (XEN) [] F pv_hypercall+0x5ac/0x6af (XEN) [] F lstar_enter+0x143/0x150 In read_pci_mem_bar(), the VF obtains the struct pci_dev pointer for its associated PF to access the vf_rlen array. This array is initialized in pci_add_device() and is only populated in the associated PF's struct pci_dev. Access the vf_rlen array via the link to the PF, and remove the troublesome call to pci_get_pdev(). Fixes: 4f78438b45e2 ("vpci: use per-domain PCI lock to protect vpci structu= re") Reported-by: Teddy Astie Signed-off-by: Stewart Hildebrand Reviewed-by: Jan Beulich --- Candidate for backport to 4.19 Patch #1 ("xen/pci: introduce PF<->VF links") is pre-requisite v6->v7: * turn NULL-check into ASSERT * adjust commentary in struct pf_info v5->v6: * add Jan's R-b v4->v5: * split the PF<->VF links to a pre-requisite patch * pass pci_sbdf_t to read_pci_mem_bar() * use stdint.h types on changed lines * re-add NULL check for pf_info in read_pci_mem_bar(), as pf_info could be NULL v3->v4: * handle case when PF is removed with VFs enabled, then re-added with VFs disabled v2->v3: * link from VF to PF's struct pci_dev * v1->v2: * remove call to pci_get_pdev() --- xen/arch/x86/msi.c | 36 ++++++++++++++++++++--------------- xen/drivers/passthrough/pci.c | 8 +++++--- xen/include/xen/pci.h | 11 ++++++++--- 3 files changed, 34 insertions(+), 21 deletions(-) diff --git a/xen/arch/x86/msi.c b/xen/arch/x86/msi.c index fa724911928e..e2360579deda 100644 --- a/xen/arch/x86/msi.c +++ b/xen/arch/x86/msi.c @@ -663,34 +663,34 @@ static int msi_capability_init(struct pci_dev *dev, return 0; } =20 -static u64 read_pci_mem_bar(u16 seg, u8 bus, u8 slot, u8 func, u8 bir, int= vf) +static uint64_t read_pci_mem_bar(pci_sbdf_t sbdf, uint8_t bir, int vf, + const struct pf_info *pf_info) { + uint16_t seg =3D sbdf.seg; + uint8_t bus =3D sbdf.bus, slot =3D sbdf.dev, func =3D sbdf.fn; u8 limit; u32 addr, base =3D PCI_BASE_ADDRESS_0; u64 disp =3D 0; =20 if ( vf >=3D 0 ) { - struct pci_dev *pdev =3D pci_get_pdev(NULL, - PCI_SBDF(seg, bus, slot, func)= ); unsigned int pos; uint16_t ctrl, num_vf, offset, stride; =20 - if ( !pdev ) - return 0; + ASSERT(pf_info); =20 - pos =3D pci_find_ext_capability(pdev->sbdf, PCI_EXT_CAP_ID_SRIOV); - ctrl =3D pci_conf_read16(pdev->sbdf, pos + PCI_SRIOV_CTRL); - num_vf =3D pci_conf_read16(pdev->sbdf, pos + PCI_SRIOV_NUM_VF); - offset =3D pci_conf_read16(pdev->sbdf, pos + PCI_SRIOV_VF_OFFSET); - stride =3D pci_conf_read16(pdev->sbdf, pos + PCI_SRIOV_VF_STRIDE); + pos =3D pci_find_ext_capability(sbdf, PCI_EXT_CAP_ID_SRIOV); + ctrl =3D pci_conf_read16(sbdf, pos + PCI_SRIOV_CTRL); + num_vf =3D pci_conf_read16(sbdf, pos + PCI_SRIOV_NUM_VF); + offset =3D pci_conf_read16(sbdf, pos + PCI_SRIOV_VF_OFFSET); + stride =3D pci_conf_read16(sbdf, pos + PCI_SRIOV_VF_STRIDE); =20 if ( !pos || !(ctrl & PCI_SRIOV_CTRL_VFE) || !(ctrl & PCI_SRIOV_CTRL_MSE) || !num_vf || !offset || (num_vf > 1 && !stride) || bir >=3D PCI_SRIOV_NUM_BARS || - !pdev->vf_rlen[bir] ) + !pf_info->vf_rlen[bir] ) return 0; base =3D pos + PCI_SRIOV_BAR; vf -=3D PCI_BDF(bus, slot, func) + offset; @@ -704,8 +704,8 @@ static u64 read_pci_mem_bar(u16 seg, u8 bus, u8 slot, u= 8 func, u8 bir, int vf) } if ( vf >=3D num_vf ) return 0; - BUILD_BUG_ON(ARRAY_SIZE(pdev->vf_rlen) !=3D PCI_SRIOV_NUM_BARS); - disp =3D vf * pdev->vf_rlen[bir]; + BUILD_BUG_ON(ARRAY_SIZE(pf_info->vf_rlen) !=3D PCI_SRIOV_NUM_BARS); + disp =3D vf * pf_info->vf_rlen[bir]; limit =3D PCI_SRIOV_NUM_BARS; } else switch ( pci_conf_read8(PCI_SBDF(seg, bus, slot, func), @@ -814,6 +814,7 @@ static int msix_capability_init(struct pci_dev *dev, int vf; paddr_t pba_paddr; unsigned int pba_offset; + const struct pf_info *pf_info; =20 if ( !dev->info.is_virtfn ) { @@ -821,6 +822,7 @@ static int msix_capability_init(struct pci_dev *dev, pslot =3D slot; pfunc =3D func; vf =3D -1; + pf_info =3D NULL; } else { @@ -828,9 +830,12 @@ static int msix_capability_init(struct pci_dev *dev, pslot =3D PCI_SLOT(dev->info.physfn.devfn); pfunc =3D PCI_FUNC(dev->info.physfn.devfn); vf =3D dev->sbdf.bdf; + ASSERT(dev->pf_pdev); + pf_info =3D &dev->pf_pdev->physfn; } =20 - table_paddr =3D read_pci_mem_bar(seg, pbus, pslot, pfunc, bir, vf); + table_paddr =3D read_pci_mem_bar(PCI_SBDF(seg, pbus, pslot, pfunc)= , bir, + vf, pf_info); WARN_ON(msi && msi->table_base !=3D table_paddr); if ( !table_paddr ) { @@ -853,7 +858,8 @@ static int msix_capability_init(struct pci_dev *dev, =20 pba_offset =3D pci_conf_read32(dev->sbdf, msix_pba_offset_reg(pos)= ); bir =3D (u8)(pba_offset & PCI_MSIX_BIRMASK); - pba_paddr =3D read_pci_mem_bar(seg, pbus, pslot, pfunc, bir, vf); + pba_paddr =3D read_pci_mem_bar(PCI_SBDF(seg, pbus, pslot, pfunc), = bir, vf, + pf_info); WARN_ON(!pba_paddr); pba_paddr +=3D pba_offset & ~PCI_MSIX_BIRMASK; =20 diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index d4167cea09c0..2cd595f75ebf 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -730,7 +730,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, } } =20 - if ( !pdev->info.is_virtfn && !pdev->vf_rlen[0] ) + if ( !pdev->info.is_virtfn && !pdev->physfn.vf_rlen[0] ) { unsigned int pos =3D pci_find_ext_capability(pdev->sbdf, PCI_EXT_CAP_ID_SRIOV); @@ -742,7 +742,9 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, { unsigned int i; =20 - BUILD_BUG_ON(ARRAY_SIZE(pdev->vf_rlen) !=3D PCI_SRIOV_NUM_BARS= ); + BUILD_BUG_ON(ARRAY_SIZE(pdev->physfn.vf_rlen) !=3D + PCI_SRIOV_NUM_BARS); + for ( i =3D 0; i < PCI_SRIOV_NUM_BARS; ) { unsigned int idx =3D pos + PCI_SRIOV_BAR + i * 4; @@ -757,7 +759,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, continue; } ret =3D pci_size_mem_bar(pdev->sbdf, idx, NULL, - &pdev->vf_rlen[i], + &pdev->physfn.vf_rlen[i], PCI_BAR_VF | ((i =3D=3D PCI_SRIOV_NUM_BARS - 1) ? PCI_BAR_LAST : 0)); diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index 977c0d08f78a..f784e9116059 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -159,9 +159,14 @@ struct pci_dev { * List entry if VF. */ struct list_head vf_list; - u64 vf_rlen[6]; - /* Link from VF to PF. Only populated for VFs. */ - const struct pci_dev *pf_pdev; + union { + struct pf_info { + /* Only populated for PFs. */ + uint64_t vf_rlen[PCI_SRIOV_NUM_BARS]; + } physfn; + /* Link from VF to PF. Only populated for VFs. */ + const struct pci_dev *pf_pdev; + }; =20 /* Data for vPCI. */ struct vpci *vpci; --=20 2.47.0