From nobody Sat Nov 23 09:34:58 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass(p=quarantine dis=none) header.from=amd.com ARC-Seal: i=2; a=rsa-sha256; t=1731686982; cv=pass; d=zohomail.com; s=zohoarc; b=UPWwwofQSu6KjqZxWSxZEZZdZ0eiPY52tm5cxD8d5Q02MnajsRP5nSskgHv8B/NMm6ORHHQrvNygC3+BZHRlbP5eQDL5wigZoz6cnn5pmgdnQ2G/6UD2t8Ame1qjPqRPGHG3Q/sZL3hQaHyBBVSr3tM+nbKFk9MZ352iuzfyvK4= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1731686982; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=6dgSRDSguain7FanHlOnJmwBRlTv1DEkQEKN/zq4jkg=; b=js4u0Kdls9moQbNKawDh7FRf8W2wsZwtchdp3gOZVpDOpoZSkxqqTFaVlPGDNmHL9WeoFV4L6Lkedk+zIjfHVK2MG6Tig8cm3sv5kfdcQt7Y+vJyPPxG7AB+9ZCtLRe0lLXa9bbTUWkaRW3QFSQ0zCgPvK09Snns3Y+sr/nSFuk= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1731686982305267.69232792107186; Fri, 15 Nov 2024 08:09:42 -0800 (PST) Received: from list by lists.xenproject.org with outflank-mailman.837778.1253679 (Exim 4.92) (envelope-from ) id 1tBysy-0002AH-Ak; Fri, 15 Nov 2024 16:09:28 +0000 Received: by outflank-mailman (output) from mailman id 837778.1253679; Fri, 15 Nov 2024 16:09:28 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tBysy-0002AA-6i; Fri, 15 Nov 2024 16:09:28 +0000 Received: by outflank-mailman (input) for mailman id 837778; Fri, 15 Nov 2024 16:09:26 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tBysw-0001uH-N2 for xen-devel@lists.xenproject.org; Fri, 15 Nov 2024 16:09:26 +0000 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2060a.outbound.protection.outlook.com [2a01:111:f403:2009::60a]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id f9d25211-a36b-11ef-99a3-01e77a169b0f; Fri, 15 Nov 2024 17:09:23 +0100 (CET) Received: from PH7PR02CA0007.namprd02.prod.outlook.com (2603:10b6:510:33d::13) by SJ0PR12MB8113.namprd12.prod.outlook.com (2603:10b6:a03:4e0::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8158.17; Fri, 15 Nov 2024 16:09:15 +0000 Received: from SN1PEPF00036F40.namprd05.prod.outlook.com (2603:10b6:510:33d:cafe::6c) by PH7PR02CA0007.outlook.office365.com (2603:10b6:510:33d::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8158.18 via Frontend Transport; Fri, 15 Nov 2024 16:09:15 +0000 Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF00036F40.mail.protection.outlook.com (10.167.248.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8158.14 via Frontend Transport; Fri, 15 Nov 2024 16:09:15 +0000 Received: from SATLEXMB04.amd.com (10.181.40.145) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 15 Nov 2024 10:09:14 -0600 Received: from ubuntu.mshome.net (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Fri, 15 Nov 2024 10:09:13 -0600 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f9d25211-a36b-11ef-99a3-01e77a169b0f X-Custom-Connection: eyJyZW1vdGVpcCI6IjJhMDE6MTExOmY0MDM6MjAwOTo6NjBhIiwiaGVsbyI6Ik5BTTEwLUJONy1vYmUub3V0Ym91bmQucHJvdGVjdGlvbi5vdXRsb29rLmNvbSJ9 X-Custom-Transaction: eyJpZCI6ImY5ZDI1MjExLWEzNmItMTFlZi05OWEzLTAxZTc3YTE2OWIwZiIsInRzIjoxNzMxNjg2OTYzLjEzNjQ4OCwic2VuZGVyIjoic3Rld2FydC5oaWxkZWJyYW5kQGFtZC5jb20iLCJyZWNpcGllbnQiOiJ4ZW4tZGV2ZWxAbGlzdHMueGVucHJvamVjdC5vcmcifQ== ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=uBCNrG6U1b+v3/QO+kl5cqvJx/qJP4V2pyB2DJljkXrnnfx0tRHKmDYNxATU2tOnZadwvRgGAOpApOeQr89sLTMjhWQWcjbLGAtpDd1xt72aStMROmev1HTMtxeIVvmza3uyvA3Etwy5t9fDDyfWeXJJWkB73Wzqmw/oQmsgjtTyAVQuO5xDNk+9cy5r5wX3AGLdjyWscyJ3W5aUYJexzP6NMZz1cvjEdH5Js6A99tBlgLKZjIAVLa0p5VIUJ2UN9koFsXgWnVaHMa4/CANKzTaYuI1jVzwLkMLPuqAImZN3c9nQ+wpbM1qjeW1v5DF3Sy4DIcB1xxc8LFwpV/TQPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6dgSRDSguain7FanHlOnJmwBRlTv1DEkQEKN/zq4jkg=; b=sbU5XtrJdb4LENwGLuuWVhPAhYT/BeLZz+wawz9FmzwOeaH4tigOR9desar0B9badCewz75FaqQqr2B7eeac3htgnJI0/NL9BYJyoIn3rorfGXCX+9i0VA5APu4mNiJ+yBsnyoR2d7/JiqLsvf4gLFcaD8hHN0XR/Qy/hoc4BqnIC3R6MZqtwhCW/8RZKTWUZ6SJA6LuZEJJwwnA2UK0R9btml1PmImZrF6S2INjVonPhxgAtp3tju6XtUOcAQi3iuXO1xrcojxAYFB9o3S8qV8SALcYVqMWG8VE3+ANvrxyj7D47LbYbJm19if+Dvc1Xkru5jQJc4eO4bFwyLdrMg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6dgSRDSguain7FanHlOnJmwBRlTv1DEkQEKN/zq4jkg=; b=o1MXuTh+W/0lZ8yfBv9+KlXGjdAPH9ARcaLsBvQKmFYrY3bLBvVgUpK+EO9W93av5rPA1xkgJgczMaZcutVhuk53EOXOLYlLAZdezOy7ugEzFE2IAEzL9xrL/dV0i7OdXl6VW4jUNClHywZyQMDfKbA1im8/W3dkJvLnGu7apnU= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C From: Stewart Hildebrand To: CC: Stewart Hildebrand , Jan Beulich , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Andrew Cooper , "Julien Grall" , Stefano Stabellini Subject: [PATCH v8 1/3] xen/pci: introduce PF<->VF links Date: Fri, 15 Nov 2024 11:09:02 -0500 Message-ID: <20241115160905.551224-2-stewart.hildebrand@amd.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241115160905.551224-1-stewart.hildebrand@amd.com> References: <20241115160905.551224-1-stewart.hildebrand@amd.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: None (SATLEXMB04.amd.com: stewart.hildebrand@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF00036F40:EE_|SJ0PR12MB8113:EE_ X-MS-Office365-Filtering-Correlation-Id: a654b0a1-3dcd-48f0-bfc2-08dd058fd9fc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|82310400026|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?bM5+wxRFzWXXJGfox39smoWvak3z9q/ckpo4SZf9EDdzYdB6XtI5tMfEh7oJ?= =?us-ascii?Q?AuVkFULuPLGKZOMOiOvtvMOYki66i5GhA09bXnpCqTQZvViiVXi1AjeYZugR?= =?us-ascii?Q?KolFzcC2i8vh7BQ3T+lOBsBL6JnVWEoy3OfgM2gPd0VkM+zVYPRXaZldemn3?= =?us-ascii?Q?4QJT9J2w9Z9CUcIlXvuRcXGifRSHc8iJmf8Vs6UEVh8CnE8ANsvx0mfesesA?= =?us-ascii?Q?kWeeRxj0jurN7hRVefUWw3UZKPj7fg/wTslYB9yNWeUCwuljk8hkbMj6cXAf?= =?us-ascii?Q?d+/yrL4VWD1E1Gm6avzc8L17sujYdDkZzKFJhuOIIuJYRTrlBBAPI9hWxPvZ?= =?us-ascii?Q?O7WlBkl+Q4EkXpQH+1p6o1AUw9SRvwLirlpYUvmBUxO6FcW9yqUUQiMKVynx?= =?us-ascii?Q?EgIMlWtM3n6uaXglCLRf1pLQk3Kdj0eegMW/2zSvihYYHzCwikJtu0giHtCV?= =?us-ascii?Q?3xzo40UpsDoAVUB6zPANAeRRgYrs14qeElkUNRJ9U8VRuvMFN8dpvb+Ps20e?= =?us-ascii?Q?vOTKMEaiF5qThFBpXn3EPkqPA3+sDvwF5AH6a8a9aqXoW9eUFA2pC5KCvXN9?= =?us-ascii?Q?GqQOsuibCogrtVxfd/kJNJu/d9PYoFjoeqihPj9fhM+WGVgM0gsRxa4ODl5U?= =?us-ascii?Q?1Ya7x7exhNOU+0hgnVdvXXmizUIZ2KwGDzPDUWFGtQpzCGt3VVPh1LDKehYA?= =?us-ascii?Q?fw5SQmD2+1cPnFkIv3djOPUyi/8G4Vy+VhQ5ijljEe07OfGL/5Eh2wbKryNz?= =?us-ascii?Q?jQ+gq5pcJT0vZKEyhRuuqEMaToQ2ycOVRV8u3gHi+8RphqRoY0Mj1Xuv9j2T?= =?us-ascii?Q?Tbjv+Ora1pCPfjYqIGURsu+54xIy19fhrog3LxkL50kk3gUB6/cZ5JmuYI8z?= =?us-ascii?Q?HJz0HwvhdtzgNpUUJwGftUn+Sy7SNTuWPPoJHJx1VVnZcja3ya1K0/iCCxGm?= =?us-ascii?Q?iyA5ejWbqAtuule50A8reKc58Z54YoUYJBcF0sEXVy2KpCpRYI5YM6cGbIHH?= =?us-ascii?Q?SYHIl+uIXFa0jX8PArJaDaiQN/EgwsbSyJT1Ipe9k3YzIGkgz/GDLD/1NWTv?= =?us-ascii?Q?WFGOUsoWveavMgKPqhBzCcCZ2WsSqyI6nbXZzxbd+sxi02YKd29JEwqXpVh0?= =?us-ascii?Q?EcWB0JJ1qyAG0VLP/+BXlSbv92OkBK1ri0B5cR8YeIejNSensZgz4RnwaClF?= =?us-ascii?Q?mNxFWi29trEXv3y2NWoTedj45kMMFgOjA4w4hJxwPsdOO96gxTavx2kIbki0?= =?us-ascii?Q?bBdGatU+YkbBRVBBuF0NIFt4S5HIJnqlCAWB5hQ8Gz417f4+3mG4ciYRJKHv?= =?us-ascii?Q?22ZypdoFwML2J/f4e4fNJlY4bxZnMdkW+qL7TvyRXTx1RpLwzcznnpTfHoC7?= =?us-ascii?Q?8Nn5HfKLSl9qjLex4fC+CL8sWIraHB9BdB0SngT2aPXR5QvkKA=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(82310400026)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Nov 2024 16:09:15.1223 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a654b0a1-3dcd-48f0-bfc2-08dd058fd9fc X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF00036F40.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB8113 X-ZohoMail-DKIM: pass (identity @amd.com) X-ZM-MESSAGEID: 1731686984405116600 Content-Type: text/plain; charset="utf-8" Add links between a VF's struct pci_dev and its associated PF struct pci_dev. The hardware domain is expected to remove the associated VFs before removing the PF. If removal happens out of order, print a warning and return an error. This means that VFs can only exist with an associated PF. Additionally, if the hardware domain attempts to remove a PF with VFs still present, mark the PF and VFs broken, because Linux Dom0 has been observed to not respect the error returned. Move the calls to pci_get_pdev() and pci_add_device() down to avoid dropping and re-acquiring the pcidevs_lock(). Check !pdev->pf_pdev before adding the VF to the list to guard against adding it multiple times. Signed-off-by: Stewart Hildebrand --- Candidate for backport to 4.19 (the next patch depends on this one) v7->v8: * re-add logic catering to adding VFs without PF for backporting (split removal to separate patch) * move & reword comment about is_extfn * check pdev->pf_pdev to guard against multiple list addition * re-indent pf_pdev initialization v6->v7: * cope with multiple invocations of pci_add_device for VFs * get rid of enclosing struct for single member * during PF removal attempt with VFs still present: * keep PF * mark broken * don't unlink * return error * during VF add: * initialize pf_pdev in declaration * remove logic catering to adding VFs without PF v5->v6: * move printk() before ASSERT_UNREACHABLE() * warn about PF removal with VFs still present * clarify commit message v4->v5: * new patch, split from ("x86/msi: fix locking for SR-IOV devices") * move INIT_LIST_HEAD(&pdev->vf_list); earlier * collapse struct list_head instances * retain error code from pci_add_device() * unlink (and mark broken) VFs instead of removing them * const-ify VF->PF link --- xen/drivers/passthrough/pci.c | 87 +++++++++++++++++++++++++++-------- xen/include/xen/pci.h | 8 ++++ 2 files changed, 77 insertions(+), 18 deletions(-) diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 74d3895e1ef6..3fe2e7884f45 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -333,6 +333,8 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg,= u8 bus, u8 devfn) *((u8*) &pdev->devfn) =3D devfn; pdev->domain =3D NULL; =20 + INIT_LIST_HEAD(&pdev->vf_list); + arch_pci_init_pdev(pdev); =20 rc =3D pdev_msi_init(pdev); @@ -449,6 +451,10 @@ static void free_pdev(struct pci_seg *pseg, struct pci= _dev *pdev) =20 list_del(&pdev->alldevs_list); pdev_msi_deinit(pdev); + + if ( pdev->info.is_virtfn ) + list_del(&pdev->vf_list); + xfree(pdev); } =20 @@ -656,24 +662,11 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, unsigned int slot =3D PCI_SLOT(devfn), func =3D PCI_FUNC(devfn); const char *type; int ret; - bool pf_is_extfn =3D false; =20 if ( !info ) type =3D "device"; else if ( info->is_virtfn ) - { - pcidevs_lock(); - pdev =3D pci_get_pdev(NULL, - PCI_SBDF(seg, info->physfn.bus, - info->physfn.devfn)); - if ( pdev ) - pf_is_extfn =3D pdev->info.is_extfn; - pcidevs_unlock(); - if ( !pdev ) - pci_add_device(seg, info->physfn.bus, info->physfn.devfn, - NULL, node); type =3D "virtual function"; - } else if ( info->is_extfn ) type =3D "extended function"; else @@ -698,12 +691,48 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, if ( info ) { pdev->info =3D *info; - /* - * VF's 'is_extfn' field is used to indicate whether its PF is an - * extended function. - */ if ( pdev->info.is_virtfn ) - pdev->info.is_extfn =3D pf_is_extfn; + { + struct pci_dev *pf_pdev =3D + pci_get_pdev(NULL, PCI_SBDF(seg, info->physfn.bus, + info->physfn.devfn)); + + if ( !pf_pdev ) + { + ret =3D pci_add_device(seg, info->physfn.bus, info->physfn= .devfn, + NULL, node); + if ( ret ) + { + printk(XENLOG_WARNING + "Failed to add SR-IOV device PF %pp for VF %pp\= n", + &PCI_SBDF(seg, info->physfn.bus, info->physfn.d= evfn), + &pdev->sbdf); + free_pdev(pseg, pdev); + goto out; + } + pf_pdev =3D pci_get_pdev(NULL, PCI_SBDF(seg, info->physfn.= bus, + info->physfn.devfn)); + if ( !pf_pdev ) + { + printk(XENLOG_ERR + "Inconsistent PCI state: failed to find newly a= dded PF %pp for VF %pp\n", + &PCI_SBDF(seg, info->physfn.bus, info->physfn.d= evfn), + &pdev->sbdf); + ASSERT_UNREACHABLE(); + free_pdev(pseg, pdev); + ret =3D -EILSEQ; + goto out; + } + } + + if ( !pdev->pf_pdev ) + { + /* VF inherits its 'is_extfn' from PF */ + pdev->info.is_extfn =3D pf_pdev->info.is_extfn; + list_add(&pdev->vf_list, &pf_pdev->vf_list); + pdev->pf_pdev =3D pf_pdev; + } + } } =20 if ( !pdev->info.is_virtfn && !pdev->vf_rlen[0] ) @@ -821,6 +850,28 @@ int pci_remove_device(u16 seg, u8 bus, u8 devfn) list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list ) if ( pdev->bus =3D=3D bus && pdev->devfn =3D=3D devfn ) { + if ( !pdev->info.is_virtfn && !list_empty(&pdev->vf_list) ) + { + struct pci_dev *vf_pdev; + + /* + * Linux Dom0 has been observed to not respect an error co= de + * returned from PHYSDEVOP_pci_device_remove. Mark VFs and= PF + * broken. + */ + list_for_each_entry(vf_pdev, &pdev->vf_list, vf_list) + vf_pdev->broken =3D true; + + pdev->broken =3D true; + + printk(XENLOG_WARNING + "Attempted to remove PCI SR-IOV PF %pp with VFs sti= ll present\n", + &pdev->sbdf); + + ret =3D -EBUSY; + break; + } + if ( pdev->domain ) { write_lock(&pdev->domain->pci_lock); diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index 1e4fe68c60fb..977c0d08f78a 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -153,7 +153,15 @@ struct pci_dev { unsigned int count; #define PT_FAULT_THRESHOLD 10 } fault; + + /* + * List head if PF. + * List entry if VF. + */ + struct list_head vf_list; u64 vf_rlen[6]; + /* Link from VF to PF. Only populated for VFs. */ + const struct pci_dev *pf_pdev; =20 /* Data for vPCI. */ struct vpci *vpci; --=20 2.47.0 From nobody Sat Nov 23 09:34:58 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass(p=quarantine dis=none) header.from=amd.com ARC-Seal: i=2; a=rsa-sha256; t=1731686996; cv=pass; d=zohomail.com; s=zohoarc; b=GMvPi7TqsfaPtRLy/SRr2Ny+6q5gT9vz1WMPd7n1jcehqyTYSqm/XpYyXCOez5GwMMYuS1pbhyitLXTrsmZq/pGMsuHD0nEKbqByWHdLfSnXsFgkiOjZ95XxnKeX9t8pW54sQ7ESM8CfYdp7fJh3rfIAQutu2e92e/L5Tj5Swps= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1731686996; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=dc/5DRjqwA6xYxjlvFn4FWVgoiHtyGlzz0R5Ezc9BLU=; b=A7FxM8gd9Gjtgc0E6vE1Le+ysL0N25+ZulFQNmwKvPyRCWtPR5QbBYwS++aNQSu5c6cDJfEPVz1fLGKd0fhToGMeDies1uAALw+5KEqwIzMSn/YFZ4xnLHba70GdW2CmXWSAkAChycYQkdjZ7myc8wxqYj87tnuFnZ7wq3vcIc8= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1731686996119518.4921340751501; Fri, 15 Nov 2024 08:09:56 -0800 (PST) Received: from list by lists.xenproject.org with outflank-mailman.837780.1253689 (Exim 4.92) (envelope-from ) id 1tByt6-0002UN-Mx; Fri, 15 Nov 2024 16:09:36 +0000 Received: by outflank-mailman (output) from mailman id 837780.1253689; Fri, 15 Nov 2024 16:09:36 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tByt6-0002UB-Im; Fri, 15 Nov 2024 16:09:36 +0000 Received: by outflank-mailman (input) for mailman id 837780; Fri, 15 Nov 2024 16:09:35 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tByt4-0001uN-U3 for xen-devel@lists.xenproject.org; Fri, 15 Nov 2024 16:09:35 +0000 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2062c.outbound.protection.outlook.com [2a01:111:f403:2416::62c]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id feca6756-a36b-11ef-a0c7-8be0dac302b0; Fri, 15 Nov 2024 17:09:32 +0100 (CET) Received: from PH8PR07CA0035.namprd07.prod.outlook.com (2603:10b6:510:2cf::22) by SA3PR12MB9179.namprd12.prod.outlook.com (2603:10b6:806:3a1::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8158.18; Fri, 15 Nov 2024 16:09:25 +0000 Received: from SN1PEPF00036F3C.namprd05.prod.outlook.com (2603:10b6:510:2cf:cafe::5d) by PH8PR07CA0035.outlook.office365.com (2603:10b6:510:2cf::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8158.18 via Frontend Transport; Fri, 15 Nov 2024 16:09:24 +0000 Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF00036F3C.mail.protection.outlook.com (10.167.248.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8158.14 via Frontend Transport; Fri, 15 Nov 2024 16:09:22 +0000 Received: from SATLEXMB04.amd.com (10.181.40.145) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 15 Nov 2024 10:09:21 -0600 Received: from ubuntu.mshome.net (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Fri, 15 Nov 2024 10:09:20 -0600 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: feca6756-a36b-11ef-a0c7-8be0dac302b0 X-Custom-Connection: eyJyZW1vdGVpcCI6IjJhMDE6MTExOmY0MDM6MjQxNjo6NjJjIiwiaGVsbyI6Ik5BTTExLUNPMS1vYmUub3V0Ym91bmQucHJvdGVjdGlvbi5vdXRsb29rLmNvbSJ9 X-Custom-Transaction: eyJpZCI6ImZlY2E2NzU2LWEzNmItMTFlZi1hMGM3LThiZTBkYWMzMDJiMCIsInRzIjoxNzMxNjg2OTcyLjE5MjQ2OCwic2VuZGVyIjoic3Rld2FydC5oaWxkZWJyYW5kQGFtZC5jb20iLCJyZWNpcGllbnQiOiJ4ZW4tZGV2ZWxAbGlzdHMueGVucHJvamVjdC5vcmcifQ== ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=fAiXlm31YZH7uhosGjn+dZbcA/39X+vTk25+UA7ioXKsYS49bIxYpzV9aKhGT90OpxKV+4CFIyIRTCJq0KsUTV/mEkkqEwj6CjKtnUWL9jwZBx9OqtTZLpNzpLg5+lAwSw0f0fXwkFaH3qbH8ecwzyh1gbnvmI6WIbzYtEembFnwewDFnmU/gljqyQ3ZXTk9cx2udBxGN3jw0VvbHzi4m+t76UhlBwe5hDIZ97fjD47Jf5OmVXvrv0w9i63J+QbckAILc006SGtaOPPYzANR6ieGKmGDVg8UT8qeGgQUhJiLug82wH5lTMOjtIaShFuO2w/uDwkTjG402kX0w3iWOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dc/5DRjqwA6xYxjlvFn4FWVgoiHtyGlzz0R5Ezc9BLU=; b=psgN8ZWsIkMtfJAyA25Qn4y5nRNCg+29oVQ5Z6+zZLJ1/dO2hrWZhRMUNfbRXCmXOC+UL0QFCyFXfJh2RN8PiVSy91KUh8533NoLeaqXhBlo8GZ3Jqu+2TxFIZFwU6Yhqpr1jKNh9kafzYIH4gI/ZnkL5+Ky7EPtNdsTNtziVEJdOh4puBs6AytubLFK4/tX7ZFs0jUE6+MXXOm6CpPc8mFN4bw2zwGCx0p2d5YILxmfv5AxaN0e6xUek8j68a9pgXx1LT9A3JTy31edHlMy9M1DWNbJjx99RCj//bfnpYxGrsUXKJ3iTc08nKfvQPox2eMt3/1vsc29itTy0mi1tg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dc/5DRjqwA6xYxjlvFn4FWVgoiHtyGlzz0R5Ezc9BLU=; b=hYg0pUmxOg1AGQYnptiNOoNpQfVvo6y5vNRdAryYs+Ut/dPutiBEpocxvgGwUoDRx/rl949nuuobp8IQ6LKi2D6rz0qvpyaNghb4Tq7HZJAVuqIAMH3Wq2NUBcFV2jyiTKHtUNLDKpFjokabW7KOPlyFGbiIaDKhkmKUck0TUXA= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C From: Stewart Hildebrand To: CC: Stewart Hildebrand , Jan Beulich , Andrew Cooper , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Julien Grall , Stefano Stabellini , Teddy Astie Subject: [PATCH v8 2/3] x86/msi: fix locking for SR-IOV devices Date: Fri, 15 Nov 2024 11:09:03 -0500 Message-ID: <20241115160905.551224-3-stewart.hildebrand@amd.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241115160905.551224-1-stewart.hildebrand@amd.com> References: <20241115160905.551224-1-stewart.hildebrand@amd.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: None (SATLEXMB04.amd.com: stewart.hildebrand@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF00036F3C:EE_|SA3PR12MB9179:EE_ X-MS-Office365-Filtering-Correlation-Id: ae102f74-b2c1-4093-9fab-08dd058fde9c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|82310400026|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?CubGn6RcsopyhL1Yor2h03Bh11s6+4lar56+TdvJftFTOSAckCLCIDRkAmxy?= =?us-ascii?Q?J5nUXbQIVeoubZqM0Zo+Fel7dRb0ptXCQX2QMI4t9iLuT5APNh3KpQNMBsOW?= =?us-ascii?Q?me1j7FnCM225gq+EXdtHk30i6+CVzBYS7p9skVuned67tnX2iCz6Ru5PDr1E?= =?us-ascii?Q?uqj2Uz8zYJ781X3eZaxa2LzZlxYsXVXK0SbRum/MZWEuIDEV4vZZL/5ZLnDi?= =?us-ascii?Q?dCLZhd/aqHZSoUOzZV3S/1bvOHQU7aCpkqwhjPe8l6ds7VhBC7aZfTVWIm4R?= =?us-ascii?Q?sMu0F4Y4ETDGTQWKAxJxM58a4zsUO6QFqD58ILbqinhyM1zCn7FiXvMmNzjG?= =?us-ascii?Q?WXQZPx6d9EGfcYAmTFYJPu84sdWJCfsO9voqPb0IgrnZKRBxG02hV7xTP8+Y?= =?us-ascii?Q?wILVUbpuN/I9l7s6eUTIAztC4/plAEy/zqZIeuhsJNol0Hy4f47c0tjm/EEh?= =?us-ascii?Q?aEg0MZhT3IFLiht8uldMtx2+EelfDq17XUZqPM+8xiGg/wma0tSbdtmN15dn?= =?us-ascii?Q?ckxyGymHEaJhh7n7tpbeE0czTCEUw2dadG+S3YKwk3c57u1bpHCRfLC/VoSv?= =?us-ascii?Q?0K+lcs0eeNlJ4JkPh0wCyA2hYvXO+TNkQSQAU6hUCbm8ERUBxzmi4j7iCONI?= =?us-ascii?Q?WrEMdf3rVlRrX9gqTNMbKufQtmpCC5kjCoNFHyQ8nFBjKyuVBLOvfXiN/8eA?= =?us-ascii?Q?OBTuQq8RGMgUkTLoCStIqXS5OPUSCaekGWwPnXGXO9yzJvSeaovosrjeUljL?= =?us-ascii?Q?X6HndrFuMAl4+WnEKxp2+SQWmpPj/QupPQdNHhpe0rvcwQPGtu34CJb7seB0?= =?us-ascii?Q?q0aG3+Ui/imLZ+hVMBBf78R5GDQn9dmYxZPT4aZ7tidm+G0ZCZX9s/nECk7z?= =?us-ascii?Q?u5OqCYlovWcUTE/tI7aHPo3i51+906wx7VTLEeQCg6MmA99RSUn2meg9Cykm?= =?us-ascii?Q?XnMxpDrbmS5ItvfvU2Apbqz0FyexwJY65QvoLMJpeJJjiUpFDTHyKqJrr8yF?= =?us-ascii?Q?CE9SEsEWVz8J4EzoRfxTVIcLmtlpqzk0uKlYf50HKmZeEaS6D0XRgrI1SKet?= =?us-ascii?Q?edu22+/qk+20+3Im6UCnnar8OasgCn+FO4WXXZMgMZ3gkjSFbLSC4U0ZRctg?= =?us-ascii?Q?XYV/jq/Br0t0zNtU0GMZ5gX2aOTdxSBKewjMiz7zsPg/FNRhuuCsbfygnJ53?= =?us-ascii?Q?XVTnYIgSh3aUeYyn4lzrctyq0ocKrmthC7tnWtUXPazNPanZlJXye1IDy0Or?= =?us-ascii?Q?d3xV6gSZBwMJ40qfb04QyhZX2SUn5fc/xjbUd1rWoS9TdgXgDPedPQHuJpCT?= =?us-ascii?Q?pPKXAW60SU+ClBrOUtAQb66Wx0Qg/mkTc9d3F8vQDfRRJdCD4ah2xPGJxYbf?= =?us-ascii?Q?OSLfkzakGlvHPjZ/nCpfkgz1jEj08V0HveIo9kRVvLLwmmn2fw=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(82310400026)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Nov 2024 16:09:22.8965 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ae102f74-b2c1-4093-9fab-08dd058fde9c X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF00036F3C.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB9179 X-ZohoMail-DKIM: pass (identity @amd.com) X-ZM-MESSAGEID: 1731686998222116600 Content-Type: text/plain; charset="utf-8" In commit 4f78438b45e2 ("vpci: use per-domain PCI lock to protect vpci structure") a lock was moved from allocate_and_map_msi_pirq() to the caller and changed from pcidevs_lock() to read_lock(&d->pci_lock). However, one call path wasn't updated to reflect the change, leading to a failed assertion observed under the following conditions: * PV dom0 * Debug build (CONFIG_DEBUG=3Dy) of Xen * There is an SR-IOV device in the system with one or more VFs enabled * Dom0 has loaded the driver for the VF and enabled MSI-X (XEN) Assertion 'd || pcidevs_locked()' failed at drivers/passthrough/pci.c= :535 (XEN) ----[ Xen-4.20-unstable x86_64 debug=3Dy Not tainted ]---- ... (XEN) Xen call trace: (XEN) [] R pci_get_pdev+0x4c/0xab (XEN) [] F arch/x86/msi.c#read_pci_mem_bar+0x58/0x272 (XEN) [] F arch/x86/msi.c#msix_capability_init+0x198/0= x755 (XEN) [] F arch/x86/msi.c#__pci_enable_msix+0x82/0xe8 (XEN) [] F pci_enable_msi+0x3f/0x78 (XEN) [] F map_domain_pirq+0x2a4/0x6dc (XEN) [] F allocate_and_map_msi_pirq+0x103/0x262 (XEN) [] F physdev_map_pirq+0x210/0x259 (XEN) [] F do_physdev_op+0x9c3/0x1454 (XEN) [] F pv_hypercall+0x5ac/0x6af (XEN) [] F lstar_enter+0x143/0x150 In read_pci_mem_bar(), the VF obtains the struct pci_dev pointer for its associated PF to access the vf_rlen array. This array is initialized in pci_add_device() and is only populated in the associated PF's struct pci_dev. Access the vf_rlen array via the link to the PF, and remove the troublesome call to pci_get_pdev(). Fixes: 4f78438b45e2 ("vpci: use per-domain PCI lock to protect vpci structu= re") Reported-by: Teddy Astie Signed-off-by: Stewart Hildebrand Reviewed-by: Jan Beulich --- Candidate for backport to 4.19 Patch #1 ("xen/pci: introduce PF<->VF links") is pre-requisite v7->v8: * no change v6->v7: * turn NULL-check into ASSERT * adjust commentary in struct pf_info v5->v6: * add Jan's R-b v4->v5: * split the PF<->VF links to a pre-requisite patch * pass pci_sbdf_t to read_pci_mem_bar() * use stdint.h types on changed lines * re-add NULL check for pf_info in read_pci_mem_bar(), as pf_info could be NULL v3->v4: * handle case when PF is removed with VFs enabled, then re-added with VFs disabled v2->v3: * link from VF to PF's struct pci_dev * v1->v2: * remove call to pci_get_pdev() --- xen/arch/x86/msi.c | 36 ++++++++++++++++++++--------------- xen/drivers/passthrough/pci.c | 8 +++++--- xen/include/xen/pci.h | 11 ++++++++--- 3 files changed, 34 insertions(+), 21 deletions(-) diff --git a/xen/arch/x86/msi.c b/xen/arch/x86/msi.c index fa724911928e..e2360579deda 100644 --- a/xen/arch/x86/msi.c +++ b/xen/arch/x86/msi.c @@ -663,34 +663,34 @@ static int msi_capability_init(struct pci_dev *dev, return 0; } =20 -static u64 read_pci_mem_bar(u16 seg, u8 bus, u8 slot, u8 func, u8 bir, int= vf) +static uint64_t read_pci_mem_bar(pci_sbdf_t sbdf, uint8_t bir, int vf, + const struct pf_info *pf_info) { + uint16_t seg =3D sbdf.seg; + uint8_t bus =3D sbdf.bus, slot =3D sbdf.dev, func =3D sbdf.fn; u8 limit; u32 addr, base =3D PCI_BASE_ADDRESS_0; u64 disp =3D 0; =20 if ( vf >=3D 0 ) { - struct pci_dev *pdev =3D pci_get_pdev(NULL, - PCI_SBDF(seg, bus, slot, func)= ); unsigned int pos; uint16_t ctrl, num_vf, offset, stride; =20 - if ( !pdev ) - return 0; + ASSERT(pf_info); =20 - pos =3D pci_find_ext_capability(pdev->sbdf, PCI_EXT_CAP_ID_SRIOV); - ctrl =3D pci_conf_read16(pdev->sbdf, pos + PCI_SRIOV_CTRL); - num_vf =3D pci_conf_read16(pdev->sbdf, pos + PCI_SRIOV_NUM_VF); - offset =3D pci_conf_read16(pdev->sbdf, pos + PCI_SRIOV_VF_OFFSET); - stride =3D pci_conf_read16(pdev->sbdf, pos + PCI_SRIOV_VF_STRIDE); + pos =3D pci_find_ext_capability(sbdf, PCI_EXT_CAP_ID_SRIOV); + ctrl =3D pci_conf_read16(sbdf, pos + PCI_SRIOV_CTRL); + num_vf =3D pci_conf_read16(sbdf, pos + PCI_SRIOV_NUM_VF); + offset =3D pci_conf_read16(sbdf, pos + PCI_SRIOV_VF_OFFSET); + stride =3D pci_conf_read16(sbdf, pos + PCI_SRIOV_VF_STRIDE); =20 if ( !pos || !(ctrl & PCI_SRIOV_CTRL_VFE) || !(ctrl & PCI_SRIOV_CTRL_MSE) || !num_vf || !offset || (num_vf > 1 && !stride) || bir >=3D PCI_SRIOV_NUM_BARS || - !pdev->vf_rlen[bir] ) + !pf_info->vf_rlen[bir] ) return 0; base =3D pos + PCI_SRIOV_BAR; vf -=3D PCI_BDF(bus, slot, func) + offset; @@ -704,8 +704,8 @@ static u64 read_pci_mem_bar(u16 seg, u8 bus, u8 slot, u= 8 func, u8 bir, int vf) } if ( vf >=3D num_vf ) return 0; - BUILD_BUG_ON(ARRAY_SIZE(pdev->vf_rlen) !=3D PCI_SRIOV_NUM_BARS); - disp =3D vf * pdev->vf_rlen[bir]; + BUILD_BUG_ON(ARRAY_SIZE(pf_info->vf_rlen) !=3D PCI_SRIOV_NUM_BARS); + disp =3D vf * pf_info->vf_rlen[bir]; limit =3D PCI_SRIOV_NUM_BARS; } else switch ( pci_conf_read8(PCI_SBDF(seg, bus, slot, func), @@ -814,6 +814,7 @@ static int msix_capability_init(struct pci_dev *dev, int vf; paddr_t pba_paddr; unsigned int pba_offset; + const struct pf_info *pf_info; =20 if ( !dev->info.is_virtfn ) { @@ -821,6 +822,7 @@ static int msix_capability_init(struct pci_dev *dev, pslot =3D slot; pfunc =3D func; vf =3D -1; + pf_info =3D NULL; } else { @@ -828,9 +830,12 @@ static int msix_capability_init(struct pci_dev *dev, pslot =3D PCI_SLOT(dev->info.physfn.devfn); pfunc =3D PCI_FUNC(dev->info.physfn.devfn); vf =3D dev->sbdf.bdf; + ASSERT(dev->pf_pdev); + pf_info =3D &dev->pf_pdev->physfn; } =20 - table_paddr =3D read_pci_mem_bar(seg, pbus, pslot, pfunc, bir, vf); + table_paddr =3D read_pci_mem_bar(PCI_SBDF(seg, pbus, pslot, pfunc)= , bir, + vf, pf_info); WARN_ON(msi && msi->table_base !=3D table_paddr); if ( !table_paddr ) { @@ -853,7 +858,8 @@ static int msix_capability_init(struct pci_dev *dev, =20 pba_offset =3D pci_conf_read32(dev->sbdf, msix_pba_offset_reg(pos)= ); bir =3D (u8)(pba_offset & PCI_MSIX_BIRMASK); - pba_paddr =3D read_pci_mem_bar(seg, pbus, pslot, pfunc, bir, vf); + pba_paddr =3D read_pci_mem_bar(PCI_SBDF(seg, pbus, pslot, pfunc), = bir, vf, + pf_info); WARN_ON(!pba_paddr); pba_paddr +=3D pba_offset & ~PCI_MSIX_BIRMASK; =20 diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 3fe2e7884f45..6af1136b6350 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -735,7 +735,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, } } =20 - if ( !pdev->info.is_virtfn && !pdev->vf_rlen[0] ) + if ( !pdev->info.is_virtfn && !pdev->physfn.vf_rlen[0] ) { unsigned int pos =3D pci_find_ext_capability(pdev->sbdf, PCI_EXT_CAP_ID_SRIOV); @@ -747,7 +747,9 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, { unsigned int i; =20 - BUILD_BUG_ON(ARRAY_SIZE(pdev->vf_rlen) !=3D PCI_SRIOV_NUM_BARS= ); + BUILD_BUG_ON(ARRAY_SIZE(pdev->physfn.vf_rlen) !=3D + PCI_SRIOV_NUM_BARS); + for ( i =3D 0; i < PCI_SRIOV_NUM_BARS; ) { unsigned int idx =3D pos + PCI_SRIOV_BAR + i * 4; @@ -762,7 +764,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, continue; } ret =3D pci_size_mem_bar(pdev->sbdf, idx, NULL, - &pdev->vf_rlen[i], + &pdev->physfn.vf_rlen[i], PCI_BAR_VF | ((i =3D=3D PCI_SRIOV_NUM_BARS - 1) ? PCI_BAR_LAST : 0)); diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index 977c0d08f78a..f784e9116059 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -159,9 +159,14 @@ struct pci_dev { * List entry if VF. */ struct list_head vf_list; - u64 vf_rlen[6]; - /* Link from VF to PF. Only populated for VFs. */ - const struct pci_dev *pf_pdev; + union { + struct pf_info { + /* Only populated for PFs. */ + uint64_t vf_rlen[PCI_SRIOV_NUM_BARS]; + } physfn; + /* Link from VF to PF. Only populated for VFs. */ + const struct pci_dev *pf_pdev; + }; =20 /* Data for vPCI. */ struct vpci *vpci; --=20 2.47.0 From nobody Sat Nov 23 09:34:58 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass(p=quarantine dis=none) header.from=amd.com ARC-Seal: i=2; a=rsa-sha256; t=1731687001; cv=pass; d=zohomail.com; s=zohoarc; b=mex6VubPjcr+ixtrs0sAcT8dCjXV/j0ImVFTPonue70wSpvgtAvLstHmCOl3Eec+VmtJduOsfpJShTcrUJWkIj5FVEfB+O3Kj+2K7WCQWAOv8QoZvCgsO0mmdX38lwBPvpSXvoX2u3CUpaPC18CpFM+W7Z87qZ8MRXIt1CbOQhc= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1731687001; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=h7wZt1IMcuMFrVXQ5VvCHPZJN8BgmZL4SJb/J4aUccI=; b=Elclo7KdsWWZdayCNut9p7J/e1lPepokWCD6+TZ8klX5x40OsVy4Y3E55iD67Ci5z3aE7FwinAsntz0wqz7S4VF8MEq+A4Ls8zvzGusm3ZF2ER+/FnhgsE4Bdb7VlSRER82qDlv0bngCdPn9dT1ryd99NvMh3xLl9XfkU0iernc= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 173168700193055.15613974289113; Fri, 15 Nov 2024 08:10:01 -0800 (PST) Received: from list by lists.xenproject.org with outflank-mailman.837781.1253699 (Exim 4.92) (envelope-from ) id 1tByt7-0002km-VL; Fri, 15 Nov 2024 16:09:37 +0000 Received: by outflank-mailman (output) from mailman id 837781.1253699; Fri, 15 Nov 2024 16:09:37 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tByt7-0002kd-S3; Fri, 15 Nov 2024 16:09:37 +0000 Received: by outflank-mailman (input) for mailman id 837781; Fri, 15 Nov 2024 16:09:36 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tByt6-0001uH-D0 for xen-devel@lists.xenproject.org; Fri, 15 Nov 2024 16:09:36 +0000 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on20611.outbound.protection.outlook.com [2a01:111:f403:2416::611]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id ff78e720-a36b-11ef-99a3-01e77a169b0f; Fri, 15 Nov 2024 17:09:33 +0100 (CET) Received: from SA1P222CA0063.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:2c1::12) by CH3PR12MB7523.namprd12.prod.outlook.com (2603:10b6:610:148::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8158.17; Fri, 15 Nov 2024 16:09:29 +0000 Received: from SN1PEPF00036F3D.namprd05.prod.outlook.com (2603:10b6:806:2c1:cafe::c6) by SA1P222CA0063.outlook.office365.com (2603:10b6:806:2c1::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8158.18 via Frontend Transport; Fri, 15 Nov 2024 16:09:28 +0000 Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF00036F3D.mail.protection.outlook.com (10.167.248.21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8158.14 via Frontend Transport; Fri, 15 Nov 2024 16:09:28 +0000 Received: from SATLEXMB04.amd.com (10.181.40.145) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 15 Nov 2024 10:09:27 -0600 Received: from ubuntu.mshome.net (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Fri, 15 Nov 2024 10:09:27 -0600 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: ff78e720-a36b-11ef-99a3-01e77a169b0f X-Custom-Connection: eyJyZW1vdGVpcCI6IjJhMDE6MTExOmY0MDM6MjQxNjo6NjExIiwiaGVsbyI6Ik5BTTExLUNPMS1vYmUub3V0Ym91bmQucHJvdGVjdGlvbi5vdXRsb29rLmNvbSJ9 X-Custom-Transaction: eyJpZCI6ImZmNzhlNzIwLWEzNmItMTFlZi05OWEzLTAxZTc3YTE2OWIwZiIsInRzIjoxNzMxNjg2OTczLjA1MTA4Nywic2VuZGVyIjoic3Rld2FydC5oaWxkZWJyYW5kQGFtZC5jb20iLCJyZWNpcGllbnQiOiJ4ZW4tZGV2ZWxAbGlzdHMueGVucHJvamVjdC5vcmcifQ== ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=s7PNeZvildj1e/UGOukygLOJYFvd8+M49rvjZf8tQsQ17fskhNSbp9gpjizWmrWz99K4YhMtYs7jMaQ3r7zSmOmXlIcflGCuxD/4YdKoqtH2UwfmYuLEfJU48dEyzzsZz92zXZI6/t44r4FCJ+6TfXJ7LXWC3v3oQVNy3gCXHrOMueqHWpFtFk0cpvlL7GE063LnPjJ93UlWxRqvUjE0/IwrUzL+r62PlKA76NGOStAioub9v92MNTsKqJZ21rdFUWrSXCP8Cw33sV5ZDCJBUu+I/jO6ytnMDLUONo+s4Vr+yPSeWgv/p2bkvKwIJ15uOtvMXWcmlfChog+8b0Wtew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=h7wZt1IMcuMFrVXQ5VvCHPZJN8BgmZL4SJb/J4aUccI=; b=E+6MKcwchzuMCUlQLjlDTl+vCookiq+mlQ80eEgWTtx9Fv0Z1/4k881kvx5ZGuc/bIxgIWQgXDcoYqLiswI9hnukHp2BkXDjd2xjSR/8WPJC5ARAJj+nXmobCZvR9igrW1lsFLfHlT/ajVxZIgRd9uBot4456/hvfEgCgk52M+olrnmCSEqvjxmeHK3Y79hWvoViquDVcYFurguEGsS3Y2PnGYWX1rUWKXgmaRZvgamMtw2hMwkR9NJQ85odBoQEqA0KoaH6fO3ftUGsJ335644R6iImcy71689p6GSD6KDdgYnp7ZKIsv9RnEB1wdx4JCxvGSsU0LHzbgWei1BL0A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=h7wZt1IMcuMFrVXQ5VvCHPZJN8BgmZL4SJb/J4aUccI=; b=B3vdZkst+KgWmTW6mhHDlt0AXkdPz1h1J2Dikw/320XJ1FX7gBuAn7HwY4s+T731wx08cpY82x0Xg8JYS/G4eIz5STaLsRYmT59Hs6GkTLICKY2KGSxHqXb4g/+Hos7nOVcKxRRPdlTsMbvIaPVizHR4EndEasC499xlWBdShwM= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C From: Stewart Hildebrand To: CC: Stewart Hildebrand , Jan Beulich , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= Subject: [PATCH v8 3/3] xen/pci: remove logic catering to adding VF without PF Date: Fri, 15 Nov 2024 11:09:04 -0500 Message-ID: <20241115160905.551224-4-stewart.hildebrand@amd.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241115160905.551224-1-stewart.hildebrand@amd.com> References: <20241115160905.551224-1-stewart.hildebrand@amd.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: None (SATLEXMB04.amd.com: stewart.hildebrand@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF00036F3D:EE_|CH3PR12MB7523:EE_ X-MS-Office365-Filtering-Correlation-Id: 005b4062-2761-42a9-1e97-08dd058fe1cc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?XRQ5Is108KhBajSMrMht++jzfH3XEO3c3DYNVTb8pj13aJmLI6Kdy8ZyNtDv?= =?us-ascii?Q?RqG+4oyZFk20dPYzi2+itImYDIeaMXgTx9iOBwfqMGfefBQEwSfbtUDCwi2b?= =?us-ascii?Q?nZWhwoqRw2Si+PKhEVRzyqzAbMPAP0YBz32d3f4bgzEuD14V/JQdnl6ekNFz?= =?us-ascii?Q?tQhoiWu3/ZiAkfyft4j8GYmc1PXvQ4PNgcOn69JurSlr6dypjop+EUdhJEQq?= =?us-ascii?Q?Yve5cpuejh/Oql2F3AtezmTwoagrhY5SiLswZYood5MZasIpDYgF6tJQ53fc?= =?us-ascii?Q?dhTr+tJO9r1zvAPGFarTEI3WloA5UHdPDdFzsBwmd/MQ2xkHyV3jRAaetkkK?= =?us-ascii?Q?zB1300EODkc22yyoINwxncjBjYWBQHgQtcqGuKnSnwZZUs9+TnVX+HrtbM0f?= =?us-ascii?Q?m5LDQcMmGM8s7PBDZhnU5W5EKEVZtPYkp6hGdMJ8PffZ7Hmg8cUihcL5vKO7?= =?us-ascii?Q?FAZ/kFzlMdkOMUCy9YcH6SUDpgV1IxjyNjm/wLKRfe6OatYtJHDaNmn6+ARp?= =?us-ascii?Q?b7LLS88hGvWoiOciGoFwHcWXTLRuiDgt6tRwAYFaZlXUSjGA1iPuoGsjs9DU?= =?us-ascii?Q?4/+is2GYSesweoqCrgHAZjK+XMN3p8Xn3g8RGpx6wrcHL2znuqd85DdoC8p+?= =?us-ascii?Q?fGDkvR2ujzMq/hHEbLVUPq+ombKyQCZIPg4tpSYudKLkxUOnx4pE1SEE6MNo?= =?us-ascii?Q?n2wfEcZuGGfP5/Q/R9+i6Sl6t57oXswwmZV60O5iWZzEnhcNC6qF8icoFYQZ?= =?us-ascii?Q?bzG3aCFZX5L450lZ3wcpVes1TYWxi6TxYDotMUkfE+f8HtW5/CvGPd7SRF9c?= =?us-ascii?Q?cDoYDDSx1Q3tCUPEk3eJlbUPcImWORoiq/RhOcDccPDVKpvc6ZtEWiIbMlO5?= =?us-ascii?Q?DMLHbYRi67BrL75hRkYrQs50F62ZIxMyXDFtb1yB2JCz3yMfIdRUL6lMfCZy?= =?us-ascii?Q?WkuBqn6N38utHm4vI37a92igG68UY1Z9St2TdP/Fo2fZSxkrWJiLqNZgEDbV?= =?us-ascii?Q?q1jspQdXg7XceB+NHKqnCovtXFJXYPMajTEJjPrDf7UI5bCGL6AYtG3BOjLm?= =?us-ascii?Q?qROd/HmSMlmR0vKX5AbWutxPCyIqzHzrmDjUXebXEOrg46eXdjcltGHn7KH7?= =?us-ascii?Q?8jZ8roO3OJ9yfPYLTc016jfpXmh4jpHKmvvzeYeTtN6tOooy+I9EEJUaJsNy?= =?us-ascii?Q?CmLXRE1svf46DO0bhO6CZT5M/LXm+13/g/mRaDNNfXgS/QNraNMvlTfRSbev?= =?us-ascii?Q?go0NdUq7615SncKSFSFDSGJbgGYtpsqXNqREja6hORGaOdI87cR2lHL30/lk?= =?us-ascii?Q?8mvbzcrOgMOKVZOHcg5VK2cq6n7zew6fnz47G8R2LCZX/soy8NqP9Bl3iNw5?= =?us-ascii?Q?9Tvlepg5PY+YIFBheT1diKGc6GWTvk2fbSoiz6n6Iq9f23h90w=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(1800799024)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Nov 2024 16:09:28.1785 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 005b4062-2761-42a9-1e97-08dd058fe1cc X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF00036F3D.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7523 X-ZohoMail-DKIM: pass (identity @amd.com) X-ZM-MESSAGEID: 1731687004088116600 Content-Type: text/plain; charset="utf-8" The hardware domain is expected to add a PF first before adding associated VFs. If adding happens out of order, print a warning and return an error. Drop the recursive call to pci_add_device(). Signed-off-by: Stewart Hildebrand --- Not for backport v7->v8: * new patch, split from PF<->VF links --- xen/drivers/passthrough/pci.c | 31 +++++++------------------------ 1 file changed, 7 insertions(+), 24 deletions(-) diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 6af1136b6350..f50dfaed75ba 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -699,30 +699,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, =20 if ( !pf_pdev ) { - ret =3D pci_add_device(seg, info->physfn.bus, info->physfn= .devfn, - NULL, node); - if ( ret ) - { - printk(XENLOG_WARNING - "Failed to add SR-IOV device PF %pp for VF %pp\= n", - &PCI_SBDF(seg, info->physfn.bus, info->physfn.d= evfn), - &pdev->sbdf); - free_pdev(pseg, pdev); - goto out; - } - pf_pdev =3D pci_get_pdev(NULL, PCI_SBDF(seg, info->physfn.= bus, - info->physfn.devfn)); - if ( !pf_pdev ) - { - printk(XENLOG_ERR - "Inconsistent PCI state: failed to find newly a= dded PF %pp for VF %pp\n", - &PCI_SBDF(seg, info->physfn.bus, info->physfn.d= evfn), - &pdev->sbdf); - ASSERT_UNREACHABLE(); - free_pdev(pseg, pdev); - ret =3D -EILSEQ; - goto out; - } + printk(XENLOG_WARNING + "Attempted to add SR-IOV VF %pp without PF %pp\n", + &pdev->sbdf, + &PCI_SBDF(seg, info->physfn.bus, info->physfn.devfn= )); + free_pdev(pseg, pdev); + ret =3D -ENODEV; + goto out; } =20 if ( !pdev->pf_pdev ) --=20 2.47.0