From nobody Sat Oct 4 22:04:04 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass(p=quarantine dis=none) header.from=amd.com ARC-Seal: i=2; a=rsa-sha256; t=1758873067; cv=pass; d=zohomail.com; s=zohoarc; b=KZjLgo00hrNvjERxnusObcD7XzdJK9DnD7eYHF+wzaBD7P9aWsCMAsf0fynxSkoiHmj83KYsINR6rmpKLG7c8CUfdcWc7h++x8ejyWFlJkXTEo5bQ2hd5c8k1pBSadKA7N1RC1j7YZkELz0rQIJRPF3QsBXgUOJJBbHUlMcN17U= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1758873067; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=ludIB4tomAkFHdVYMp26SsnX4Cy76mwEPAwccNGqB8k=; b=RzraBAQ+Acw/9yXuSawpMdo9kPtmbgMb+4dl+BthDR9f9dLYwbaQYja7Fx9grw2uCMIOLe584jCs8Spfr6GyHqImYE69gcyHeAJ4IkWKh+FJnzjuU3Whc2FzyBtL5dHS3zYDUC8Y0w0FeeK89h+SJi9iQAA0mYeid/+WqPjgY8s= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1758873067413892.4612795135916; Fri, 26 Sep 2025 00:51:07 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1131214.1470398 (Exim 4.92) (envelope-from ) id 1v23EA-0004Q0-0w; Fri, 26 Sep 2025 07:50:50 +0000 Received: by outflank-mailman (output) from mailman id 1131214.1470398; Fri, 26 Sep 2025 07:50:49 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v23E9-0004Pt-U6; Fri, 26 Sep 2025 07:50:49 +0000 Received: by outflank-mailman (input) for mailman id 1131214; Fri, 26 Sep 2025 07:50:48 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v23E8-00049y-Gz for xen-devel@lists.xenproject.org; Fri, 26 Sep 2025 07:50:48 +0000 Received: from SA9PR02CU001.outbound.protection.outlook.com (mail-southcentralusazlp170130001.outbound.protection.outlook.com [2a01:111:f403:c10c::1]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 82ef4daa-9aad-11f0-9809-7dc792cee155; Fri, 26 Sep 2025 09:50:46 +0200 (CEST) Received: from CH2PR05CA0030.namprd05.prod.outlook.com (2603:10b6:610::43) by MW4PR12MB6731.namprd12.prod.outlook.com (2603:10b6:303:1eb::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9160.10; Fri, 26 Sep 2025 07:50:42 +0000 Received: from DS2PEPF0000343C.namprd02.prod.outlook.com (2603:10b6:610:0:cafe::13) by CH2PR05CA0030.outlook.office365.com (2603:10b6:610::43) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9182.5 via Frontend Transport; Fri, 26 Sep 2025 07:50:41 +0000 Received: from satlexmb07.amd.com (165.204.84.17) by DS2PEPF0000343C.mail.protection.outlook.com (10.167.18.39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9160.9 via Frontend Transport; Fri, 26 Sep 2025 07:50:41 +0000 Received: from cjq-desktop.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Fri, 26 Sep 2025 00:50:39 -0700 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 82ef4daa-9aad-11f0-9809-7dc792cee155 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=SGpJf5pvpCncbXnoNtXZGHFR9xn0pC4ViyjwUojvZ5R2PZuqMIQpmiaM3ryVKNwYcllh2C4IKn/0ahUryEltqsOV536E0OP63RSfn4MUGbfJOQf3ncmFtZ/DhY2ValX54EWs72k3BsKadlYUFQyp8NyPoaEQS7CG4J51Nlmrba7YoUOsudGdl38TtMpv7Y5QOSVHtnZIHLAVP7k4/w9d0Nh69npTX5g5tAr73isZXWeOFExd/Zlf7wce2KusrScuwc55FS3RN7X9KiSzTesEKlDMPumBhbiifDwB8cYBFacmTNib6pJJwqgFuLcaIKgL2R3ie6OGb52dt9teR4QnfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ludIB4tomAkFHdVYMp26SsnX4Cy76mwEPAwccNGqB8k=; b=RhoaHPKkTdsrt/oL46qlRI0qYj6YlmaXlHnspeWLOVJpt8pOh/hORWs2hpb601+cqruvfBGCR4YrSlLZ2zx46JLwN200lmkr3wdAXYIZmbzS2ALQQWnWET5Ez/U43wLAdVqSP4GW1U4U/JTTCntxyxSsUwteCVP5Ahkc3pMEuT0SY6AbayOdk/l36mh9eaT6wXRdCxU+/AmWNQhN/XB8GXMDegBfolVNb111GIfSFviVeiCaXyW9cjnK/gL509QMK8rlvyG5q31QnZB2GzAvXPtMfutrextHgd7Lu2HjmofT0fCNNQxC6taDGlSTJjs3FpwzJpWXh3Dld1mcIfPx+w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ludIB4tomAkFHdVYMp26SsnX4Cy76mwEPAwccNGqB8k=; b=rmW4kcRPA/jUKemZ4LQjudr9/+rEwCRnzV667C77Jm3Fx7ODLs4u6z6wQYGa1uM6yhe5Eu1MkTnx/AExJoOdCgGJtUPQGkJoiD+5B380EDmp7RXbrgGsfMuUunVy5F4C5ktnZxzdI2L6tgEIGMKFUYNh0Einms8eQEUK4WZIZdI= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C From: Jiqian Chen To: CC: Huang Rui , Jiqian Chen , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= Subject: [PATCH v12 1/3] vpci/rebar: Implement cleanup function for Rebar Date: Fri, 26 Sep 2025 15:50:19 +0800 Message-ID: <20250926075021.27967-2-Jiqian.Chen@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250926075021.27967-1-Jiqian.Chen@amd.com> References: <20250926075021.27967-1-Jiqian.Chen@amd.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PEPF0000343C:EE_|MW4PR12MB6731:EE_ X-MS-Office365-Filtering-Correlation-Id: 8a13f36e-261b-4e89-ffdb-08ddfcd163f7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|376014|1800799024|36860700013; X-Microsoft-Antispam-Message-Info: =?utf-8?B?TnlOQklycEZDd2E4Ujh2ejlodmNWMHFTU0VyeGFjSytzUWZJMnpKUGhqU25J?= =?utf-8?B?WkI1WnUzRHBxbTQ0NzFONEEydklLQkVXd3BqbkFTV3F0RGh6Qi9QdlVITHY4?= =?utf-8?B?SmxzTDdZeUloQnJuck1PSGo0ZCtNUkNtUGlTWWkzNVg0OERBaXZuUlczUlR3?= =?utf-8?B?SVpNMDg2TUtTWGVsVDgxNm5vZHRHYk1uSm0yY0h0TCtmTk9JRWdvV1l5OEY1?= =?utf-8?B?T2lJaDFoYzJEd1dzUHBPQ0ZwTVVnNXFKT2I2bmVlQkJjcFcyaWRpQjlzS29E?= =?utf-8?B?S2dMcnZsOVJrOERJNTY5WWFHcVlqcGdZMkxoOEJ4ODZ3Q1B5U1pVdGxYTlpv?= =?utf-8?B?NitYZEN4YTFzRmRmaG8vazhnelRDTlpOVlJMenNKczZDSG5RWXFuY2l2RWxB?= =?utf-8?B?VndSK3JYUnpBL1R6bTlISlFmV3hORVJjZ2tBV1FoRVJZMnRoRnppenI4aDhE?= =?utf-8?B?Mko0bFAxZHRnbXVob1RwR045RHcvV0p5TWNSK0V4WjVNYjEvVG82dlBVWG95?= =?utf-8?B?bjIwM0JoZythYVNSb1RVN2JtU3F4bDNqZU1tMlRGS0tOZExuSG8yQm5lTzRC?= =?utf-8?B?UWZCTzlNdWNrRDJyM0J0OFkzU0V0RDgwdW1jcUNKVzVhaXBKb1VTNVo1aDNW?= =?utf-8?B?ZmxsMlE3cXIwdEEzalYzejlsWlhlS2pGK0RBNWc2WE9jTW9hMmxuMjk2YTE2?= =?utf-8?B?d2ZWTGp6bXlZa21BTVhUUzBtQnVZenZyZCtGakxHa1lxTnRvMXlVVXhWcWlI?= =?utf-8?B?Yi9WcnNFV3VhSC9vTjFKZUNGQk1JV2dxSUU0WTZmNXk0Rm9WUXhlOWMwTTRu?= =?utf-8?B?NUJ4WWs4K0d3Wmx2SklzNFY1cncrWEFVRVVTeThFRXIxbVpmSklEUWsyc1Jj?= =?utf-8?B?TE9SLyt6RDNWd2gzMWp3Z2FrUHM2M0tYdDQxdjdVcUVPbkVZdjdoQVN1RXpL?= =?utf-8?B?bUdsTUc3OS9jei9vZVBjOTV6cEZuL25yaFJFczljdEtpTWE2Wm9DckVMYTBI?= =?utf-8?B?K2FNUEdXS0R1YWE0ZFB6ejQzTVRXRkkrdllySSsvWUZEZkh1Q2crQnI1anNv?= =?utf-8?B?dWFoc2dqYWtZQjk0akxmbHlZK1U2czQzbFZ2N01vbldkT1kxWURQSjJ3MW5u?= =?utf-8?B?TDUzejljN1lVcjZKQndEMjJDNFQ5NVR5KytrZzFVd3Z4YnlPaTlsNktMZnpY?= =?utf-8?B?V3ZWaldKZG9rTWdnaGFiVEc2dnJGT1Ixd1FuNXhTS1NZTjFzREtVWU9OcUVi?= =?utf-8?B?U0NJY2JOK2xkWjdNM2dyaW1ERUUzVGI3Y29JakU2ZVBwbHFRUGJTb0V1a3pl?= =?utf-8?B?SFVIai8wNzZqKzdrNGZieldmWDVZSURJMEduNk9GZ2Fucklhb1JzVWpsRmlG?= =?utf-8?B?N0U4eUhQZkhqd1I2WGFsWTFoLy9tbEUvQlk1bkxGOHlWdXV3STg2aTN1ZUph?= =?utf-8?B?SkV0NnM2RjFyeDkwLzJFRnNHaVZQTmJFL1hXSkQwMnZYSnNEWXVtTnk4ZHd2?= =?utf-8?B?VkJBakNwdWMrc1kvazAxMGI0THRrb2p3dGN1VWlWek1xVzM4MnNsaTJjU3o5?= =?utf-8?B?RkNBWllTcUVFUS9EdWI5YzBhdXNEYXZmaDdHMStLalhrZlhuanI1SGJQcEtQ?= =?utf-8?B?clRCUkFRM2tjWjlDVXZqM0xwb1VZdlA0Z0JsbHpKTGVibmhnak1qNE51YUk3?= =?utf-8?B?N2tkNWFhdGhaMktyMGZ2alBieXBDUEpoMWNLOVJuSmFrZlBsOTNXTFhJQUdF?= =?utf-8?B?ZmpJZzR5QkZBbTJwc3U2dE9rR0RkckF1aWV2Zmk4VWpoR3pNZGh4aTFZVWMv?= =?utf-8?B?UHN6MVZ4RURFd1JRbGQ4eHBHb1dFS00rVjAra05sUHBPSndvbHJvZVZLN0NI?= =?utf-8?B?a3VvTnpEb21SMGRZaUJxK013TUx6L3hLNlVCZ2JoQlpob3llc1JMNTJxdXVB?= =?utf-8?B?ZmpaMUlhZzhQNjNuY2NNYmtSS3lsRW5EbU1nWW11bHRVZjlmaThlVlFSd3Z2?= =?utf-8?B?NTAzRVd0aGtocmMxRlZEV2hwUjlMdzVma2xHcXVqM2ZFVlA4RVBGekJIc3A3?= =?utf-8?B?OHpNNksyN2IwZ3VIYmhaZ3BXZmdhU3Y3WW1Ydz09?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(376014)(1800799024)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Sep 2025 07:50:41.1241 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8a13f36e-261b-4e89-ffdb-08ddfcd163f7 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS2PEPF0000343C.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB6731 X-ZohoMail-DKIM: pass (identity @amd.com) X-ZM-MESSAGEID: 1758873068519116600 When Rebar initialization fails, vPCI hides the capability, but removing handlers and datas won't be performed until the device is deassigned. So, implement Rebar cleanup hook that will be called to cleanup Rebar related handlers and free it's associated data when initialization fails. Signed-off-by: Jiqian Chen --- cc: "Roger Pau Monn=C3=A9" --- v11->v12 changes: * In cleanup_rebar(), move the check "if ( !hide )" above the vpci_remove_r= egisters(). * In init_rebar(), change return rc to continue when "if ( index >=3D PCI_H= EADER_NORMAL_NR_BARS )" and "if ( bar->type !=3D VPCI_BAR_MEM64_LO && bar->type !=3D VPCI_BAR_MEM32 )" v10->v11 changes: * Add ASSERT_UNREACHABLE() when vpci_remove_registers() fails * When hide =3D=3D true, add handlers to let Rebar ctrl be RO. * Remove Roger's Reviewed-by since patch change. v9->v10 changes: v8->v9 changes: No. v7->v8 changes: * Add Roger's Reviewed-by. v6->v7 changes: * Change the pointer parameter of cleanup_rebar() to be const. * Print error when vpci_remove_registers() fail in cleanup_rebar(). v5->v6 changes: No. v4->v5 changes: * Change definition "static void cleanup_rebar" to "static int cf_check cle= anup_rebar" since cleanup hook is changed to be int. v3->v4 changes: * Change function name from fini_rebar() to cleanup_rebar(). * Change the error number to be E2BIG and ENXIO in init_rebar(). v2->v3 changes: * Use fini_rebar() to remove all register instead of in the failure path of= init_rebar(); v1->v2 changes: * Called vpci_remove_registers() to remove all possible registered register= s instead of using a array to record all registered register. Best regards, Jiqian Chen. --- xen/drivers/vpci/rebar.c | 62 ++++++++++++++++++++++++++++++++++------ 1 file changed, 53 insertions(+), 9 deletions(-) diff --git a/xen/drivers/vpci/rebar.c b/xen/drivers/vpci/rebar.c index 3c18792d9bcd..3651e9613b4c 100644 --- a/xen/drivers/vpci/rebar.c +++ b/xen/drivers/vpci/rebar.c @@ -49,6 +49,57 @@ static void cf_check rebar_ctrl_write(const struct pci_d= ev *pdev, bar->guest_addr =3D bar->addr; } =20 +static int cf_check cleanup_rebar(const struct pci_dev *pdev, bool hide) +{ + int rc; + uint32_t ctrl; + unsigned int nbars; + unsigned int rebar_offset =3D pci_find_ext_capability(pdev->sbdf, + PCI_EXT_CAP_ID_REB= AR); + + if ( !rebar_offset || !is_hardware_domain(pdev->domain) ) + { + ASSERT_UNREACHABLE(); + return 0; + } + + if ( !hide ) + return 0; + + ctrl =3D pci_conf_read32(pdev->sbdf, rebar_offset + PCI_REBAR_CTRL(0)); + nbars =3D MASK_EXTR(ctrl, PCI_REBAR_CTRL_NBAR_MASK); + + rc =3D vpci_remove_registers(pdev->vpci, rebar_offset + PCI_REBAR_CAP(= 0), + PCI_REBAR_CTRL(nbars - 1)); + if ( rc ) + { + printk(XENLOG_ERR "%pd %pp: fail to remove Rebar handlers rc=3D%d\= n", + pdev->domain, &pdev->sbdf, rc); + ASSERT_UNREACHABLE(); + return rc; + } + + /* + * The driver may not traverse the capability list and think device + * supports Rebar by default. So here let the control register of Rebar + * be Read-Only is to ensure Rebar disabled. + */ + for ( unsigned int i =3D 0; i < nbars; i++ ) + { + rc =3D vpci_add_register(pdev->vpci, vpci_hw_read32, NULL, + rebar_offset + PCI_REBAR_CTRL(i), 4, NULL); + if ( rc ) + { + printk(XENLOG_ERR + "%pd %pp: fail to add Rebar ctrl handler rc=3D%d\n", + pdev->domain, &pdev->sbdf, rc); + return rc; + } + } + + return 0; +} + static int cf_check init_rebar(struct pci_dev *pdev) { uint32_t ctrl; @@ -97,14 +148,7 @@ static int cf_check init_rebar(struct pci_dev *pdev) { printk(XENLOG_ERR "%pd %pp: BAR%u fail to add reg of REBAR_CTR= L rc=3D%d\n", pdev->domain, &pdev->sbdf, index, rc); - /* - * Ideally we would hide the ReBar capability on error, but co= de - * for doing so still needs to be written. Use continue instead - * to keep any already setup register hooks, as returning an - * error will cause the hardware domain to get unmediated acce= ss - * to all device registers. - */ - continue; + return rc; } =20 bar->resizable_sizes =3D @@ -118,7 +162,7 @@ static int cf_check init_rebar(struct pci_dev *pdev) =20 return 0; } -REGISTER_VPCI_EXTCAP(REBAR, init_rebar, NULL); +REGISTER_VPCI_EXTCAP(REBAR, init_rebar, cleanup_rebar); =20 /* * Local variables: --=20 2.34.1 From nobody Sat Oct 4 22:04:04 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass(p=quarantine dis=none) header.from=amd.com ARC-Seal: i=2; a=rsa-sha256; t=1758873066; cv=pass; d=zohomail.com; s=zohoarc; b=Kb4rWVI/+Yjp6PZLJHiaRZPDbVv+5I+ZlI8sOyuOG9UtS4QnqMHrx3pk4OPoObx0mtSUqQlGp2+OBQXosIKK0YEmEJIKXwvH6zJ29q/wZoy87YJMFG1/4iKtofwxcQCE8aabl0sj410pkfCHX9SOP0q5k83LDwBBU8yiLRcKZxA= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1758873066; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=e4ll3zGyqnoS22DY6zoekAmF0Bra4jeTbZ+pxTmziEM=; b=JVPM4wOxrVMUjuWXEhwDi+mOQd1xPRT+20kkldi6SSFRf09HSUdEYbjrYVCt6JMoWxIAobSMst1jcC6/CD0Bs2ulkiiDl2CbVKjNQdWTceWtSH5r+Fe1J+gx52DOHlz9SjoCegup6jkpNNSgLYrj8Ckrs3LEnIxt5tmAO029sNs= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1758873066064300.2795507289254; Fri, 26 Sep 2025 00:51:06 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1131215.1470402 (Exim 4.92) (envelope-from ) id 1v23EA-0004SA-9U; Fri, 26 Sep 2025 07:50:50 +0000 Received: by outflank-mailman (output) from mailman id 1131215.1470402; Fri, 26 Sep 2025 07:50:50 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v23EA-0004RR-4X; Fri, 26 Sep 2025 07:50:50 +0000 Received: by outflank-mailman (input) for mailman id 1131215; Fri, 26 Sep 2025 07:50:49 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v23E9-0004PV-Bt for xen-devel@lists.xenproject.org; Fri, 26 Sep 2025 07:50:49 +0000 Received: from MW6PR02CU001.outbound.protection.outlook.com (mail-westus2azlp170120002.outbound.protection.outlook.com [2a01:111:f403:c007::2]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 82df3347-9aad-11f0-9d14-b5c5bf9af7f9; Fri, 26 Sep 2025 09:50:47 +0200 (CEST) Received: from CH2PR05CA0031.namprd05.prod.outlook.com (2603:10b6:610::44) by DM6PR12MB4170.namprd12.prod.outlook.com (2603:10b6:5:219::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9160.9; Fri, 26 Sep 2025 07:50:42 +0000 Received: from DS2PEPF0000343C.namprd02.prod.outlook.com (2603:10b6:610:0:cafe::15) by CH2PR05CA0031.outlook.office365.com (2603:10b6:610::44) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9182.4 via Frontend Transport; Fri, 26 Sep 2025 07:50:42 +0000 Received: from satlexmb07.amd.com (165.204.84.17) by DS2PEPF0000343C.mail.protection.outlook.com (10.167.18.39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9160.9 via Frontend Transport; Fri, 26 Sep 2025 07:50:42 +0000 Received: from cjq-desktop.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Fri, 26 Sep 2025 00:50:40 -0700 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 82df3347-9aad-11f0-9d14-b5c5bf9af7f9 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=NxUIvesYJ7WINpQTTE3gdfDsN61h1JpgpMdEwYlMhMUuCOqHhyIwOtiWItYoYtjDcwQ5P5WRSfk8LcaCtIZYmQcpLvOV08KbaQGFlDTuZAyaxKFZNmf68WjA5XFN4WHmETLKANzRFYdQF1npD7l3IaigNwMNdbYLwiRKWZqs2ZjX/DVUXF1f2zuCj8KyHYb4kt8m2SdPwMy7Nsvl6NQ026fAwJp8XGOc4Y/+iQXBDgwI6BCrXF/f7Lw/4CJD/6dkMVjgduGwMMHkWal7SxCM1Gh02tri2IPE/YSnTpuFQUYYC+NKqrTFxf8/+3XfGZ6iOoz9hGCigamCYMI7PYvT8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=e4ll3zGyqnoS22DY6zoekAmF0Bra4jeTbZ+pxTmziEM=; b=r0jZ8PXzeO3EOMtOruWZMdbLj4+GVIywbb96zz3I+tvg0qZBpLZbVe54RazYLkWHpz1ghDTZc73N8FkrbJpTmqiEjxW+bEiDQu7OhgT+N617axPMT5zX22tJnLLi5BcQHMHUwHt2Y+sd/nuUNeW0kERnqD4yasnMfGXAhYh42pagol4blr5m4Vn8ASYPujoxnIksWNCLQPgRgFhS+/nkAsYc2l++oivDLrdbUAs/BOnO8jNqbyu1Kw0zfmswbmb6vh0OIVqxqhmmD0thH+XcHPdi1IZJ8dcyXbTMB4fWc0jwPeyGO+y3oMV45n4xLixVIPgXLAIQclg0+8qbob7cag== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=e4ll3zGyqnoS22DY6zoekAmF0Bra4jeTbZ+pxTmziEM=; b=xyot2SmJjJUMbpbA5KD2T6K+Vk3eCIFyHu+n8XjviNJn/cU53V7Ry/ffVQ53tH/Syt4L/bX0FQJM2eBnyKJaPIl9E+xPcDxIIYqHfsc63thlrB28WUH+I+MNhpopgnjdTZWVTYbrrEAo4oeZ2QgHn5ZJHc5to/TMgr9aRagNzLE= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C From: Jiqian Chen To: CC: Huang Rui , Jiqian Chen , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= Subject: [PATCH v12 2/3] vpci/msi: Implement cleanup function for MSI Date: Fri, 26 Sep 2025 15:50:20 +0800 Message-ID: <20250926075021.27967-3-Jiqian.Chen@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250926075021.27967-1-Jiqian.Chen@amd.com> References: <20250926075021.27967-1-Jiqian.Chen@amd.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PEPF0000343C:EE_|DM6PR12MB4170:EE_ X-MS-Office365-Filtering-Correlation-Id: 3a2d4d50-4def-4740-018b-08ddfcd164d0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|82310400026|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?B?RXNGZnNpSzloQkwzK2xjcEhjdFNvZy9NY0ZzTzR3U0xPMS9ndmdxOUdlNFd1?= =?utf-8?B?YkZweTdjSHFib2FaR2VaNkFkbE55QXFzaXdKV1RFUVNmVE5lVkJoQm9OUGtJ?= =?utf-8?B?SnhPbVNxakpweGhESkswZnhBWG5xRkdhbFlrN3c1Q0c4OWV2VHM1N1lHWFFL?= =?utf-8?B?MFVLcWY5K21aZENCYVVqc2lVWGJ6MGpicDliV1dabktjTFFJSmt2eUtTQkdN?= =?utf-8?B?M2pHTit1K2ZXTE51Q1Q4YVJEOUZQdlhEdHgxdC9lczNzeTNvUEJZdnV4Vm9p?= =?utf-8?B?bDY2VWIzbmVuWFd3bmhXUThKL29NZ21jQkRISkhLVWlCdTZ1RFNrQnlTRlpv?= =?utf-8?B?TGYycFlUNnUxekVhd0NZdEFrblMzU01XOGFzKzF0Z2k5TVQzaXVuSkk4Rk1i?= =?utf-8?B?TS90WktGLzhvY3VQUlUvZW43NHlOV2JseUFqVmt0SUpnYStzbVZpVGk5dXQ5?= =?utf-8?B?S0dSSGJtMUgvZ0dscmRRNUk2TU9aWW5BSVdWWWNsL0N2RjlSRUMwcDhsWGhO?= =?utf-8?B?aHpHaGJsa2VHVkhHNWVPcFpuVGFCbDJKUGRwRWJ0dXlMVGdtUlRFV1pNbjAw?= =?utf-8?B?VDFlQlJBVTVWOEJBc3BQdThRTHFVakh4UmUrYkUwQzZaRmpCMDVsWXcrR1RB?= =?utf-8?B?NHhITFUyYTZoa09LajlpaXlVRHFyRDRJWEJwWWpIUG45Wm8wMWliU1JQdFM3?= =?utf-8?B?Vmk1WEFZK1ZKTHF1WE1MeWk1eWRGdXRPcVJjNmFrOU40cExsb3d4Rjh5NEVF?= =?utf-8?B?WEYxVXFGVUJuWlFpUWRuVW10S20vd2xCN010YmRnVGMzUHhES1AzbU5VaWJZ?= =?utf-8?B?Ri85V1YvaEkvUjRwaWZ3dDlDUmFHUmJKZ0lIR2FYbzl3bTRYbzBmbGRqNXVl?= =?utf-8?B?Y0pSaGZPbVVVb0szaFl2aEJZQkVsNGFwN1BrODVMSDVUSVBCZTNRSG5iei9R?= =?utf-8?B?ZGV3a3pIRGUwRzR6SGNpenFVR1p4RGZLNENVU1VLOVJaejM0Y21EeEFSZUZP?= =?utf-8?B?aGJzc0J6dlBEckttQ0RLQWIwNVdycFkvZjVTUFNFZWNKeW5aMWV6K3BVd2FV?= =?utf-8?B?Qytaa1FuNklhdFpoTWJOMUhpYzN0QS9HN1ZBU0R3eXNNRHVPcWpwSjRuUWZR?= =?utf-8?B?Q0hPa2wrZzFkMXlqQ0dWbVRXbXNmYy9uanA3Zkp0alpzRkRVZmF6cUtvUWhW?= =?utf-8?B?akhqZFNERFlWWk5paytndnBlUFJKNlpyWHFTRDZVeVpTZVpsRlVETHJHYXE1?= =?utf-8?B?QThtVDhIajdjTjBjT21SSXFLY3NlNHZobWJTdXJaaGdzYVNYZXM1b3pWUkFD?= =?utf-8?B?Y3o5NEk1WTBGYTFBRm5iL3NXakpZS0pTT0tGVzdWQ1JyNHdJZDNtTnhLTU9x?= =?utf-8?B?MGZHRDM3VU9nT0tJMGlVdjJSZURJSVpXdHBOUzJwVDJrbkIwSWpJaWZHZDd6?= =?utf-8?B?VUV1LzVPZ1FxM243RUw0OWxvUDU3YWpQNjAxcXpJZmplaUkxdWVHeDA3RWpR?= =?utf-8?B?N3Jia2FYdkQ0VVoxWlBxWXF0K2JUQThVcVhVSklxaVNmR3YyVGhBNTViUExk?= =?utf-8?B?aHM3dzU2NGhESm1Pd1ZjTXFWWEU5MXpHcy9pNmpCOEpwQkkyYm1idWZ1VjE3?= =?utf-8?B?N2JqNUpIdmd5a1BaYXR2RUlycHpCOUhTMnpxM0lhMnUxVklBK1hOZWtyaXA3?= =?utf-8?B?dlU1Vy9YV2Z2THNxT1JNendyWEdUd0F0OTN4TzhTckMwQzVvV2t1dlh5L3pr?= =?utf-8?B?OU9NV0p5MitYTFVyd2ZZWDhYVjliSi9LVGRHV3M4TndYeVhnc0U4NENlT05S?= =?utf-8?B?eloyam1iUlA5RkRkbjBmdXB4N0wzR2M1QVFlNEhZV1MzbXBIWUFOcWZ6bitC?= =?utf-8?B?Q1ZNV20ycTd2MXRpbDRoaUdiRE9iT3pMcGNSWGFiemk5eW54NzFNb0x0UGNB?= =?utf-8?B?dDkzRXFQQUFyTHgrdzQ5REcybGNEdFd5TVg5TVFnYXlPckEvcHJhbFh0bGR6?= =?utf-8?B?Kzlid283Szl0aGFteDRsajNKMHh2cXQzV21wV1lPRlFsSUxRckIwSitDUDVJ?= =?utf-8?B?dFRzN2pBV3ZwSXFTUGIxT3l3azJQYmhLYU91R2RkN1JONXpla1oxc1FJVWlZ?= =?utf-8?Q?Y7dw=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(82310400026)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Sep 2025 07:50:42.5392 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3a2d4d50-4def-4740-018b-08ddfcd164d0 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS2PEPF0000343C.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4170 X-ZohoMail-DKIM: pass (identity @amd.com) X-ZM-MESSAGEID: 1758873066557116600 When MSI initialization fails, vPCI hides the capability, but removing handlers and datas won't be performed until the device is deassigned. So, implement MSI cleanup hook that will be called to cleanup MSI related handlers and free it's associated data when initialization fails. Since cleanup function of MSI is implemented, delete the open-code in vpci_deassign_device(). Signed-off-by: Jiqian Chen --- cc: "Roger Pau Monn=C3=A9" --- v11->v12 changes: * In cleanup_msi(), move "if ( !hide )" above vpci_remove_registers() since deassign device will do removing registers itself. * Read address64 and mask info from hardware since they are not reliable when init_msi fails. v10->v11 changes: * Add hide paratemer to cleanup_msi(). * Check hide, if false return directly instead of letting ctrl RO. * Delete xfree(pdev->vpci->msi); in vpci_deassign_device(). * Remove Roger's Reviewed-by since patch change. v9->v10 changes: No. v8->v9 changes: * Add Roger's Reviewed-by. v7->v8 changes: * Add a comment to describe why "-2" in cleanup_msi(). * Given the code in vpci_remove_registers() an error in the removal of registers would likely imply memory corruption, at which point it's best to fully disable the device. So, Rollback the last two modifications= of v7. v6->v7 changes: * Change the pointer parameter of cleanup_msi() to be const. * When vpci_remove_registers() in cleanup_msi() fails, not to return directly, instead try to free msi and re-add ctrl handler. * Pass pdev->vpci into vpci_add_register() instead of pdev->vpci->msi in init_msi() since we need that every handler realize that msi is NULL when msi is free but handlers are still in there. v5->v6 changes: No. v4->v5 changes: * Change definition "static void cleanup_msi" to "static int cf_check clean= up_msi" since cleanup hook is changed to be int. * Add a read-only register for MSI Control Register in the end of cleanup_m= si. v3->v4 changes: * Change function name from fini_msi() to cleanup_msi(). * Remove unnecessary comment. * Change to use XFREE to free vpci->msi. v2->v3 changes: * Remove all fail path, and use fini_msi() hook instead. * Change the method to calculating the size of msi registers. v1->v2 changes: * Added a new function fini_msi to free all MSI resources instead of using = an array to record registered registers. Best regards, Jiqian Chen. --- xen/drivers/vpci/msi.c | 55 ++++++++++++++++++++++++++++++++++++++++- xen/drivers/vpci/vpci.c | 1 - 2 files changed, 54 insertions(+), 2 deletions(-) diff --git a/xen/drivers/vpci/msi.c b/xen/drivers/vpci/msi.c index c3eba4e14870..181ec902dffb 100644 --- a/xen/drivers/vpci/msi.c +++ b/xen/drivers/vpci/msi.c @@ -193,6 +193,59 @@ static void cf_check mask_write( msi->mask =3D val; } =20 +static int cf_check cleanup_msi(const struct pci_dev *pdev, bool hide) +{ + int rc; + unsigned int end; + struct vpci *vpci =3D pdev->vpci; + const unsigned int msi_pos =3D pdev->msi_pos; + const unsigned int ctrl =3D msi_control_reg(msi_pos); + + if ( !hide ) + { + XFREE(vpci->msi); + return 0; + } + + if ( vpci->msi ) + { + uint16_t control =3D pci_conf_read16(pdev->sbdf, ctrl); + bool address64 =3D is_64bit_address(control); + + if ( is_mask_bit_support(control) ) + end =3D msi_pending_bits_reg(msi_pos, address64); + else + /* + * "-2" here is to cut the reserved 2 bytes of Message Data when + * there is no masking support. + */ + end =3D msi_mask_bits_reg(msi_pos, address64) - 2; + + rc =3D vpci_remove_registers(vpci, ctrl, end - ctrl); + if ( rc ) + { + printk(XENLOG_ERR "%pd %pp: fail to remove MSI handlers rc=3D%= d\n", + pdev->domain, &pdev->sbdf, rc); + ASSERT_UNREACHABLE(); + return rc; + } + + XFREE(vpci->msi); + } + + /* + * The driver may not traverse the capability list and think device + * supports MSI by default. So here let the control register of MSI + * be Read-Only is to ensure MSI disabled. + */ + rc =3D vpci_add_register(vpci, vpci_hw_read16, NULL, ctrl, 2, NULL); + if ( rc ) + printk(XENLOG_ERR "%pd %pp: fail to add MSI ctrl handler rc=3D%d\n= ", + pdev->domain, &pdev->sbdf, rc); + + return rc; +} + static int cf_check init_msi(struct pci_dev *pdev) { unsigned int pos =3D pdev->msi_pos; @@ -270,7 +323,7 @@ static int cf_check init_msi(struct pci_dev *pdev) =20 return 0; } -REGISTER_VPCI_CAP(MSI, init_msi, NULL); +REGISTER_VPCI_CAP(MSI, init_msi, cleanup_msi); =20 void vpci_dump_msi(void) { diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 07c7071d0a17..7aaf015f63d4 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -368,7 +368,6 @@ void vpci_deassign_device(struct pci_dev *pdev) rangeset_destroy(pdev->vpci->header.bars[i].mem); =20 xfree(pdev->vpci->msix); - xfree(pdev->vpci->msi); xfree(pdev->vpci); pdev->vpci =3D NULL; } --=20 2.34.1 From nobody Sat Oct 4 22:04:04 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass(p=quarantine dis=none) header.from=amd.com ARC-Seal: i=2; a=rsa-sha256; t=1758873073; cv=pass; d=zohomail.com; s=zohoarc; b=FyPldof626cjmMWSuBOMNT185GA9JncF6LvmP7RiggdJo9s6uD8/v4NwppaYJOTRlA03aEiGjkRtdLUe6jntqPtEzFFhP2Vy9B7RiUX7u7Dm68GhiVilIJj0dSDgQR2hXShJo9T+C4bRt10N1+cUqEsO4sfSCjGVPzPXV8iNqJ4= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1758873073; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=MaLJ7+pe1aPBvQLtTo9+A9wNYi42Nr55akKCwiM69gw=; b=Rqb7lfwPKfqux9zxYvfkAILGSWNbK8LJofhwW4T+SqLax++M4Ogw9tfyAMFTJUDNAyoEIm5cwzXch6vZdefL1Tp3hBRw8Dda7dUAA/hPrZSSHICgxQxTl/4ZawNrb271WdegzBza1PjooX2ofdOazL7smuErLmHiEShjevoNVf4= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=amd.com); dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1758873073044293.32164570190037; Fri, 26 Sep 2025 00:51:13 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1131216.1470418 (Exim 4.92) (envelope-from ) id 1v23EC-0004u8-N9; Fri, 26 Sep 2025 07:50:52 +0000 Received: by outflank-mailman (output) from mailman id 1131216.1470418; Fri, 26 Sep 2025 07:50:52 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v23EC-0004u1-JO; Fri, 26 Sep 2025 07:50:52 +0000 Received: by outflank-mailman (input) for mailman id 1131216; Fri, 26 Sep 2025 07:50:50 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v23EA-0004PV-IS for xen-devel@lists.xenproject.org; Fri, 26 Sep 2025 07:50:50 +0000 Received: from SN4PR0501CU005.outbound.protection.outlook.com (mail-southcentralusazlp170110003.outbound.protection.outlook.com [2a01:111:f403:c10d::3]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 842b5f11-9aad-11f0-9d14-b5c5bf9af7f9; Fri, 26 Sep 2025 09:50:49 +0200 (CEST) Received: from CH2PR05CA0004.namprd05.prod.outlook.com (2603:10b6:610::17) by LV3PR12MB9403.namprd12.prod.outlook.com (2603:10b6:408:217::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9160.13; Fri, 26 Sep 2025 07:50:44 +0000 Received: from DS2PEPF0000343C.namprd02.prod.outlook.com (2603:10b6:610:0:cafe::ea) by CH2PR05CA0004.outlook.office365.com (2603:10b6:610::17) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9182.4 via Frontend Transport; Fri, 26 Sep 2025 07:50:44 +0000 Received: from satlexmb07.amd.com (165.204.84.17) by DS2PEPF0000343C.mail.protection.outlook.com (10.167.18.39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9160.9 via Frontend Transport; Fri, 26 Sep 2025 07:50:44 +0000 Received: from cjq-desktop.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Fri, 26 Sep 2025 00:50:42 -0700 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 842b5f11-9aad-11f0-9d14-b5c5bf9af7f9 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=cr21t+ej57ajwAedNluHt0xaBz6MH4WAIwDAo5ckvOEdEwUXkfoiAs3N2tRpLcc8RGxIeoVjFyT/VtTj4iydiCHrPl0GdpBw2E28a8gTWkHl56X/KP8E7MJe4CBtxIDlWkRxEQTecjsmO86Fjs6QaMGheuiU3rK5RXGH4kiLAQJszsl/ExTFGJwR/6+AhSGXrDUjUUvScxW+YXF3P+ImzgNhBbsag14+Y8xbOMqLnayFgTg7DpweXjJ7Dx67h1Sm4MoY2pka7zXAf4hZc+wdc2iN6k7ApgnxU9qUBE69Cjxby5AKeR3aNuGg38G8PicS9v2RXqUvxd2m0BLTdXqQBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MaLJ7+pe1aPBvQLtTo9+A9wNYi42Nr55akKCwiM69gw=; b=kU+T8MNkYUOHUcjmIKkp65H2yE1dk+1aTTctSSNvF4yTIOezAMHOHXBUP3XpPrP0qt9vlN1bh8BWrkBEl3YfbOAT59icdPKFxRQhfaGM0mh23eJSTJ1EncTICFyBFMAke3avYEVW77DAdJH2mtV0w0jMf2arWqPB0Dj5X1oMHI+GnRmC9OGnZjg9e1ovt8xr8q54+pL3QLP2q1H++yD02UzzbHpXq9ZfT14D7w3ZBjjaeyvbZHJkGXUbEiqNuMg/6u8TvKnBKtqiHeIhBV24Reb/JmJWx4Z1D2dLCe/m5PRoGloIdQ38R8VlyVJaQLaMC/uPQFUXX5u6jAY9ESbKnQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MaLJ7+pe1aPBvQLtTo9+A9wNYi42Nr55akKCwiM69gw=; b=amhUsKrADTz3gJKQHLszK5Ba6/Bdi+eEb+iWNKaZDrVfMTAkfPFP8/Gvs+5iCsWWbbouoJX6hdT3F5AAYw+MX2zWBYgu+yhfFIKA0JfjQ7HWESq+CQwKmCnOHuRLi7Fo/5Gyn5tSEvLUJ9WkkumhykEHW5/4OY+ibYHq9/BeTts= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C From: Jiqian Chen To: CC: Huang Rui , Jiqian Chen , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= Subject: [PATCH v12 3/3] vpci/msix: Implement cleanup function for MSI-X Date: Fri, 26 Sep 2025 15:50:21 +0800 Message-ID: <20250926075021.27967-4-Jiqian.Chen@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250926075021.27967-1-Jiqian.Chen@amd.com> References: <20250926075021.27967-1-Jiqian.Chen@amd.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PEPF0000343C:EE_|LV3PR12MB9403:EE_ X-MS-Office365-Filtering-Correlation-Id: 4e354947-da3b-4df5-df94-08ddfcd165b5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|36860700013|376014; X-Microsoft-Antispam-Message-Info: =?utf-8?B?aXdGTThpSGpYNG1oR0g1NUdJMzgzZlUrMXZML2dsbkhoSVc1aG5DMjhBWVlG?= =?utf-8?B?T25rNWdUVkN3dTBuQUU4bytJWmx0czZITHNkQTBGNEROaUdpbDZ0NXhJNTNZ?= =?utf-8?B?NWFwb0RpS053ZWN4S1h2SXRxNzJVNmpicnhaSnV4MGRsNjUwbVZReDlTOFlr?= =?utf-8?B?U2NYaUtoNExHbGJrZ24rTnM1Z2tPbEVPeDZ2Mjd0czNEeVBPbGxSR3AwM1RB?= =?utf-8?B?N2FxUDBaRGl1NVBOQVpSVE04NDlXcUtyZS9maFF4Z2plejRiSytaaWE5MG5w?= =?utf-8?B?WTNldnhDQkk1SDRsWXVmaUpybDEzU1VkcW9QZWRFd29UeWhyUDg4dGZEL0Ur?= =?utf-8?B?TjNKL1NqcFRHeXVJVmV1cTlIaXVhOTc5QzNIK20xbUJuWmNoQ3dDb05qZFhR?= =?utf-8?B?SWpycE42eVRKZk1nTGRGSGxlMzgvd2JWMnoxYnZML0pPZlJHY3VPcjM1ZzNI?= =?utf-8?B?aVRrTFAzVUVyWmRBRkZwd2tyZFNDRWFqQXA5Y2NNQ1ZZb1QwNzJaWVVHQWlQ?= =?utf-8?B?UFN6Qjl6b1krYklXSG5EY0N5VTJmWlZkYTVNVXVwSGJyZERXeGQzQ2ZrNWhZ?= =?utf-8?B?dExjY3RKdkIwYmhldnNXYVFHcDA1WndkamVhcDlTWXFUUHJrRVptRG5kQk1G?= =?utf-8?B?c2RFSy9RbEovMU9oaGxBV3BNY2htZGVuc000Ly96aFQ3U3JPMDR2dFFlRG52?= =?utf-8?B?YU05NmRRdEVKTEZ4VVQzc0doTkRKekhYTnBwWW5jOENaaFNxeHcwc05UUCtT?= =?utf-8?B?K2FBanlhRThWaGpkUVJRSGNGMlJFWEZmQTNIQ1VhWWlNQit3MzAwbzdGSDRF?= =?utf-8?B?R3poakNVQmFoNkp5MEJad3duc2Q2NGRtcWJtZDJzaFN6OEZrTzFpamkraHBi?= =?utf-8?B?MmRHV3dIbFY1TzRyaDhnVGZqYnlVcTNuNmRzYmN1TmpSbDdHNVhXd1Q0T1Fa?= =?utf-8?B?ck9SY0JkcENaV2ZTT2dZTzNNd0RpVmFEM0V5T2dDZ2VHQXJkL3JPY1l6TjQv?= =?utf-8?B?NG9pYlkzVThNSXJPd1BqMVFiV0xBSlBzOGJRVnlncVQ5am9USWFCaGRPeGx4?= =?utf-8?B?WEt1OVV6c1YvWkltV1J3d2FNRWVlcDhaVTdjaC9qcS9jbnlERlBESkpzMkFj?= =?utf-8?B?Y2ltN01PVVJJNlR2KzAveUJlcGh3a3hmdzRkeVFuL2ZIb3NGaDd5VEdUUklx?= =?utf-8?B?Y01pQ1M2eUdSckwwa0EzRWdGU3FyT3J5bEIrUWZsR0szRVZlMU90alhqWWZN?= =?utf-8?B?V3JCdG4yZWF1YmtadHNpT3BuNkxOQTJ2eTFpNkNGV3NpVkVtSnZ3T1hRQW1y?= =?utf-8?B?T0N6aGx1aTZtNHhWNGxQVVBZYW16QStZOURzbjlZdGxTalUzYzFMWHlXU05j?= =?utf-8?B?K0xURDZrVTEzTDF1MkR3aDVJNGlsNXlQMzNZd0luMkNaUTZ0NnJvUnJuVkVC?= =?utf-8?B?YWlZOHcva1ZFOURNTWRpeE9jZXc2QkEwNXJ2K09EYXRWd0oyWW9NQ2VOa25s?= =?utf-8?B?aHZTZG90bkN3aTM5Q1NWODUzUzhsYnZ1Zm1mWVZGU1lOOWdZUFZwVjVpY0c0?= =?utf-8?B?WjY3dCtQRUJGckVpcWJWMkdldFZWZHk3TXpTTGc5MzdWQWhIVS9nWEtTMkpq?= =?utf-8?B?bkYzMFBwUmxreTVlMXFzbndtWnRldFd3dWVTbWlMckg4U0tVMDZVdWpTNXJ1?= =?utf-8?B?WHkwTFFOVW42NHdkZkdVcUVzdUdVTXdxaXEvY1B6TCt6dHFsRVZTWXRoeWQ2?= =?utf-8?B?OXprYUF4d21QU0JIaDRaR05UcUpzajNGcUJqYkNzUG00MCtNV0xETGd2Um5l?= =?utf-8?B?dDdveXJHYmxLWlFBdTZ2L0Nia3E3N2VPRy9nNkpFaWhESTZHVVZJVm05RzhU?= =?utf-8?B?NnZKbENHaTJLTmN6cjFMOHFjdXRLeloxWVZsd3FuYUNINTBRN1pWdHlHUDNr?= =?utf-8?B?bWg1aldsVm1NMUVibXd6STNWNzYzVFVMQldtcmVCYWFPaTNibnA2eUlza3lB?= =?utf-8?B?RHZCZmFhL0NZV1hzdEwrbnJpUnVNUGJRRFpmSUNIb01mWlVnMHBhRnVhU0tU?= =?utf-8?Q?2VrcqD?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(36860700013)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Sep 2025 07:50:44.0458 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4e354947-da3b-4df5-df94-08ddfcd165b5 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS2PEPF0000343C.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR12MB9403 X-ZohoMail-DKIM: pass (identity @amd.com) X-ZM-MESSAGEID: 1758873074614116600 When MSI-X initialization fails, vPCI hides the capability, but removing handlers and datas won't be performed until the device is deassigned. So, implement MSI-X cleanup hook that will be called to cleanup MSI-X related handlers and free it's associated data when initialization fails. Since cleanup function of MSI-X is implemented, delete the open-code in vpci_deassign_device(). Signed-off-by: Jiqian Chen --- cc: "Roger Pau Monn=C3=A9" --- v11->v12 changes: * In cleanup_msix(), move check "if ( !hide )" above vpci_remove_registers(= ). * Remove the check "!pdev->msix_pos" since current callers already do that. v10->v11 changes: * Move calling all cleanup hook in vpci_deassign_device() out of this patch. * Add hide parameter to cleanup_msix(). * Check hide, if it is false, return directly instead of letting ctrl RO. v9->v10 changes: * Call all cleanup hook in vpci_deassign_device() instead of cleanup_msix(). v8->v9 changes: * Modify commit message. * Call cleanup_msix() in vpci_deassign_device() to remove the open-code to = cleanup msix datas. * In cleanup_msix(), move "list_del(&vpci->msix->next);" above for loop of = iounmap msix tables. v7->v8 changes: * Given the code in vpci_remove_registers() an error in the removal of registers would likely imply memory corruption, at which point it's best to fully disable the device. So, Rollback the last two modifications= of v7. v6->v7 changes: * Change the pointer parameter of cleanup_msix() to be const. * When vpci_remove_registers() in cleanup_msix() fails, not to return directly, instead try to free msix and re-add ctrl handler. * Pass pdev->vpci into vpci_add_register() instead of pdev->vpci->msix in init_msix() since we need that every handler realize that msix is NULL when msix is freed but handlers are still in there. v5->v6 changes: * Change the logic to add dummy handler when !vpci->msix in cleanup_msix(). v4->v5 changes: * Change definition "static void cleanup_msix" to "static int cf_check clea= nup_msix" since cleanup hook is changed to be int. * Add a read-only register for MSIX Control Register in the end of cleanup_= msix(). v3->v4 changes: * Change function name from fini_msix() to cleanup_msix(). * Change to use XFREE to free vpci->msix. * In cleanup function, change the sequence of check and remove action accor= ding to init_msix(). v2->v3 changes: * Remove unnecessary clean operations in fini_msix(). v1->v2 changes: new patch. Best regards, Jiqian Chen. --- xen/drivers/vpci/msix.c | 44 ++++++++++++++++++++++++++++++++++++++++- xen/drivers/vpci/vpci.c | 8 -------- 2 files changed, 43 insertions(+), 9 deletions(-) diff --git a/xen/drivers/vpci/msix.c b/xen/drivers/vpci/msix.c index 54a5070733aa..38a14c0ded05 100644 --- a/xen/drivers/vpci/msix.c +++ b/xen/drivers/vpci/msix.c @@ -655,6 +655,48 @@ int vpci_make_msix_hole(const struct pci_dev *pdev) return 0; } =20 +static int cf_check cleanup_msix(const struct pci_dev *pdev, bool hide) +{ + int rc; + struct vpci *vpci =3D pdev->vpci; + const unsigned int msix_pos =3D pdev->msix_pos; + + if ( vpci->msix ) + { + list_del(&vpci->msix->next); + for ( unsigned int i =3D 0; i < ARRAY_SIZE(vpci->msix->table); i++= ) + if ( vpci->msix->table[i] ) + iounmap(vpci->msix->table[i]); + + XFREE(vpci->msix); + } + + if ( !hide ) + return 0; + + rc =3D vpci_remove_registers(vpci, msix_control_reg(msix_pos), 2); + if ( rc ) + { + printk(XENLOG_ERR "%pd %pp: fail to remove MSIX handlers rc=3D%d\n= ", + pdev->domain, &pdev->sbdf, rc); + ASSERT_UNREACHABLE(); + return rc; + } + + /* + * The driver may not traverse the capability list and think device + * supports MSIX by default. So here let the control register of MSIX + * be Read-Only is to ensure MSIX disabled. + */ + rc =3D vpci_add_register(vpci, vpci_hw_read16, NULL, + msix_control_reg(msix_pos), 2, NULL); + if ( rc ) + printk(XENLOG_ERR "%pd %pp: fail to add MSIX ctrl handler rc=3D%d\= n", + pdev->domain, &pdev->sbdf, rc); + + return rc; +} + static int cf_check init_msix(struct pci_dev *pdev) { struct domain *d =3D pdev->domain; @@ -710,7 +752,7 @@ static int cf_check init_msix(struct pci_dev *pdev) */ return vpci_make_msix_hole(pdev); } -REGISTER_VPCI_CAP(MSIX, init_msix, NULL); +REGISTER_VPCI_CAP(MSIX, init_msix, cleanup_msix); =20 /* * Local variables: diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 7aaf015f63d4..3c9bebcbe977 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -356,18 +356,10 @@ void vpci_deassign_device(struct pci_dev *pdev) xfree(r); } spin_unlock(&pdev->vpci->lock); - if ( pdev->vpci->msix ) - { - list_del(&pdev->vpci->msix->next); - for ( i =3D 0; i < ARRAY_SIZE(pdev->vpci->msix->table); i++ ) - if ( pdev->vpci->msix->table[i] ) - iounmap(pdev->vpci->msix->table[i]); - } =20 for ( i =3D 0; i < ARRAY_SIZE(pdev->vpci->header.bars); i++ ) rangeset_destroy(pdev->vpci->header.bars[i].mem); =20 - xfree(pdev->vpci->msix); xfree(pdev->vpci); pdev->vpci =3D NULL; } --=20 2.34.1