From nobody Mon May 13 07:44:14 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1711090142; cv=pass; d=zohomail.com; s=zohoarc; b=KZDoKIqKdNPyG18Ttpzf37E3QYwW3e7kjLEo4350R/p6n3ea3kLU2YcrUuMxZWpusbzCzvli5yHcapvlKwJu7kwHhP0UF+ySbqDNYd+GG8q+z07VgUwpKcHzE4NrZc9FsBOxDNLFNE846h2lm2huavdaodbqL8DlGItRLoWFIf0= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1711090142; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=mbbxXrcwvRYtjYm83Z3muw0qSHQbsXQVykRIti2Lau0=; b=a8Buf21ICaT5ONSDriVOH19p1xgza/beHms12PvZvpnS3LuQPA5SadagsanOvw7s7IhtHn+EE5Y/u6OHBXeAt5Y6OtyiSxuhcBWknOXU/K/oxd4AEuHkpVKeyBIp26V9BpINvE8DGgifswrXj/IAGMtlLtqNoy15dGXSO55aO6Y= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1711090142379209.28750221371934; Thu, 21 Mar 2024 23:49:02 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rnYh5-0003oW-G5; Fri, 22 Mar 2024 02:47:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rnYh2-0003oH-DH for qemu-devel@nongnu.org; Fri, 22 Mar 2024 02:47:57 -0400 Received: from mail-bn8nam12on2068.outbound.protection.outlook.com ([40.107.237.68] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rnYgx-00013S-9d for qemu-devel@nongnu.org; Fri, 22 Mar 2024 02:47:54 -0400 Received: from SJ0PR03CA0107.namprd03.prod.outlook.com (2603:10b6:a03:333::22) by CH3PR12MB8259.namprd12.prod.outlook.com (2603:10b6:610:124::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7386.31; Fri, 22 Mar 2024 06:42:44 +0000 Received: from SJ1PEPF00001CE7.namprd03.prod.outlook.com (2603:10b6:a03:333:cafe::99) by SJ0PR03CA0107.outlook.office365.com (2603:10b6:a03:333::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7386.28 via Frontend Transport; Fri, 22 Mar 2024 06:42:43 +0000 Received: from mail.nvidia.com (216.228.118.232) by SJ1PEPF00001CE7.mail.protection.outlook.com (10.167.242.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.10 via Frontend Transport; Fri, 22 Mar 2024 06:42:43 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 21 Mar 2024 23:42:33 -0700 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.12; Thu, 21 Mar 2024 23:42:33 -0700 Received: from vkale-dev-linux.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.12 via Frontend Transport; Thu, 21 Mar 2024 23:42:30 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KLMGoXvbme2xrZQMR8SqZW4mWTzFcdh/wxMtb5SKSIzED00mArQwd+XYJGTDqap6unEOFrx0fj+Jy8RykhGbNMy0sv8VPrzq8gwLjAAN3LRp9vDt1mc7iznWz8ZYrOrG4yHihtXMiUpLy6LGxt4kMBlKAdyylXhUhcbWydgJK4F4dwNRrle/eUONc2sjpq6/jG3mqA1QOYLY5c6JWCKjoHADkl5MZj3CeuyuJVO0LL6UtfAPjdyK4LDOlf6RRbZVZDAjVRlfZeG+wmOXzJokf/5092tA2qM7MKFrY/JmV7mfktaSOV2duyvUbFSyEJC2kdv/kxwgT/VkqxsNgDubUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mbbxXrcwvRYtjYm83Z3muw0qSHQbsXQVykRIti2Lau0=; b=B6kX0L9B4y0f61KUmTt8w63ppQkFhIdMQ59cPyoIcH7xVHOKob/vqL7fSmcUb8svyFxmxRQyIFLAECoHc2Lf6pPWioUTUKNnBgKHCmd6gzwY1nWjuUtuz3bfRl+cDfVcvO+NjvqfBTzmm4RCRTLeswbp77JQsYgIGdoBi1cit369sP/TAQ87/7k3X21u5P4+MBrhC/V7QhhKrN0z2XHIjClGLe8+JIWPV0bIaWQYKFLSi49UA37GPERT8MMgEmZ4DFW/YkldCzN5loa80OMr0mSl0zh13j6aQrSrXE4o8y1ZIjg4+ID9PCQ70WmM9IwhWr9nTIrz8Ejn3rSfuZXMvA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mbbxXrcwvRYtjYm83Z3muw0qSHQbsXQVykRIti2Lau0=; b=HsAXL0Aa2R1xMBDY1zfOYDySpXENg5iojwDaY3A+VqvWyyhipINIYwWfQeHpLYcWObTGhZLBp5Pz+dTx0kWH7GOwhOsiOtxTPSSnBkr7g1abyAGegtl/exHR1VIKd/FEAZqncPzl3nc9gz8CBM5No1AYOVaOWKSPYu1sQRG7ho+gp4qIuuzUtlgQpT9iyfsppCKQ1LOlryqzI+LTTmjRwe4f0uFvwcW4azierQsXoDBbJ1S0yMeoe5VF3j4bYqcR80JjGCibZ62FVHr7vBBdWKEMAdmIQsgT9aXY/7hC+NiOTmW3rnqn3YfIjPDaxnwLNOeXlqa8DmiCcr6OARJRzQ== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C From: Vinayak Kale To: CC: , , , , , , , , , Vinayak Kale Subject: [PATCH v3] vfio/pci: migration: Skip config space check for Vendor Specific Information in VSC during restore/load Date: Fri, 22 Mar 2024 12:12:10 +0530 Message-ID: <20240322064210.1520394-1-vkale@nvidia.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-NVConfidentiality: public Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF00001CE7:EE_|CH3PR12MB8259:EE_ X-MS-Office365-Filtering-Correlation-Id: 21273839-0d7d-42f0-43f7-08dc4a3b46fc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ePcYB+IfMUqbPjnGOa8R/WF2Vul0nIOPbEZ8dU2D5kbImBFxeBbr8KPGLKBhjKB/WNuIE8bbLMciCZjUhrdEfgoRvr/OyE5UM6O9xY9CNXGMxxWD5zPCGgsCLyQoXX+XI0jygwvCgdUlAiPVc9HjGAaEXP6B+X2eco16ZkXQdg2BNEhzIriSlG/qdUvZwiyhoLhOCIRal7y+KGUZ71vD9XR9wrclEFOA6IHeiadC223ovt/dtj9apjYsR+4H5SMRn7cZsfZVKGS0zaIpbbM+x2iF1luj3SrYJz7CaDBmbxaTOwFrM9ZYDoMGmUe573gGBYC/btaLBHvvvrYa1QzpiseBKNFNKQiy1okUrdYZCjQWWfLSYxrp3Z4cR7tk1gKC+bDe+R+bb98hEtvG/iQV+Bt0d7sdYgNwBPm3ipEXOBPbYT1LHYJ2kqfUMN1G9jKH6JKKM5S7w6BgLhRadE+k2Qs+On16T4yh2ysf9fgPJDlZsuG9ccmJy3nNQCwEW8uZCAQrq4O8+mGqS8vo3g4QQyAnQgPYGuOM0FUS1p3MX9fpFC/McYLRLwCOa/v47p6nlRiZO4n3Req3/LSeP5Ri9PrPFC2xCBKaxPPDL1Hlv2oxNpCOWijGG+JCaikRyMI/c8l/jhTR1ytXGvTu5jxaW0ylu/AbYZDbF/qOyArEMHwr+J1IGhE15S/Kx3LvAeFfuwtHEVeO08EKplY9j/0AZfyNmhYETUS2NzxFA+26wYH9+B9evskLzv0AwdgWrDVO X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230031)(82310400014)(376005)(36860700004)(1800799015); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2024 06:42:43.3918 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 21273839-0d7d-42f0-43f7-08dc4a3b46fc X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF00001CE7.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8259 Received-SPF: softfail client-ip=40.107.237.68; envelope-from=vkale@nvidia.com; helo=NAM12-BN8-obe.outbound.protection.outlook.com X-Spam_score_int: -24 X-Spam_score: -2.5 X-Spam_bar: -- X-Spam_report: (-2.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.372, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1711090144522100003 Content-Type: text/plain; charset="utf-8" In case of migration, during restore operation, qemu checks config space of= the pci device with the config space in the migration stream captured during sa= ve operation. In case of config space data mismatch, restore operation is fail= ed. config space check is done in function get_pci_config_device(). By default = VSC (vendor-specific-capability) in config space is checked. Due to qemu's config space check for VSC, live migration is broken across N= VIDIA vGPU devices in situation where source and destination host driver is diffe= rent. In this situation, Vendor Specific Information in VSC varies on the destina= tion to ensure vGPU feature capabilities exposed to the guest driver are compati= ble with destination host. If a vfio-pci device is migration capable and vfio-pci vendor driver is OK = with volatile Vendor Specific Info in VSC then qemu should exempt config space c= heck for Vendor Specific Info. It is vendor driver's responsibility to ensure th= at VSC is consistent across migration. Here consistency could mean that VSC fo= rmat should be same on source and destination, however actual Vendor Specific In= fo may not be byte-to-byte identical. This patch skips the check for Vendor Specific Information in VSC for VFIO-= PCI device by clearing pdev->cmask[] offsets. Config space check is still enfor= ced for 3 byte VSC header. If cmask[] is not set for an offset, then qemu skips config space check for that offset. Signed-off-by: Vinayak Kale Acked-by: Alex Williamson Reviewed-by: Michael S. Tsirkin --- Version History v2->v3: - Config space check skipped only for Vendor Specific Info in VSC, chec= k is still enforced for 3 byte VSC header. - Updated commit description with live migration failure scenario. v1->v2: - Limited scope of change to vfio-pci devices instead of all pci device= s. hw/vfio/pci.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index d7fe06715c..1026cdba18 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2132,6 +2132,27 @@ static void vfio_check_af_flr(VFIOPCIDevice *vdev, u= int8_t pos) } } =20 +static int vfio_add_vendor_specific_cap(VFIOPCIDevice *vdev, int pos, + uint8_t size, Error **errp) +{ + PCIDevice *pdev =3D &vdev->pdev; + + pos =3D pci_add_capability(pdev, PCI_CAP_ID_VNDR, pos, size, errp); + if (pos < 0) { + return pos; + } + + /* + * Exempt config space check for Vendor Specific Information during re= store/load. + * Config space check is still enforced for 3 byte VSC header. + */ + if (size > 3) { + memset(pdev->cmask + pos + 3, 0, size - 3); + } + + return pos; +} + static int vfio_add_std_cap(VFIOPCIDevice *vdev, uint8_t pos, Error **errp) { PCIDevice *pdev =3D &vdev->pdev; @@ -2199,6 +2220,9 @@ static int vfio_add_std_cap(VFIOPCIDevice *vdev, uint= 8_t pos, Error **errp) vfio_check_af_flr(vdev, pos); ret =3D pci_add_capability(pdev, cap_id, pos, size, errp); break; + case PCI_CAP_ID_VNDR: + ret =3D vfio_add_vendor_specific_cap(vdev, pos, size, errp); + break; default: ret =3D pci_add_capability(pdev, cap_id, pos, size, errp); break; --=20 2.34.1