From nobody Sun Apr 5 18:20:02 2026 Received: from DM1PR04CU001.outbound.protection.outlook.com (mail-centralusazon11010068.outbound.protection.outlook.com [52.101.61.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6311A2EDD7D; Mon, 23 Feb 2026 15:55:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.61.68 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862137; cv=fail; b=oduGI3o6nxYzxK9rGm3cW1wn86v6Hj3OVeQTZIlB/VdXXKozfZM0oVD7F3GlqvTwn26Mqhh58U6uhiJhBQlZpHi1AOYhQln7MtPYafgGL1eF8Iczd834rGo6D+Zs04xcjxfmYEOD5DbEWC3qH8k1R9I2GjxuB9uLQEw/Me7xnFE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862137; c=relaxed/simple; bh=QjwWbBzDV0jfkjPkTHLvn6f6lLy2egMzHRkJmWO+qnE=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=d3H9JokP55Rjwl3/1C8AfBMEw2pnhrblN+nGG45LIb6s68BGFp1duaQ2gSyzHuNRHWb40fmpK1y94ftCC25x6iu3KEDqomAql94Kacldl+RGMGjlePL5wxBfAsbJs36KklDofReMLeBlBPycs1RXcPiEKdKWRwJOgWzwvzS2cyQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=D9lxcf8/; arc=fail smtp.client-ip=52.101.61.68 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="D9lxcf8/" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ujtj5G6Y3iUb3KetNXtfItehcA7EThwNOlw3PRaQSDh0OmmDF7+EqgtDYRzSOXTLqC3UsXdCN0xbXKUUzBiNko6ly0LR/khEvLVhoHaJmDiGJUF9saNBPBfpCIh79pXOuw/3beBPBe8IJIVsx3GJvzMt4kLNqi6bgU8ttcCAIDIGAurUBfcoGlh7Czb3K+NBsU2tGclsKvKIIzHd5IXtonPgTg35ld0co7ZO/kEYFb+a5k9AKd3FZBHwP79FMsRtlzfEeJ3nNSls+cc9UeG7Vn9Cqc3fcT8SmG2jqUJDYoXgCsavtJekRPSHP8W4RTEFk7W3TLhuv1FmltxAN6GXWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rvsmo3F4hPWSLCxKrNqc6n4M99BGnkzxCoeujR3MgRc=; b=TYSDVHCdUMRNH18zzNb0ahVDHurH/7133DUllWZdXwDYLx32/kHlEbxEt0M4+dpB+kkvHA6wXQNOzhDPKqt327PHHWJR2kIcaIQ3y3KauLWs22UNEbskOucU1Z5OzRoUaUqsvBAeAtMNoH6XAYB7Oj5Om68g7nGIKaXodevII/tNIrrIFBynDiUJWEaAACUR5zyXRIpZvuZYsnpvB01jYg5bGFuHI12rOsbcONwzbjcQvTzFBsRmEXRVVnyuL2NOrq1HJCRx2bN3PZqwfDkftTroyQqRICl2XjOmTGBThNjhyAx/i3YVi332h0oWorXLa1uEqNY5DwgQBOoR/axxIA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rvsmo3F4hPWSLCxKrNqc6n4M99BGnkzxCoeujR3MgRc=; b=D9lxcf8/orUPQvr5lpTTGDGZ1xR9J3YdzC1o7Tg++9u9iwmxl5BmkTMvepbp//DFwddtn9mzd/8nB6oMUcM8eQXjbw5k8HuznpU+6+Gqfr5Jln+us3nSwK55M3UiPeP18mNGLyo5shMCZ9Jj6XnsBxKuJhzHmEFG3UGoVERGgQm8X2rA4/JmVrau0c65pCKZA1FsfWEke4TTv89CFU9oqZf/AYuu5BduV+WEa49GY4EIXkLvVfEkSf5N2cYJev1ykLbuk/vJ+cYiSC/a9G/McaQn/n+wCc+3Nx8AZIqawTjA8hIIm3+4z1wu1stC3cPgElAXKcWIXem15hHZLG9ZGQ== Received: from DM6PR14CA0046.namprd14.prod.outlook.com (2603:10b6:5:18f::23) by IA0PR12MB8302.namprd12.prod.outlook.com (2603:10b6:208:40f::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.21; Mon, 23 Feb 2026 15:55:32 +0000 Received: from DS3PEPF0000C37A.namprd04.prod.outlook.com (2603:10b6:5:18f:cafe::c4) by DM6PR14CA0046.outlook.office365.com (2603:10b6:5:18f::23) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9632.21 via Frontend Transport; Mon, 23 Feb 2026 15:55:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by DS3PEPF0000C37A.mail.protection.outlook.com (10.167.23.4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.12 via Frontend Transport; Mon, 23 Feb 2026 15:55:32 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:15 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:14 -0800 Received: from localhost.nvidia.com (10.127.8.12) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Mon, 23 Feb 2026 07:55:14 -0800 From: To: , , , , , , CC: , , , , , , Subject: [PATCH RFC v2 01/15] vfio/nvgrace-gpu: Expand module_pci_driver to allow custom module init Date: Mon, 23 Feb 2026 15:55:00 +0000 Message-ID: <20260223155514.152435-2-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260223155514.152435-1-ankita@nvidia.com> References: <20260223155514.152435-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF0000C37A:EE_|IA0PR12MB8302:EE_ X-MS-Office365-Filtering-Correlation-Id: d22c1ea9-d190-45c8-cc71-08de72f3f98e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?iJZbGftxDxkUHlx+0j6UItEJwswH9EqciCkYfGmfC0nDJi3a9WaAhI9EdUro?= =?us-ascii?Q?yMwsLbcfb+vXzUC6ljuDvL01XOx/W3rDX3BY8rHXNQVINrP1U4skkRXTk/NL?= =?us-ascii?Q?oObbGS5MZueHkhj+bGVmDg2rTNybM9lOp5FqDJt0/jJLmslyJp+ww/xUE616?= =?us-ascii?Q?qZQU2FzJE2Jggf0Mc5nh+OJv2c/lFR+aP0OBoqose1z+1RP1JZRnfhUWQRbl?= =?us-ascii?Q?ahS6o0Y0PfQJ2OjuKxAie1X5lSafaLcJSY4q6C3nMSpTqyycLb3tuoWXzYX8?= =?us-ascii?Q?BhoZemVygHgRjgZ8L6sqB6nvifoB7j4oxXoE3vZkt0tBjrxNg5rXmMfqFSQ0?= =?us-ascii?Q?f69R3/+sUxwSzH2OyoBrNIpdJmyaTfGNMUi5+XveAfIyIcn+wJQleEom+PbU?= =?us-ascii?Q?zqQ6fI5VqfwwjI7CuGv+ji0mvAwtr+OcrNSHR9yiDnvsIWvbjPpwhKxkCZqa?= =?us-ascii?Q?HthXX8nzqFIvC++Qha9vx+mownTtYgVTtrMy3udTwbR1eZG5vKupe85NfD1c?= =?us-ascii?Q?afjd+C8BjeTRuv0jW1hKaMOTAQe7ta9qU2eT7APR0mpFS0rzi3Sc6x2n2wkK?= =?us-ascii?Q?kMIIf33Ba3rQuXq0dNZs+Kv/NOdFIvd3aORE+B1Nbe5a4cwtjZOy1jFs14nb?= =?us-ascii?Q?q5XlgBuEE/KkcDCR+z7wXPN9qgX+9jRmx7L1H8zUSDmivE7bUW31HNhRJ/XY?= =?us-ascii?Q?QFi0K4GnAdyaokfZfE+dqCplpmoLqvrrRl1y9WruLGtOCn4EhCXZClRptEN8?= =?us-ascii?Q?ozvqX9vBmmkb7Ibw10tdOhR5Mno7AOdAu2EgviT+l18od7tvRmWaocWzyJMV?= =?us-ascii?Q?i2qcsW5LFSiZ/mzliUsyZxxHulke5qPZ3Xx5gJy4lTnYImVLWgCTBFUZC3PT?= =?us-ascii?Q?p92xHSrG0uvGgoxVYFhuphTDlWgtIx20jlmtVjEA5TJJwefZKlx1YBWFyB5d?= =?us-ascii?Q?FrePGrai4C48ePN2JLonIf0ZmjQzSED4vkuSSrgUgJnZF9Mc1mCwJDZb1T9Y?= =?us-ascii?Q?bhGl4p4nop/moOcQG8OUfNz9PE4tWarHCeayvr/1MH48u/jSL/YF7zhDghIH?= =?us-ascii?Q?ULeAA9crdjzYuSn4SQI2PCZE2U9rSAXDnmmYlFVy0XSaJDz66WPKv8bJJSZ8?= =?us-ascii?Q?+z2BXqWoAKWR4ntQpz/PWB7DY8bvdsBtel8TK/4xkpC5ZGLXxUkJhTLfexbG?= =?us-ascii?Q?qzMYb+rGOOExlUPMGN8lztG5pignJDyXsw6Cj9vG3XV3JZtWKz+BpLOQUzzN?= =?us-ascii?Q?tUHhOTCYPgOwWpr7UVbYU4PrYNH8XGJeiWWktU0iVSu+sA0yijjW+kNazn5X?= =?us-ascii?Q?55pT5S2T0+9tc9Rbui2e4cpiCQLQLbyBDuFDzddJj/rCe74GMzzah4s3IeEe?= =?us-ascii?Q?aHrHkWKUWExlCmITQmFaIClo8v9I8bWEjdcKnqfhC0dFGPGyO9mPJfoZ/9tV?= =?us-ascii?Q?5potw9VaIKSGa3KwrSDYi48izXwfvWmOTCjJC8WG7wNMDm4pf60Rj2N5FhrF?= =?us-ascii?Q?vNauKZ93R26sogkVk1buPw7dq76Su2A4AMnk/CldG6d8xutot5xv08izC7ZO?= =?us-ascii?Q?NwBMAJfspvTjbdYJODNmDgPYO4gKnOG/c1hc+0c79A6v1GWhUcL0iB4JhBlx?= =?us-ascii?Q?QylegT0urbWuFrkayItIvCAK49GPvM9BuRfCa5cbDTpgquwsTZXXOhnC2rFT?= =?us-ascii?Q?+GQIuw=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(376014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 9qqkJWzp56Fi96+gw824T0IzOgdwTwv1jkbqEg9vGA8kTqlzFTqUe3RV0BvvDFq4L2rJlqUH+WrqSCBIjiExZIR+MQxRy6o0+nXSV7bG4WPY0Gx0jjsIPhR+jGnEC71t6jbXoQ1fq6FYvm/IkBC46A0deRLpOAI76C6FqpJCnTRzXO8xrnPaEGooNUuL3hy4HtFB+7GKEI4IwiZ9Cpt+sYNLXdPN27hoZuQyzjyeWBY0EUtLs5CB14Kpn60kym+GPTSwP5d0L+z0KuCKxo39sXHxHJ5hoRtwTzV61XTbQQ4FamwOkaR4NohMzykT7vxvLw0ih5a+hf19zqjjbBNaKKmmB9AQi4XUELBFlLwDw++mUBYOruaQH83CvcJnP7uf3iNJ4jc9q+TycJgBTvvSyVipSIBj8A+B22oil28vpF/Bs2tU2vk0PKWTXq7sS0wu X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2026 15:55:32.0211 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d22c1ea9-d190-45c8-cc71-08de72f3f98e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF0000C37A.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8302 Content-Type: text/plain; charset="utf-8" From: Ankit Agrawal Allow custom changes to the nvgrace-gpu module init functions by expanding definition of module_pci_driver. Signed-off-by: Ankit Agrawal --- drivers/vfio/pci/nvgrace-gpu/main.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace= -gpu/main.c index d3e5fee29180..7c4d51f5c701 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -1287,7 +1287,17 @@ static struct pci_driver nvgrace_gpu_vfio_pci_driver= =3D { .driver_managed_dma =3D true, }; =20 -module_pci_driver(nvgrace_gpu_vfio_pci_driver); +static int __init nvgrace_gpu_vfio_pci_init(void) +{ + return pci_register_driver(&nvgrace_gpu_vfio_pci_driver); +} +module_init(nvgrace_gpu_vfio_pci_init); + +static void __exit nvgrace_gpu_vfio_pci_cleanup(void) +{ + pci_unregister_driver(&nvgrace_gpu_vfio_pci_driver); +} +module_exit(nvgrace_gpu_vfio_pci_cleanup); =20 MODULE_LICENSE("GPL"); MODULE_AUTHOR("Ankit Agrawal "); --=20 2.34.1 From nobody Sun Apr 5 18:20:02 2026 Received: from MW6PR02CU001.outbound.protection.outlook.com (mail-westus2azon11012004.outbound.protection.outlook.com [52.101.48.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C4D23043DB; Mon, 23 Feb 2026 15:55:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.48.4 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862144; cv=fail; b=sTsKoKV7wRMriJu+rN6HLMUhnFErJw/V3c/uc6hLIILiDd/tEpb2erRcZcxmMaOqC/6GGzPEjHRarFm5/dVZYo7niCrSG1WhdyQkFD7saYuV8UR6XtJ01qApXL6c5znXeme4e5KGsmj2wSVDkXhU0HGEtYZzFlMYg4QfmqcUc+Q= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862144; c=relaxed/simple; bh=uuWHGNSkAP94jR/fgsXeb9oI1YhjojJJgRJNLCGhess=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Bv7U9ESLiTUg1vyFLrDlHJMIX+P2eej/pOiNahegxlqJsT5ey/aqBGX/sBsYcSfKsmweljUSjUwE0XuxBI2vKLeR9YIzcIBuWAixdk6QDc7KCMsTHTHswhdZh+akNrqL3bwPYVBti+FNPB5cLu++jV/ET3+oS8f34ppEMtzb0O8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=FH2W0DUx; arc=fail smtp.client-ip=52.101.48.4 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="FH2W0DUx" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Qc/WsehQLKLGlEYMbBrgos30fjd8Nqqs2/WPH+WqLYnHSJ5XT+c7fba+B0yJU37+9enY3rTYandFfngR30uZCEPZHPZlh0Mey0L0dSmG0Kt5urNQRzmOzX+lR05qR/jaswwJHfeEkFTIfW4SfooPA9VA7GFZLKnrbIIx4d/ih1nB2qysSYGpcvC+Rqpc5ZteR3FBoDZQDJkn2Io9OZQri04Cmrlw31pl9brgt8xB9eExC4TIN76QvdgDDEpXv51hFJn0Mb8LIMGkZQ73FJjNyY4VtUzpdqDOdDVkRoqvPT3jrVQuSOFDlQA5jooxDyhJ88Xfk+TUS4nMol6VFskFuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1OfFU20yP01Dw8OSK9Q+I6UX8mB9z4UZamfT8Vu8x+A=; b=L8eav1agQLed00u4rMU5x/Pn1FAF4BfEGb/Skdfw1/0bJyuzMxILkLJdGw4oNtpo1WtWoAcowsTqEUeaxBEx1pOKVsyw/lneObxG4/fEHBXbeZM6pB67T+MKSmGHYSasnDlPygeWOBwubnUFAyR7wM5FgV53vDkw9E5+SYroVq+1JH8le/uOztcDpGBMyyUYTMrQCojcf8dOD5nHD7lDrn/qPG6CPnuVD1G2NRbNFmUhRad3pfg2oJhDenaXlJaY7pr4MPIbuzheOrMD7I4cTJU8jhyIFkbiJb1XwsQ+mkQ4NhJWrNGAWZdcQsUSpm3WPga4PY6VL6Jey6MgjoQTpA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1OfFU20yP01Dw8OSK9Q+I6UX8mB9z4UZamfT8Vu8x+A=; b=FH2W0DUxW+n/uokoFtlh0t6dMXnC42nOdmKNw8HJVWPg5SpywDYGH+rFQo/dwMGoErEC+ouUr+iocoqC7P0O+LBiNRtDdPiJBLoXoemgpc4jase9n/bM8Prlf0beqaQe9wL4Ew7GY0JGXYa4ttusRQVEXoGM5I2W7M10dgmqaPXqYTJHDp+aZvmOXQ6W2h1eIOtXna98mQanmH2qSlidziJeQor2Rl9rhcUVmlPtEebSOU7TGLEA002k0X1+73BmMWr7vcFePn/KVHRXmddvbEvIeXaKU4sP7y0vlRtVNagUz9jhp7yYG4YfvWAHl2FiOzewLGTkoWmifC4RqUeksw== Received: from CH0PR04CA0044.namprd04.prod.outlook.com (2603:10b6:610:77::19) by IA0PR12MB8225.namprd12.prod.outlook.com (2603:10b6:208:408::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.21; Mon, 23 Feb 2026 15:55:34 +0000 Received: from CH2PEPF00000099.namprd02.prod.outlook.com (2603:10b6:610:77:cafe::2c) by CH0PR04CA0044.outlook.office365.com (2603:10b6:610:77::19) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9632.21 via Frontend Transport; Mon, 23 Feb 2026 15:55:27 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CH2PEPF00000099.mail.protection.outlook.com (10.167.244.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.12 via Frontend Transport; Mon, 23 Feb 2026 15:55:32 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:16 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:15 -0800 Received: from localhost.nvidia.com (10.127.8.12) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Mon, 23 Feb 2026 07:55:15 -0800 From: To: , , , , , , CC: , , , , , , Subject: [PATCH RFC v2 02/15] vfio/nvgrace-gpu: Create auxiliary device for EGM Date: Mon, 23 Feb 2026 15:55:01 +0000 Message-ID: <20260223155514.152435-3-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260223155514.152435-1-ankita@nvidia.com> References: <20260223155514.152435-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000099:EE_|IA0PR12MB8225:EE_ X-MS-Office365-Filtering-Correlation-Id: 5e76e54c-4bbe-4606-b0ac-08de72f3f9f9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?dkyR7aaf1fIWMsvjFlTVf68jQ4NfDou480QafN3fjiEqCYTTbMS8/RFUr3qq?= =?us-ascii?Q?COSgNJ1LT0HnejWj1NF43EU5WPUMYNHeWhpLyVgvRFhyAYMp+zkMA3Pf7kfH?= =?us-ascii?Q?Dqxla10koJHv3P1sd+8H1y0uJJYQdNZozQt9wUHB9ys+nKpwpTrWnSKK8AdI?= =?us-ascii?Q?7DjcDZ7rqMxiwr6IF1v1lNcJ1ZgqK4VFmjF5RsBzVUtpERSMIhp824CXJQhJ?= =?us-ascii?Q?gvBRLobW5p09i0WnSuFPOwApNGEBnX7MJkMhWYQNV9BOu0s9C6+dQj+GWhF6?= =?us-ascii?Q?Jgo9nLts1P9KEcRPbk7w9lgrI/prQU3y9MWNbVyZrrDFyJegvua+wMUiDV2C?= =?us-ascii?Q?apLXXndL1NbHkH8uzswZA0eU1I63LKZG0Pu22whEGpCusDfbh+N1IF/7SrGS?= =?us-ascii?Q?fpQ4XZiysoK67h0h+UCim6bCHklx43o7GnP/G0BjWQhcxMExSgcsXlWOPaP/?= =?us-ascii?Q?jE1OawjOsyIqGfxxU6T8aLBXwHMGh1CR6KQyfT2h8ccZKCk+3fqf+TzpyaB8?= =?us-ascii?Q?DV9CYwqNJ6s978vyu69ABKjJ9W3niedMgXyas1SteIbZMriLi8yhQ6ONR3zt?= =?us-ascii?Q?eF6uSe9j5XFf15HEfM/CHs3q4LzdgPZmBR6AqH80u5xoE05S6NtDcJL8l10F?= =?us-ascii?Q?yc/dijL7Tq0rJnl7FUEvTBhffX7OA66JxXkWjsw6kFLoBhDWJ7nNPl0O0baU?= =?us-ascii?Q?HIroOsyOWSHCtSX8o4zEa1Jr5y7h+QCaZ9AJAaXxrF0hBN7sQbvn1EurxwDI?= =?us-ascii?Q?wO7TGEECRvlsjTywLLGsAmUNIoG0oQ/iK7AbPoZoMwtKFcGMG0m6WEL1OafX?= =?us-ascii?Q?ao7EWOPauL0ngPJilanC8rQhIhG8RBHAQeVjvw/0EfsBQzTcN5yz8LiH/RVa?= =?us-ascii?Q?jVH2Mo4822aOt3RgvtOTCN484ILnAMdXgiZF722cH4aE9iT2+Cq6dZDHLU/c?= =?us-ascii?Q?S/FV6kmX0P613aGzQBiaFgNbNyF3YhnGxN9KVLxKMfBrblRJRmLX51k0IZCm?= =?us-ascii?Q?7f/HNblb3ogQpTPkiIUMaiQyxoOz9N4TwpA/vfCpm3rF76P9Prr8A2pf5iJl?= =?us-ascii?Q?m5fAMPRRx8eIh8thmFLFVyuFteHvo8AtJDaSSrWM6RB7EpVumDYYkLK+QX9q?= =?us-ascii?Q?sh74VvWJqYV8yi2DPKvFlDndhMnQYYCHEFmlp83Td1LPOWwcHD772GBqj2HK?= =?us-ascii?Q?II8HmpCle8tbq4yAVZZ6r/1UWNpUIHIjzqV44aZ59zQmlkSMF+L5p4vE+wH+?= =?us-ascii?Q?6bRIvrwECs2/lPiyKRmLRz5dSjoJpEpWJLbXQI/XSUqdxxjU7CsZ8jPbzM41?= =?us-ascii?Q?CMfGUW1HnPI4gZqZdzkIcKGOtQUHwycr8EVfaiJKr8fbxhOfHzjH9YLdGQPq?= =?us-ascii?Q?3Mi+7tS0/Tl6HXyNYRwUSR3F/PBp8pMgPprXZpAuYI6GrkWMcgNuNLzTazMU?= =?us-ascii?Q?Mh97X54Y52hZenZMMzMY6z4r9bAydvZQdIQ/bqwNG+8n+qjdQPHnMYluNv8x?= =?us-ascii?Q?ojcsbNMtVwHisaDGXFE6VOohbESKxZljhu4QzEDmaxipGMxTBvj22Wqhki4c?= =?us-ascii?Q?yaqlSK7dBh71MYeK2cf9ycllmU7stj5tnrv0tz9knOa27HEf90GZ/Y5o9i4D?= =?us-ascii?Q?OuyRYxodS4mNyDhTE3hfYCNl9c67s3M+MILH5gdDYnCB7ltX3Jf1LgosJ9uD?= =?us-ascii?Q?AghEww=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(376014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: pNysskSZ8cUSDa2fW2p2GfrFY4jjIh4jz+c78GsmHVwSTQ+hYvlo2n1bgw9HoTEEjk50xXoriRz3RB58l9TJYPn6HnqeblaTy7jUYulhxtoaKioDcm9wMnm08ekU8EAgdRfr/deHFk2I3bBLdnnzp7MnfbGEIzo8TUsW/DnInfGfbDGfGkzL749Y0Ca1DkDXZh+iyhTUAFvjVhA3HOlG7OWLHL9ZjI5OW679YlPXuDmto7SYpJMjI0/LhUKLlHS9TGEre7jnJwVU2/ZtJyxW61Iz2m5Z7yXjYSYA4pnLTBj5gxy30NPimwMwVKGCEOCztEOzA9WEN1lA6moK+6oAkepYgRuqeM/Onms6/pZlWQVx2H7S8V+1/DGT9FEPKO12fCfU80Qlic87o97Zlvieo0SOXKjpcWitRX9c/XiNl8t/VJJSW/RRsoYZmwOJadqU X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2026 15:55:32.8250 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5e76e54c-4bbe-4606-b0ac-08de72f3f9f9 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000099.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8225 Content-Type: text/plain; charset="utf-8" From: Ankit Agrawal The Extended GPU Memory (EGM) feature enables the GPU access to the system memory across sockets and physical systems on the Grace Hopper and Grace Blackwell systems. When the feature is enabled through SBIOS, part of the system memory is made available to the GPU for access through EGM path. The EGM functionality is separate and largely independent from the core GPU device functionality. However, the EGM region information of base SPA and size is associated with the GPU on the ACPI tables. An architecture wih EGM represented as an auxiliary device suits well in this context. The parent GPU device creates an EGM auxiliary device to be managed independently by an auxiliary EGM driver. The EGM region information is kept as part of the shared struct nvgrace_egm_dev along with the auxiliary device handle. Each socket has a separate EGM region and hence a multi-socket system have multiple EGM regions. Each EGM region has a separate nvgrace_egm_dev and the nvgrace-gpu keeps the EGM regions as part of a list. Note that EGM is an optional feature enabled through SBIOS. The EGM properties are only populated in ACPI tables if the feature is enabled; they are absent otherwise. The absence of the properties is thus not considered fatal. The presence of improper set of values however are considered fatal. It is also noteworthy that there may also be multiple GPUs present per socket and have duplicate EGM region information with them. Make sure the duplicate data does not get added. Suggested-by: Jason Gunthorpe Signed-off-by: Ankit Agrawal --- MAINTAINERS | 5 +- drivers/vfio/pci/nvgrace-gpu/Makefile | 2 +- drivers/vfio/pci/nvgrace-gpu/egm_dev.c | 61 +++++++++++++++++++++ drivers/vfio/pci/nvgrace-gpu/egm_dev.h | 17 ++++++ drivers/vfio/pci/nvgrace-gpu/main.c | 76 +++++++++++++++++++++++++- include/linux/nvgrace-egm.h | 23 ++++++++ 6 files changed, 181 insertions(+), 3 deletions(-) create mode 100644 drivers/vfio/pci/nvgrace-gpu/egm_dev.c create mode 100644 drivers/vfio/pci/nvgrace-gpu/egm_dev.h create mode 100644 include/linux/nvgrace-egm.h diff --git a/MAINTAINERS b/MAINTAINERS index 765ad2daa218..5b3d86de9ec0 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -27379,7 +27379,10 @@ VFIO NVIDIA GRACE GPU DRIVER M: Ankit Agrawal L: kvm@vger.kernel.org S: Supported -F: drivers/vfio/pci/nvgrace-gpu/ +F: drivers/vfio/pci/nvgrace-gpu/egm_dev.c +F: drivers/vfio/pci/nvgrace-gpu/egm_dev.h +F: drivers/vfio/pci/nvgrace-gpu/main.c +F: include/linux/nvgrace-egm.h =20 VFIO PCI DEVICE SPECIFIC DRIVERS R: Jason Gunthorpe diff --git a/drivers/vfio/pci/nvgrace-gpu/Makefile b/drivers/vfio/pci/nvgra= ce-gpu/Makefile index 3ca8c187897a..e72cc6739ef8 100644 --- a/drivers/vfio/pci/nvgrace-gpu/Makefile +++ b/drivers/vfio/pci/nvgrace-gpu/Makefile @@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) +=3D nvgrace-gpu-vfio-pci.o -nvgrace-gpu-vfio-pci-y :=3D main.o +nvgrace-gpu-vfio-pci-y :=3D main.o egm_dev.o diff --git a/drivers/vfio/pci/nvgrace-gpu/egm_dev.c b/drivers/vfio/pci/nvgr= ace-gpu/egm_dev.c new file mode 100644 index 000000000000..faf658723f7a --- /dev/null +++ b/drivers/vfio/pci/nvgrace-gpu/egm_dev.c @@ -0,0 +1,61 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#include +#include "egm_dev.h" + +/* + * Determine if the EGM feature is enabled. If disabled, there + * will be no EGM properties populated in the ACPI tables and this + * fetch would fail. + */ +int nvgrace_gpu_has_egm_property(struct pci_dev *pdev, u64 *pegmpxm) +{ + return device_property_read_u64(&pdev->dev, "nvidia,egm-pxm", + pegmpxm); +} + +static void nvgrace_gpu_release_aux_device(struct device *device) +{ + struct auxiliary_device *aux_dev =3D container_of(device, struct auxiliar= y_device, dev); + struct nvgrace_egm_dev *egm_dev =3D container_of(aux_dev, struct nvgrace_= egm_dev, aux_dev); + + kvfree(egm_dev); +} + +struct nvgrace_egm_dev * +nvgrace_gpu_create_aux_device(struct pci_dev *pdev, const char *name, + u64 egmpxm) +{ + struct nvgrace_egm_dev *egm_dev; + int ret; + + egm_dev =3D kzalloc(sizeof(*egm_dev), GFP_KERNEL); + if (!egm_dev) + goto create_err; + + egm_dev->egmpxm =3D egmpxm; + egm_dev->aux_dev.id =3D egmpxm; + egm_dev->aux_dev.name =3D name; + egm_dev->aux_dev.dev.release =3D nvgrace_gpu_release_aux_device; + egm_dev->aux_dev.dev.parent =3D &pdev->dev; + + ret =3D auxiliary_device_init(&egm_dev->aux_dev); + if (ret) + goto free_dev; + + ret =3D auxiliary_device_add(&egm_dev->aux_dev); + if (ret) { + auxiliary_device_uninit(&egm_dev->aux_dev); + goto free_dev; + } + + return egm_dev; + +free_dev: + kvfree(egm_dev); +create_err: + return NULL; +} diff --git a/drivers/vfio/pci/nvgrace-gpu/egm_dev.h b/drivers/vfio/pci/nvgr= ace-gpu/egm_dev.h new file mode 100644 index 000000000000..c00f5288f4e7 --- /dev/null +++ b/drivers/vfio/pci/nvgrace-gpu/egm_dev.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#ifndef EGM_DEV_H +#define EGM_DEV_H + +#include + +int nvgrace_gpu_has_egm_property(struct pci_dev *pdev, u64 *pegmpxm); + +struct nvgrace_egm_dev * +nvgrace_gpu_create_aux_device(struct pci_dev *pdev, const char *name, + u64 egmphys); + +#endif /* EGM_DEV_H */ diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace= -gpu/main.c index 7c4d51f5c701..23028e6e7192 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -10,6 +10,8 @@ #include #include #include +#include +#include "egm_dev.h" =20 /* * The device memory usable to the workloads running in the VM is cached @@ -66,6 +68,68 @@ struct nvgrace_gpu_pci_core_device { bool reset_done; }; =20 +/* + * Track egm device lists. Note that there is one device per socket. + * All the GPUs belonging to the same sockets are associated with + * the EGM device for that socket. + */ +static struct list_head egm_dev_list; + +static int nvgrace_gpu_create_egm_aux_device(struct pci_dev *pdev) +{ + struct nvgrace_egm_dev_entry *egm_entry; + u64 egmpxm; + int ret =3D 0; + + /* + * EGM is an optional feature enabled in SBIOS. If disabled, there + * will be no EGM properties populated in the ACPI tables and this + * fetch would fail. Treat this failure as non-fatal and return + * early. + */ + if (nvgrace_gpu_has_egm_property(pdev, &egmpxm)) + goto exit; + + egm_entry =3D kzalloc(sizeof(*egm_entry), GFP_KERNEL); + if (!egm_entry) + return -ENOMEM; + + egm_entry->egm_dev =3D + nvgrace_gpu_create_aux_device(pdev, NVGRACE_EGM_DEV_NAME, + egmpxm); + if (!egm_entry->egm_dev) { + kvfree(egm_entry); + ret =3D -EINVAL; + goto exit; + } + + list_add_tail(&egm_entry->list, &egm_dev_list); + +exit: + return ret; +} + +static void nvgrace_gpu_destroy_egm_aux_device(struct pci_dev *pdev) +{ + struct nvgrace_egm_dev_entry *egm_entry, *temp_egm_entry; + u64 egmpxm; + + if (nvgrace_gpu_has_egm_property(pdev, &egmpxm)) + return; + + list_for_each_entry_safe(egm_entry, temp_egm_entry, &egm_dev_list, list) { + /* + * Free the EGM region corresponding to the input GPU + * device. + */ + if (egm_entry->egm_dev->egmpxm =3D=3D egmpxm) { + auxiliary_device_destroy(&egm_entry->egm_dev->aux_dev); + list_del(&egm_entry->list); + kvfree(egm_entry); + } + } +} + static void nvgrace_gpu_init_fake_bar_emu_regs(struct vfio_device *core_vd= ev) { struct nvgrace_gpu_pci_core_device *nvdev =3D @@ -1212,6 +1276,11 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, memphys, memlength); if (ret) goto out_put_vdev; + + ret =3D nvgrace_gpu_create_egm_aux_device(pdev); + if (ret) + goto out_put_vdev; + nvdev->core_device.pci_ops =3D &nvgrace_gpu_pci_dev_ops; } else { nvdev->core_device.pci_ops =3D &nvgrace_gpu_pci_dev_core_ops; @@ -1219,10 +1288,12 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, =20 ret =3D vfio_pci_core_register_device(&nvdev->core_device); if (ret) - goto out_put_vdev; + goto out_reg; =20 return ret; =20 +out_reg: + nvgrace_gpu_destroy_egm_aux_device(pdev); out_put_vdev: vfio_put_device(&nvdev->core_device.vdev); return ret; @@ -1232,6 +1303,7 @@ static void nvgrace_gpu_remove(struct pci_dev *pdev) { struct vfio_pci_core_device *core_device =3D dev_get_drvdata(&pdev->dev); =20 + nvgrace_gpu_destroy_egm_aux_device(pdev); vfio_pci_core_unregister_device(core_device); vfio_put_device(&core_device->vdev); } @@ -1289,6 +1361,8 @@ static struct pci_driver nvgrace_gpu_vfio_pci_driver = =3D { =20 static int __init nvgrace_gpu_vfio_pci_init(void) { + INIT_LIST_HEAD(&egm_dev_list); + return pci_register_driver(&nvgrace_gpu_vfio_pci_driver); } module_init(nvgrace_gpu_vfio_pci_init); diff --git a/include/linux/nvgrace-egm.h b/include/linux/nvgrace-egm.h new file mode 100644 index 000000000000..9575d4ad4338 --- /dev/null +++ b/include/linux/nvgrace-egm.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#ifndef NVGRACE_EGM_H +#define NVGRACE_EGM_H + +#include + +#define NVGRACE_EGM_DEV_NAME "egm" + +struct nvgrace_egm_dev { + struct auxiliary_device aux_dev; + u64 egmpxm; +}; + +struct nvgrace_egm_dev_entry { + struct list_head list; + struct nvgrace_egm_dev *egm_dev; +}; + +#endif /* NVGRACE_EGM_H */ --=20 2.34.1 From nobody Sun Apr 5 18:20:02 2026 Received: from CH4PR04CU002.outbound.protection.outlook.com (mail-northcentralusazon11013051.outbound.protection.outlook.com [40.107.201.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A14922258C; Mon, 23 Feb 2026 15:55:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.201.51 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862141; cv=fail; b=RK5VW9Vuy+w4PzeGVdAXMZJfH/1N/S2u2FwHsB+/I6Xkpe9uHTMdo3Y2XdydbF+O82dyAFFhw9kX4msVdCmhmB2LrqVi9brILjRAZOe6tpuFdtxAjChGlpbvJ0/oELn1CHfROO4o9somX9twve5om4rMLkqPDcJkGC3UkcIDNxg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862141; c=relaxed/simple; bh=1tRSaDJxDXlWaH50xDQocNfJh2OXlHOu1F/uBJt/28o=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=I9/qeewztFrDtKMS5gjXqbl6KbPJ6L4h+btpBQvez9ai39w8DbLsJTNVLoe4l3SHc0P7jnZ6c3waZbo2hlRatFp/OZdUtMoZ+jmxPvhS7KaSG7c2e2v2a3cA/x6Z2qDtpyIR8X+2slVNAo9e0S4R2wiFa2v+tzoVYjOhAjphvJs= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=gMeitxXw; arc=fail smtp.client-ip=40.107.201.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="gMeitxXw" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=OjmespkPCtRahzZ25IIXmJIjEnd6B1FxAv4/HUvxUU1CL+U3TBH8EaMhFGQHeAGbfmWy/GLi4EbRoR+u0LoMA83gHbsl0XO8ERKXeyi1n3Tbgrc0p1aVrCVmwrP7eGTZEwDG41KHwj7MDBLBI9hsz2cjGz4NR1TUNl7gIlvb9t2ffFchU5ywxmhPUWNLx/R/YwzSq7RpFD2PxOlfGijVil7B0NRnZPYXoRXVzNki5MhGODexbzp1VhOGvljCkFH1o5ML/VOcsGm19kOibnQdvfpocyoNGsq0g9BtAbdD9lW9zPS0S2YRhKdVDvPU9m70pjc4/GEffBWBdfzPF5ox+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cyiZEjCLFK5HCbJXW61JmcWvQfYKyHqfvmPG8ovdSDM=; b=GWlrJ0ocVlJIAGIp7EljTc2Ktb94vcA6dHEESnkKpc2Uz7L1K8Syqb8wmq+8Sy8ack5NrjegrQAkMnc3rCrGFbTkqqk8eBfoBkNrkflUegKaYTnapnyZV4qrVUMDTZ8LEJ+JKbolUgYVCgXXFOYpt0hFDVommo7E2lFjJ3/en5eIggMuw4WzqofIpjN/Vi5v0gMdxjxKabuihg1uQEkl5bgss2eX27qU0+1sacs4Ja39KanJdBSQ6BFd7NVuN0deJkhwqDgmVHgkkI4hRyJ0ko7tKB+dj/Rs8ERUY4WNOPj9wO2NgyzBUSNpiM3Sgk1/uZmEIIVZGmKlkfGsH6gEUg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cyiZEjCLFK5HCbJXW61JmcWvQfYKyHqfvmPG8ovdSDM=; b=gMeitxXw11m+KmbRNsftRhozrE6AUdUgVAzDyGYWW3OLalP3rnAN9knqF7K/hASuR3GiV6yX7eSCZa7voK6DhtBBk2Q/QIavIYlD2NKApKl/Q7Ak4IM8c8VJ3+EuQbKAVboX5feDw9j7wVl6owwnsjgXtGcThLiM2ed2sT7aK8sqC+Lv8PXs0GAhXCQgQ1sCejxV/zC1i50LKeZMEFIDQtPHVR8jmQTO4fitOCz4cPyvv1Ecalgd4d8i1du7oRh5KD3VhIYj4ze5zlLCOJ9iAh0pa372+eUZ+IEN+lyMEsLoVbMXjYWf7GG0iEPl/y2G+xETJrotRsXGs+hgnvjfew== Received: from DM5PR08CA0030.namprd08.prod.outlook.com (2603:10b6:4:60::19) by DS2PR12MB9637.namprd12.prod.outlook.com (2603:10b6:8:27b::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.21; Mon, 23 Feb 2026 15:55:33 +0000 Received: from DS3PEPF0000C381.namprd04.prod.outlook.com (2603:10b6:4:60:cafe::1f) by DM5PR08CA0030.outlook.office365.com (2603:10b6:4:60::19) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9632.21 via Frontend Transport; Mon, 23 Feb 2026 15:55:34 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by DS3PEPF0000C381.mail.protection.outlook.com (10.167.23.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.12 via Frontend Transport; Mon, 23 Feb 2026 15:55:32 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:16 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:15 -0800 Received: from localhost.nvidia.com (10.127.8.12) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Mon, 23 Feb 2026 07:55:15 -0800 From: To: , , , , , , CC: , , , , , , Subject: [PATCH RFC v2 03/15] vfio/nvgrace-gpu: track GPUs associated with the EGM regions Date: Mon, 23 Feb 2026 15:55:02 +0000 Message-ID: <20260223155514.152435-4-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260223155514.152435-1-ankita@nvidia.com> References: <20260223155514.152435-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF0000C381:EE_|DS2PR12MB9637:EE_ X-MS-Office365-Filtering-Correlation-Id: 5ab9ce93-9712-467e-8d08-08de72f3fa09 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|36860700013|82310400026|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Mruc+PDF8zUQUA86pMoB2etMkflvhFaJqH55eZ+bMfy5/EGb0MIovgvNCHaV?= =?us-ascii?Q?qwCYIq24MU2LsG5Zg0GpT9MVDnvoJd+R+2+/RDag2/wJPEZ7x3PuYhlkMJDq?= =?us-ascii?Q?SxtkDWmJTN43lKL1en0W4bCfVMB6EksQIEfungBFGOXQFI8pCwPYquNK5zP6?= =?us-ascii?Q?FmJDOyk9nkx1z+r50lhxf9wW/GeeTmxDvI9DS0SvWux68RAOX6yVulVDZtLK?= =?us-ascii?Q?nHlJOEoIVlzq92Uf9Zp9B2DUc55UZSfQe66bHzSkqT7oyCYawUyucvRLUlrU?= =?us-ascii?Q?k6kSxUmCn5QpCGPVnyPPuD/AYEJZPLHePMNKivawFirvn2kocqoLslcj7r5s?= =?us-ascii?Q?6u5c/GxFU+DkHVPtCdIplTHqHoBhBR+tVn+wkcvsWO+PUMHEXr/FwDFiSxOj?= =?us-ascii?Q?c/EGivsmrFaCuUkA88HvwsQRhFV/zbagm/ngKzQ31d/DOkj7zGkRGJNoDkMl?= =?us-ascii?Q?wq4Bzbn2PIfFuYvRzqq3QfA6S1HO0KFL+lJo0VixoXLL2kaOJ6UnBi+74Obd?= =?us-ascii?Q?NqHI03DL6ajBErSVMYpAzEpHJMERJTQK3QVEpFBvrFYkrA7Yc5tVUFS4vt3h?= =?us-ascii?Q?5XettBejpBvZD401wYqr8v0tDa5EQHsMjYe9xrCgGHbfvkDaxpgT1EhyXkjD?= =?us-ascii?Q?82EQeSK/UiyY1xpM6K+6ERZ5117jXftjMuB3Grl3IIntzGG1FhiN0NgWITBK?= =?us-ascii?Q?/dNh0PZebiqV9cHCBT0OdtnPfnK4JkpfQT0dv+uFEkathFiPNApb5eV69Pvk?= =?us-ascii?Q?8FuVTzU76wm9b2ccmqgCs7nRnMHBggAHjKfONjqLeJL/CUB41zEWx/hGx1DF?= =?us-ascii?Q?SF722IX4ShGlUMhXTTebOr5Z8jmREWl3bw5KLK/W+WA9CkkW3s2y9b5XRQoo?= =?us-ascii?Q?kry5brRj3cq6yd8ICkwllvsAT8LBmWm4U3j0G4SYLnysi1SsPqA6BUl1rCF5?= =?us-ascii?Q?j+a/dqNfWWU5/SxUs78KN22BEXp3qxPEl/pqzq7IAWp++FovsXBT9KxQOhPB?= =?us-ascii?Q?8FonljUg/+m0Owj/Jts0jv2ZKE+8zdzsOFHwWlkw3p8KdQxGZRf7WGwcMltR?= =?us-ascii?Q?FvBQvRE86XH9y+26pChN4k8i0H7+miaTb8PaxrZDxgjq7OCDbfR0ETgZefbt?= =?us-ascii?Q?V+exejbB7kqHQs3tND+XwwZmDWd2J4/Uf17BboQWOKcdu79U2SC0TyWyNjp3?= =?us-ascii?Q?Y2nL+fiW5yJ7lNsd7AOdgj4xNY4PPO0LgmZSriub/anHm607qkRDbiAog3aw?= =?us-ascii?Q?f4TpE10TlWPM4D/xHEJEAV1bEvnltdDcOxiasZyMliz6C8Zem/JZRu7l53fG?= =?us-ascii?Q?wOD8dvYW5SYU0oj/KEZwVKKuGamWi/3vP+QwkcbjqfFQk05NcD2Xjrqiu39v?= =?us-ascii?Q?FNXLVqdQyKDIL9NGB5RHUiVasv6D+QtjHsreZYgWhw9uCJ/fzBHC2pVqBLYU?= =?us-ascii?Q?U1WYXcW9yg0PW6F7dFOhYzURMV9b6PDa6/sqS4As5s92F4jcdDy7IqpuNsyB?= =?us-ascii?Q?Jb7diHvsT+G156ezNemeEVeCKWbbG2JuWNjjAX2xP4241v7NmDAyeaB5gr5Y?= =?us-ascii?Q?sCIYITucdNOEpyJB4qHaamkhzOTUKKXhj4p6St5VYgFXm6dxWNLeJpWCqU2+?= =?us-ascii?Q?yn7ZwoGsna9/TDhutO/+BSHp8r9Wb2iRxBop7+nEXiL6UC3Sjb1W7W1Pca0v?= =?us-ascii?Q?ghj+EQ=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(36860700013)(82310400026)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ibRp+NS+IGJqdeT0IyFhCQ9SfS4bFk58lA8b0H1djHY8U8ROx4GRgit7es29LJuUU4qtxHRp1zbyAVWbqyvdC9udxtCz3USg4G/0eUpt6x7u+qRP3Bg1qdepwnrDCBQHO+aiFbeEUZgZsD2HAEsFmFQ2nQDrBCwLGOrFyRZDhe+iyZcZXosMnKoRAXyFazdWoQxRrwlH2TKRXzuK1c1O0Q2YkeQddvgNgpNPkM56Y9jeSM7ZMbUawCkz+h7Hwpy5UOpFySr4JP0JM+vjmSVum6F/kqaYLHo2rXBfMYrIb97/mBRmjaaGl3aS85LQpDMxMN8mzo0y+3/ScVHLbnIie6Hu7v26ZtSGbI0aHuxsF7oZclprl60sZS6rsOI5llsT7GyXLwyWjULm5dYdRD+yeW3pr1OZYJz0hoHyhiM4GPqJGa/7lZo43uvXVERS6ljo X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2026 15:55:32.9361 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5ab9ce93-9712-467e-8d08-08de72f3fa09 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF0000C381.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS2PR12MB9637 Content-Type: text/plain; charset="utf-8" From: Ankit Agrawal Grace Blackwell systems could have multiple GPUs on a socket and thus are associated with the corresponding EGM region for that socket. Track the GPUs as a list. On the device probe, the device pci_dev struct is added to a linked list of the appropriate EGM region. Similarly on device remove, the pci_dev struct for the GPU is removed from the EGM region. Since the GPUs on a socket have the same EGM region, they have the have the same set of EGM region information. Skip the EGM region information fetch if already done through a differnt GPU on the same socket. Signed-off-by: Ankit Agrawal --- drivers/vfio/pci/nvgrace-gpu/egm_dev.c | 29 ++++++++++++++++++++ drivers/vfio/pci/nvgrace-gpu/egm_dev.h | 4 +++ drivers/vfio/pci/nvgrace-gpu/main.c | 37 +++++++++++++++++++++++--- include/linux/nvgrace-egm.h | 6 +++++ 4 files changed, 72 insertions(+), 4 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm_dev.c b/drivers/vfio/pci/nvgr= ace-gpu/egm_dev.c index faf658723f7a..0bf95688a486 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm_dev.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm_dev.c @@ -17,6 +17,33 @@ int nvgrace_gpu_has_egm_property(struct pci_dev *pdev, u= 64 *pegmpxm) pegmpxm); } =20 +int add_gpu(struct nvgrace_egm_dev *egm_dev, struct pci_dev *pdev) +{ + struct gpu_node *node; + + node =3D kzalloc(sizeof(*node), GFP_KERNEL); + if (!node) + return -ENOMEM; + + node->pdev =3D pdev; + + list_add_tail(&node->list, &egm_dev->gpus); + + return 0; +} + +void remove_gpu(struct nvgrace_egm_dev *egm_dev, struct pci_dev *pdev) +{ + struct gpu_node *node, *tmp; + + list_for_each_entry_safe(node, tmp, &egm_dev->gpus, list) { + if (node->pdev =3D=3D pdev) { + list_del(&node->list); + kfree(node); + } + } +} + static void nvgrace_gpu_release_aux_device(struct device *device) { struct auxiliary_device *aux_dev =3D container_of(device, struct auxiliar= y_device, dev); @@ -37,6 +64,8 @@ nvgrace_gpu_create_aux_device(struct pci_dev *pdev, const= char *name, goto create_err; =20 egm_dev->egmpxm =3D egmpxm; + INIT_LIST_HEAD(&egm_dev->gpus); + egm_dev->aux_dev.id =3D egmpxm; egm_dev->aux_dev.name =3D name; egm_dev->aux_dev.dev.release =3D nvgrace_gpu_release_aux_device; diff --git a/drivers/vfio/pci/nvgrace-gpu/egm_dev.h b/drivers/vfio/pci/nvgr= ace-gpu/egm_dev.h index c00f5288f4e7..1635753c9e50 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm_dev.h +++ b/drivers/vfio/pci/nvgrace-gpu/egm_dev.h @@ -10,6 +10,10 @@ =20 int nvgrace_gpu_has_egm_property(struct pci_dev *pdev, u64 *pegmpxm); =20 +int add_gpu(struct nvgrace_egm_dev *egm_dev, struct pci_dev *pdev); + +void remove_gpu(struct nvgrace_egm_dev *egm_dev, struct pci_dev *pdev); + struct nvgrace_egm_dev * nvgrace_gpu_create_aux_device(struct pci_dev *pdev, const char *name, u64 egmphys); diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace= -gpu/main.c index 23028e6e7192..3dd0c57e5789 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -77,9 +77,10 @@ static struct list_head egm_dev_list; =20 static int nvgrace_gpu_create_egm_aux_device(struct pci_dev *pdev) { - struct nvgrace_egm_dev_entry *egm_entry; + struct nvgrace_egm_dev_entry *egm_entry =3D NULL; u64 egmpxm; int ret =3D 0; + bool is_new_region =3D false; =20 /* * EGM is an optional feature enabled in SBIOS. If disabled, there @@ -90,6 +91,19 @@ static int nvgrace_gpu_create_egm_aux_device(struct pci_= dev *pdev) if (nvgrace_gpu_has_egm_property(pdev, &egmpxm)) goto exit; =20 + list_for_each_entry(egm_entry, &egm_dev_list, list) { + /* + * A system could have multiple GPUs associated with an + * EGM region and will have the same set of EGM region + * information. Skip the EGM region information fetch if + * already done through a differnt GPU on the same socket. + */ + if (egm_entry->egm_dev->egmpxm =3D=3D egmpxm) + goto add_gpu; + } + + is_new_region =3D true; + egm_entry =3D kzalloc(sizeof(*egm_entry), GFP_KERNEL); if (!egm_entry) return -ENOMEM; @@ -98,13 +112,24 @@ static int nvgrace_gpu_create_egm_aux_device(struct pc= i_dev *pdev) nvgrace_gpu_create_aux_device(pdev, NVGRACE_EGM_DEV_NAME, egmpxm); if (!egm_entry->egm_dev) { - kvfree(egm_entry); ret =3D -EINVAL; - goto exit; + goto free_egm_entry; } =20 - list_add_tail(&egm_entry->list, &egm_dev_list); +add_gpu: + ret =3D add_gpu(egm_entry->egm_dev, pdev); + if (ret) + goto free_dev; =20 + if (is_new_region) + list_add_tail(&egm_entry->list, &egm_dev_list); + return 0; + +free_dev: + if (is_new_region) + auxiliary_device_destroy(&egm_entry->egm_dev->aux_dev); +free_egm_entry: + kvfree(egm_entry); exit: return ret; } @@ -123,6 +148,10 @@ static void nvgrace_gpu_destroy_egm_aux_device(struct = pci_dev *pdev) * device. */ if (egm_entry->egm_dev->egmpxm =3D=3D egmpxm) { + remove_gpu(egm_entry->egm_dev, pdev); + if (!list_empty(&egm_entry->egm_dev->gpus)) + break; + auxiliary_device_destroy(&egm_entry->egm_dev->aux_dev); list_del(&egm_entry->list); kvfree(egm_entry); diff --git a/include/linux/nvgrace-egm.h b/include/linux/nvgrace-egm.h index 9575d4ad4338..e42494a2b1a6 100644 --- a/include/linux/nvgrace-egm.h +++ b/include/linux/nvgrace-egm.h @@ -10,9 +10,15 @@ =20 #define NVGRACE_EGM_DEV_NAME "egm" =20 +struct gpu_node { + struct list_head list; + struct pci_dev *pdev; +}; + struct nvgrace_egm_dev { struct auxiliary_device aux_dev; u64 egmpxm; + struct list_head gpus; }; =20 struct nvgrace_egm_dev_entry { --=20 2.34.1 From nobody Sun Apr 5 18:20:02 2026 Received: from BYAPR05CU005.outbound.protection.outlook.com (mail-westusazon11010015.outbound.protection.outlook.com [52.101.85.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 014F82F1FE3; Mon, 23 Feb 2026 15:55:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.85.15 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862140; cv=fail; b=W4gwITO/qackabxZBWFgxX9YbuG+EwV7nH+pfFbm/UboSoxdydbj8y2roquLT8x9pq9WQJjBaDZmsVzUPKv8DPASIs2Mw3gDR/lQ0l/nC3iE8JvYf5UKaw6q2qWRKzqpIErmtZbnAKmiu9aNotrWdZYyhueRBPHgrVmyhOKG4CE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862140; c=relaxed/simple; bh=ph5fq6sR/n1flyACgmeAbMWikJGXQTVUDDTEoJUoOAc=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=PU5Ncs/G6UUjg9Chc2qE6mWK6ec/qSZeZ5yaTYbJNMON4ZRhYI0TlZRCbu/mqXmal7Dab/u4VV4cazH0ts7QEuY/jtHH/darq5AskkX8kpoEIwQ7yAdu66H30tEv74pmpP/vmex+pjadbXNJ8Nrudj6YXZQuau4zuJNfMTg7VfE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=WaQBtVPA; arc=fail smtp.client-ip=52.101.85.15 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="WaQBtVPA" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=NQ/dBssRASf3hY8My0pKfQP+kt0vEgEGy6fyv/IrZN/J5DrvdMOzo+WEzXQjIWWf2rjE6vxyoLRzZ6MoeJSTcvlL2cw67dd9AJX/uS36nZS1QwaeLnopqybXzlldnlo05TrIvgoaBnJnezCpWAVXnE3tr3M7AO8HyCOBAxSf0j4nakPVJc2pxnBY/8kUIgLCpADhPcoNKk7bknb+DPDZWLzjGYPnoi0VhQwA/tSAOKEAo17q6WXkU1CYGUV5+iYKP1ig7AShzVbVeSXq/UFcVWXM4u0ZBrMHVkYWFbBq3h7gTCqo25J+RzE1vyirOn1cOrISuxIrVrRx7P1Bm3kdRQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9iPhUybabJYrtNLgQR6RxIiA+CT14TDEPTttbpKRYcg=; b=wrQkwmII29iP7umpCUGqwBBPS7U565SexDy8a0jLALskvxVaBEpe4trEyufEo7ftWGvEWpHIA0DxXjQO5ho3Zl8XBMq0PRRQC626muo8ILMxJxM/ChiReelpew2kfLz7VmcAd4/mfe37a5Pa3oyQ5Zh5y5H0ezpYgvExWAZB6abZ5yUn52bF7yYM5R5J3AL/YB+YaZE25lJT7v25bmUtYXRGs/DRlWML+RUg1RkhD5YZBlCYUih7SnHHm4caF5zvpcOWWN1E6BaBr9mVhvOuYtYDY912v6cRZAvVB5TAHVBQa0TY6Q1xNpcJi8iSz6UfL+jP99/0vxFduJ/mGnVpEg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9iPhUybabJYrtNLgQR6RxIiA+CT14TDEPTttbpKRYcg=; b=WaQBtVPAUkTSkaHbg1uRuAn0ZiqxThb3zpoQl83sKjKtBBg4qipjBdjDtkC5+Q1+5MPST8CbysTkctl4hUFBXZe+IEqeKiK/3Pc9QMmlwvDHLPqRIX/msKEdoZQP+GgbNsFNE9vSaJ+omksRUUf+FyNVPNucQYFOfKOVrHU1JNo+HC78eZ6/XhYyZdH0ohJsVdeoeU5ZmjxTgUb6GX4KyEBMwjw0Bi6c2KTy7fDzOHnYOvWBJGOQAEh3o1pNbSSHf2JwSEJviguVQbeMZBOWqeLcsFdw4HFGI/aUUoMZk2woH+HZQh3nKyuVYpweHOMeej0mW0HG45tasg+/YYUoHA== Received: from DM6PR14CA0050.namprd14.prod.outlook.com (2603:10b6:5:18f::27) by CH0PR12MB8578.namprd12.prod.outlook.com (2603:10b6:610:18e::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.21; Mon, 23 Feb 2026 15:55:33 +0000 Received: from DS3PEPF0000C37A.namprd04.prod.outlook.com (2603:10b6:5:18f:cafe::6b) by DM6PR14CA0050.outlook.office365.com (2603:10b6:5:18f::27) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9632.21 via Frontend Transport; Mon, 23 Feb 2026 15:55:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by DS3PEPF0000C37A.mail.protection.outlook.com (10.167.23.4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.12 via Frontend Transport; Mon, 23 Feb 2026 15:55:33 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:17 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:16 -0800 Received: from localhost.nvidia.com (10.127.8.12) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Mon, 23 Feb 2026 07:55:16 -0800 From: To: , , , , , , CC: , , , , , , Subject: [PATCH RFC v2 04/15] vfio/nvgrace-gpu: Introduce functions to fetch and save EGM info Date: Mon, 23 Feb 2026 15:55:03 +0000 Message-ID: <20260223155514.152435-5-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260223155514.152435-1-ankita@nvidia.com> References: <20260223155514.152435-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF0000C37A:EE_|CH0PR12MB8578:EE_ X-MS-Office365-Filtering-Correlation-Id: 4423ffbc-884e-444b-b4f1-08de72f3fa6a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|82310400026|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?bdTi9Kn5hPYH0WXMqrB/7h4zg6cOmUAPyoKUBO4ytCyY+sOFhGoocry4Ivmh?= =?us-ascii?Q?aFtpGgF5EUTHmeMuiGwzaliJWY4NQF6OLJfdxqZne9h4t61d3U0sYWOpv1LQ?= =?us-ascii?Q?OYgePeAdovSoF02S0F6iMR0yXiZBCgvSwcZb4LZle207hAErGDxbHmKSGXB2?= =?us-ascii?Q?A7JEumybCQvGblXdy8d1GzU5n49skYOYqO9MWCjjnHyyby1FWFD422zhw78t?= =?us-ascii?Q?YD1ICm3syGNSZuL5jTMhvE9d9v0ZQnPrYkmSijnf3XjWHCfBnDi/JVPzWYNi?= =?us-ascii?Q?qNKT1fiTaU/9Yd77q8CZng5wYL1dIsAUYfdhQuzylgX2pRMi7O9XYpJNkxKV?= =?us-ascii?Q?5XyfJxMVIYbb5VDNuRubw1ZcdIvr1AfHOPIzFp5LXTwUJ1QhT0fYrO/aOMRx?= =?us-ascii?Q?nBhC0IaGhxqGh+2AYc6l6s8E1h+xe+DrEmqvFl2v6f4ugfgKNHcEZm6Ds+lX?= =?us-ascii?Q?ogWAZRWbppf1K7w6GbZA3mZ9FHUeZcwArU+aLl08hbaoKBiaBUhVj+h0sH7h?= =?us-ascii?Q?/fS10ZiECw8sAs8S5I4jVloNfgZpkbY9IdhBY9fowg2ZbOssQzQjpa1jurht?= =?us-ascii?Q?keej966p5f2218E2wzdZ1Xfxm9stzIGZmAUIShOBmIpJyaaapZlsfmgFdJfo?= =?us-ascii?Q?1lRcNTOEAKKQvJYgM/X0M2Cl7GusRSfkQ7aY/JUIeme04fqRToRZWwB1dCkm?= =?us-ascii?Q?4e4+EWVCTp1bA9zVwwCoVHJ6sstD9gm8skw87AlPycY6/IfI9z25+8pW+PgQ?= =?us-ascii?Q?UOKjz5gYOH8rAGF28JTok0f6/VxJf2BJoUydCNxkLQtgIAdBOOx53XWoZ34S?= =?us-ascii?Q?R53jvZdX2l00s+LUWT88pbjq5j8An99GZh40Ylvm56srZKzNpOU63kcjw/zY?= =?us-ascii?Q?rArRdfZUrkXd0Ui56vyV5hoVj8wYyUE75CLg6mtAcGNqKoUbUH1OT6dLHuj9?= =?us-ascii?Q?D2Cs3jSLmHgGRZ30HNXsaSNfMf3mhwBffxYZKCWbBOoK6iwc46rLRhALqeJl?= =?us-ascii?Q?jU8qJsCn86/oZ+AovnkR8S0/q9wdoXnrLmXemxywrpP5Wbxi0qmuy1xF67zd?= =?us-ascii?Q?XqDWY7/Lq7+8uizoqr17kZ22QeYoG+tlPBlh69bf9HE4Y0DnporkazpBQt0V?= =?us-ascii?Q?+fYtkQrUofI/EhTOtiuali2DitSOIth+5bWrNagQ3J+r+nR0qo5tqrlme8ec?= =?us-ascii?Q?Xt3cTD65gFhMQcZCHLyoYt4vpyNsHCmc5hXJEbIeTEztUDNRjDfizm1tg23/?= =?us-ascii?Q?qfP1OHaFgap8T+3h1yaxlojd5TYFnUXEH89xl7z+nqzBzaXEHf2GbsRBq+uG?= =?us-ascii?Q?jfxTG3fUw31H/Yw4L3qPEisQbAFag84OtWXt0R6CY8EDGMkqg/5xNf1VTSnN?= =?us-ascii?Q?klmSax3uhHiy3EuWx7RA6chubdpg8v8CoNyDPviFnmTdEtVYaXNYYGCiFDK/?= =?us-ascii?Q?uVGR5bs9HRplkCMYURMZsXeptiZsnnLuAFtaxZD5LrwlFxBDD2Wf3gx6jx1O?= =?us-ascii?Q?wFpUnu4a5HFGL6rRqWJ6iR3QcIuUfaH7HDi9zFDtTELRtx5S1SRNIpvpVR2H?= =?us-ascii?Q?moWL3L5zdxoTFudx3bjwCV5hDEdfUP1lmPbt99N+Ge4Qv19g4vSlzbihFD/S?= =?us-ascii?Q?k+P8zfbxxzSoFc0E+PhSe6AYwIwk8eF1030cnDc7N5Yppak+ImZiynd7ajY6?= =?us-ascii?Q?MLW0iQ=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(82310400026)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: rUFliEwiGRUAfcrCqkFmK7mtI8shbZZOYCaC7Q6dOX4+NwC9VpnSynQyR43xhII6G4fKb9NJNV9JwUNb+OBelV6ldcIJuwO4rN/QavZFIYzJ03jTfjBp/pypVKMSId1Ozdj+E9BsG6w7ZMWoJ4JoMtgeUuk8qlQAhXdowvtQMFVIn1qZWDMVEKtK9TM2hGvq2JEBKzQ1yOA1Sesq0s9QNXa8qLVmo3efLSTmKfVJoJygizev5WcGRnUJ6v88HX3xqPfc1JtMYmraqg/sdgnzG8fb98SgIpuE7z+6OIQpVdl6cDSkcJzlyfS1ps9lxNgf3UWKEPnodLTcKPmzy78aT9vq3/xnnCV2XidB/phTWlAF+S6ys754us3zcqkplZrqV951vzAm22XPO4w7B4hDNowit/RZcV/NyIN+VPLegPrsATd52DIEqBd2/yzydTXo X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2026 15:55:33.5688 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4423ffbc-884e-444b-b4f1-08de72f3fa6a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF0000C37A.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8578 Content-Type: text/plain; charset="utf-8" From: Ankit Agrawal The nvgrace-gpu module tracks the various EGM regions on the system. The EGM region information - Base SPA and size - are part of the ACPI tables. This can be fetched from the DSD table using the GPU handle. When the GPUs are bound to the nvgrace-gpu module, it fetches the EGM region information from the ACPI table using the GPU's pci_dev. The EGM regions are tracked in a list and the information per region is maintained in the nvgrace_egm_dev. Signed-off-by: Ankit Agrawal --- drivers/vfio/pci/nvgrace-gpu/egm_dev.c | 24 +++++++++++++++++++++++- drivers/vfio/pci/nvgrace-gpu/egm_dev.h | 4 +++- drivers/vfio/pci/nvgrace-gpu/main.c | 8 ++++++-- include/linux/nvgrace-egm.h | 2 ++ 4 files changed, 34 insertions(+), 4 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm_dev.c b/drivers/vfio/pci/nvgr= ace-gpu/egm_dev.c index 0bf95688a486..20291504aca8 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm_dev.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm_dev.c @@ -17,6 +17,26 @@ int nvgrace_gpu_has_egm_property(struct pci_dev *pdev, u= 64 *pegmpxm) pegmpxm); } =20 +int nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, + u64 *pegmlength) +{ + int ret; + + /* + * The memory information is present in the system ACPI tables as DSD + * properties nvidia,egm-base-pa and nvidia,egm-size. + */ + ret =3D device_property_read_u64(&pdev->dev, "nvidia,egm-size", + pegmlength); + if (ret) + return ret; + + ret =3D device_property_read_u64(&pdev->dev, "nvidia,egm-base-pa", + pegmphys); + + return ret; +} + int add_gpu(struct nvgrace_egm_dev *egm_dev, struct pci_dev *pdev) { struct gpu_node *node; @@ -54,7 +74,7 @@ static void nvgrace_gpu_release_aux_device(struct device = *device) =20 struct nvgrace_egm_dev * nvgrace_gpu_create_aux_device(struct pci_dev *pdev, const char *name, - u64 egmpxm) + u64 egmphys, u64 egmlength, u64 egmpxm) { struct nvgrace_egm_dev *egm_dev; int ret; @@ -64,6 +84,8 @@ nvgrace_gpu_create_aux_device(struct pci_dev *pdev, const= char *name, goto create_err; =20 egm_dev->egmpxm =3D egmpxm; + egm_dev->egmphys =3D egmphys; + egm_dev->egmlength =3D egmlength; INIT_LIST_HEAD(&egm_dev->gpus); =20 egm_dev->aux_dev.id =3D egmpxm; diff --git a/drivers/vfio/pci/nvgrace-gpu/egm_dev.h b/drivers/vfio/pci/nvgr= ace-gpu/egm_dev.h index 1635753c9e50..2e1612445898 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm_dev.h +++ b/drivers/vfio/pci/nvgrace-gpu/egm_dev.h @@ -16,6 +16,8 @@ void remove_gpu(struct nvgrace_egm_dev *egm_dev, struct p= ci_dev *pdev); =20 struct nvgrace_egm_dev * nvgrace_gpu_create_aux_device(struct pci_dev *pdev, const char *name, - u64 egmphys); + u64 egmphys, u64 egmlength, u64 egmpxm); =20 +int nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, + u64 *pegmlength); #endif /* EGM_DEV_H */ diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace= -gpu/main.c index 3dd0c57e5789..b356e941340a 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -78,7 +78,7 @@ static struct list_head egm_dev_list; static int nvgrace_gpu_create_egm_aux_device(struct pci_dev *pdev) { struct nvgrace_egm_dev_entry *egm_entry =3D NULL; - u64 egmpxm; + u64 egmphys, egmlength, egmpxm; int ret =3D 0; bool is_new_region =3D false; =20 @@ -91,6 +91,10 @@ static int nvgrace_gpu_create_egm_aux_device(struct pci_= dev *pdev) if (nvgrace_gpu_has_egm_property(pdev, &egmpxm)) goto exit; =20 + ret =3D nvgrace_gpu_fetch_egm_property(pdev, &egmphys, &egmlength); + if (ret) + goto exit; + list_for_each_entry(egm_entry, &egm_dev_list, list) { /* * A system could have multiple GPUs associated with an @@ -110,7 +114,7 @@ static int nvgrace_gpu_create_egm_aux_device(struct pci= _dev *pdev) =20 egm_entry->egm_dev =3D nvgrace_gpu_create_aux_device(pdev, NVGRACE_EGM_DEV_NAME, - egmpxm); + egmphys, egmlength, egmpxm); if (!egm_entry->egm_dev) { ret =3D -EINVAL; goto free_egm_entry; diff --git a/include/linux/nvgrace-egm.h b/include/linux/nvgrace-egm.h index e42494a2b1a6..a66906753267 100644 --- a/include/linux/nvgrace-egm.h +++ b/include/linux/nvgrace-egm.h @@ -17,6 +17,8 @@ struct gpu_node { =20 struct nvgrace_egm_dev { struct auxiliary_device aux_dev; + phys_addr_t egmphys; + size_t egmlength; u64 egmpxm; struct list_head gpus; }; --=20 2.34.1 From nobody Sun Apr 5 18:20:02 2026 Received: from CH5PR02CU005.outbound.protection.outlook.com (mail-northcentralusazon11012054.outbound.protection.outlook.com [40.107.200.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B3C63315D3E; Mon, 23 Feb 2026 15:55:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.200.54 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862154; cv=fail; b=Vp1r81pI0VfgBSwWpFwTzbyFJWdHMijNPyAfgwedfHBjejnVgUcV8vFQbV8Rd4HfVc8mP6C0TOuRxs5PgK3vGDVIky1Mze5SkUQxcaXDdwB6ZaR7Pvm2wKN9xauQCR8kWTcyNdx+R85/ygwA/ZoWgktWini34VLkZ/KfSZHSSVI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862154; c=relaxed/simple; bh=s5xQ52f8j1WAQKXX7vPkyUFMqYYAgVuwyoO9SDf9STs=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=WULK/ZQ56Y9U7j/hgvQa+K9m8N9MZpdpid8sf/HaTTF/xkSUra795LX2FI/cRrFk3TKydpTJx7whdiIGpJsUqmx78YotPRW/xnD/zI9GgMnqU5pp41WmkM1JZqeOrTvEq4aHXuijFSAmzESNZRTdcazjE9u2xqFvagENWzKbmSc= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Y3yOkCpv; arc=fail smtp.client-ip=40.107.200.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Y3yOkCpv" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=JBu6Znpoc6GGGJt4fZoR2cJsKo5j+mjahVWY543vlHyIv//LZSrEzAY20Qmh2XVmbUmCMPzF1x4njxGb83fNj/y6w1vBhMFuI1N5HexqZ7E9UhIYuZbUVDm/weY+DXmoT18sa4iIJlpc3vonemH++goU+gEWnDLNv7EQz+rEC6uGMqDbzlS74g2ROOnxceaeZLVtPfs3Sa9vbkrW3H75xQRTOfcuzoNvTimxtAgO+UXUf2hpci7dWAuJwTHSCl1x8dk8GiDOiVQ2F+Q9Dpv/ToUgAH85w5bRCRydTCTiwyPcZM89MSEZUf2FBTZm7YcC8gTwPyAj/yDLv5Fy4TQVSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=i4iqXniX24XY64FcqMsmliuq2LgQIqSAQMBweKNbvJA=; b=YGTNKdjzBeLud7PrHnEfGQXlRXRdjeZ/P7pjFnJDiVDpvCvS/xqs5oOKaFTOiLG3MF+FEUwQouZwLVYZOLHCbcXPb+vbPuPIMYx9mIA1j/nmYB+TxkMXfkRiL4D1jNcPQcyyj/z3T/pRBHZK/0FVjMw/OGehSu0bz+4zNopwJbUryCGsd5yCOYgQhSmuyp1yf+3ZAB/J/TSujDOjEQdhB6ivhdR2ktiSB2Qc09wmBEYFlNAnXHyBBoHv1OZz5d2pTaoTvvOcQG7Yfj/wFBC/ee7vAhAkfDt3Jm11j2CpoBCkcMpdhhyuJmSZ4z40KC9JQrqdlJnz712G4NjlcFssyg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=i4iqXniX24XY64FcqMsmliuq2LgQIqSAQMBweKNbvJA=; b=Y3yOkCpvxydxaBWn+xmYn/8ZJ07JGYgdy2xFo2auk+B410pxG2JIU/yIwb6CczU61mRyqHdvj5xPVehWDXDH76ObeiSzkkYIjkRgAXV2Di0/99/2v/Rn48KLW2RGDOOkAX+wWqsAg6ahWenPA+zlxWFoTN0WpcCEtcna/+Z7KzlglV+cXM6gAo+dE1ZubmYF7MRvZ5LfeD/I7IVepAk49+soKIZvReFZj0oMOYxHoPfXG5xy+1TQuy5YX/GBG/Q5dPg+hOXUz+9et1VC88qPZBDxzBIvIYeb4MrhiKZ4nK2VmAJAuDvvZ3COkPqGUA2C8XLInNAaLBhNDBsf/YdtPw== Received: from DM6PR14CA0059.namprd14.prod.outlook.com (2603:10b6:5:18f::36) by CY5PR12MB6227.namprd12.prod.outlook.com (2603:10b6:930:21::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.21; Mon, 23 Feb 2026 15:55:34 +0000 Received: from DS3PEPF0000C37A.namprd04.prod.outlook.com (2603:10b6:5:18f:cafe::2c) by DM6PR14CA0059.outlook.office365.com (2603:10b6:5:18f::36) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9632.21 via Frontend Transport; Mon, 23 Feb 2026 15:55:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by DS3PEPF0000C37A.mail.protection.outlook.com (10.167.23.4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.12 via Frontend Transport; Mon, 23 Feb 2026 15:55:34 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:17 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:16 -0800 Received: from localhost.nvidia.com (10.127.8.12) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Mon, 23 Feb 2026 07:55:16 -0800 From: To: , , , , , , CC: , , , , , , Subject: [PATCH RFC v2 05/15] vfio/nvgrace-egm: Introduce module to manage EGM Date: Mon, 23 Feb 2026 15:55:04 +0000 Message-ID: <20260223155514.152435-6-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260223155514.152435-1-ankita@nvidia.com> References: <20260223155514.152435-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF0000C37A:EE_|CY5PR12MB6227:EE_ X-MS-Office365-Filtering-Correlation-Id: cee8bfdd-ca34-4780-d7cd-08de72f3fada X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?PnVo0pkMk0tfUulQNWgd1CH+WWnWFJprNqyS27WM3Y4PfD4AVoNqJC3LZZJZ?= =?us-ascii?Q?81znRJyfdDqUQjFatX4m3snDSiGqv1D99Tr74KUtx78NKAVZXXwsOZx6w020?= =?us-ascii?Q?4IpyXufnbnH0WqfuoSrif3yyWXXBR/Th56nYBP/GusugdsjFyLK4lh5kHii8?= =?us-ascii?Q?mrlfv6ih4+xscYkq4oGkIoNIiKZwi4vD/smXikM+R4likEIjwU403UqCaSY6?= =?us-ascii?Q?Xc1Xi2KIWZ4R+y2eBhu7BysgiUq+MSXxsSgqAWdRx4AjrrhYbwNE+WebYpL9?= =?us-ascii?Q?exq/2O4vJ28+Tx2P/fOiIK3dUANEUUwSFX6+9HWn/tFHGAE/abPHMfRpF9g8?= =?us-ascii?Q?vtVUAHCt7L7RkKvGyevTIqyI1F4lWgJO12DDnP0sJF+wqrA80xsYVdhPndyd?= =?us-ascii?Q?GzH5nD2vonSCR82cRCWszZgZk/s/i8WLyt9qpnuFBVAjZeM1JUHNLzuGxSIf?= =?us-ascii?Q?1W2coCdPOVH0LJXWQRqq/2/WmoC5kDJDhTlROroZRLdpPy7NT7bFxTX3rB2i?= =?us-ascii?Q?FLcnXta9zM7Sn1NK4n1dAJ557SsctoxZkbyuyc20KPfoQ4KWw8pfyNeuHRcZ?= =?us-ascii?Q?7RRYMD2RNBhSOZZOGzl8J3qIj+qwONAV4epc0GcvLURjtCW0uO/ew578irEN?= =?us-ascii?Q?MdWd/pQJCijQOb5GT+HHum38kJnj90G8iICmWysj3XSc/u9BosQXmmuEFOdy?= =?us-ascii?Q?3BS0hrWbdWRbOcBAHg98hLTZrM2IjDXP2bN9sBmMM2ybfSnfSIvHooCSd1kT?= =?us-ascii?Q?PhbqaigXe7Rl0O+knBXH6K+jLMKRatsGEWVsurbDFx76g03yN1Zdv6WRf2vR?= =?us-ascii?Q?bfmi6Ci00ahlHleq7GYZc9m1I0rvLllgG2gvXl96c/jXTE9kqdgOQe8J7ksf?= =?us-ascii?Q?uKNJqry8NRAayvYmZ75Uyj2W5ydNYYH1Erbqa2NV3yKYpcK3cxPiEsaOdhe1?= =?us-ascii?Q?H8kL4S90kYi3Ssu4wIb1PwgkROM1PXR28qQjqf1zwOP5fMkM9RJuEaBJTx5H?= =?us-ascii?Q?3IeveFRFB3jlNsnfO0QhQKjwmcBl45gc8Q5/DkLfGgVx400QlitwNDHhsAOL?= =?us-ascii?Q?qy67NzeZyvEr39aa31y+vjEf2rJJlvnnc+icPS1RV0ogsY5QPaJ2exBUPW53?= =?us-ascii?Q?kxsMmzVwyI8Pcw1YpZgD+83PZ9ZeWi5Wo49D/PLisfXbuAGquArWP8xSxEKA?= =?us-ascii?Q?QDJgyw6Ki7gBqF1H/G3H7dl7P6Rzj05peFOQs7tNovy5VWtuEBO0Zvd+XKF1?= =?us-ascii?Q?PKk2eMSwR3+pilKu2MhkavuvcXUoIYQjpzgY9CP8ZFKyQ+jkbVStzfwKNLhC?= =?us-ascii?Q?RCHk5PI4e4XRQr6kmpFKxAThogLx36s81AiSDa3N1rnWlxqQB5DIR9xMy2K6?= =?us-ascii?Q?JgnMmk3KVF/vJqSmdmgowcg3umSJjaHnXrdhM97k/yzIVowf6uL5cgJ5ZALM?= =?us-ascii?Q?dxiD19gsnQ7R2z+IxO0uf2ni8Z/za9Uq6sItWvNLa5tQXv9/2CJqxgqz69jU?= =?us-ascii?Q?2/rOgGNcwOeXX86RkWlMt49Wr4nYCytrOd8FC6XQmaPXNyFZLrbcsuCVn9Rw?= =?us-ascii?Q?xxIyMl1V5UUcoxK+kkxTJPkXUSwp3CnfnnT+PtCH3tI60+2lnBbZtpnpLlUF?= =?us-ascii?Q?o0vDX6JfCUMyQTPLd+moNOrQdrbwAMFAbNHCCLYsvlKq/NrUwcJZ8A5llf0a?= =?us-ascii?Q?Ga/nJA=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(376014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: x5KOabRb5mNhkpxiKEhWboeG7Jo2EcsdAyUfyxKltGp1q+i3Bu8mv5JRyp/mh38Gr+RjLCaa47TnQhUo0Bqiqf7I/uXdWd/9Ll2imAAWoVE3bVaABILa7x7HBS9DF5Hx4tzYQGnbvwh5aUSFaA0lMhbY8HIkAfTMow+WR/mqflvAnfFBmyzTAMFu1HsOdvKSPJrrivQGm7cCpRnwraZMUWm9nFxNVNPl7J9BK3kQQxUjQM0XomTLVsH/grk40/rNPJsshJV/Ct6sbXd0hDU2Eda/6caiKneGrommfhCpQD/Zm05EX6lCOSqdHEXznWDET/JzjJjmJXW5VcaoGoIlJFiaOX7JM0WNxPvmGQ8/qQZbNSjjn0IxaB05jQXh/umNoePGXTyF4fn07GFWP+BTIengUCSGK0g/mdoQEolYr89L/k2fMLUPdLvkkQtOP3ws X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2026 15:55:34.2998 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cee8bfdd-ca34-4780-d7cd-08de72f3fada X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF0000C37A.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6227 Content-Type: text/plain; charset="utf-8" From: Ankit Agrawal The Extended GPU Memory (EGM) feature that enables the GPU to access the system memory allocations within and across nodes through high bandwidth path on Grace Based systems. The GPU can utilize the system memory located on the same socket or from a different socket or even on a different node in a multi-node system [1]. When the EGM mode is enabled through SBIOS, the host system memory is partitioned into 2 parts: One partition for the Host OS usage called Hypervisor region, and a second Hypervisor-Invisible (HI) region for the VM. Only the hypervisor region is part of the host EFI map and is thus visible to the host OS on bootup. Since the entire VM sysmem is eligible for EGM allocations within the VM, the HI partition is interchangeably called as EGM region in the series. This HI/EGM region range base SPA and size is exposed through the ACPI DSDT properties. Whilst the EGM region is accessible on the host, it is not added to the kernel. The HI region is assigned to a VM by mapping the QEMU VMA to the SPA using remap_pfn_range(). The following figure shows the memory map in the virtualization environment. |---- Sysmem ----| |--- GPU mem ---| VM Memory | | | | |IPA <-> SPA map | |IPA <-> SPA map| | | | | |--- HI / EGM ---|-- Host Mem --| |--- GPU mem ---| Host Memory Introduce a new nvgrace-egm auxiliary driver module to manage and map the HI/EGM region in the Grace Blackwell systems. This binds to the auxiliary device created by the parent nvgrace-gpu (in-tree module for device assignment) / nvidia-vgpu-vfio (out-of-tree open source module for SRIOV vGPU) to manage the EGM region for the VM. Note that there is a unique EGM region per socket and the auxiliary device gets created for every region. The parent module fetches the EGM region information from the ACPI tables and populate to the data structures shared with the auxiliary nvgrace-egm module. nvgrace-egm module handles the following: 1. Fetch the EGM memory properties (base HPA, length, proximity domain) from the parent device shared EGM region structure. 2. Create a char device that can be used as memory-backend-file by Qemu for the VM and implement file operations. The char device is /dev/egmX, where X is the PXM node ID of the EGM being mapped fetched in 1. 3. Zero the EGM memory on first device open(). 4. Map the QEMU VMA to the EGM region using remap_pfn_range. 5. Cleaning up state and destroying the chardev on device unbind. 6. Handle presence of retired ECC pages on the EGM region. Suggested-by: Jason Gunthorpe Signed-off-by: Ankit Agrawal --- MAINTAINERS | 6 ++++++ drivers/vfio/pci/nvgrace-gpu/Kconfig | 12 ++++++++++++ drivers/vfio/pci/nvgrace-gpu/Makefile | 3 +++ drivers/vfio/pci/nvgrace-gpu/egm.c | 22 ++++++++++++++++++++++ drivers/vfio/pci/nvgrace-gpu/main.c | 1 + 5 files changed, 44 insertions(+) create mode 100644 drivers/vfio/pci/nvgrace-gpu/egm.c diff --git a/MAINTAINERS b/MAINTAINERS index 5b3d86de9ec0..1fc551d7d667 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -27384,6 +27384,12 @@ F: drivers/vfio/pci/nvgrace-gpu/egm_dev.h F: drivers/vfio/pci/nvgrace-gpu/main.c F: include/linux/nvgrace-egm.h =20 +VFIO NVIDIA GRACE EGM DRIVER +M: Ankit Agrawal +L: kvm@vger.kernel.org +S: Supported +F: drivers/vfio/pci/nvgrace-gpu/egm.c + VFIO PCI DEVICE SPECIFIC DRIVERS R: Jason Gunthorpe R: Yishai Hadas diff --git a/drivers/vfio/pci/nvgrace-gpu/Kconfig b/drivers/vfio/pci/nvgrac= e-gpu/Kconfig index a7f624b37e41..7989d8d1c377 100644 --- a/drivers/vfio/pci/nvgrace-gpu/Kconfig +++ b/drivers/vfio/pci/nvgrace-gpu/Kconfig @@ -1,8 +1,20 @@ # SPDX-License-Identifier: GPL-2.0-only +config NVGRACE_EGM + tristate "EGM driver for NVIDIA Grace Hopper and Blackwell Superchip" + depends on ARM64 || (COMPILE_TEST && 64BIT) + depends on NVGRACE_GPU_VFIO_PCI + help + Extended GPU Memory (EGM) support for the GPU in the NVIDIA Grace + based chips required to avail the CPU memory as additional + cross-node/cross-socket memory for GPU using KVM/qemu. + + If you don't know what to do here, say N. + config NVGRACE_GPU_VFIO_PCI tristate "VFIO support for the GPU in the NVIDIA Grace Hopper Superchip" depends on ARM64 || (COMPILE_TEST && 64BIT) select VFIO_PCI_CORE + select NVGRACE_EGM help VFIO support for the GPU in the NVIDIA Grace Hopper Superchip is required to assign the GPU device to userspace using KVM/qemu/etc. diff --git a/drivers/vfio/pci/nvgrace-gpu/Makefile b/drivers/vfio/pci/nvgra= ce-gpu/Makefile index e72cc6739ef8..d0d191be56b9 100644 --- a/drivers/vfio/pci/nvgrace-gpu/Makefile +++ b/drivers/vfio/pci/nvgrace-gpu/Makefile @@ -1,3 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) +=3D nvgrace-gpu-vfio-pci.o nvgrace-gpu-vfio-pci-y :=3D main.o egm_dev.o + +obj-$(CONFIG_NVGRACE_EGM) +=3D nvgrace-egm.o +nvgrace-egm-y :=3D egm.o diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-= gpu/egm.c new file mode 100644 index 000000000000..999808807019 --- /dev/null +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -0,0 +1,22 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#include + +static int __init nvgrace_egm_init(void) +{ + return 0; +} + +static void __exit nvgrace_egm_cleanup(void) +{ +} + +module_init(nvgrace_egm_init); +module_exit(nvgrace_egm_cleanup); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ankit Agrawal "); +MODULE_DESCRIPTION("NVGRACE EGM - Module to support Extended GPU Memory on= NVIDIA Grace Based systems"); diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace= -gpu/main.c index b356e941340a..0bb427cca31f 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -1410,3 +1410,4 @@ MODULE_LICENSE("GPL"); MODULE_AUTHOR("Ankit Agrawal "); MODULE_AUTHOR("Aniket Agashe "); MODULE_DESCRIPTION("VFIO NVGRACE GPU PF - User Level driver for NVIDIA dev= ices with CPU coherently accessible device memory"); +MODULE_SOFTDEP("pre: nvgrace-egm"); --=20 2.34.1 From nobody Sun Apr 5 18:20:02 2026 Received: from PH7PR06CU001.outbound.protection.outlook.com (mail-westus3azon11010059.outbound.protection.outlook.com [52.101.201.59]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 825D830C632; Mon, 23 Feb 2026 15:55:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.201.59 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862148; cv=fail; b=AANHIuxAbZbALvMMFEpomM+yScidMYythKFNrNSDHhBOJhan6UI6zK9r5Z4+swJW6GMLq0nqPmctUpf7pGZdANt96ajcY/0mtjskAGkkJsBJbiua4slTCkY6Yli8+D/K2zlQSoGZ0tjTcCwbEaivOvOkdJvRARmwq6NUv72RsqA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862148; c=relaxed/simple; bh=t/x3PIIH49dIA2r4AkyvHyyrIpBnkaK3FkdB7qGrKyo=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ah+Q4hsnZuXioN/ZsG44C9Ly29ektFwHjQd1zeL32WkfzUkmuruwEC7S0aqIyAHSlRVqpsJqG7b2BpTssEDvo30bh0/iU4rYajEE/aWnqB/M6e6BiVfmVFRvUYbBwe1iIXuMa5/tLgSDeiFfnVuIiVQ+ExDMOM+Ux0ZGHA3eNSg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Z+gsA97Y; arc=fail smtp.client-ip=52.101.201.59 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Z+gsA97Y" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=DEy6gJYCBHgVvM8EpBLg148EPceN5eXX+kYRHkGOBK5B29s4oMc00FrEd7Gz57ydNHkc+VTd6XS9bJievoriqA9qKth6RE4hKOqVBcW5vtHLLL/Frn3nkigX+xQ8YBOrUi+lB+v1jSW21NtYW+jSjK9yBcDt6ng0TyFDfG/KPyxN/Utn9XzWXefazgYgkQIvF0/3gTMCc9ITio0NWGL190Y8FyAmuUz/pQZ6VPIDHPd5sBdbguLsmDNpcqoNFek1gybrVI3Ynrujt6MUp0Y3tehC9dK0K3RYP6i702s+yqtQYCm9CaZZscJBtNA1Tn/G48F7A0/Tw2U3QTyejtuddA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MzJ+e4pr6eIhdtuH+3dJ2Ii5jpfG5RP1f6Yx5bd3C50=; b=FOrNXVJ+I1nEdX7Yo5St27p8bf511bnpXIVtkaK/PCZoevBQ2QFUsDAYdvusjGaEJ+Jiehf0bNqqvTctXD8M7RLQXvPFEhi7FfA/e8/8+2c6oF1AR0n5EcNs/o4xKDp5CvaWKsZWHxxL5xseK77tuJglUHReRriXFNxFx09VTAnSz7TPP5hszy6SykOx+K5djyH3cO5XFvuh3rJlJYYSsPOAuR8dNYaCMyJXmP5fetInmbL+5EbHcHgekapBEIul6S8gRaVvOW97BN7rndB27jTblJq/T86RFnSfr/6hLuukos2fCoXZz889NRA933TpSCOsgHNNY+kjUY1poII1DA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MzJ+e4pr6eIhdtuH+3dJ2Ii5jpfG5RP1f6Yx5bd3C50=; b=Z+gsA97YcrRIIvEfZIuGz4iFIniW4IhU5hSpHQGd2aeuR5v4xPMZgrHp0uJ1Ltk/79WDneamdE+XhgJxrS8boEDgZXZj2Trr+nT+TjpbhscEaibpR0d6ubDRwhoBiQ8dIxq+TF9aUXcX87Ucst6M+W6U618D5M3QCNhtYNuisivOXhU7e7uvEvli0+vTXkMCRXVUGLySjeO+z7XcLHgvP6Q8NXoYBqPXLUL77bfWcAzPLlTQbuy/F1Do6O4aXCR9EyLAwGt5BR8hq/KeFCk7se+xqHLMObwNwOPqEZ3AHOW0nZsvOWXZaNeQh6ZihBXi3lvqGSMaop8TXP+Zv5SwoA== Received: from DM5PR08CA0060.namprd08.prod.outlook.com (2603:10b6:4:60::49) by PH7PR12MB9176.namprd12.prod.outlook.com (2603:10b6:510:2e9::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.21; Mon, 23 Feb 2026 15:55:37 +0000 Received: from DS3PEPF0000C37B.namprd04.prod.outlook.com (2603:10b6:4:60:cafe::e0) by DM5PR08CA0060.outlook.office365.com (2603:10b6:4:60::49) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9632.21 via Frontend Transport; Mon, 23 Feb 2026 15:55:38 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by DS3PEPF0000C37B.mail.protection.outlook.com (10.167.23.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.12 via Frontend Transport; Mon, 23 Feb 2026 15:55:36 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:17 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:17 -0800 Received: from localhost.nvidia.com (10.127.8.12) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Mon, 23 Feb 2026 07:55:17 -0800 From: To: , , , , , , CC: , , , , , , Subject: [PATCH RFC v2 06/15] vfio/nvgrace-egm: Introduce egm class and register char device numbers Date: Mon, 23 Feb 2026 15:55:05 +0000 Message-ID: <20260223155514.152435-7-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260223155514.152435-1-ankita@nvidia.com> References: <20260223155514.152435-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF0000C37B:EE_|PH7PR12MB9176:EE_ X-MS-Office365-Filtering-Correlation-Id: fa89116b-3ef8-45b6-6f6c-08de72f3fc33 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|36860700013|82310400026|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?kbzkcn5Rv+EX1Von6ruC6cvQ8PMJO4q8/jJoGGC5s9YRTV+2iu/4phzvOBJC?= =?us-ascii?Q?YM44WnthO4Quv6mBYCfwna/ErwVh5CQcc0FUu4wSiZ1OQmUCRriul1F0qF0s?= =?us-ascii?Q?dfuMigqvnyCLvt3M2MYoCNKskgXv7NjUW7C4Siow16Ilr0270njv7ADOPfFa?= =?us-ascii?Q?9uWiMLqB9qnhRsyyht9KcYDb5LBxik7iLwKGLn0DdcSc0m7K7laSOW7MOwhV?= =?us-ascii?Q?qs5TrT+5aX88ZlKqodvCnSYyPJ/qpPbV+QKmvgyhGC9FHUj9cqW7nBAIz4QX?= =?us-ascii?Q?4o+otbRe2ibyLcVd7jaOMVg1x27StGgGBB7T48/Jkj97UaSP8onWHWW23LBW?= =?us-ascii?Q?5/QWZJi/3LVrNqw5yUptYP/jGbCBE0qiC3VflgKQtOMJcb3taUnnA8xB2j5I?= =?us-ascii?Q?4EmIKLE4zsuFfUk1Vzl1+eZeLinCP94/1/pIeGe67Zp0N4aFTeQGJRR5FKg6?= =?us-ascii?Q?NoCUh8xq76uL6uLVw9ndaRSUr2FZm7ZcLq2wy+LugPbK1T62uj8eRKV4/2Ao?= =?us-ascii?Q?V4aJWVXK4+4HD/Gij6c/VrspbuaaAjCbXe3C0ikcNs54XQvmee3au0TaAwbG?= =?us-ascii?Q?FMUUZWaxUk/QZZGYiXEqB9xShFAeytv0e3KfwqboFGsn90TLEO8CoUmUDoz6?= =?us-ascii?Q?VPGHnpbO099HLiv2WnSJkxSvPmDUQp/6knc0RU7sF2TWutWG8r7CDHhzSzZh?= =?us-ascii?Q?HCrpPsvr1yna8LWOgAq3EPBn5GTVK4GhlPjQsrR+eUzPUCNCkQj7LJRRd8hw?= =?us-ascii?Q?UZRqcNji/zueKOufWLKrF8cyZgB/mbbXhHMfmYfkI8t9RSwSF56kzth5qp86?= =?us-ascii?Q?LOsk1H176ksIRDPfqfclWdMV5UMrdeDd5O7ImCR+HZPG0wjTIhmUTDz5cDLS?= =?us-ascii?Q?3DObH2J0NCF5P/+x3PiPQgctczHtH3sjVq0objQ4Hjvlw7xmBnnNfXi7JN0e?= =?us-ascii?Q?3n1bAtMmTidRovWC6unOO5zsDkBxGwLKDMW+HIOoT0Gcv4FXp9PcYeV2Lp/8?= =?us-ascii?Q?rbOcYQ9g5epGvvQfQhoPD/jXfTzdmsnxMBdEJi2y7lGSIV5eTzckt5vGiCGo?= =?us-ascii?Q?rTuUi6PIh5b70I07pTJrTX+5iOsEGM2HruuoGD9PIP7wx5CMjlfwgTt2aNLh?= =?us-ascii?Q?iicgMTZuoG+Jy7MZ/A9SXwOmlU5SHlACzfRjbzBkaoaQoE6fW4hCqsYZkHhS?= =?us-ascii?Q?O1BGpz8bRNZxDdA875BVAVHbvpWvAk2TwK2XB8miEqD+of5Y+cnly9VLSf05?= =?us-ascii?Q?zTesxJpgf3Nx7QnB5x6BOtcWqUQACffNYmsW8bYMblFhOru9MOVMrG4MxgNI?= =?us-ascii?Q?dHmuCKtGH+XwCH7Mz9vlbnqzVkw3BP+a7Tm4CA/dTLJTeM3Kpzfx5Dbfl3Zk?= =?us-ascii?Q?pq+ynNpyvOtPHIyN+itJh0jF0co/HSzxhQ7F4ZztpaZmOn8Y90JszLezELTj?= =?us-ascii?Q?mWE+2n2V6oHv7IJwJrVTBhrfTmNo/74b4vt7wyjJCLYxx1IfVIksuHGmzo7K?= =?us-ascii?Q?oVhtR8n+6p7hJNEB3fKG6QsZ3GqHa95Wb55ZmXaGYmsfyerXoi2UaMlh6R2I?= =?us-ascii?Q?VNd9nf0NNePkBE6lP23MUE0TelR52zQbc7egwecEklzS2+sncCfl4Wgz6rym?= =?us-ascii?Q?tbFSpp/WBOYWirjtlr8oxm0BdQ9PCCDDMd/HjS/XKgFaalpwr2ebLOZ0X5lC?= =?us-ascii?Q?0YdZ6Q=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(36860700013)(82310400026)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: TDYgxJkUnmc7WwOvSDTLUyNOF0uNCWdUA92k6JdETjMkYmzT5KRMyJM4Z0CcS09ZyatYjb8eteZ9mAq5Z1t2HVDzi3mHrS3xsZAq1dyvBlNNCs2F9FdILQLb5h/PoUmSGd6wksmGyzzGBNhZHCeCFggtX8z9++diGTUQytv4/6kKsyA2nIb3N48vrTwN5fi68rNIy9U4r7kfunsT5WGbN3RdvuRso+4cGo3DsbCvn90wWbevsZ3KTaJda2IRf4LZgHW70UK16TZ6jQ92BxAYOw+SGYq3yOtcWrWIPtWhJ3zb+F1TmPlnadyn+0pRMakUJrV37OYOBmstN939C370X/DFvv0Appfz1Wsymga4U7NMtIIrOpoZ/wrSfIMnYesk6lq0nhXCssf801BN2Htj6lF/3kfPQl6BFZP5GbS7hhVX44tY6wB0GrwfCNH0MkSk X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2026 15:55:36.4650 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: fa89116b-3ef8-45b6-6f6c-08de72f3fc33 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF0000C37B.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB9176 Content-Type: text/plain; charset="utf-8" From: Ankit Agrawal The EGM regions are exposed to the userspace as char devices. A unique char device with a different minor number is assigned to EGM region belonging to a different Grace socket. Add a new egm class and register a range of char device numbers for the same. Setting MAX_EGM_NODES as 4 as the 4-socket is the largest configuration on Grace based systems. Suggested-by: Aniket Agashe Signed-off-by: Ankit Agrawal --- drivers/vfio/pci/nvgrace-gpu/egm.c | 36 ++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-= gpu/egm.c index 999808807019..6bab4d94cb99 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -4,14 +4,50 @@ */ =20 #include +#include + +#define MAX_EGM_NODES 4 + +static dev_t dev; +static struct class *class; + +static char *egm_devnode(const struct device *device, umode_t *mode) +{ + if (mode) + *mode =3D 0600; + + return NULL; +} =20 static int __init nvgrace_egm_init(void) { + int ret; + + /* + * Each EGM region on a system is represented with a unique + * char device with a different minor number. Allow a range + * of char device creation. + */ + ret =3D alloc_chrdev_region(&dev, 0, MAX_EGM_NODES, + NVGRACE_EGM_DEV_NAME); + if (ret < 0) + return ret; + + class =3D class_create(NVGRACE_EGM_DEV_NAME); + if (IS_ERR(class)) { + unregister_chrdev_region(dev, MAX_EGM_NODES); + return PTR_ERR(class); + } + + class->devnode =3D egm_devnode; + return 0; } =20 static void __exit nvgrace_egm_cleanup(void) { + class_destroy(class); + unregister_chrdev_region(dev, MAX_EGM_NODES); } =20 module_init(nvgrace_egm_init); --=20 2.34.1 From nobody Sun Apr 5 18:20:02 2026 Received: from SN4PR2101CU001.outbound.protection.outlook.com (mail-southcentralusazon11012041.outbound.protection.outlook.com [40.93.195.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5409D2F363F; Mon, 23 Feb 2026 15:55:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.195.41 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862140; cv=fail; b=qC7Q8zdq5UxDN7NBQt5n5MXAzMEo8Hzmy4iE0zkiD2kUvlbNmPglzk5Y0m+sZUTKCmkQN2v2ltfDxrAQYbquhy0B3WuruuY1GrXDL67WgieIIjMxRhVw76FGI/8GZhKAHzD1OuQeccPRxGHU1SGLPYptt7O4p7WGbWYD5Jg0vV4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862140; c=relaxed/simple; bh=YO8moe29foWCl3XIUM3N3zbxtanwcSlho464Xzcr4as=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=A+FKOvUezt7UCKehCot8fFhBX8L5/3HTgvzFtcj4LJyvhVfEFCtQmrDguqTXztQ6WW/l40m63LK2BTogzg1HIhOwaf/F/oRuIfAD+TKVjlfT85KIPHdrPgP8X1qGEQ3oP7lk4qKiLSABkY/uWj3C/2Z7dVXzMODqbzR7icQojJg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=MLOQ8roD; arc=fail smtp.client-ip=40.93.195.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="MLOQ8roD" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=GHAtBp7zX1eUiFsl1tMsnGOLWhxKrdJuaZRI+f28o9i6I22WKFI1XLqemqc30kcnsAiZH8nccFPpm74Ylzmn8g4nbC+rflZASlfhdQUX1VLf2HyUxsArAL3AuDtjSyl8D8GZQHVgUFoO47mY9V0Ky3HIK9ece6fBBgy5kuTF9ZueITY7uQqp7znAuyuvfDPl4/gtbcr9Y5m54c2RZrmyLhLbx8z1eoN5Kg3S1eQgV7oILQqEuhGFHTzo1CTF534JCUbASSo36ZS+NIInHI5X9xKNBRdHjF596ZvR/dtF55LWKZH0ZGPuS33Da1NuZMDRvdzJFJXAzXoUsP3EjBQVfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mo8VLK1SUlVSQGIwKVqulaxuYfV522PjTVGPrkJfYhw=; b=KWcwTDeKBxp0abocJKnEACbU9PBZof6YN9pjiTux3vlDzm9W7Ccd2afP5c1UTpkSrKK7q8nu3MzQOccBFJSXEBTRKZluL1eSCu+sRCsDPzWQLC9L3o7D3cg47kaykE3QEfgjrkkAk4M2tS8XyTxkO7HMiqwZQXjzSVIKZQQ0iFM7Xk4lD/4l21yroilq+vIDM1Vq/bz9UCkXhZqu7Kt3T1cb+tgkWhPG5xb6qhIJ/F/95UpCmDDXOkVLiP4ng1eIqKHkE+JgaYVLx//0jG6H0yT2kfg+mt6ymoCM0FZbEDD9Y/+mO4wEHKtt7HQYFRemYF8GNZTS79ZB4zmilj1WPA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mo8VLK1SUlVSQGIwKVqulaxuYfV522PjTVGPrkJfYhw=; b=MLOQ8roDYFlsK5vV0S/vYlI4Obovr2HpA9CMBVdOEuJMHwhF9Vfm0O7jAz3NhBpyz+oJaECeP9iTKKaZYQGBVSzHOgGiU46T4yYjDMHoxFERvBZX4F0Uexwqsp2au+0P4iRWpMWk3VqtcsiMLxGrKtY5qd1Iy+GKUn6awVbmT8jLK94JzHwz0WOoCqSv1drAVPBqD82GNhC9tieV9cCnQRrCOqdu47CZSiCJmamViEIXllF+6KCc/wou9yK33LB0kBo9fnKf2NnGM7YLR0nYTWk/PI2A7+FHSbpb+cVqEUR6FzlH+NyJWIql8/SpxqHzC+qVS6OeM3mlsP2O3F6HeA== Received: from CH0PR04CA0040.namprd04.prod.outlook.com (2603:10b6:610:77::15) by CY8PR12MB7634.namprd12.prod.outlook.com (2603:10b6:930:9d::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.21; Mon, 23 Feb 2026 15:55:36 +0000 Received: from CH2PEPF00000099.namprd02.prod.outlook.com (2603:10b6:610:77:cafe::42) by CH0PR04CA0040.outlook.office365.com (2603:10b6:610:77::15) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9632.21 via Frontend Transport; Mon, 23 Feb 2026 15:55:29 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CH2PEPF00000099.mail.protection.outlook.com (10.167.244.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.12 via Frontend Transport; Mon, 23 Feb 2026 15:55:35 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:18 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:17 -0800 Received: from localhost.nvidia.com (10.127.8.12) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Mon, 23 Feb 2026 07:55:17 -0800 From: To: , , , , , , CC: , , , , , , Subject: [PATCH RFC v2 07/15] vfio/nvgrace-egm: Register auxiliary driver ops Date: Mon, 23 Feb 2026 15:55:06 +0000 Message-ID: <20260223155514.152435-8-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260223155514.152435-1-ankita@nvidia.com> References: <20260223155514.152435-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000099:EE_|CY8PR12MB7634:EE_ X-MS-Office365-Filtering-Correlation-Id: 1385ea5c-2b10-4566-2a85-08de72f3fbc1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|36860700013|82310400026|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Iby/Jb1605ccQhd8yUdnDk9SEz4BQ5FS6rqum03IT0AGBHxXDB9uX0iTqrnf?= =?us-ascii?Q?TD6msLPkUe42VfRrO3vBGLwz6F5Z/t0yJiv7Rp5TspS10YidQpP5je4QG9pE?= =?us-ascii?Q?x9/II15UHGZGXYj0ODF7WQoi535jeE2YZ6Pn/vqK/TAoxCmUTJaxk1vsXr5B?= =?us-ascii?Q?W/pv70ztDlpzXVPP9Rme6IsgxKoNxhsKMJxen3rF55l/yKOCYjhNBPq/maju?= =?us-ascii?Q?kDCOaUkboA7gpxTXsvFWqFZX0PUsBUQ7kXVEJvfx5Qna6znvvfrAfJzL6hFg?= =?us-ascii?Q?JcjkNdK4JPVHIIXkTgCCNgaFn1I3NgZkOPAZBWxYooELdBjzEjEb8eZAM8ti?= =?us-ascii?Q?F1dOmpYsC9CKi/YLlgjYhFb4r/zW++im8lpMpl6erHP9NDc7I8jnZt57/14m?= =?us-ascii?Q?8lI3SzygcoaogAeL0jx/YGuA1o9xgFysb+8jRawO7UYGKk5NWTu4nCJaPrHV?= =?us-ascii?Q?pMtFX4vmi5g8IdMeUgR7mVgd2v6yqLMTAhwK55KAxUDWxrJeCQLAd5ywvjnE?= =?us-ascii?Q?usYLdr4dWwyx8c76f1xU17hs1Lcy2o0hOXqNC+cjQUHo3ABXDmI0nqjH8BLK?= =?us-ascii?Q?NIZ05yrvE9NqpQVrFZM3KF8aFrPha4JECkUd0puLFmZdoUUZDlD5sqW+49OA?= =?us-ascii?Q?0j4Pp5yZA/ViC/r5R6FY6L3zeRc8jNj9YFUTsEIQbYbB247ATauPiwhbcYBa?= =?us-ascii?Q?RTSc/NmjrNWLUNvg78dWwwlIT+XZB4fncDLHsIUi3zQ7qoHW/AYr9H3tukWY?= =?us-ascii?Q?q3A4Ugi8ki+vkxFcvPyfFkCchrkpg4GGaKnCBsc0AyxDVhe6xStdhtUl0AI2?= =?us-ascii?Q?Ht/XVGhKTUx4sWVna72z2Sw01C7mDdKujgDDpTHEMjbrhDHK1hqBuIdjluFo?= =?us-ascii?Q?1EX0nhQzLDiqT9vuRe1WG6+cUv1PNk/8Aegbxf7MWT+3RCzlhZq0+uS6RAA1?= =?us-ascii?Q?pVnguCTN45xOjIlOrlJfAg9w52Rw39GuLAN5ufXabOIznjGOK20DYJ6brftr?= =?us-ascii?Q?uFpS02SGbNquDPWVq0cM4uRNl8zTl4dZBd8rq2bIOTrLgpb+YD1vG9N4pzOO?= =?us-ascii?Q?gzk3NJktHmC5q852gNsrd6TneLKw/vUUHNSxsnJiHo87D3sFOM9l6iNJYHJJ?= =?us-ascii?Q?TmhAo7Dtbk+4rEYj2P+LpZO6IOZrRpR6SNSgBR1djveRXm1BMQS5/1F0CtAN?= =?us-ascii?Q?c56uEJA2CCHczIsa4iVsF/QKrfKKfAlbbWBGC1MvMuZYsQj3mzIpSa/eRkTo?= =?us-ascii?Q?iO5fs/hn8ZjnZxdj8mwJ5zB5jmKXzljLM/avK+J8UMuiodA8Iq6NyPGcUxEO?= =?us-ascii?Q?K2t8N/LCDo/9HudxjqquLdn5pJH6vHYLwRqxBoXlQtb7fw0fvCPdy9KbxYd/?= =?us-ascii?Q?goL3dd/YFmekC0Ktnh6kOPBzlrJnoVulf2L47f2sCgTPL5jybtG1nEkEg3ly?= =?us-ascii?Q?OIuI5Pg21MHwod+vxIMXZr7TpmIGJsSSTEPji/L2CNBuutq8XQeIChS0M+hQ?= =?us-ascii?Q?7dhnbtkc31zTtGXVK3oMTlwGmwLcAWmj0vo1Rlqt65CzRQX2swytCj1sWNCq?= =?us-ascii?Q?SkuRB0XOcF5QCuIOHhbuzxxcCri+SOa8MeBDuOJFmarcXj18Ag36TGR1Slip?= =?us-ascii?Q?vaBPcNBiVc2LK2rBFOEExQU9WMUe6jOPQJHIkSGctFkT0m24qFpIhJHQ2uoH?= =?us-ascii?Q?+oOwlg=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(36860700013)(82310400026)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: tTtndsj40g2S1KPElDstLu0DF2eEGJginW4+t9Ghba3JFGVvXjsP5BNXf1MvjwoeAbNcotAcGz+l7jO9o1xSTiEoUcMCkOmjBuKQxbwSGWWMyEBTVhYK0eH8v4yIAQIJ8UJsQEscQa81Q4Z9hUzyO9YQR6/Gg01/34WXWN+ZyNecBO77sZYkmVC8SXyj7QaCk4C9S8kKHFCAjfTYw78NDgxrsPjYHQmhbqPX6N6ZqoPIZVImbisF8Ej9fAclXsJnXCkj5WUeq1HWSmyL6vaz9PK9S1ov9S2zLoLIjjfnCU/UuhNqWYMmqqE+83Hju7QeYlVVQRbXVB6W1ru/u427cl/O48aMyVlQjN668ZDbFirXfO77prlTpVC/4jfZceGJCeqo0aRtSUKuZXXtYEGOZVtOrBdRdQzRgFaReITIiRHiHa9ZA8jxEqKK3F14XJMr X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2026 15:55:35.8072 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1385ea5c-2b10-4566-2a85-08de72f3fbc1 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000099.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7634 Content-Type: text/plain; charset="utf-8" From: Ankit Agrawal Setup dummy auxiliary device ops to be able to get probed by the nvgrace-egm auxiliary driver. Both nvgrace-gpu and the out-of-tree nvidia-vgpu-vfio will make use of the EGM for device assignment and the SRIOV vGPU virtualization solutions respectively. Hence allow auxiliary device probing for both. Signed-off-by: Ankit Agrawal --- drivers/vfio/pci/nvgrace-gpu/egm.c | 38 +++++++++++++++++++++++++++--- 1 file changed, 35 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-= gpu/egm.c index 6bab4d94cb99..6fd6302a004a 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -11,6 +11,29 @@ static dev_t dev; static struct class *class; =20 +static int egm_driver_probe(struct auxiliary_device *aux_dev, + const struct auxiliary_device_id *id) +{ + return 0; +} + +static void egm_driver_remove(struct auxiliary_device *aux_dev) +{ +} + +static const struct auxiliary_device_id egm_id_table[] =3D { + { .name =3D "nvgrace_gpu_vfio_pci.egm" }, + { }, +}; +MODULE_DEVICE_TABLE(auxiliary, egm_id_table); + +static struct auxiliary_driver egm_driver =3D { + .name =3D KBUILD_MODNAME, + .id_table =3D egm_id_table, + .probe =3D egm_driver_probe, + .remove =3D egm_driver_remove, +}; + static char *egm_devnode(const struct device *device, umode_t *mode) { if (mode) @@ -35,17 +58,26 @@ static int __init nvgrace_egm_init(void) =20 class =3D class_create(NVGRACE_EGM_DEV_NAME); if (IS_ERR(class)) { - unregister_chrdev_region(dev, MAX_EGM_NODES); - return PTR_ERR(class); + ret =3D PTR_ERR(class); + goto unregister_chrdev; } =20 class->devnode =3D egm_devnode; =20 - return 0; + ret =3D auxiliary_driver_register(&egm_driver); + if (!ret) + goto fn_exit; + + class_destroy(class); +unregister_chrdev: + unregister_chrdev_region(dev, MAX_EGM_NODES); +fn_exit: + return ret; } =20 static void __exit nvgrace_egm_cleanup(void) { + auxiliary_driver_unregister(&egm_driver); class_destroy(class); unregister_chrdev_region(dev, MAX_EGM_NODES); } --=20 2.34.1 From nobody Sun Apr 5 18:20:02 2026 Received: from PH8PR06CU001.outbound.protection.outlook.com (mail-westus3azon11012001.outbound.protection.outlook.com [40.107.209.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E945930BF4E; Mon, 23 Feb 2026 15:55:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.209.1 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862149; cv=fail; b=uv07MbN6NKUTqgDb7OzgFfAH70QuVgUokJZ8htyMxfFtlYxu2PD+BXGyaASL4KIZ+AqInZmbIOFmtGD3IBj9ioL0OGOv4v1g5ZAPZFFMHuyGB/9uZHcCSPaMhlGYeHiIoxImk7Zvbu6ROmHLqepq/T3Wr3okt5Ab3L2cQ8oZN4I= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862149; c=relaxed/simple; bh=0nAnMzpd4HZk6HLjn3GC6O2yU1yT9WOtR/w7lr7UGZU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=AQI2RDRKwxTmqHTywGpMyTZcDVimKN6YPiXbuX1Y0rmewBWR/B7ICm/ID2C5A0m2JDljM/dpJWbS1FboGdNMpOzgtXH1UwfI6MCzLm4iQfWfXGI3FYOZ8nplQZQBbQyI5cojLLHVKAe8iqAT7Tl1cvvaYGspWpCZxecYMirl6uM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=lVasQ/b3; arc=fail smtp.client-ip=40.107.209.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="lVasQ/b3" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=h1gSelVqFAuaqjpQF7L/k9m8QvwLX9jXoA9vIfKky8pPbTu56skw5I6Zo0tCilMANFsmaFk+XNQLviH6kz9hz1R/N6SRZWjSggSyZoGwBQ44sff6fg8GolZyGIWNGQl/1EuzvaQmcSDhMdAA3DVioPHri+oLocfspqKAGyXw1TTggSch54zu7CyiHVcCHTw1/PMZ7JzNjhc9UL3QbvA7mKonDVMbvtuuOdQ/nWqgsbvqkPNL2ZuUI7oiDv8E/zKsjTg7XqWrfiJkY7FXTmq+klw8QcDOyfiadecCWAj+zWUd7CuuEBOlMd0iHNcYZs1lZQP3UptJeiEsN8diNJlkWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+I25YG4LtbfsLnMnjEVrZbpE+juFFCg8XJlnCv4ZLVU=; b=lW+8Dh7jImjN0fTtZQaC101DWhq9MunJBOknlDVUQZnVl+8kQ6pO60ia6phIIHDmMeIdfE91Ls+YxbdnNZasedTk1xlHbVFYxMe3uapGwv3vYtiEEV6vvUHdxNOP10a9c/wj0cfk20Lpt5lnKI26UQVRQX4BatFTDqc/Bwhud3ws9E/ADZwZ2FWMPbUOvddSuXFWmrwY3/vzNyXJ4BKRgssuL1uYZCVygaD+FQT28iQzaljQybEr1UVBkA7YCEW3yQ8jSIdfoZgK64oGyMuxdGzMpFk/Gs0iDVuwOkQXsqhw0Rj8jD4mD/hE7OmV3TpJsFtUcMmv8VtjktvzUqUe7A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+I25YG4LtbfsLnMnjEVrZbpE+juFFCg8XJlnCv4ZLVU=; b=lVasQ/b35qgEIJZIooEuQjIPuDC7B4KTooOehQ38mJklA1Dy08988YBsCVdF8cg0ZIciZx93iEmf3gPO1cw3TD0k//SKcwRhnCwLvrJ9ZKCNnV3RYrEVlDgV5TTRHg23aHpX4mJ5xTIL56tQO4P8QB1wNR6RidQdr2rmVJd9PfQyyjvtV1NDxVg1itJOR7sxX3jwbgltOFG1OGx2MiojoyVx1mcULdkdqF5PjwwOuNMK5a5m5N2NfP2qK2qORuy74HpMF7mQ3+Jdd6a2Dni7ac8UibQZMMsT+h6Hmrikbu/PrMrTguDIn5ZiMZK42UIoR3HLmkZ6uFScw4Ojx3aloQ== Received: from CH2PR05CA0050.namprd05.prod.outlook.com (2603:10b6:610:38::27) by SJ0PR12MB5634.namprd12.prod.outlook.com (2603:10b6:a03:429::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9611.15; Mon, 23 Feb 2026 15:55:38 +0000 Received: from CH2PEPF0000009D.namprd02.prod.outlook.com (2603:10b6:610:38:cafe::8f) by CH2PR05CA0050.outlook.office365.com (2603:10b6:610:38::27) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9632.21 via Frontend Transport; Mon, 23 Feb 2026 15:55:37 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CH2PEPF0000009D.mail.protection.outlook.com (10.167.244.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.12 via Frontend Transport; Mon, 23 Feb 2026 15:55:37 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:18 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:18 -0800 Received: from localhost.nvidia.com (10.127.8.12) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Mon, 23 Feb 2026 07:55:18 -0800 From: To: , , , , , , CC: , , , , , , Subject: [PATCH RFC v2 08/15] vfio/nvgrace-egm: Expose EGM region as char device Date: Mon, 23 Feb 2026 15:55:07 +0000 Message-ID: <20260223155514.152435-9-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260223155514.152435-1-ankita@nvidia.com> References: <20260223155514.152435-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF0000009D:EE_|SJ0PR12MB5634:EE_ X-MS-Office365-Filtering-Correlation-Id: 3845b4f7-7aa1-448c-f2df-08de72f3fcdc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|36860700013|82310400026|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?vAVEM4KCrUw/Ab7PQNmDyA2J3SsuebhkvVaIN+crqNefOVuBQ2GR4Gw/iJlG?= =?us-ascii?Q?Gr/Q/nd7GHGuM8blEnQYCrPFscOjkU86mvuHxdQt79WAynJYg/El6fCL6rka?= =?us-ascii?Q?H04bUn0zDOIww7aTePfk//vbtVwLXmYfV+ue2uMV39QUmTDYJpl59J3Sde0g?= =?us-ascii?Q?+4y1dWiQerpbWjm4IUVqlAYsdkJ/5I82tVP3EQje6Bv8osAG6DSuCvNxsweU?= =?us-ascii?Q?1IrjZi30ER3z3hp3IVoMFMxsgi0O5CIbUZt0Nq/iSubzTvmgjVlZE3CbVqkj?= =?us-ascii?Q?hI1/wvjwca3s/Gz97+o0TlTxjuw7i/HjpvEmpwWbH07h8sZJuHFNSd3zE0/5?= =?us-ascii?Q?h7TafMssIz9o4w00/nW9Dx+SUUGImxAPirI4dvRln566P4mDzQsYO7sAsgvu?= =?us-ascii?Q?0oZ5CcJ7/ZgMOAM0PEBQL31l/5TdQl/khIruoVfLQoJROd+G24lwcuVahW91?= =?us-ascii?Q?keUmuH1m3d2GFLDZAD1FTFtEsx/kKvsxOL8vQX/6BHSPlc83iSgU2RtbpeFk?= =?us-ascii?Q?wqnVDrGyijQaYSFeLBFPOcBOa4KT7ePjLiAinOsrnvCgLbkTEXSszcF/TjpW?= =?us-ascii?Q?Xf4VzX5Qp4QKRWedRJhRN1s3dDdukFUP9ZKrwSlQpJWy6aa1r+I7R0vlVC0s?= =?us-ascii?Q?iJinVFI+YJ1ofabxlGu9bpqaDE9xFR+qYAT3GI6WA38E28d+HYOHGw/D72zr?= =?us-ascii?Q?bpKF6kKrSwtDRe7W3X850Xz3gY9gx23/z9HhccAFSZ1289TUJflBM/PKU3E2?= =?us-ascii?Q?BcYx2Z2JTqwTnIrPRTX201uhHH81OHJ0KgLtN85hlwhHnxkrRxsadJf/6j0x?= =?us-ascii?Q?3Nv0U8QvJVNRY2J8/Y9vhLqHiCL6HjcxfZxFKfqHTaECeuyI1dU3CYJBMDWJ?= =?us-ascii?Q?hbHu7oUztgwXbhX+SWCbOd2n9stIPQAjnKtnmdCSYtZLyHyV79rt0ZqnGMmr?= =?us-ascii?Q?Ez2szK1FyJN7/QHR474wFdjc5mG9R+kBbuF2bQcXIzUdzRHAiC15sGypztMu?= =?us-ascii?Q?iEA2A46nJQXioUlLXF0DwZok9NU2RVwxuZJlkRwOIpIpoeHc3UxwM2RPE37m?= =?us-ascii?Q?2c7mM6q4EZhwqLUUHl9+hDQv8aed6Ut66Z+8h/ef9ERQpIATdP9eg9WVy2ZE?= =?us-ascii?Q?re0XFv68RrMVpexcwXHZW9MZRfB68oVHoLNxwEI4ByV0AW5fU8UF1o+ZNzFU?= =?us-ascii?Q?SE54EEgjXF6fDWBRK4fOkcYGBjCLekKrANQCa0wU24EbIT+uXBHZm1wOJP6P?= =?us-ascii?Q?zsN8CEDnSkZTg102RfiktEzy2H3H3zJxPMJbThhGRPoJohbIp+h98chqt6GF?= =?us-ascii?Q?JwPBgHK8BBRth/e92Wtr4jZYpX+DOmwEHRveKwmrRUYKha00Cbx/pBM5lceh?= =?us-ascii?Q?ingrVnIXAMNcCKPF0s+KaZc5053GwLosNxE41kdLIeE0ZqbcQ5Y+QDTl/ZQ+?= =?us-ascii?Q?6770pgUFLmY0Ks9VoQ5PMT5Vs6VszlwF68JoYeNI9X33AZh8FX4TeEXyKwbT?= =?us-ascii?Q?R8iu3N1GGp9SaSgQ15fzk6axFvBajPkkMod7aaZHCHseG28oVa+vPUJWSBX0?= =?us-ascii?Q?BwP9F+fhYBPmyAf68623MHHc/GWT0JRqIFPl64oq1njd8eWsATjjMA/a1oQN?= =?us-ascii?Q?W1O6Bm61q1u4uhTg6193FTheIL8Mw8SrwQaoBKE01qcCgMzdAX5D8jiQGwNm?= =?us-ascii?Q?LOVLKQ=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(36860700013)(82310400026)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: qNq7v5EjV/iElsuPUOpBlI2NvtGGGM0SvPDF1nOe+ZvedZRFOqOxu10FOTBd4UvFrPXje8nWufT9bc32HrcCTVnUJbpHWtXi/Q3LFnZSk2Kiksb0p3qwXk09SXZ9YalLe1rD7JzOfC2l3yAXFsQm9HESmXRGekjTouOl9PXN3qaYvoYkxrleQEyE4R5L8EDFZCa5FgiIzCcwO0C4mHZ68LeQEewb99QqCLvKormNFcMg83YDfXm8Mu8NyjivY2hfvZFI2lnwfXQDZsRr3PIEsK438yTX31qdhckHZxL4Ndp0J/nNfzNoRU4+uyB53CwxbyC7xaKG7wt9OOMQzwssyBKhFUiE6PTV4+O2Qm9APUoc9iTeKmrJf/MMZmO7jIudaMOndioB/XVvGJBlCLS3HKhgNkcRkD6AQwnnWtm5Ey5Bhp2hGElME5VRQUzvh8// X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2026 15:55:37.6683 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3845b4f7-7aa1-448c-f2df-08de72f3fcdc X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF0000009D.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5634 Content-Type: text/plain; charset="utf-8" From: Ankit Agrawal The EGM module expose the various EGM regions as a char device. A usermode app such as Qemu may mmap to the region and use as VM sysmem. Each EGM region is represented with a unique char device /dev/egmX bearing a distinct minor number. EGM module implements the mmap file_ops to manage the usermode app's VMA mapping to the EGM region. The appropriate region is determined from the minor number. Note that the EGM memory region is invisible to the host kernel as it is not present in the host EFI map. The host Linux MM thus cannot manage the memory, even though it is accessible on the host SPA. The EGM module thus use remap_pfn_range() to perform the VMA mapping to the EGM region. Suggested-by: Aniket Agashe Signed-off-by: Ankit Agrawal --- drivers/vfio/pci/nvgrace-gpu/egm.c | 99 ++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-= gpu/egm.c index 6fd6302a004a..d7e4f61a241c 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -10,15 +10,114 @@ =20 static dev_t dev; static struct class *class; +static DEFINE_XARRAY(egm_chardevs); + +struct chardev { + struct device device; + struct cdev cdev; +}; + +static int nvgrace_egm_open(struct inode *inode, struct file *file) +{ + return 0; +} + +static int nvgrace_egm_release(struct inode *inode, struct file *file) +{ + return 0; +} + +static int nvgrace_egm_mmap(struct file *file, struct vm_area_struct *vma) +{ + return 0; +} + +static const struct file_operations file_ops =3D { + .owner =3D THIS_MODULE, + .open =3D nvgrace_egm_open, + .release =3D nvgrace_egm_release, + .mmap =3D nvgrace_egm_mmap, +}; + +static void egm_chardev_release(struct device *dev) +{ + struct chardev *egm_chardev =3D container_of(dev, struct chardev, device); + + kfree(egm_chardev); +} + +static struct chardev * +setup_egm_chardev(struct nvgrace_egm_dev *egm_dev) +{ + struct chardev *egm_chardev; + int ret; + + egm_chardev =3D kzalloc(sizeof(*egm_chardev), GFP_KERNEL); + if (!egm_chardev) + goto create_err; + + device_initialize(&egm_chardev->device); + + /* + * Use the proximity domain number as the device minor + * number. So the EGM corresponding to node X would be + * /dev/egmX. + */ + egm_chardev->device.devt =3D MKDEV(MAJOR(dev), egm_dev->egmpxm); + egm_chardev->device.class =3D class; + egm_chardev->device.release =3D egm_chardev_release; + egm_chardev->device.parent =3D &egm_dev->aux_dev.dev; + cdev_init(&egm_chardev->cdev, &file_ops); + egm_chardev->cdev.owner =3D THIS_MODULE; + + ret =3D dev_set_name(&egm_chardev->device, "egm%lld", egm_dev->egmpxm); + if (ret) + goto error_exit; + + ret =3D cdev_device_add(&egm_chardev->cdev, &egm_chardev->device); + if (ret) + goto error_exit; + + return egm_chardev; + +error_exit: + put_device(&egm_chardev->device); +create_err: + return NULL; +} + +static void del_egm_chardev(struct chardev *egm_chardev) +{ + cdev_device_del(&egm_chardev->cdev, &egm_chardev->device); + put_device(&egm_chardev->device); +} =20 static int egm_driver_probe(struct auxiliary_device *aux_dev, const struct auxiliary_device_id *id) { + struct nvgrace_egm_dev *egm_dev =3D + container_of(aux_dev, struct nvgrace_egm_dev, aux_dev); + struct chardev *egm_chardev; + + egm_chardev =3D setup_egm_chardev(egm_dev); + if (!egm_chardev) + return -EINVAL; + + xa_store(&egm_chardevs, egm_dev->egmpxm, egm_chardev, GFP_KERNEL); + return 0; } =20 static void egm_driver_remove(struct auxiliary_device *aux_dev) { + struct nvgrace_egm_dev *egm_dev =3D + container_of(aux_dev, struct nvgrace_egm_dev, aux_dev); + struct chardev *egm_chardev =3D xa_erase(&egm_chardevs, egm_dev->egmpxm); + + if (!egm_chardev) + return; + + del_egm_chardev(egm_chardev); } =20 static const struct auxiliary_device_id egm_id_table[] =3D { --=20 2.34.1 From nobody Sun Apr 5 18:20:02 2026 Received: from BN1PR04CU002.outbound.protection.outlook.com (mail-eastus2azon11010026.outbound.protection.outlook.com [52.101.56.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E3CF30AD1C; Mon, 23 Feb 2026 15:55:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.56.26 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862148; cv=fail; b=SpIa8z2OKuhdBi6Lz7P9UzvesoxEA4lAe9J6s1gMSeFu1Nxr3o2Y6/ttaer7R+jZlm/5Ym1eBW2Gt+xui5Z9HGpULaqzhY3iqE5bSDaadgvIsctUM6LJHKn2b0c06LwNdrnwzEOUtFmcm2BMZjnw9bodknsf3mm2FnHn1TlHgHw= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862148; c=relaxed/simple; bh=bOGbzPt8MfbN/aZ31VmNMTcjTO2DVU8Ic/EGhVEvRds=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=bPFj6VbSF0eRuu5t0inU+TAIq7pOo8+tRO2JJKZ4z/x365cxAusAqqCWnAaOwpRq5vzlZ6/nFwBlrFEqHASAtarKuu4/PSIijzIq6bxrmIEDv3XC3895YF6HWfD1VqkhpNA4h+gF69of/S8t0LX/CKn+lwxtVK7GGrLtQFFOlPo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=kI3cHlx5; arc=fail smtp.client-ip=52.101.56.26 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="kI3cHlx5" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=XIdjSqQHdrVkfNVSQLY+ncWmLypdToLCxPzucT3eUL2MiHVNCZj7AUHxi0j4o+2HPSf9aozJQJiqlMzpFDmV4n+MFB8ORM7tYB88+17Bnoylb0ldFzmZq5Cclj2Gz4CYaLxb51i0aMw/scv1h51gV5AKm3XfiiFPUK9YJVqyYuoDboV+Exhx3p1TKN50tFNz7ZhVM2UqIRA9w8pmRqH+1bE0Erwr/70yj3N/70l0fiKJhZF00//ghyF1S98+8GAuiougolNP2dKs8RWB3Cc3wPoOe9MLRodYAIVCz0jTl+zcZbznMoEQa1Q4YrpQIeB3/42ZhhaMIc1TREKRI6iZ2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=30IoT7g6Kq7hzEuadLEEVccSblTtGObHoN1VGPOrVmQ=; b=s2il03oFw9r2GKLnHKLCeAy7eTs6KA6gZcQyLMy+QZLsVrDPMOi8aKebiXCGolRoZWWSedMUqGc4JuBB3WjaToVlpluX0dXDgxptrlcnrNkF4B70i7pkVQF9mRG33pahNlHMHxOjiDSimQlxlZxXF5GKwoklfNywpmzbKcaZ1xT7kQhiWo82IyDT7rmGGBkKpmHa5kv2hgYJXEcOznxQSLbqYmDUWW/9q5u/4CbUVqv/ErhLkbXFSqXaJkTgI43HmQr3SmrBNyJwRVOtCGwEoh37SG8hzwjmacjD2vGqFLruSD0+abDjmNbs4HfdSULMcsgV4iUbrcn1Ot2DrmbgXA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=30IoT7g6Kq7hzEuadLEEVccSblTtGObHoN1VGPOrVmQ=; b=kI3cHlx5Y5fJuffDmM2mnb/wh4scnL9sVdSNq4NY2JnHocAUuslaUW0vLTuDrZhcK1yhNZj17bIhskaxRZMU4wjyMA4y+aJsMhQOE90h9l1JTobEfHOApfF80cB9Em9Nk8HaWKabgG8AVlBP0DG7BywOOezlnuYBRQnIMqfBcqglQg4RFz0pLOCjLna3Jmm3KW6YR+/Bh4tsqoOwUOW3zbFYQQ6JECcMItSbjs0sTM/cL7pbL6fTaWXWKmuIC98vtxryIZUxlxCfdzwsOTpvJ3KO31fihxNrM8wjuffA0PeYgkvbaai/r/qm5N+R6c+buPWIvpyf38c7s84FboqXJw== Received: from DM6PR14CA0067.namprd14.prod.outlook.com (2603:10b6:5:18f::44) by IA0PR12MB8975.namprd12.prod.outlook.com (2603:10b6:208:48f::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.21; Mon, 23 Feb 2026 15:55:38 +0000 Received: from DS3PEPF0000C37A.namprd04.prod.outlook.com (2603:10b6:5:18f:cafe::c0) by DM6PR14CA0067.outlook.office365.com (2603:10b6:5:18f::44) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9632.21 via Frontend Transport; Mon, 23 Feb 2026 15:55:37 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by DS3PEPF0000C37A.mail.protection.outlook.com (10.167.23.4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.12 via Frontend Transport; Mon, 23 Feb 2026 15:55:37 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:19 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:18 -0800 Received: from localhost.nvidia.com (10.127.8.12) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Mon, 23 Feb 2026 07:55:18 -0800 From: To: , , , , , , CC: , , , , , , Subject: [PATCH RFC v2 09/15] vfio/nvgrace-egm: Add chardev ops for EGM management Date: Mon, 23 Feb 2026 15:55:08 +0000 Message-ID: <20260223155514.152435-10-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260223155514.152435-1-ankita@nvidia.com> References: <20260223155514.152435-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF0000C37A:EE_|IA0PR12MB8975:EE_ X-MS-Office365-Filtering-Correlation-Id: 9de9ecab-c1da-418a-2343-08de72f3fccd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|376014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Y9/SQLhRAHw/dXQ8geoIpsesfQw4sn8Xdmgx9JFLK2e0jCBHoTfSrOL6M+H/?= =?us-ascii?Q?Yvv9FEz9cyaizQft0BLpZG5O+axz8KmXj3XtYohkf0PPvAS0RHe+1XG/RQrm?= =?us-ascii?Q?blR0zrfKzT+TY5Jrx/qzR2kgDGf8QnuQmfBpQZJ1PGtmzoMQj98xniUmU2+J?= =?us-ascii?Q?SHn+0DnCaaCr8zhKrd4FVEI9eDfPKlQyZwY+5Yp3sOU40xE1VeHw+D2Qt0MC?= =?us-ascii?Q?jgbirYwJissk8e2l9PKfayKeRUG+y5rlQwjcSRM8MIuUe9ZhFxRQaxG+xV+L?= =?us-ascii?Q?tbon1kfWu6NA2I9QxIS9TBqSbfU9rWdPbvEOBniCBlz7shW2HSS5qqcGbqjt?= =?us-ascii?Q?eOEMnr7eFWD+FAZhZq93/wSEg560Wv8hBnImuw4QasUMvdTtsyeKWR+TkNVE?= =?us-ascii?Q?NrGAoYFtzVfdTppD0BUFxZY40V7uaHboZoCK46DHoZOG+rzAYD90FW9rkQ65?= =?us-ascii?Q?SXRz8a5oREhYeomXBc9cG3IqUOFNu+bP6vNYsRscivNAuXGLrRPZpHTB64Zc?= =?us-ascii?Q?2vUxAxJzH042b2an8bH9HwTmysNwzDyH7hpnvBTNoNUQD+wZf/uz5tPM5gn+?= =?us-ascii?Q?d6oquVIZPLFbcnrILfVNnon5Pt1dFPEKt9K079Zi7DN8qjWNpIl0QyjLMtD5?= =?us-ascii?Q?oascHpjk4EssAsMe+o9n5xH10oQ50n6fUo5R24ITdP0mcKsK/+LLW+hx8zLd?= =?us-ascii?Q?IctVYuc30n2zJDVceMo5QBCg4Eeg3ke5U/Vo6Pt4me9WUipa1n6vY3hTRdOx?= =?us-ascii?Q?OrBHSv9xXG1iBI4rJSc0DFZk7oQ0pH54U44eEs8X0McIbhXzv5vHsxFyPky8?= =?us-ascii?Q?IH7SYvh9N2MWoWhIdotSNhBej7OexANjTvlwdtFgMOplQudY155vKz/YNX0d?= =?us-ascii?Q?DFzKQYNmYuy8A3IcPx5iZh97aVzaJEHwurjc+smKbqFmn4+9poTD+1eMTuOD?= =?us-ascii?Q?sKTaZyBc1l4HWdzryjaoJM+cpquCj2pnSyHmUHhV2XaJ9snRn5nrSmnHk6Vw?= =?us-ascii?Q?txDD4QGKfaz/rBTzspB8qrt7GTq5FFtjM3oUKYkfxhJLHJhBbctCpLVIwXiB?= =?us-ascii?Q?IZePQeLieZaRVvyYwMrdJHo7+jLjZWu6LKWfhwcdCUGnBrF7FmVwGlnDlGaT?= =?us-ascii?Q?Q/pZwn7C2FdNa3luB4Qqd2h2DPqidapBRyqfe8QdRBWbwiOJgAkyCZHDJiWz?= =?us-ascii?Q?C/ITZrfyPIkngn1rEHSjs96UvfSifjOu1f3rSmot9IJTlXYDm8i/wXRgD6t7?= =?us-ascii?Q?tZsqY+3MRP3PM6x3KjVGQWTpYwG8n4Kcp7FTCWT0vm8zsaG6TzEqcdClbKWf?= =?us-ascii?Q?FhMGKp8MdoVfMUlmGltTznNmzjFwK0AgYp5CN/4IRy3HpnufdjX8awaxMbEs?= =?us-ascii?Q?CHTIkCx7Xau4QKeDTHpaF+rFgyggiNtTlkz2ORMxZFJhR67Icf3a9UR2EAIf?= =?us-ascii?Q?raXJVMrTFClPfv4ATApuHCyElhgVE8NO1nhHhQnXwnFBZ1k/gQvd4Ezolsun?= =?us-ascii?Q?5eXNjBqjR9BfnLgccWSy/NatbGBHh+DEr0f16wf33Lz5QoaQh7SVmnQB04PL?= =?us-ascii?Q?B+vLdtnM1BGy1LCnvhoKV1PSMFZUvrRt3fL2E0GsIoAWrEhENtb1VURXNLYV?= =?us-ascii?Q?PyZXVKJuT4IQ1sLDA/rYcc2rrmYHHZk9FxdgdRkRohIwONUw/YRpjQylp+Ib?= =?us-ascii?Q?E6HOyQ=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(376014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 4Z30cAC4cu64saco+EtATyWQe08wB8GNjU5YZwlEfWz/Q7mYxgjAo4y/+RDEvrGsCRMB1sM62VeZVwNw30lgzAy7BEHI7S+cBxkXKXuMl8CkthbAkbq4oJUhDfAWi5it8FRc5dXaq5mv8sxGm80bAtkq6ju40NMdN3om1boLeK8+9tcbwacRlhJTKKhwrD5dzcIjbkUTwzmErcQ8rN1WAn20jrvTr2T5Tdp4LqTUvsDEtL1Rtx0S6bnAh+b/5dgcXvsgKemVy74rIPdwMaO8dQTrToNl8YUPSlzv5TQssnKpP00HwdgPw3kx5kVKiG+Mi9F8YqJGLqjRbNSD/nEaKmIFPsWGwCBZq8PnVRseZteQV0fMMdWNy1EGP7SARAiU0XjVOtux6Cxc8aOuA2DFfYbmVoCTbIk4e9Fh3zy0Wf1/bUBA1iL1+mMNXRL8QmtM X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2026 15:55:37.5815 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9de9ecab-c1da-418a-2343-08de72f3fccd X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF0000C37A.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8975 Content-Type: text/plain; charset="utf-8" From: Ankit Agrawal EGM module implements the mmap file_ops to manage the usermode app's VMA mapping to the EGM region. The appropriate region is determined from the minor number. Note that the EGM memory region is invisible to the host kernel as it is not present in the host EFI map. The host Linux MM thus cannot manage the memory, even though it is accessible on the host SPA. The EGM module thus use remap_pfn_range() to perform the VMA mapping to the EGM region. Suggested-by: Aniket Agashe Signed-off-by: Ankit Agrawal --- drivers/vfio/pci/nvgrace-gpu/egm.c | 41 +++++++++++++++++++++++++++++- include/linux/nvgrace-egm.h | 1 + 2 files changed, 41 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-= gpu/egm.c index d7e4f61a241c..5786ebe374a5 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -17,19 +17,58 @@ struct chardev { struct cdev cdev; }; =20 +static struct nvgrace_egm_dev * +egm_chardev_to_nvgrace_egm_dev(struct chardev *egm_chardev) +{ + struct auxiliary_device *aux_dev =3D + container_of(egm_chardev->device.parent, struct auxiliary_device, dev); + + return container_of(aux_dev, struct nvgrace_egm_dev, aux_dev); +} + static int nvgrace_egm_open(struct inode *inode, struct file *file) { + struct chardev *egm_chardev =3D + container_of(inode->i_cdev, struct chardev, cdev); + + file->private_data =3D egm_chardev; + return 0; } =20 static int nvgrace_egm_release(struct inode *inode, struct file *file) { + file->private_data =3D NULL; + return 0; } =20 static int nvgrace_egm_mmap(struct file *file, struct vm_area_struct *vma) { - return 0; + struct chardev *egm_chardev =3D file->private_data; + struct nvgrace_egm_dev *egm_dev =3D + egm_chardev_to_nvgrace_egm_dev(egm_chardev); + u64 req_len, pgoff, end; + unsigned long start_pfn; + + pgoff =3D vma->vm_pgoff & + ((1U << (EGM_OFFSET_SHIFT - PAGE_SHIFT)) - 1); + + if (check_sub_overflow(vma->vm_end, vma->vm_start, &req_len) || + check_add_overflow(PHYS_PFN(egm_dev->egmphys), pgoff, &start_pfn) || + check_add_overflow(PFN_PHYS(pgoff), req_len, &end)) + return -EOVERFLOW; + + if (end > egm_dev->egmlength) + return -EINVAL; + + /* + * EGM memory is invisible to the host kernel and is not managed + * by it. Map the usermode VMA to the EGM region. + */ + return remap_pfn_range(vma, vma->vm_start, + start_pfn, req_len, + vma->vm_page_prot); } =20 static const struct file_operations file_ops =3D { diff --git a/include/linux/nvgrace-egm.h b/include/linux/nvgrace-egm.h index a66906753267..b9956e7e5a0e 100644 --- a/include/linux/nvgrace-egm.h +++ b/include/linux/nvgrace-egm.h @@ -9,6 +9,7 @@ #include =20 #define NVGRACE_EGM_DEV_NAME "egm" +#define EGM_OFFSET_SHIFT 40 =20 struct gpu_node { struct list_head list; --=20 2.34.1 From nobody Sun Apr 5 18:20:02 2026 Received: from SN4PR0501CU005.outbound.protection.outlook.com (mail-southcentralusazon11011030.outbound.protection.outlook.com [40.93.194.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A12D23033FB; Mon, 23 Feb 2026 15:55:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.194.30 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862149; cv=fail; b=p01GO4/fT6RBOyT+TLYhmkWYoKQKq2OY2Kr84rTRj3BxwwJNU2gt0bcZffjnt6+XCIIIm6n5oclXw7ybF4GhLitTjjJlCD06s7DJgiFswH1qizPP4NBHG/2aq+qXvNwy63HiRUasUbaPnFmUlGgWTQUOjD+dp+xX6pcY5mHmMAY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862149; c=relaxed/simple; bh=g9WGhvD9ngoFTjnzO0NsB0gWhptpyW6E12+YgNdnhGU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=a0sjuQcWimYV+wVVpxLwIClRjkljQAWYKhJN97tNGaxQZclafOEcm1bGNlJvos2V1KAuO07hHchC4IcAMabMdgbaaEEKnnnf6J1ifXJX4XVSxfKQpb7W8nnm4gghC6xT+sHzM765VFTeucI9pjAXdtr9TglzHVcbvaGXnvsfRRo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=GpuvIa9S; arc=fail smtp.client-ip=40.93.194.30 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="GpuvIa9S" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=CgM2MmLdCi5gyb0IGeR2dt+VLe9Ua0aJdzCKJXq3acgdex/LQqNKVPb9B5F1izg6SUMQS3w5367zK5qUWeGtgSpiM6P3s7odQeBdWHHoWMOjz4xaKVeR2SFgpgh56bRSSzfzeppDzL456MTr7+MYK0bPfIHuYFFLiUt8bR+ToHOFOhMQWRPbAQCg7JBgU7pSLVeLTWgqmIRgGPS6Guik4Zv8t3GQ8V2UhbIleQCL3c7z5Lqg2Et6acBQy8dbqXQmJ4KoVX1l6FnxSCGCbdxoaiq6RoExlqHm62pSyAvCXkvRvu8cV6BNO6RUjkz65BQu61YsIMLohsuBxwcwhtaAbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wE57MkPHKMH6S3M6xJWU835IV+am9ffEU6YlWoEmW78=; b=mbfPVTcXaBLm1pTcNlYRhVSUjJroi2e0Z4jSYptOD1QZmm2ZKTDNmZGafR2THYFaGInW1RWMtW5Z6WR2e626qZ4MO4L5drMWzdTWX+TdNYIMP3eIoRmWiH6g64hRVegEFnwsuXO7sxn8ObSN++snFK41a+4SRJiBOsbkbEH5H9ukg2ZO92rWTzqUbn1AixGJNbbhlfYNmm1CVnYZAtKy70TAvBfi+zDW4NnUflggkh7KYpu6qSmPGCvWGY1YbG1f2kZHGsgqhB/6PzYspfn0fR9wn23nEqNSgOVtsiXOYZSAI+FERaFNlJ3cDriRLBOtupautt20hPMsbWATTE3ycg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wE57MkPHKMH6S3M6xJWU835IV+am9ffEU6YlWoEmW78=; b=GpuvIa9SVMUiP0yTogbc/0eKcT56Sjz4hs5JJafJ4RMYdJ0Srx++bYJP6i0Atc1KyrGpaXCb6gQ/BlaKuS4o8vHz+ButpkKqcQOvLXrw8RASNG/BazjlSFfalTW6+ywFMOe3DNUFgtgIWRCbLQfl6YDcktl24A5r94TMcUQbOk9uW1UtzWPhaZKRzLKlXTH4e3cNJV74+w4LaX4Vzttj2uzJwdeq1PtdaojIb2cfOtDfVZ2bi0yD5QGRjrf9CvYY4yZz8n/LwIfwc2nBPi7qvl1gylXS/BKG+iACv6SyrANIlf6Z7Cc7hYQkClqP4luV8rV3z7Mt0fdX9sdPWFFJ1g== Received: from DM5PR08CA0027.namprd08.prod.outlook.com (2603:10b6:4:60::16) by DS0PR12MB8044.namprd12.prod.outlook.com (2603:10b6:8:148::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.21; Mon, 23 Feb 2026 15:55:38 +0000 Received: from DS3PEPF0000C37B.namprd04.prod.outlook.com (2603:10b6:4:60:cafe::6f) by DM5PR08CA0027.outlook.office365.com (2603:10b6:4:60::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9632.21 via Frontend Transport; Mon, 23 Feb 2026 15:55:38 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by DS3PEPF0000C37B.mail.protection.outlook.com (10.167.23.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.12 via Frontend Transport; Mon, 23 Feb 2026 15:55:38 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:19 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:18 -0800 Received: from localhost.nvidia.com (10.127.8.12) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Mon, 23 Feb 2026 07:55:18 -0800 From: To: , , , , , , CC: , , , , , , Subject: [PATCH RFC v2 10/15] vfio/nvgrace-egm: Clear Memory before handing out to VM Date: Mon, 23 Feb 2026 15:55:09 +0000 Message-ID: <20260223155514.152435-11-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260223155514.152435-1-ankita@nvidia.com> References: <20260223155514.152435-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF0000C37B:EE_|DS0PR12MB8044:EE_ X-MS-Office365-Filtering-Correlation-Id: 6c802257-a8da-45f1-8f07-08de72f3fd59 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?IOY9QG53JE+lt/ZaYYKw7beFTrGMLWaVZcREiP96Tm5kD6MaIbYRDKfLW+EG?= =?us-ascii?Q?Y/FKB0nFx171phm7yfhxBEmfvCmcGaD+48k8Gww7AV4W2x8gqjSvgOBFkF7a?= =?us-ascii?Q?SXOHMPRGdoirBMxyqGvRzRzuvRSYyLmBnK87Pt0gfpwZrpedH5k2ACg8uuo8?= =?us-ascii?Q?I3nhXryO1RszGk8Kcy5HX1a9+PthlwOhjyridtpt5Ng29+BSoAN0cGQnoYJD?= =?us-ascii?Q?ufTm07Rf90MARDjEzRLrdX9gTP6mtv00AKxGrKuvhaRFhGj7M633fwiFUL8i?= =?us-ascii?Q?x/IIlXXxz7m8LE7Semw8grS1leYrp2d+dN6JPy5T+YU4Xt2K2yM6AWX8k9W1?= =?us-ascii?Q?RINx957oei8g9S/lOzk2qLMVoPHbYgQxPxTnghBQGx/+ZkEToIszVuSPBbcp?= =?us-ascii?Q?VMTeQ9yhY8N9qpHWIHqEWXy7JmYjcs/KaAHbkUyOxOd9xUkcwmMb+e+1zRLp?= =?us-ascii?Q?2jH+HXPhrJ/ljp5qs4MMWGPETYsk0GWTLLJnbeyu5HkCDAR53vNgo3UhkvUF?= =?us-ascii?Q?dfueBmBJPz6Ibc8fdwfJWpvMUJpyhNboUbc2IrJdV4nA2tPNL1DMTVdappnr?= =?us-ascii?Q?H8YZXNZNM4W8wWSWkaDRt/R3nkfND3FVkTMF+1CT2bl8gOU1AGH2XyFEQTJ3?= =?us-ascii?Q?aur06NzZ5u14J7nH5KyGWV87h2z5MQ+8ziDM1fQQWkQAmhUTuj88L4B6CLbq?= =?us-ascii?Q?VBvFib9sZ5/mOvpCTB2ea89+4JcXqrWT07NbYoi5rZnLeLiTYVpoxPPD0g8R?= =?us-ascii?Q?AAFLBLOSCn4F3FxistEwran5wwRHd8aN4PfYy8JbyokejK5T5Mo8t3Asrj7I?= =?us-ascii?Q?Smc7PhwifVlcjvwRbsIlLkImomsLtvXzDXnKlAuZso5IUPudoKFGv9UBRAsg?= =?us-ascii?Q?4zi9/Q5CGos2Tb3QC8+0BWPAme0soT9FxYKkiN5VQCTfAlXRv0341zFsyhKM?= =?us-ascii?Q?HUqq84XmW7XXSW0VuMTpGAZ6RQGkBx8pJ3eYUzXgqyVy4WYCz8m9/7T+/LZp?= =?us-ascii?Q?rdjScTQJYvA8eZp/KBCu/+JX4kQCiZJB8zIF0CKmCyV2DCI/Esk9UcoWYdG0?= =?us-ascii?Q?fmaB/Fe2SKLE+w32/7qtTHhEdR82dK+f2B/GCCAE36sIvE1x0tq9raOWEi6n?= =?us-ascii?Q?y1uUlv9nrtlD0+hX7k+w2AX0xCpbkMXrprbhaB7E0SlNMWWM1Q0Cv/Q+tFnr?= =?us-ascii?Q?CYLkT7BdBOXiMFNl11ODTUMk34oyhE04HZizApbNqUgoafsQkjsI98oo/fpn?= =?us-ascii?Q?B/V2bOB5v34R/15OZI55qngp4Ft8Rad4quW2pqKFfakhx+L+K5J91sdOVPgf?= =?us-ascii?Q?L53UnkL16Yv1FhfQWRQU+mzEnK+RtjtvWKt+EijoLD2USwOE60K5QKqHSxi3?= =?us-ascii?Q?Ha+WgOZqRU3JT9h4kruHeAJIsni+azpaBa7bCUZMs5tna9WqPK0xNQPoQ4kG?= =?us-ascii?Q?3+0TIxx5Hf13niuNXBzlWyy9J3zd345UTnP0IATaCB66dnnK1v9OEBu0WVAs?= =?us-ascii?Q?/xmu4io7FllSoq2GVPzAYafpx31vjbtnCcHhi3S/Lwy5hFX8Lg9zu3FZFUSE?= =?us-ascii?Q?ZJCk6SRRDEdyh2pJ5UT2DBvckvp0av6znDTGjJHpp3/2zPAftwV/EIHdJF8X?= =?us-ascii?Q?a69Pe2ZBa6kBKnaVo9WUjlTUpm1FGwvkngvZmIb5zGGlicqBg2Z73FPHBwGG?= =?us-ascii?Q?spJD4Q=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(376014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: m/oCOBcoo6Vt3BLohvRkupBC7Y3y+0ZvkClAMaUnb+eMO8Ltc5Av41NJSp9Z+lKJdjfo2qrEMl/bHcx3xtPWDCkENcZMoQjGku5qsZvahVLHqao653osHsY5DMMWoRIT2UOg2omXLODVAtBdtwI86P2l9AzBs3q22WfIOm2mX//+/aIG3mZt0lPKEG6flkpPQmOuKWtwqJDmXHMO5+sOOY2gkUFKoWmd4PCZKDxJnuG/AchMoWLdNf03zdDbrLTwLk/6Cfxm6LMgA5R84ZfHHny5MwnRpCNztihhbLckmgxuTsHX1ytHooUcq9ziclr/ZYG2EaMjwky1szJi6Zwi8NbfRxGW3MyxuenaGeAj+h0bFd6CTt+AtvamT9GKp/Z+bhSueFU9psWySgu0faJf2xVLTikZMMIf64uSkMcWlP0I0alzVE7YIesDbYdjddqm X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2026 15:55:38.4758 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6c802257-a8da-45f1-8f07-08de72f3fd59 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF0000C37B.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8044 Content-Type: text/plain; charset="utf-8" From: Ankit Agrawal The EGM region is invisible to the host Linux kernel and it does not manage the region. The EGM module manages the EGM memory and thus is responsible to clear out the region before handing out to the VM. Clear EGM region on EGM chardev open. To avoid CPU lockup logs, zap the region in 1G chunks. Suggested-by: Vikram Sethi Signed-off-by: Ankit Agrawal --- drivers/vfio/pci/nvgrace-gpu/egm.c | 43 ++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-= gpu/egm.c index 5786ebe374a5..de7771a4145d 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -15,6 +15,7 @@ static DEFINE_XARRAY(egm_chardevs); struct chardev { struct device device; struct cdev cdev; + atomic_t open_count; }; =20 static struct nvgrace_egm_dev * @@ -30,6 +31,42 @@ static int nvgrace_egm_open(struct inode *inode, struct = file *file) { struct chardev *egm_chardev =3D container_of(inode->i_cdev, struct chardev, cdev); + struct nvgrace_egm_dev *egm_dev =3D + egm_chardev_to_nvgrace_egm_dev(egm_chardev); + void *memaddr; + + if (atomic_cmpxchg(&egm_chardev->open_count, 0, 1) !=3D 0) + return -EBUSY; + + /* + * nvgrace-egm module is responsible to manage the EGM memory as + * the host kernel has no knowledge of it. Clear the region before + * handing over to userspace. + */ + memaddr =3D memremap(egm_dev->egmphys, egm_dev->egmlength, MEMREMAP_WB); + if (!memaddr) { + atomic_dec(&egm_chardev->open_count); + return -ENOMEM; + } + + /* + * Clear in chunks of 1G to avoid CPU lockup logs. + */ + { + size_t remaining =3D egm_dev->egmlength; + u8 *chunk_addr =3D (u8 *)memaddr; + size_t chunk_size; + + while (remaining > 0) { + chunk_size =3D min(remaining, SZ_1G); + memset(chunk_addr, 0, chunk_size); + cond_resched(); + chunk_addr +=3D chunk_size; + remaining -=3D chunk_size; + } + } + + memunmap(memaddr); =20 file->private_data =3D egm_chardev; =20 @@ -38,8 +75,13 @@ static int nvgrace_egm_open(struct inode *inode, struct = file *file) =20 static int nvgrace_egm_release(struct inode *inode, struct file *file) { + struct chardev *egm_chardev =3D + container_of(inode->i_cdev, struct chardev, cdev); + file->private_data =3D NULL; =20 + atomic_dec(&egm_chardev->open_count); + return 0; } =20 @@ -108,6 +150,7 @@ setup_egm_chardev(struct nvgrace_egm_dev *egm_dev) egm_chardev->device.parent =3D &egm_dev->aux_dev.dev; cdev_init(&egm_chardev->cdev, &file_ops); egm_chardev->cdev.owner =3D THIS_MODULE; + atomic_set(&egm_chardev->open_count, 0); =20 ret =3D dev_set_name(&egm_chardev->device, "egm%lld", egm_dev->egmpxm); if (ret) --=20 2.34.1 From nobody Sun Apr 5 18:20:02 2026 Received: from DM1PR04CU001.outbound.protection.outlook.com (mail-centralusazon11010058.outbound.protection.outlook.com [52.101.61.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF5BD30DD12; Mon, 23 Feb 2026 15:55:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.61.58 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862151; cv=fail; b=DglZtQIYyIpe5PbygMZRwVpHA9IYmkbtIb9Y3Sc728NTZubMHnm+/cXlf36hvGiRHeP7xuj7RNktGv3UQWI/DqkdL07MzwK45w4O0oqBZLXxR187jCUiI+fKcO1nfQGz9eP2JzvpuYRdFLVuYrZl5tKE5dpuhmhOG/Gk4uE5zSU= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862151; c=relaxed/simple; bh=xpV1mVTEgJUny4G6PFv/Cnh511/X3QbQmlhfgVMVazM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=CW5mBAgupx/p3uDFnuaAiqu0vz8Z16xyk/KyOcVApCWzwyPmGzz4yWdtG5v/f8LONUIz/vh1InDURFliT5O9aMmV/pfr5chx4z8iBa34rA+a86i8Yd9qK9GaEpP0MhVDm6UmU5Y48XoGZf7ZoPuznSIW27FePKCtrckMOCpBqII= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=BnhQu/wB; arc=fail smtp.client-ip=52.101.61.58 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="BnhQu/wB" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Bmv446DyNwW0XqgaOxUxnmNDXB8pyrDl+bW5K387KRrZeX1ZxVmVvLtyxOj9k0FO33RTClKyQB0OTk6Hh5hfy+TEigtoP/bj8d5I9Tt2kXgHsG4r+MMmhSeDvJgWTsYV/9dB3EsqWLmeb6YSvvuNBpSxhPFRWO22SEIQlp241sE4wfxHVswXBCjth4Q1HeCY1Api1+0w9EtEHlwBsIAMpPaNflJx1qu0kN1sm+wTmLZmj3igWP7aXWObIHbRhcdbm8B3FEVrlgY6J8fkwnck+pfRUXyjH2yx+BYuHoOrJuif37r7O0MRCcMML85yEd18w175FkKNTjxDM8lkT9lFOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iU9SNhUgBxp63sHFsBQEly8YO1b8ESgHB71rDFRRUjY=; b=PyzADHmdVc12crk8gOMJqIUlSL2j+t2cx1Kx0WZ70KHzYZTuR2veWL3GD9N3o99S8ZFSxlL4qG3kxF7AuQKiqvAttaEBxTza2Fyjt2ox+PHYjo3m6te4q+SQAzzpe+KnuH7/e0tuIp0roDaE+QBevaHYY3n5jW1/Y7pJ79KXCaJpD8NxlCr6kWUIn/8Ub1epOgiXuzDJzPJ1qDwBfQDxz3Aaj8d5BWTleBXqWKX2+eaW/j+A5njNwvylJLC88q5obAquORSJtOrkY+JBNt0JEg2uPflE/62v5mRFEGEkj26niiyIWHYEdlaJL2H4otG32sf1GOIyVTPqPTO51crpiQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iU9SNhUgBxp63sHFsBQEly8YO1b8ESgHB71rDFRRUjY=; b=BnhQu/wBMoQmynYc9p3z8ZoJ/TBb/SVnWbmYngGauaLarZRjhXueiAHIF2THg8GKy+qnp2QuuNLIy6vYhiCMpRphyP/c2yDs3ejVFziNWvasWuwQ3xvG3y8u5kkUi9qJsyUkCInl7ut65HkfmJQjGEAH2QHfj3toE7D4Z/mOdm5uuJoAapH5kCwxXz4ViJ8ClXgny5g6tAquW/mmYDDZsusy5xUoOF6szMu3jB1b1lp8qBBLIV9Qy7uhYU27h2W6VhtJkqpv9rG3V0YjoR+lhj2TOTHDMKTFIzvYWWYE+46uLJ0tP1Nh2R59pG4cML8sC1ILth/hu1mTCp0CGa9FfA== Received: from CH0PR04CA0058.namprd04.prod.outlook.com (2603:10b6:610:77::33) by DS7PR12MB8369.namprd12.prod.outlook.com (2603:10b6:8:eb::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.21; Mon, 23 Feb 2026 15:55:38 +0000 Received: from CH2PEPF00000099.namprd02.prod.outlook.com (2603:10b6:610:77:cafe::30) by CH0PR04CA0058.outlook.office365.com (2603:10b6:610:77::33) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9632.21 via Frontend Transport; Mon, 23 Feb 2026 15:55:38 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CH2PEPF00000099.mail.protection.outlook.com (10.167.244.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.12 via Frontend Transport; Mon, 23 Feb 2026 15:55:38 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:19 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:19 -0800 Received: from localhost.nvidia.com (10.127.8.12) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Mon, 23 Feb 2026 07:55:19 -0800 From: To: , , , , , , CC: , , , , , , Subject: [PATCH RFC v2 11/15] vfio/nvgrace-egm: Fetch EGM region retired pages list Date: Mon, 23 Feb 2026 15:55:10 +0000 Message-ID: <20260223155514.152435-12-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260223155514.152435-1-ankita@nvidia.com> References: <20260223155514.152435-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000099:EE_|DS7PR12MB8369:EE_ X-MS-Office365-Filtering-Correlation-Id: 0173a496-98d3-4e38-9004-08de72f3fd4c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|36860700013|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?5Kn5RMgWngNKvOzrv2tdjK0gGNNdnnBGetWX2TxaFUBfq3BYr0nuoc1UZ88x?= =?us-ascii?Q?H4EtErJbybbzGlOdHqmA1llLb2DTdWa7fw9Lktilangm/+wvbX2osXnWh+sJ?= =?us-ascii?Q?GPVUKw9UVA6BZqdtMK8nGY/lRmV1xQv/8/R4FzmUb1Ikn1htM8LKfB+R6RG0?= =?us-ascii?Q?Tldpscd6QaaEUgs6aYXvFJ2R/1iZyBa3+ZHeTLdNrF/Au6KU/VBDxMVwd6TV?= =?us-ascii?Q?MsqaTrLh2dzOmsi8y90w8qLD3JUr3HH5hcSjOrsiw92FgPT0ZLWDgc2KbTrV?= =?us-ascii?Q?zI7thARv3TAJeLIm57DtjivnIL/gLVXLggAN2/K5VHkZk3u7M2jpOEKTG0yo?= =?us-ascii?Q?4A+ZSneeqCZFWb1u6LytN0s9jHuDWqBgRWoJEek1eIswwinzx6SM9Lhow6Hs?= =?us-ascii?Q?746qetyd2s2HhcB9UJIZP31/4//ARuDBatUDn2z/xM2VNQzEOffrIUgB/smV?= =?us-ascii?Q?N6gQlWlQ8wHJonta00upLzCkYkBjixv+Ansbpr5KF13d/Q+e1msdlb2sd7ep?= =?us-ascii?Q?k9z7lAQm2rAwBhQ9N6E5rLe+b/s7MtYnB3hw3lnLdZbqBEstx2aoAZ6h2tZk?= =?us-ascii?Q?wqODUHphHYJ7zhLRMxaaUiK/PVAeknaDvFRNvEbQCbUbW/Tbefi2t4SKtjD9?= =?us-ascii?Q?zvOGz5QJaMaOzUmZEdXP6VOUhsQB44yT0gcEIbkGSA9vra8io4DqEYCZCg+j?= =?us-ascii?Q?kR1cxe7g0DU3t/qiCl+pdORb52EwD8KQ87hDgMw6rTVIPhaYbMl775TeDRin?= =?us-ascii?Q?rMiI1yOAJb8T5PQ+DmNi1lW3vjMuFAhv3dPn4xjxNpbSWSYrpFc8zrQYJgeS?= =?us-ascii?Q?k3CFKYmUwjrPwTD7a2rhmoueURzulB8B20wZki6Au3AEgqXz6WOGcSzYaltS?= =?us-ascii?Q?82DCjVDhjMlkUPkcDRXBE6XmstopSqkSEw7/rt5fmQExwp44puR2cETmIUMF?= =?us-ascii?Q?AJRt7v/DVLwoJ5znqEs4cYrd3M4zvT8b7saMe/xrK94bslvugelhJ3/pGB8T?= =?us-ascii?Q?blcHVkbW3u5sjtdVx44DvqYXr2PHA5zEGvwNpCSchLcr8MXSpVjLRyBy1hsc?= =?us-ascii?Q?8+u0SGxPzKezPbqOG5V+h9Kfnfy2xvrYZil36lTXpUw5tdrG3eCkhS73An4p?= =?us-ascii?Q?avjWXEKymPyH8IQBZp5sWnk6+zY52G4iBakcRjoi+Cz4yxJ0viBjvksZgZwD?= =?us-ascii?Q?zi43NuaI2+JdMs9zOTbqvT/oipaRF6W1vRsW1bL/EnJYIN36V4BswoCxNo/M?= =?us-ascii?Q?NBvNosoQqprlga8g0+FLJnC5cqrJru0uOHJXtyoRHrvPEstz44jXjT3p+Ew1?= =?us-ascii?Q?LlGbaQDHcqu+tU2XY3uC6AIEL31aGM1A2gQnw6wItFYV3St37neaL9lr8rCK?= =?us-ascii?Q?UUkgG6XRpsZH8lQNQD1o3jImdh4A9NA+0lUw2+0pf8cLRZ0WWLybKSrM2MtF?= =?us-ascii?Q?XNvU52v5bb2l4K/REj8nBhOvgxHymxL1YonJ2dfpq739spDzRme9z5YxEQSk?= =?us-ascii?Q?eWa56yMN7BGwv3tKvf/9p5F1/NH4EPOsrIqbBoZOlFMRFm+xjFLT5XsNcxyp?= =?us-ascii?Q?vjgqCRkZNdN2hPVAkB0uA2bYAibBlCNLGfVOcTPRmJH4BtLGf+73Qb4NWh7m?= =?us-ascii?Q?jaqm/Y9gOgF9xztLRX3NLgViGoj7thfqcKvtUDYnbQHBG/oVGvrWRncMPUJV?= =?us-ascii?Q?JZj4Jg=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(36860700013)(376014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: lzFKqsDjC6/3+YaGqkKEjjZD8RC+GUw3nTRxi1rC3WypjaqbAks/nZjLi6pBWUSU1LeNZTBtAqfbaRrfy/KE4FokZCJQeGphg6fqIId3ED541OADygOrmhwEeSRw7M9HQLMh2IpiOJCCv+dehzH6kD9t32GXBCCs/QDthR/N2FBrU/xRcfnJ2RjwGgBsSyiPxXGtzS5vO4UwCetDeMVE6mBrGQ71XcjyTZWEbYZi3vHJaYv0xK36vaBbiIwN6/3bsS632BnXAxlGQBT+jqRt1hJPEt+ercBLiUfLkrgcFUlBmpMvRySrieswGgm0fs+uLyg2JMPhHK61f8qIyBhRXiLUMlsKRSKA6/Lo5o4SCyXi2GiTqffYmhXYtRxrWk8Cg44KZjMjLQzDSA5Z7W0su7G3+DvNieRFVtOggP78ae6S7aisonR3LSxSzt9tDmq+ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2026 15:55:38.3872 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0173a496-98d3-4e38-9004-08de72f3fd4c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000099.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB8369 Content-Type: text/plain; charset="utf-8" From: Ankit Agrawal It is possible for some system memory pages on the EGM to have retired pages with uncorrectable ECC errors. A list of pages known with such errors (referred as retired pages) are maintained by the Host UEFI. The Host UEFI populates such list in a reserved region. It communicates the SPA of this region through a ACPI DSDT property. nvgrace-egm module is responsible to store the list of retired page offsets to be made available for usermode processes. The module: 1. Get the reserved memory region SPA and maps to it to fetch the list of bad pages. 2. Calculate the retired page offsets in the EGM and stores it. Signed-off-by: Ankit Agrawal --- drivers/vfio/pci/nvgrace-gpu/egm.c | 75 ++++++++++++++++++++++++++ drivers/vfio/pci/nvgrace-gpu/egm_dev.c | 32 +++++++++-- drivers/vfio/pci/nvgrace-gpu/egm_dev.h | 5 +- drivers/vfio/pci/nvgrace-gpu/main.c | 8 +-- include/linux/nvgrace-egm.h | 2 + 5 files changed, 112 insertions(+), 10 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-= gpu/egm.c index de7771a4145d..077de3833046 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -8,6 +8,11 @@ =20 #define MAX_EGM_NODES 4 =20 +struct h_node { + unsigned long mem_offset; + struct hlist_node node; +}; + static dev_t dev; static struct class *class; static DEFINE_XARRAY(egm_chardevs); @@ -16,6 +21,7 @@ struct chardev { struct device device; struct cdev cdev; atomic_t open_count; + DECLARE_HASHTABLE(htbl, 0x10); }; =20 static struct nvgrace_egm_dev * @@ -174,20 +180,88 @@ static void del_egm_chardev(struct chardev *egm_chard= ev) put_device(&egm_chardev->device); } =20 +static void cleanup_retired_pages(struct chardev *egm_chardev) +{ + struct h_node *cur_page; + unsigned long bkt; + struct hlist_node *temp_node; + + hash_for_each_safe(egm_chardev->htbl, bkt, temp_node, cur_page, node) { + hash_del(&cur_page->node); + kvfree(cur_page); + } +} + +static int nvgrace_egm_fetch_retired_pages(struct nvgrace_egm_dev *egm_dev, + struct chardev *egm_chardev) +{ + u64 count; + void *memaddr; + int index, ret =3D 0; + + memaddr =3D memremap(egm_dev->retiredpagesphys, PAGE_SIZE, MEMREMAP_WB); + if (!memaddr) + return -ENOMEM; + + count =3D *(u64 *)memaddr; + if (count > PAGE_SIZE / sizeof(count)) + return -EINVAL; + + for (index =3D 0; index < count; index++) { + struct h_node *retired_page; + + /* + * Since the EGM is linearly mapped, the offset in the + * carveout is the same offset in the VM system memory. + * + * Calculate the offset to communicate to the usermode + * apps. + */ + retired_page =3D kzalloc(sizeof(*retired_page), GFP_KERNEL); + if (!retired_page) { + ret =3D -ENOMEM; + break; + } + + retired_page->mem_offset =3D *((u64 *)memaddr + index + 1) - + egm_dev->egmphys; + hash_add(egm_chardev->htbl, &retired_page->node, + retired_page->mem_offset); + } + + memunmap(memaddr); + + if (ret) + cleanup_retired_pages(egm_chardev); + + return ret; +} + static int egm_driver_probe(struct auxiliary_device *aux_dev, const struct auxiliary_device_id *id) { struct nvgrace_egm_dev *egm_dev =3D container_of(aux_dev, struct nvgrace_egm_dev, aux_dev); struct chardev *egm_chardev; + int ret; =20 egm_chardev =3D setup_egm_chardev(egm_dev); if (!egm_chardev) return -EINVAL; =20 + hash_init(egm_chardev->htbl); + + ret =3D nvgrace_egm_fetch_retired_pages(egm_dev, egm_chardev); + if (ret) + goto error_exit; + xa_store(&egm_chardevs, egm_dev->egmpxm, egm_chardev, GFP_KERNEL); =20 return 0; + +error_exit: + del_egm_chardev(egm_chardev); + return ret; } =20 static void egm_driver_remove(struct auxiliary_device *aux_dev) @@ -199,6 +273,7 @@ static void egm_driver_remove(struct auxiliary_device *= aux_dev) if (!egm_chardev) return; =20 + cleanup_retired_pages(egm_chardev); del_egm_chardev(egm_chardev); } =20 diff --git a/drivers/vfio/pci/nvgrace-gpu/egm_dev.c b/drivers/vfio/pci/nvgr= ace-gpu/egm_dev.c index 20291504aca8..6d716c3a3257 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm_dev.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm_dev.c @@ -18,22 +18,41 @@ int nvgrace_gpu_has_egm_property(struct pci_dev *pdev, = u64 *pegmpxm) } =20 int nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, - u64 *pegmlength) + u64 *pegmlength, u64 *pretiredpagesphys) { int ret; =20 /* - * The memory information is present in the system ACPI tables as DSD - * properties nvidia,egm-base-pa and nvidia,egm-size. + * The EGM memory information is present in the system ACPI tables + * as DSD properties nvidia,egm-base-pa and nvidia,egm-size. */ ret =3D device_property_read_u64(&pdev->dev, "nvidia,egm-size", pegmlength); if (ret) - return ret; + goto error_exit; =20 ret =3D device_property_read_u64(&pdev->dev, "nvidia,egm-base-pa", pegmphys); + if (ret) + goto error_exit; + + /* + * SBIOS puts the list of retired pages on a region. The region + * SPA is exposed as "nvidia,egm-retired-pages-data-base". + */ + ret =3D device_property_read_u64(&pdev->dev, + "nvidia,egm-retired-pages-data-base", + pretiredpagesphys); + if (ret) + goto error_exit; + + /* Catch firmware bug and avoid a crash */ + if (*pretiredpagesphys =3D=3D 0) { + dev_err(&pdev->dev, "Retired pages region is not setup\n"); + ret =3D -EINVAL; + } =20 +error_exit: return ret; } =20 @@ -74,7 +93,8 @@ static void nvgrace_gpu_release_aux_device(struct device = *device) =20 struct nvgrace_egm_dev * nvgrace_gpu_create_aux_device(struct pci_dev *pdev, const char *name, - u64 egmphys, u64 egmlength, u64 egmpxm) + u64 egmphys, u64 egmlength, u64 egmpxm, + u64 retiredpagesphys) { struct nvgrace_egm_dev *egm_dev; int ret; @@ -86,6 +106,8 @@ nvgrace_gpu_create_aux_device(struct pci_dev *pdev, cons= t char *name, egm_dev->egmpxm =3D egmpxm; egm_dev->egmphys =3D egmphys; egm_dev->egmlength =3D egmlength; + egm_dev->retiredpagesphys =3D retiredpagesphys; + INIT_LIST_HEAD(&egm_dev->gpus); =20 egm_dev->aux_dev.id =3D egmpxm; diff --git a/drivers/vfio/pci/nvgrace-gpu/egm_dev.h b/drivers/vfio/pci/nvgr= ace-gpu/egm_dev.h index 2e1612445898..2f329a05685d 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm_dev.h +++ b/drivers/vfio/pci/nvgrace-gpu/egm_dev.h @@ -16,8 +16,9 @@ void remove_gpu(struct nvgrace_egm_dev *egm_dev, struct p= ci_dev *pdev); =20 struct nvgrace_egm_dev * nvgrace_gpu_create_aux_device(struct pci_dev *pdev, const char *name, - u64 egmphys, u64 egmlength, u64 egmpxm); + u64 egmphys, u64 egmlength, u64 egmpxm, + u64 retiredpagesphys); =20 int nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, - u64 *pegmlength); + u64 *pegmlength, u64 *pretiredpagesphys); #endif /* EGM_DEV_H */ diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace= -gpu/main.c index 0bb427cca31f..11bbecda1ad2 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -78,7 +78,7 @@ static struct list_head egm_dev_list; static int nvgrace_gpu_create_egm_aux_device(struct pci_dev *pdev) { struct nvgrace_egm_dev_entry *egm_entry =3D NULL; - u64 egmphys, egmlength, egmpxm; + u64 egmphys, egmlength, egmpxm, retiredpagesphys; int ret =3D 0; bool is_new_region =3D false; =20 @@ -91,7 +91,8 @@ static int nvgrace_gpu_create_egm_aux_device(struct pci_d= ev *pdev) if (nvgrace_gpu_has_egm_property(pdev, &egmpxm)) goto exit; =20 - ret =3D nvgrace_gpu_fetch_egm_property(pdev, &egmphys, &egmlength); + ret =3D nvgrace_gpu_fetch_egm_property(pdev, &egmphys, &egmlength, + &retiredpagesphys); if (ret) goto exit; =20 @@ -114,7 +115,8 @@ static int nvgrace_gpu_create_egm_aux_device(struct pci= _dev *pdev) =20 egm_entry->egm_dev =3D nvgrace_gpu_create_aux_device(pdev, NVGRACE_EGM_DEV_NAME, - egmphys, egmlength, egmpxm); + egmphys, egmlength, egmpxm, + retiredpagesphys); if (!egm_entry->egm_dev) { ret =3D -EINVAL; goto free_egm_entry; diff --git a/include/linux/nvgrace-egm.h b/include/linux/nvgrace-egm.h index b9956e7e5a0e..9e0d190c7da0 100644 --- a/include/linux/nvgrace-egm.h +++ b/include/linux/nvgrace-egm.h @@ -7,6 +7,7 @@ #define NVGRACE_EGM_H =20 #include +#include =20 #define NVGRACE_EGM_DEV_NAME "egm" #define EGM_OFFSET_SHIFT 40 @@ -20,6 +21,7 @@ struct nvgrace_egm_dev { struct auxiliary_device aux_dev; phys_addr_t egmphys; size_t egmlength; + phys_addr_t retiredpagesphys; u64 egmpxm; struct list_head gpus; }; --=20 2.34.1 From nobody Sun Apr 5 18:20:02 2026 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11012051.outbound.protection.outlook.com [52.101.53.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0694130DEA5; Mon, 23 Feb 2026 15:55:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.51 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862150; cv=fail; b=te3R0eTrzZwiZl2hsOZbTWKCgiW8WxqTa2LpxgiWNLZs/YEEM05gUOnlqz683WShenbNczFuxwJsZL6fPSb6rjoeWbgo0G0WrJEbYWawAoJ5DaXrDhWYGF2FXgD/OBV0P8oyFogBG42xvPWB6Sd9T/byhciE1/tCeeISRp2SC2Q= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862150; c=relaxed/simple; bh=clOsIYpjBZFOCHZhqZ7UzYD0rE0bHaXlQftHPfx91uQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=gefP20I+iSix1eY2LNEH0jaZkbdA2zvro+v2a9CLYD4WnrZudPyOzeVImLtZ495JQVJuEZGG2MogNdOdJBIWvof6U5sRqVNzZ0hZAgTuy2UR97z+2BgMf9DubNPglvlhxHN0b6l/rfmspd2kg72d96XlO+g2ZWlwkcleUxKV9As= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=JE20e0au; arc=fail smtp.client-ip=52.101.53.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="JE20e0au" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ox3r/bqZNpbFoDZ0gOI7sjFxT8scbG54Zx3kRTIdNcHZdO27X8BUs9dj0p2T3TX+vpchMls/sAVeRfrYuzahfhvHFOBZ4zzBFqcTgFTZLhRXfhr2S6NaC8hBxTAvq79bKZC897JdsTyg8phh9LSKt7u0yF7UEx3/5e7awqqcBMldgKP6Hwn/hqtQvJPfkjT8yPiCrGP+USjX5xwcpvNQjHDXuyj8OepT3a3bK30wx6H6dQzf+e7yJ4QnLRN6YsERR+S0pqLVycOWbmRT9xHA5E1uC6o8RK6r/8HNdj4c2pd+cpS4oJ+Ubr4QllBYKVC9pRUAB4Ygw1aUATPekAyH2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=z84dVII1echT0RJNM7JQ/br7x45aqd+bmTWnuqhB4bA=; b=YNVct1gwjczQ9cvVybHE9aMCpsXthlNEmHgF/c3xvz8MQk5BSgDh+WJh0unAEi/Qq5Ht5oR/gMSMO3Q7iEtCAN8Cp1i8n7/Yh22TxQWCiv6oV4ufJMM5FV747mTnBMd2Z573OtAiaBwmiPZWEtLPPJkddzTep69+Zbeb1ScanxcF5WiViWorLZXJkjQMuIyOFW+exEEEgdMkL/5MuURxB59A7GSw1/aCusjEHo+ZGSNRks6/DGvJLyVUGXx+n/DKhiuoD7cYDWxiCN24c72ZER/oal+gw+sxic00c/bYRmU8UpgNwA65yk158SCulRIvuh64RRML79IutTJ8f7QC4g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=z84dVII1echT0RJNM7JQ/br7x45aqd+bmTWnuqhB4bA=; b=JE20e0au5PlSTMbGVlAaB2zglgQOA734tdxdOAhbeRGxUY6CEU/BUEUjQC3Oz8WOul+aUDGoCENZNEkgsGZRz+3QvxuR8hOi3Z5pX1lkxN6czz521lBQcAPGM7LO/BtErANb0hvy56vghceDlI1Q30EogOsQ5Xe8PGnC4R659CD2BYdUSCmwVc6SN9TRW6avSPUeihYKmFCq8ut578BJ14o8XTK2TROHBamjsojsuX4i/YUqhX/VIQ89XjnQoUYlDC7g3AdFwxaFrbNTR6qT+hR3Xt8C15HOauWLN0bJXrxVNcD7iw9mHRYqsfWxKqpo/Cej3N15RcoKwmxS3DhcnA== Received: from DM5PR08CA0033.namprd08.prod.outlook.com (2603:10b6:4:60::22) by CH8PR12MB9837.namprd12.prod.outlook.com (2603:10b6:610:2b4::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.21; Mon, 23 Feb 2026 15:55:39 +0000 Received: from DS3PEPF0000C37B.namprd04.prod.outlook.com (2603:10b6:4:60:cafe::fe) by DM5PR08CA0033.outlook.office365.com (2603:10b6:4:60::22) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9632.21 via Frontend Transport; Mon, 23 Feb 2026 15:55:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by DS3PEPF0000C37B.mail.protection.outlook.com (10.167.23.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.12 via Frontend Transport; Mon, 23 Feb 2026 15:55:39 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:20 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:19 -0800 Received: from localhost.nvidia.com (10.127.8.12) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Mon, 23 Feb 2026 07:55:19 -0800 From: To: , , , , , , CC: , , , , , , Subject: [PATCH RFC v2 12/15] vfio/nvgrace-egm: Introduce ioctl to share retired pages Date: Mon, 23 Feb 2026 15:55:11 +0000 Message-ID: <20260223155514.152435-13-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260223155514.152435-1-ankita@nvidia.com> References: <20260223155514.152435-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF0000C37B:EE_|CH8PR12MB9837:EE_ X-MS-Office365-Filtering-Correlation-Id: 578c90c7-c672-4867-8284-08de72f3fdf9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|376014|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?7S9K+p4Bxmy0SJMHtvGkjRo9r/QkrFy79QTDrDBacTfIklOjpC7gGBTRspTT?= =?us-ascii?Q?ivEhYNt/WIIoYF0qzzzwzUoFNP7kFL5xsB6Ca6LLbIdhQZxiS4m8rh8ZRMmq?= =?us-ascii?Q?igQyN2bvfHsvLjRBM5T4B99gXM2JaekB7omMh79i+rV2/h5PbYGaMT74Nm3O?= =?us-ascii?Q?K6f85ZIC38FPVA1P6vGPGfet6p1634khQAJfmp7M0+rlenWEFiu1XNKauO2N?= =?us-ascii?Q?H5x1Obu/5aVQYAQnejDbm+SzX5NnkmsWnjyb0bXz0GuC2RKXYwe+G1k3Lrkr?= =?us-ascii?Q?P042SjSMzYOpTkgYdXY9UGD+su4OVe/fjP7+gD1ktjwiqvURBz1V0s/07LwR?= =?us-ascii?Q?B5KVIhnoTlDqtPVzoNhClEr+IY3RvpcxJciOWNTn5L+JjrAQw1v+r5M8kCTA?= =?us-ascii?Q?g1KdrUDzIyzOKnA9w/vp560OKckCiQdtyE8A6T/5QwnC6rMtR4Q4s4lApEED?= =?us-ascii?Q?mCT3qDHYWWDOyRQCJlGbAnmzOij4Ul/yFj+ywAOvj5lrVjUUmv+djXazDDTS?= =?us-ascii?Q?mMmO786xsD1TXTSKK3b8MUY+trIbrtC9XZWgTZgnxzSQoPQzaM+Dbyv/aalx?= =?us-ascii?Q?6eLpmWKQ/jW9OhQYm/YQ9utJT6868Z828zSjy1vwRb6RH9xUH0Y6iDDCZivo?= =?us-ascii?Q?juL5ejPOoJjKAQVMxxOB2ijBL86cr45MOzOjXKAnPsXEAoLd9/XDfSWaFKnU?= =?us-ascii?Q?OX4icBySfNzi/Xwda8DL7xhHyMYlcOIQdpwTukraMoOXhGdI9jd8ro6wsbb9?= =?us-ascii?Q?/Qm3U8Rx0sgg+ARkaHRBq5uo/WG3tP+EqEl04R7cKxg+kIQgPGUhjzIG/Lfb?= =?us-ascii?Q?XaP/P6xzg9QTIY8NKm9u/4DWjoM6l9b3fRYVxw/ErDmIjSvVOrDhwqGPExuL?= =?us-ascii?Q?nVNBC7NNz0dXCLJKw8fsOu8sD7jZbnwjzUKvPKN37ZHl3MvfxLtSlKCpjHmm?= =?us-ascii?Q?IppM7ArAbqdZBRy5/eLCRqDFbFDj9/6yM/kIHQN3U0Golde7l8Pt+ocATYU5?= =?us-ascii?Q?O8YAnBmjebvwAY9qJhec08+MeYSotZxo7/dakIifklLZseofAu9OwvS67cyn?= =?us-ascii?Q?47V6bPtQpoUJh0CFqaSlAmkIH1KFCEM0E1510cMSuY1d9fzMOwDerxfFcix1?= =?us-ascii?Q?66haUQZ8sEVDUSN3c2m98+taOM66Mc7UG9oDMG0Uoe4XhE2siDiVlwYbF6Uu?= =?us-ascii?Q?KjVqx03hjHs1wng+Ap8UYMA+nlsJxwUJq3TzGovKail86PlxH13bj1PyzvI6?= =?us-ascii?Q?Nl4hsf884dGRadtBPvvyDuAdv4gyhkhBlNmn5iseqHYE0GOisKEXFy3dOJOg?= =?us-ascii?Q?DF53X76OrN5ApN9QgOAOe513ORE1OEHPsxbl+hWnkrtdAYQx/1/LaZHIhsor?= =?us-ascii?Q?4ao6F6wcolSsK0HcXRLQyQ0NJbvfHE9ofJrARq8hfylYh1LHV9zFbMQLcbaE?= =?us-ascii?Q?W4GnoDKX7N1K4tiMIhMpJXbxSXffQOPlQw437ItmeE9c6R0TKXGuhE2YHW3o?= =?us-ascii?Q?Db/71HEzm3hmVrx4LwjdnM2oOu++UeRu1Ircl9iycdHmLsC73xnH7Rrh4z4A?= =?us-ascii?Q?B/s1pisFrXBeE4tFuph8MhY2A/UYMsrR8f/KQHDjzUTO+vV3pmG5K74G6WfX?= =?us-ascii?Q?PlwIhOu0+QmuHONHSzz9POUD25/1FG/fdA6WRkpZRB6cu4EEmppKq1wFbDE6?= =?us-ascii?Q?bl0Bmw=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(376014)(36860700013);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: NSapZZ0tLyqAwJUGYY2XPMRU1tW53iKPzlcmQKcKqryy+yFU2mMTwMLj4KItF9lgj/gPponY61g6/vLlUyAwjHHjwd2mJbZvyjACaEGWLBgXbdHXTIXj9xp2JTmXmeeRst5PGmL+GqHRyCQN8+NWeyHjkqHiVe7RilbTyvGuMsnjZJ6gvt/1WZwxCeNPo6AYkLPVK2HbHIf0pT9XjdnX8MKU60BZwCx4CZXp55c1JDwV4aolxlwqeyuze7tDWQE99CiRD1rshWaQghAf7gk/tEPhnQk8CJlcuYd2YjnqxxjamiE28ZB50EzufYU2l9aHChsEpAVatSuVoI5U/CwdWSpUYsmUPwM43H2FyoRvaQoKieZ5oys9Qkfdufhd1SYlve8sJRJR+K6Jl4nYrGAHZj8iNp/kl2x48mHHPrZMk00FjBXeUgV/Qm+gTBv5Z6yM X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2026 15:55:39.5493 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 578c90c7-c672-4867-8284-08de72f3fdf9 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF0000C37B.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH8PR12MB9837 Content-Type: text/plain; charset="utf-8" From: Ankit Agrawal nvgrace-egm module stores the list of retired page offsets to be made available for usermode processes. Introduce an ioctl to share the information with the userspace. The ioctl is called by usermode apps such as QEMU to get the retired page offsets. The usermode apps are expected to take appropriate action to communicate the list to the VM. Signed-off-by: Ankit Agrawal --- MAINTAINERS | 1 + drivers/vfio/pci/nvgrace-gpu/egm.c | 67 ++++++++++++++++++++++++++++++ include/uapi/linux/egm.h | 28 +++++++++++++ 3 files changed, 96 insertions(+) create mode 100644 include/uapi/linux/egm.h diff --git a/MAINTAINERS b/MAINTAINERS index 1fc551d7d667..94cf15a1e82c 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -27389,6 +27389,7 @@ M: Ankit Agrawal L: kvm@vger.kernel.org S: Supported F: drivers/vfio/pci/nvgrace-gpu/egm.c +F: include/uapi/linux/egm.h =20 VFIO PCI DEVICE SPECIFIC DRIVERS R: Jason Gunthorpe diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-= gpu/egm.c index 077de3833046..918979d8fcd4 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -5,6 +5,7 @@ =20 #include #include +#include =20 #define MAX_EGM_NODES 4 =20 @@ -119,11 +120,77 @@ static int nvgrace_egm_mmap(struct file *file, struct= vm_area_struct *vma) vma->vm_page_prot); } =20 +static long nvgrace_egm_ioctl(struct file *file, unsigned int cmd, unsigne= d long arg) +{ + unsigned long minsz =3D offsetofend(struct egm_retired_pages_list, count); + struct egm_retired_pages_list info; + void __user *uarg =3D (void __user *)arg; + struct chardev *egm_chardev =3D file->private_data; + + if (copy_from_user(&info, uarg, minsz)) + return -EFAULT; + + if (info.argsz < minsz || !egm_chardev) + return -EINVAL; + + switch (cmd) { + case EGM_RETIRED_PAGES_LIST: + int ret; + unsigned long retired_page_struct_size =3D sizeof(struct egm_retired_pag= es_info); + struct egm_retired_pages_info tmp; + struct h_node *cur_page; + struct hlist_node *tmp_node; + unsigned long bkt; + int count =3D 0, index =3D 0; + + hash_for_each_safe(egm_chardev->htbl, bkt, tmp_node, cur_page, node) + count++; + + if (info.argsz < (minsz + count * retired_page_struct_size)) { + info.argsz =3D minsz + count * retired_page_struct_size; + info.count =3D 0; + goto done; + } else { + hash_for_each_safe(egm_chardev->htbl, bkt, tmp_node, cur_page, node) { + /* + * This check fails if there was an ECC error + * after the usermode app read the count of + * bad pages through this ioctl. + */ + if (minsz + index * retired_page_struct_size >=3D info.argsz) { + info.argsz =3D minsz + index * retired_page_struct_size; + info.count =3D index; + goto done; + } + + tmp.offset =3D cur_page->mem_offset; + tmp.size =3D PAGE_SIZE; + + ret =3D copy_to_user(uarg + minsz + + index * retired_page_struct_size, + &tmp, retired_page_struct_size); + if (ret) + return -EFAULT; + index++; + } + + info.count =3D index; + } + break; + default: + return -EINVAL; + } + +done: + return copy_to_user(uarg, &info, minsz) ? -EFAULT : 0; +} + static const struct file_operations file_ops =3D { .owner =3D THIS_MODULE, .open =3D nvgrace_egm_open, .release =3D nvgrace_egm_release, .mmap =3D nvgrace_egm_mmap, + .unlocked_ioctl =3D nvgrace_egm_ioctl, }; =20 static void egm_chardev_release(struct device *dev) diff --git a/include/uapi/linux/egm.h b/include/uapi/linux/egm.h new file mode 100644 index 000000000000..4d3a2304d4f0 --- /dev/null +++ b/include/uapi/linux/egm.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* + * Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#ifndef _UAPI_LINUX_EGM_H +#define _UAPI_LINUX_EGM_H + +#include + +#define EGM_TYPE ('E') + +struct egm_retired_pages_info { + __aligned_u64 offset; + __aligned_u64 size; +}; + +struct egm_retired_pages_list { + __u32 argsz; + /* out */ + __u32 count; + /* out */ + struct egm_retired_pages_info retired_pages[]; +}; + +#define EGM_RETIRED_PAGES_LIST _IO(EGM_TYPE, 100) + +#endif /* _UAPI_LINUX_EGM_H */ --=20 2.34.1 From nobody Sun Apr 5 18:20:02 2026 Received: from DM5PR21CU001.outbound.protection.outlook.com (mail-centralusazon11011069.outbound.protection.outlook.com [52.101.62.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 372FB1A9F8C; Mon, 23 Feb 2026 15:55:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.62.69 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862151; cv=fail; b=lGPFy+v1cWF/9V8Dn/h231A9VkV/eZx5c8VMS/IjAP0YejGe4giMuXNyfK94u7qaGvwXrpdnhD4nqa2VH4iS1yIUdse0n48PQuhhPk7wsL5a60c9TMtYn7GzVULkAZzcWRE/yc2zTomo+RfbYF+7Ak1ZOlXsz1zyEYK+yceOcbk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862151; c=relaxed/simple; bh=PCSsEOL2JrZcrfCO86empU2fimhWIY3v6GVVycm2Yac=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=YfBeHsFoRZj/uiVoojXOAMhId849cluHfhypnYt4d8Eu5Y3qi6z3+gJL/BEJElIW41EV2r5hPSnJmj2DT8x0WDk+nio12vAP3tDSbWzTdEM9KiudEz940+tjTn4pNxxywtDWgZz2Z0deR/sVy3ZrQgJfBrfvi91GciLfalB0TQM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=iRpMEQMT; arc=fail smtp.client-ip=52.101.62.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="iRpMEQMT" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=rHOQfklqq8WUi5OPFyQrV7HoI4x5yU/i8VMT5prYW0jMrey15hyToKvUattQQhvvVV/XWWrzWl2muoSoPe0WFy+QaBJWZo3q/Mvt0Xb/jNwICO3bCcPz1YIE0fr9QdInUAr27Acog0shzSJOP55L9jgykjhEMOl5mDkEpV1y9foYGwltNIC8zz+goPU0NwA0ZHcuoKaU4liSOC4mPp+NqTmxt+uwWobXHdoX/m8VIxRD/vfmCnAgIN28iErXsedn2QEzrMJXwtSqDBDyZBwrqjclqYzJE9tWtgw9/ZPtQ1R8nAz6kSvyhwTDlvnDo9/4RWPNktX+yJLz+p8/5WFkGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=B75mlKomfm/qn+4qf6nwTAKA98LwNab5j81V56MOMdg=; b=DK1fhhnUtY7P1IkiWntIeNLa1CKPKB8D889nIQe7KidxMfltz3KzUOgSngjToMxIq1uW8r3G5Dhycq0l4zked38q3LRxr8z1AK6+rr5QRiSJ5Sm5FL08EzLztw28/MN+5rJIEGYtSkeMGMInjeq4GRlAJulWl1WS3hauKAEoKR+hHSTI4YAmpQDHmh+EuSHBWatLoGikAPozRlJRmNGX/Kdk6Q10S3yRZUat9MDlxOeDmfdY0jamho2eoJ4DDz06Df3ITyDdul7xYYJuxTqMCSwJKAlOP6yBSZ4W8d7+HVq6XVaouS9R+HNDbgN1hDjdmM+8cX87UVugKZMPT6jFFw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=B75mlKomfm/qn+4qf6nwTAKA98LwNab5j81V56MOMdg=; b=iRpMEQMTP/5yAe6OqvvhZQ3bA89RRxim7hDHbPM5FX894FYhawcAidAQohPSV3RpDFflBMDY/qwnd79oru6lkXtgVbhvKyJCi8HG3Me+Nn1NHtVg7NcDPRwCIRmZ+ji6VaSznzzTVtiOR87erlP7eV0+Y7rD1O3fKWso7JgsLs5N68vUAufo6swpXDfcnC/jbtgcFurDEgnAfa4SMZMrBoxV1mVZ+8x5+30akGQocq3uD/vZEeiTWLzxavEFIcNefpJl37fPk7PFwHI8z5DlXjppqUrIJPxRAect4/nkC41n7w+csImm5NG7+77rz32/qbQnaDo+0G4Zf2pTOvMw+A== Received: from CH0PR04CA0048.namprd04.prod.outlook.com (2603:10b6:610:77::23) by CYYPR12MB8924.namprd12.prod.outlook.com (2603:10b6:930:bd::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.21; Mon, 23 Feb 2026 15:55:39 +0000 Received: from CH2PEPF00000099.namprd02.prod.outlook.com (2603:10b6:610:77:cafe::21) by CH0PR04CA0048.outlook.office365.com (2603:10b6:610:77::23) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9632.21 via Frontend Transport; Mon, 23 Feb 2026 15:55:38 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CH2PEPF00000099.mail.protection.outlook.com (10.167.244.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.12 via Frontend Transport; Mon, 23 Feb 2026 15:55:38 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:20 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:20 -0800 Received: from localhost.nvidia.com (10.127.8.12) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Mon, 23 Feb 2026 07:55:20 -0800 From: To: , , , , , , CC: , , , , , , Subject: [PATCH RFC v2 13/15] vfio/nvgrace-egm: expose the egm size through sysfs Date: Mon, 23 Feb 2026 15:55:12 +0000 Message-ID: <20260223155514.152435-14-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260223155514.152435-1-ankita@nvidia.com> References: <20260223155514.152435-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000099:EE_|CYYPR12MB8924:EE_ X-MS-Office365-Filtering-Correlation-Id: 41bfb6be-0c08-4721-3b33-08de72f3fda7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|376014|82310400026|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?it/xNP0+P6wnL08Ha/FigY/FQVuinfgL3JjRddXpXfL5hlJv/4j1StYO5Qs5?= =?us-ascii?Q?SI4XFO5CzZoJQ71fnK9beERH5v9HbyzcVjHe9J3fwwDAvrCeZ7t4L9Y2zm1+?= =?us-ascii?Q?dRniTtvlObUT24y3HsiBBYgkTgtQ6fRTYjdGjHxx6E2Z0MX7UiaMOEPLQ2aX?= =?us-ascii?Q?mIWhcnhwGqsRGBDvFOHcX3zKqitE6Tqa41Ud2t5rkaCmUQNZdGTNAwkxocvy?= =?us-ascii?Q?hr+xVFAYHY4XROUKzHcPyt54wASS/eeyjYFsYvwzctDfjXqZiwhcbRdYoPHM?= =?us-ascii?Q?6i5c2GA9JUlWXRlAzQs7nPS+6A/h/lvRFRMjqV5LBUqJ0lBw6fQ6IthZx679?= =?us-ascii?Q?wvJC/3tTpAfB4l0YgkFKUFiIJxvVa0vC6UzdqWd7UF8S2qsQKjcgb+btqqRY?= =?us-ascii?Q?btlqIpjVKNZc6Cq20nbWDGAELqlsvkxxgxiV48ZU4p9LWsBJVyvOkaPTHJnb?= =?us-ascii?Q?mVgVzEL4XKusKpU5DCWZNXneIKjLUS/Dl4T8d4f5r3Tari5IbJd7mc76hwpb?= =?us-ascii?Q?okL77eaLjdVHZ3Vt5w4t7pR4ZoCNsLld2WDQC3dm24Lr/1d8Z9Yk6t9fjmXA?= =?us-ascii?Q?wcuIQZpV1yHOkND1+1LoxmIS1Qk91SODMfJ2tNc/hEXs9LwlSxIg/zKqDthj?= =?us-ascii?Q?Zq7KCwn+9R2BmJJvatp8nx5iHp9Aumla05P9rTD4RyZkHJ8G/5c02sYvJ/7K?= =?us-ascii?Q?kflX6MXBPyw6dl0AYeddd1jvU+CRH0uf5YmH9Lbq9yZmnOdDuZEdxnkygAT0?= =?us-ascii?Q?+DXUA+J8X5SY/z+UezFr04aXuTkru+e4teB9D0Dy3OdYM6wbJthHYd9cRQ+n?= =?us-ascii?Q?LvyaRkrXLqrJxNQVec0HaRs68c7N7Fo3nvuzkYo9fpKtZ4uNqZwsJW4SuDLX?= =?us-ascii?Q?OIvVzS24XAnN9MQiuI1uJib4OVl9vWKAXcUE7nZzffMEIeKWP5D+p9qwRJck?= =?us-ascii?Q?cj9R/UVccbQt3tR1M2etFhpdA3BOlql1cYOMjeUq2N7C1AlUVJvWbc0rTiyx?= =?us-ascii?Q?hSv5swF71KSBIYcY2xRrd7w7P6ei9Jff9h5faQFTBBqgYr0r0tqAqPoHLNSZ?= =?us-ascii?Q?XJcH/5BntytDBvsoW1kO1ny7dDvvnm8l+JilRW+hY5Zto/Co12mUZptXD8OT?= =?us-ascii?Q?1hnmo1UInU7wIquqFM3lmWCEAouNw548VWPID2Gae+qoyjL59o4NLxubNvWS?= =?us-ascii?Q?tCP4uh0iFaF90xKehx4ai1yaIS9mC7Sat/oPhK+dS83Iry+0hEtD5TxBu9lM?= =?us-ascii?Q?f69vKFjWqFuVaLsh+AT6M7AjsNH1A3KCvc3WvrpBmGw8cd5ytwGthtwO1Alf?= =?us-ascii?Q?md7XsZlumUgxUALMfWsX6dfV1Tf9vj7YAkewoAY3HrAYTOx7V5YWYef9n9v2?= =?us-ascii?Q?sfmeYPccrIUBiFhAnVuEAm26y1RyBWyllIhsCNq8H5puG/wEmYhV7ItZwV/t?= =?us-ascii?Q?YTZQ2kHk51Mjkxk5pXSxhROPcXVCIq4/2/vjjWTn6WdyQ5Bc5rYLpRIaK1RM?= =?us-ascii?Q?CK57r18OEJnyr2iAU4uM3uUUoj+vQsnkRRr5mwtgaYQ3UCH0H5yMztO6dkC1?= =?us-ascii?Q?1+34Kyq5d8yPjeNOmycNkmi1DmP6z0hEvWmsjvPTlSouptCBjU2iEkucCDjU?= =?us-ascii?Q?1F4STAykii8EQVOF+YYTuqKvy8e0WvK/y5YX3IADrc5bs0hCAVCSutzL+TrX?= =?us-ascii?Q?dA2OAQ=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(376014)(82310400026)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Pg4xyuWuY0Q86lYkxSvRhCtyPEuN5OSAyHreTRG9oSIhIPE18gLUPDB3ngDYAoymvHQVqtX/1L3NXVDUvHbx/ZH6y5NUhg6vAm0k6HuLR52UjBTMhIKfDybBV/5b2878RUiHBwjyaSUQ/6R3aYGUDTHZLDHSQyOE+UCwABCC196p3dLW8nuSxPW7vZmGRIXE0HpTDgMf0PNa7EPIrd6QJ8+HUgjvyHmi2zI7Q+lNeno32Ob7GfTwXQtZeXnD2GmBIyUZJgQQ3+aLTjQme4TGYfz4dW51EUlS9ntjLBmitkVO5u65QYsWnZpPO5RbBPB5VICPHeSU5SLEkyF9G7+OfgoWujimLe4yBoksUVKqrMBrbobN7jcFEg9OJBbAZLzoal3EXb+sxNOup6BNC/7sHEYp4m0v7/Wvrw7IQ3SbTqYERZMk6SBp5bWZSOQRwOCB X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2026 15:55:38.9958 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 41bfb6be-0c08-4721-3b33-08de72f3fda7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000099.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR12MB8924 Content-Type: text/plain; charset="utf-8" From: Ankit Agrawal To allocate the EGM, the userspace need to know its size. Currently, there is no easy way for the userspace to determine that. Make nvgrace-egm expose the size through sysfs that can be queried by the userspace from /egm_size. E.g. on a 2-socket system, it is present at /sys/class/egm/egm4 /sys/class/egm/egm5 It also shows up at /egm_size. /sys/devices/pci0008:00/0008:00:00.0/0008:01:00.0/nvgrace_gpu_vfio_pci.egm.= 4/egm/egm4/egm_size /sys/devices/pci0018:00/0018:00:00.0/0018:01:00.0/nvgrace_gpu_vfio_pci.egm.= 5/egm/egm5/egm_size Signed-off-by: Ankit Agrawal --- drivers/vfio/pci/nvgrace-gpu/egm.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-= gpu/egm.c index 918979d8fcd4..2e4024c25e8a 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -365,6 +365,32 @@ static char *egm_devnode(const struct device *device, = umode_t *mode) return NULL; } =20 +static ssize_t egm_size_show(struct device *dev, struct device_attribute *= attr, + char *buf) +{ + struct chardev *egm_chardev =3D container_of(dev, struct chardev, device); + struct nvgrace_egm_dev *egm_dev =3D + egm_chardev_to_nvgrace_egm_dev(egm_chardev); + + return sysfs_emit(buf, "0x%lx\n", egm_dev->egmlength); +} + +static DEVICE_ATTR_RO(egm_size); + +static struct attribute *attrs[] =3D { + &dev_attr_egm_size.attr, + NULL, +}; + +static struct attribute_group attr_group =3D { + .attrs =3D attrs, +}; + +static const struct attribute_group *attr_groups[2] =3D { + &attr_group, + NULL +}; + static int __init nvgrace_egm_init(void) { int ret; @@ -386,6 +412,7 @@ static int __init nvgrace_egm_init(void) } =20 class->devnode =3D egm_devnode; + class->dev_groups =3D attr_groups; =20 ret =3D auxiliary_driver_register(&egm_driver); if (!ret) --=20 2.34.1 From nobody Sun Apr 5 18:20:02 2026 Received: from PH8PR06CU001.outbound.protection.outlook.com (mail-westus3azon11012046.outbound.protection.outlook.com [40.107.209.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 185162BF00A; Mon, 23 Feb 2026 15:55:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.209.46 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862149; cv=fail; b=a63K5mgUsNY7qUWul9Dd2jGzVZiPmIWUBqaMmK7LYLDYiiOhr8JMSY3Xh8m4wyWBl/H5NkuLmM9KcIi3QbtRlyOU7eZf/bzUColc1GBsjx1OwtchmTeXJG4C5tibeyYFAhafGa5jzES2SgQ4CLV0ebm0mQ4vN0YtFK3knCFngg4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862149; c=relaxed/simple; bh=/48JkPGcwYBVGJOYTLgYwFZd9pzXcL3Lr8tqCuQndi8=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=bnN3a8N8cNbHh5RG6miXrmT21MvSIP9qErnqP/VZK2bKUOHBoQwmqM10u3fqDM7Abdg8ZsthzljDK5dspsja6QHVHw1I/leVmV22oax4TrtC3VBreD/rPAtHKnDGAvc1+0NaWmqSFhYt9jD4D5TAYDC0jif/ad70hWbdgOgtqNA= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=cYBvgsSl; arc=fail smtp.client-ip=40.107.209.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="cYBvgsSl" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=YLzSMnYXGggcINqI7+e/7u+1Eo407M7vZL6dZ6Mg2iiA5f1y/IlrLv1oKpM6UpFQDMrsLRBYWGpj2S0esJqfiKYDsqpXZE5Hascaqd1zOK/G35XppuVuTl1EHd7RyryhBEMjOddmXDD/hZ7GeLr6ycbkRnXeRsLrh5R3oOeB3/1Vk03iJFZxvc1JRmSZI6rln4Q4we0FrOS15meN+pc5Mmu1VTykJt5Y564O2jiH7QJuxNw9CV/FpY5JdNq7Kw76URy9Jr4P16MLoK2lgaPM+Zf1hKKcpi/m+4YDYBHT5NBr9fbl543+KHnLCLIoId/RpGNeYHOfN1SAISFsjMbF7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jPvQL6+3OunCG9gwbeHmCGLE+4Uf9UnBCcYpn28wJ1o=; b=o7kPre/jHOWpV24HPnXixnmlwfYKzeMRGs4JjN70h9NguwXU+iePpwAKhEpkXFgzDnD+3BI3se7n3RDa9WAukbCcn56kiHad12UkyntP44TGVbPdPXCD6m1R2ZVhkeuydS2o83DXsuSnMqvlooiieQEK099euXAegWT3IGUIrrsZCQMnldWqKrInaHoCoZFrbRa1FLNWwPaEuHGDuBJaPsV1L4VUPOfKeuNeg9s6R7xSodf+8DxBcrN3hvMdWTMG//ztMFEufozmBdeXQ7RK3hMtk6RL76t2KiKsdv3qFGdo/ZajgSlQuWge9R8Looc9EKrjMHTpKiIUvGzneJ+wwQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jPvQL6+3OunCG9gwbeHmCGLE+4Uf9UnBCcYpn28wJ1o=; b=cYBvgsSl/CBD9Xamc3NPa2vCQcDiRV7cHEEC6lDbk30IbPkBzjfIEzQqb8clqNOzTZMGodzU5aEgkHUZLEfeg4mHt5wF9HlwQnprlcntU9X7KC/R71ePaxkLddwi7O3TSrZZVpkmzeSoM0jnAfDqOzbu0ldB52IPsSDMWXW/Y34OCJOSYCtSpB9rZV0b0BEGkmAkgzUGLOtRsP1gDT4Sjr4QdgwxISdzmCnZJ1EsBx/XUdDrVyxgj9qNUno8QYsigx5hCJ0pT+HJrWwXu1AS/HLfGr+SH2eDCp8FTcS4bIB6HH2P6DCJSkkM7aolZDY5FFmp1jK3O7j7y+U2uiv1vQ== Received: from DM6PR14CA0070.namprd14.prod.outlook.com (2603:10b6:5:18f::47) by DM4PR12MB6109.namprd12.prod.outlook.com (2603:10b6:8:ae::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.21; Mon, 23 Feb 2026 15:55:40 +0000 Received: from DS3PEPF0000C37A.namprd04.prod.outlook.com (2603:10b6:5:18f:cafe::3b) by DM6PR14CA0070.outlook.office365.com (2603:10b6:5:18f::47) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9632.21 via Frontend Transport; Mon, 23 Feb 2026 15:55:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by DS3PEPF0000C37A.mail.protection.outlook.com (10.167.23.4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.12 via Frontend Transport; Mon, 23 Feb 2026 15:55:40 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:21 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:20 -0800 Received: from localhost.nvidia.com (10.127.8.12) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Mon, 23 Feb 2026 07:55:20 -0800 From: To: , , , , , , CC: , , , , , , Subject: [PATCH RFC v2 14/15] vfio/nvgrace-gpu: Add link from pci to EGM Date: Mon, 23 Feb 2026 15:55:13 +0000 Message-ID: <20260223155514.152435-15-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260223155514.152435-1-ankita@nvidia.com> References: <20260223155514.152435-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF0000C37A:EE_|DM4PR12MB6109:EE_ X-MS-Office365-Filtering-Correlation-Id: 6a424ccd-d7b0-4496-3380-08de72f3fe52 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?utf-8?B?RStEaWREYktQcnZuelY2UjM4MzBvdUszTVNCb2RhUFU0VFhhYnI3T1I3N3NU?= =?utf-8?B?Z3lyWUlqK29oa0k1Nm1FZnlHSmQxVUgxNWxjN1Q1VENvL2VNRkgzWGNzWGtt?= =?utf-8?B?aUlSMWdyWHZzckVjNkRkOGZ2emdmL1pZVTFBTlo5UVB6RE0xQVBTNjUyUHNj?= =?utf-8?B?MW5RbExvRmw2OHRGZ3hEbXVWTzdHbVVGZkVUV0RDSDBmNmhZbDdPZXZCbGdR?= =?utf-8?B?UFY1M1Z6RlFvVlZPOThzNkNYcjlDcVhvbjVja1VoUjFPdm04R0lHcnNYQ3My?= =?utf-8?B?SXhGbVZMSEYwVWx3NTZFNzBYRnFEc1hJY0VYZnA3Z0JjbUZkQXlMbG53dVBR?= =?utf-8?B?QWlwNXF5czU1Ym1tUHRrOTVHdldvbFJIN0xnSVJFeUhDNzNOd2JTWlR1SzRz?= =?utf-8?B?L1k0dUJRR1hyd01RaFNOVDlwQnd0amVReENuK2ZvM3Jmd1daMFVuekVWdTBq?= =?utf-8?B?c0pqNk54QmJOSnpkS2IzR0hQeTYrNWUrcyt2cU5uQTMwZVFReWtuRUh3N29Z?= =?utf-8?B?REpaRkZxaXNIdWw1TmV3Q2VzWElRNVhrNHhtVnlxSUdEWnpxQjlkUXEwTm1D?= =?utf-8?B?enhxZmhhZFVXZnFsQ0IzSjhzWFRzUjhXYmVmd2Q1Q09uZ2FYaUJqa1orS05n?= =?utf-8?B?RnlId2lJYmo5REQrcTdLTHF2d2t4Y1FRaS9JeHc3SnZXYnB1U3p6VzFRSTYw?= =?utf-8?B?SHpGTG5aM1lRa0cyTTBCTkV0SmFBajNhSXB5LzIrUSt4OURXeURXVHFxMjZ2?= =?utf-8?B?NzdtYTIvRTRYZ3hIUTdYNExZU2RLRHBjNWVyYUxUa3JPTHducXNEUERyKytU?= =?utf-8?B?WDVyYVFqMVNQNEV4OTJKS25HTzBlTnpLVHVsYW1SVnlkUGc1TlRDcUdHc2tW?= =?utf-8?B?ZU50eERwcUZZQXhzZzJZYUpVR1M1YnZVZkFXTUpDcHZ6ZlRmU3AwaHJtZElL?= =?utf-8?B?MzU2R2tJVjhtUVUvYmEwclQyQ1N5bE1wYW1BZFZpSEdEOTlSYWJnSWJkYnZj?= =?utf-8?B?bExYVkhwRFFOZ3NtTWlpYkpHWXMvOXQ0Zi9oWHFYaStjWUZ1bDkvY0c2Rko4?= =?utf-8?B?YlR1WjlWQ0lFL015VkN5OW45ck45Q3U2UHpQOVNCd29meTlMM1I4ZVZUVXcx?= =?utf-8?B?eE5tdE5tSWdjWmlGU3NmWGtUR2JtRmJQbE1TQlZOTzV4L1NqV2s3dDFrQlAv?= =?utf-8?B?bnRpUXUvSFpnY3Q3M2w1Vy9XZ1VSbW1XbUF5TFNtNnVFVmJzZXBvS0E4RTJQ?= =?utf-8?B?aDN0aFBSTjVoM1VBSzRqalVqQ2svME96d1cxTWQxNFg2RTE2UWhiKzVybDJR?= =?utf-8?B?bkVEanRiSUR1YnNwTmxyTDFQNngrYWUwamo5OGFwN1QvaTlwL290V1BNdy9Z?= =?utf-8?B?THgvYWhXYmU4c2dYWDhZMUZKeFYvU2hLTisxNUFmdjZPRVArODM0Q1N1RGlK?= =?utf-8?B?a2lHOXpYQ25YS2NXazZ2a3g1TVdRclZSYWcvWHhQTzVBTUJYOG5VZWZPMndK?= =?utf-8?B?MkplendFWXZuSFRsaUhVdVc4dW80aVBJWHI4SXU3R3RZb000TmdaRjVoSEZM?= =?utf-8?B?aUxYU04xeUhhMnZiU3R5SHpnY3JwdDZyK2Jld0NCc29jQ2dTdk1QMUhtY2VM?= =?utf-8?B?YjVSVTFTY2Jjbkx3TXhBNkVtUXYyMEhNdWJtYXVua1BUOHRZR3BYZXo3d280?= =?utf-8?B?WjFyNDVIUDFkMENXY0tuUVlQV2xHZ3JaZTZrNG9vbmFzVmtzaDFhWXBPaTlr?= =?utf-8?B?NEtxWHFXSUJxVzF6UHF3LzdUaHNwbkFpOUY0Q0R3NSttcE56RGRRU3lBbmo1?= =?utf-8?B?b2ZYTGIwaDczaDNMdm0xU29IV0VTaHcwdmVOYUdlcGxRaEZzOWhTcFZzbDJp?= =?utf-8?B?dG1iMnJxRFVPZUhXZ3dFQWVLVzVVekpYZ3FjbVZlTXRtSXhza3NlRndDL3dW?= =?utf-8?B?U3RPOUZJMFgzeVBnR05lOTc1YVc5M2FFczNQRjl5VVhxdUFNQUFUYTYycHZY?= =?utf-8?B?a3ZOMy8rV1R0eCtFL3FpUVl6VGhQNXNoMXpXS0FuaDZFeFFSSEV6dmxVOTZz?= =?utf-8?B?VWdITEdmeHBGRzQ4RmZxeUFVWFFWOXdxOXQzUUR2bnBVc3ZVem1KOVhnWGJt?= =?utf-8?B?UFd0czVBc2RrdnRRYlFnM21lMnFUMGtXQXJoVENHaXBiTE5jYjBLOWh0aTlq?= =?utf-8?B?VWlUNm9SRmRwS29Rb21xM1pERzAzcEFtVEUyWW1wMFpydi9DUHd0VnZhZ2ti?= =?utf-8?B?TTZpdmF6K0hCdzYvUjZXQXpSN3dBPT0=?= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(376014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: be/Yg1wsjyzR4JuddN/+CnbTDht/6taQ/OhWqpHTqQDAX0lU6RiS5ZnRTHO3Wd4qPzxiO+nyYJ4Kwec72urXI8VrK4FJtm6Whz3WPV2M+gXvls69D+mYyEdRTJNYbpwwyzqOUSCrtjDCd6GHBlQM9a6FBMyLbu5ITkSGyYgcrN8k0roxPwCJHoTSoLcSsWB+4CrEgq3UaWgGft+ZfnX5fO64qEXzcDCx/saHqkZoHSONASv9nAJTKekyCd0Da9+WqLCh40Nmf0A8tETeyb0XIWHLX1MuFI3k9X8wKcquLWR+tAIPeSKggo2KdHKRZxod0WA2YFNKky+rW7ZQ/BZ7iYyJwxH4lTtZpAD+ZWGMCDM/bIcvk6Kn9CR6ZKdNHEB14JzKlWLVA1v4ZpHqfUQm8DlvMuXBpNiFTFONE719aBUswW+7AyflGPDZvCR8AwjZ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2026 15:55:40.1317 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6a424ccd-d7b0-4496-3380-08de72f3fe52 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF0000C37A.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6109 From: Ankit Agrawal To replicate the host EGM topology in the VM in terms of the GPU affinity, the userspace need to be aware of which GPUs belong to the same socket as the EGM region. Expose the list of GPUs associated with an EGM region through sysfs. The list can be queried from the auxiliary device path. On a 2-socket, 4 GPU Grace Blackwell setup, the GPUs shows up at /sys/class/egm/egmX. E.g. ls /sys/class/egm/egm4/ 0008:01:00.0=C2=A0 0009:01:00.0=C2=A0 dev=C2=A0 device=C2=A0 egm_size=C2=A0= power=C2=A0 subsystem=C2=A0 uevent Suggested-by: Matthew R. Ochs Signed-off-by: Ankit Agrawal --- drivers/vfio/pci/nvgrace-gpu/egm_dev.c | 47 +++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm_dev.c b/drivers/vfio/pci/nvgr= ace-gpu/egm_dev.c index 6d716c3a3257..3bdd5bb41e1b 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm_dev.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm_dev.c @@ -56,6 +56,50 @@ int nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev,= u64 *pegmphys, return ret; } =20 +static struct device *egm_find_chardev(struct nvgrace_egm_dev *egm_dev) +{ + char name[32] =3D { 0 }; + + scnprintf(name, sizeof(name), "egm%lld", egm_dev->egmpxm); + return device_find_child_by_name(&egm_dev->aux_dev.dev, name); +} + +static int nvgrace_egm_create_gpu_links(struct nvgrace_egm_dev *egm_dev, + struct pci_dev *pdev) +{ + struct device *chardev_dev =3D egm_find_chardev(egm_dev); + int ret; + + if (!chardev_dev) + return 0; + + ret =3D sysfs_create_link(&chardev_dev->kobj, + &pdev->dev.kobj, + dev_name(&pdev->dev)); + + put_device(chardev_dev); + + if (ret && ret !=3D -EEXIST) + return ret; + + return 0; +} + +static void remove_egm_symlinks(struct nvgrace_egm_dev *egm_dev, + struct pci_dev *pdev) +{ + struct device *chardev_dev; + + chardev_dev =3D egm_find_chardev(egm_dev); + if (!chardev_dev) + return; + + sysfs_remove_link(&chardev_dev->kobj, + dev_name(&pdev->dev)); + + put_device(chardev_dev); +} + int add_gpu(struct nvgrace_egm_dev *egm_dev, struct pci_dev *pdev) { struct gpu_node *node; @@ -68,7 +112,7 @@ int add_gpu(struct nvgrace_egm_dev *egm_dev, struct pci_= dev *pdev) =20 list_add_tail(&node->list, &egm_dev->gpus); =20 - return 0; + return nvgrace_egm_create_gpu_links(egm_dev, pdev); } =20 void remove_gpu(struct nvgrace_egm_dev *egm_dev, struct pci_dev *pdev) @@ -77,6 +121,7 @@ void remove_gpu(struct nvgrace_egm_dev *egm_dev, struct = pci_dev *pdev) =20 list_for_each_entry_safe(node, tmp, &egm_dev->gpus, list) { if (node->pdev =3D=3D pdev) { + remove_egm_symlinks(egm_dev, pdev); list_del(&node->list); kfree(node); } --=20 2.34.1 From nobody Sun Apr 5 18:20:02 2026 Received: from BN1PR04CU002.outbound.protection.outlook.com (mail-eastus2azon11010024.outbound.protection.outlook.com [52.101.56.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64AD430AAA9; Mon, 23 Feb 2026 15:55:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.56.24 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862150; cv=fail; b=VVH3/VxP+FwMzzYJs/6qjPFqR2qd+wUK0o+zFoppYQTr8ihtqeYVIyhSEzGmuPzbkX3EzxsnfDEBQ8RhCOFlxXKd45W4A5JHed5w5YMN08McRtyTo4A+GMF7CAJXXXzz5YyILY5xe0dI8D6R0KiBiW33Q5MLUe5cw6WAu6j2lB8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771862150; c=relaxed/simple; bh=g6r/8oIAVimzZBq29YK1UI/PGWw1WlW8HsNv6BBZb4U=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=SyH3413Ze8YQ2yR2FNW6edpZ1BX4qqaG+VJNiwn/Fe+G1YrBTq/a1z5+JA4sFJTbrAyhhjNzc1oL09atectffCdSnenN62Y0bZsIHFp4UC8PEh69+/zPSLZv+ZQ1ZStQP1CQbvgcmi23xuBBjssMwtC+X2x9Kr5UqzrzdwBdh1g= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=E9xsGQP0; arc=fail smtp.client-ip=52.101.56.24 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="E9xsGQP0" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=XSYbaZUlLwI2oslnf8E4WiiS9jdJ3GfMrozPM1Wx6lfnLqOK0S6jnksoBpFFX5FV+KGIQUqpE7h6qnW/+cmhJGYXhQCdAbdPF99ywHVciiJaqDwUAEUFvJr4ERXXoUJHWNQaNiJb1U8hhAD1jQ8cIM8RbIxUEnL5LneYLk4uk6doDFF92hOBHddLt0popWMiASFqjQ+gUuRg+wDTa3YAwP33mB+suDhEbr0/Sk0nG5lZryN1sHn7YkILCXUpnqIaw97pstmtWCaLY0ufqvFLxOx01tT4+dBQb9eYm6K0WSKb09s3TvWXceu7aLqVr5iPNR3KVIWCbOAWg83poLEv2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XG1Emf5niFZeaGGr1StNoL/JjI1MdxauRNQHOsbP7r0=; b=NeBh8TdKF4+ByhQnkyIJXsQub3KIRH+ExfQneBtg642kbIXsSnfZAkLBTBlKPijEHwhK/DoYzXV4OST/vGhNTjL6S/tm2wVF2hHCCwLwy+okRHY0eGKNwHBJDBmByJQGtWAfOPqvJY4KdK/M54IF+qP0LP5natN9/CJWGlAlAmYZTGlsgAZJM3P9jxZbGLtCkSEXhYbHHLwdK623ZlKXqgX8ZuB8sxHH1tG6PCrgKIYd+iL2bdjD9tnfrUVkw3ibMaaxpXaBkurl7Z4y6S45KQx31g20+4kW1b1Sb3d3ZHpv/U4z2zqcmaDSfJkeYU092nWbDOQZ1j9b9rhixlOxWg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XG1Emf5niFZeaGGr1StNoL/JjI1MdxauRNQHOsbP7r0=; b=E9xsGQP0IFud0iTxiVJ1oh2LXSTiWdtb206Yt2L6RDp9eHTJgkUJ5jMIcotd67QJ5iAe7BEI8uNBJ8f8lPcZ9DiFRTpkbap5C4b3NsAkuXqVjrdGsgbT9j40L48UBBkeyTiIB2bBPljQu3xNKyh4HixEIGK8rYHORhShnm0HA9+pKj4bdEzCIx/7r3Sz0hCqjD3Azojq4LX3RMnAO7OcrgqVA1CwkPSTHzKMWh9niYkxl39LzOP3H2Y9HC+IFWT3gX4c8UErxFGTB7kQV5PPVPlPLJySnue+RvnoZJzWjx31fmGUdcSnovzCNR+WNPWOdMaEfzcZzaZL9R+3Jk62NQ== Received: from DM6PR14CA0069.namprd14.prod.outlook.com (2603:10b6:5:18f::46) by DS7PR12MB6071.namprd12.prod.outlook.com (2603:10b6:8:9d::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.21; Mon, 23 Feb 2026 15:55:41 +0000 Received: from DS3PEPF0000C37A.namprd04.prod.outlook.com (2603:10b6:5:18f:cafe::f) by DM6PR14CA0069.outlook.office365.com (2603:10b6:5:18f::46) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9632.21 via Frontend Transport; Mon, 23 Feb 2026 15:55:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by DS3PEPF0000C37A.mail.protection.outlook.com (10.167.23.4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9632.12 via Frontend Transport; Mon, 23 Feb 2026 15:55:41 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:21 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Mon, 23 Feb 2026 07:55:20 -0800 Received: from localhost.nvidia.com (10.127.8.12) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Mon, 23 Feb 2026 07:55:20 -0800 From: To: , , , , , , CC: , , , , , , Subject: [PATCH RFC v2 15/15] vfio/nvgrace-egm: register EGM PFNMAP range with memory_failure Date: Mon, 23 Feb 2026 15:55:14 +0000 Message-ID: <20260223155514.152435-16-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260223155514.152435-1-ankita@nvidia.com> References: <20260223155514.152435-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF0000C37A:EE_|DS7PR12MB6071:EE_ X-MS-Office365-Filtering-Correlation-Id: a118353b-f719-4c3c-e850-08de72f3fee5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?ZzjfmrpmvA8tE6fO4+ucA2kYg56b7gACHHiesfEM5VTkpasOYUrnHixaQA6e?= =?us-ascii?Q?Hp8NSijF8VZIW1WYuisojZVt1APj1/sZWOyjBjmn+Nx/lA1vavJPasSUxjCA?= =?us-ascii?Q?7i9N3HHIkXFyFbTxfEFeD7VJrG4e6yLY7j3GBzuyjarypmK/6nim/B416tIK?= =?us-ascii?Q?Vn64dU9fTEAgzH7WyqhNzfLLlCYxXMjiuE5i0NEBNOjW+YVLweXNVb1Ocjcg?= =?us-ascii?Q?fFNy2jj3E7xmBqH7A7it1JE2pWkvpCbbFJ4Kve36rMVJisS+LA4wJh6NGSrn?= =?us-ascii?Q?WXHJZ7cZi5YepU/y4pN2bkVNU94OzHHszsdFqzT32O/B1xjRfdGdHRGjZdXi?= =?us-ascii?Q?xRXq6rcnsABwTjcn36zkhMBPALddhVtGEVuOJV8XYoqhC2HH6tzPa+CwBzas?= =?us-ascii?Q?VTQeTyKAYS5VzxgthuoNzgAPLmJdYIq3LzkcCkgmJJcpeybhGauQZKdw85LB?= =?us-ascii?Q?QPl9B/UwaEQU7w4oqqh9KK2lUDd4cvJvVxfov0bpMbTDOqbdpLf9/Fp2HIUW?= =?us-ascii?Q?GxuJtzI1AH0dRTvd7nFBGTjJuvUrMpE3ahcxOm8WW8FrDiyM/KXHt2Dfuqvd?= =?us-ascii?Q?DFvTpewgM9T0hUW3/2ipSEpw+JOqs7QZuwZoJpJdbWR0jm6+TxN+ZZjQD6Fb?= =?us-ascii?Q?pRgLLGAVnQ4EiL8TQGXTBVLOAhDnuszkXBzuSYZEPSj0IBrFP50L7DJRjiHG?= =?us-ascii?Q?fm9FYb4rvoE+ZZ5tUZSaYjFdyRFPegCVhk0mi7nvBfIy9duVU9dEkhFAqkvd?= =?us-ascii?Q?x+meQaY4Xg5XrOBkHz3ww0wQd8tn/B2W1bxFLOSkwMQl57RjQL0YCosHon38?= =?us-ascii?Q?X1xUEiCwvcfzwOx3p7YpJI5lnGAqtQRLfiLPLhR0PpGlH+cZKznmDkwNgwXV?= =?us-ascii?Q?OgwqujrqjtkKVq7aXRErKO0b70bnqQ8PmMeWhtR/IMvU0ZfUntEwAeI0DZNL?= =?us-ascii?Q?HIp6SeGDSR2XNJRKwSAwk2l1QpoJDvPLHZ5hGw4n4ogFniTcdftsVSG2RDGJ?= =?us-ascii?Q?XpNye559om90NV47qRblnZz4P6Mpjc6XOGof0Irz37p4VEwhIZdIwufCQmXs?= =?us-ascii?Q?0mgI7m3YD4MqZVDwV528gYr00qOvUwA/JsW5UIf79QufM9swuoBwG7gDS78a?= =?us-ascii?Q?1eOjhjHbrqROKcMkrUjFrioTU31Yrmqm/OgP4/xGnHCq7cNv6SVQPNSOFQGJ?= =?us-ascii?Q?N+zApQEbPBnK42CubRlamU4WV7OPWdjR2A9TmFIOmcUAHWZQGYNEWieQFvIR?= =?us-ascii?Q?1o0jsN/WRvYcCbQMEyWyUnoIFsrAfXTESHj86ioL4ui30zXrBBIxySwIkhI7?= =?us-ascii?Q?4z0kqph0ucoQE4R3TqfKVBJGJgrWorxFE/s9707/sBs7eajLTgMLRff/s4vg?= =?us-ascii?Q?N2J5H6jJCRXpc96AI7BMUeBl/mENrhV1Vm2ALrO+1Juk5tLfFglJMLe8E+ri?= =?us-ascii?Q?CD0VdcNwqCpfbJYB4Hp3iS65VPj5XAhF44U9u27L1taObQh6KL0UgSNVCQru?= =?us-ascii?Q?ySJ5OfbAeuzp8eHrL3jZsM+YLugjnKLuPai/htv00nO7I4439pd/YZfGXsnO?= =?us-ascii?Q?i7nFDEfr6qL6QVWNnzE2l6mcdS4/Hii6dBMxD1NLVyGJ9QYThzcylS65hoJk?= =?us-ascii?Q?97CK9+CW+NTVqlfg9K05p3CGsPMSH980dc/utO4igzm/n5xoBb9c6e0Zw9dJ?= =?us-ascii?Q?abCNjg=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(376014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: g9r1rNs2LdmBcky2OiSDcVvL/ZnA/PbZ+AZ+bclIawuhH+vzBZiS4CtNlkBVvrmea/oELyBFC+aI9i/0tQjr7s0jbxv2kGwFrdkf0+H1zk8oVOuqRykR3hLFdG/ME8jNr75wzg7b9+pKbRW626YqX7Lcaylp04bIMwLE4RXOitqUZ8+mSibu6HQXThEvwiEv5Ba3IqekJYyYUeK9zVSKDNZfgjRzUZzq+6Kaoq08ZbmC1cIjWcfx6ljVqTvr67EEvOK0P+YEAva2LJ1pLPAPII1r3yfux4Ib7t/OnrHLGVdJn1NJqDhwvMwesn806QvY6HSY9D+RwbhtDnih/jdEuqr4dD0pFU3LWzaSgh/oaZT3g9lgqRXDyxuT1kQUN2fXd/D5qfKRygE74WZW6bnRztXR0B9BDPhXGpx8XIjWD/eiPdxDdVSn+AAgZctioVhK X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2026 15:55:41.1001 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a118353b-f719-4c3c-e850-08de72f3fee5 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF0000C37A.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB6071 Content-Type: text/plain; charset="utf-8" From: Ankit Agrawal EGM carveout memory is mapped directly into userspace (QEMU) and is not added to the kernel. It is not managed by the kernel page allocator and has no struct pages. The module can thus utilize the Linux memory manager's memory_failure mechanism for regions with no struct pages. The Linux MM code exposes register/unregister APIs allowing modules to register such memory regions for memory_failure handling. Register the EGM PFN range with the MM memory_failure infrastructure on open, and unregister it on the last close. Provide a PFN-to-VMA offset callback that validates the PFN is within the EGM region and the VMA, then converts it to a file offset and records the poisoned offset in the existing hashtable for reporting to userspace. Signed-off-by: Ankit Agrawal --- drivers/vfio/pci/nvgrace-gpu/egm.c | 100 +++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-= gpu/egm.c index 2e4024c25e8a..5b60db6294a8 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -6,6 +6,7 @@ #include #include #include +#include =20 #define MAX_EGM_NODES 4 =20 @@ -23,6 +24,7 @@ struct chardev { struct cdev cdev; atomic_t open_count; DECLARE_HASHTABLE(htbl, 0x10); + struct pfn_address_space pfn_address_space; }; =20 static struct nvgrace_egm_dev * @@ -34,6 +36,94 @@ egm_chardev_to_nvgrace_egm_dev(struct chardev *egm_chard= ev) return container_of(aux_dev, struct nvgrace_egm_dev, aux_dev); } =20 +static int pfn_memregion_offset(struct chardev *egm_chardev, + unsigned long pfn, + pgoff_t *pfn_offset_in_region) +{ + unsigned long start_pfn, num_pages; + struct nvgrace_egm_dev *egm_dev =3D + egm_chardev_to_nvgrace_egm_dev(egm_chardev); + + start_pfn =3D PHYS_PFN(egm_dev->egmphys); + num_pages =3D egm_dev->egmlength >> PAGE_SHIFT; + + if (pfn < start_pfn || pfn >=3D start_pfn + num_pages) + return -EFAULT; + + *pfn_offset_in_region =3D pfn - start_pfn; + + return 0; +} + +static int track_ecc_offset(struct chardev *egm_chardev, + unsigned long mem_offset) +{ + struct h_node *cur_page, *ecc_page; + + hash_for_each_possible(egm_chardev->htbl, cur_page, node, mem_offset) { + if (cur_page->mem_offset =3D=3D mem_offset) + return 0; + } + + ecc_page =3D kzalloc(sizeof(*ecc_page), GFP_NOFS); + if (!ecc_page) + return -ENOMEM; + + ecc_page->mem_offset =3D mem_offset; + + hash_add(egm_chardev->htbl, &ecc_page->node, ecc_page->mem_offset); + + return 0; +} + +static int nvgrace_egm_pfn_to_vma_pgoff(struct vm_area_struct *vma, + unsigned long pfn, + pgoff_t *pgoff) +{ + struct chardev *egm_chardev =3D vma->vm_file->private_data; + pgoff_t vma_offset_in_region =3D vma->vm_pgoff & + ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1); + pgoff_t pfn_offset_in_region; + int ret; + + ret =3D pfn_memregion_offset(egm_chardev, pfn, &pfn_offset_in_region); + if (ret) + return ret; + + /* Ensure PFN is not before VMA's start within the region */ + if (pfn_offset_in_region < vma_offset_in_region) + return -EFAULT; + + /* Calculate offset from VMA start */ + *pgoff =3D vma->vm_pgoff + + (pfn_offset_in_region - vma_offset_in_region); + + /* Track and save the poisoned offset */ + return track_ecc_offset(egm_chardev, *pgoff << PAGE_SHIFT); +} + +static int +nvgrace_egm_vfio_pci_register_pfn_range(struct inode *inode, + struct chardev *egm_chardev) +{ + struct nvgrace_egm_dev *egm_dev =3D + egm_chardev_to_nvgrace_egm_dev(egm_chardev); + unsigned long pfn, nr_pages; + int ret; + + pfn =3D PHYS_PFN(egm_dev->egmphys); + nr_pages =3D egm_dev->egmlength >> PAGE_SHIFT; + + egm_chardev->pfn_address_space.node.start =3D pfn; + egm_chardev->pfn_address_space.node.last =3D pfn + nr_pages - 1; + egm_chardev->pfn_address_space.mapping =3D inode->i_mapping; + egm_chardev->pfn_address_space.pfn_to_vma_pgoff =3D nvgrace_egm_pfn_to_vm= a_pgoff; + + ret =3D register_pfn_address_space(&egm_chardev->pfn_address_space); + + return ret; +} + static int nvgrace_egm_open(struct inode *inode, struct file *file) { struct chardev *egm_chardev =3D @@ -41,6 +131,7 @@ static int nvgrace_egm_open(struct inode *inode, struct = file *file) struct nvgrace_egm_dev *egm_dev =3D egm_chardev_to_nvgrace_egm_dev(egm_chardev); void *memaddr; + int ret; =20 if (atomic_cmpxchg(&egm_chardev->open_count, 0, 1) !=3D 0) return -EBUSY; @@ -77,6 +168,13 @@ static int nvgrace_egm_open(struct inode *inode, struct= file *file) =20 file->private_data =3D egm_chardev; =20 + ret =3D nvgrace_egm_vfio_pci_register_pfn_range(inode, egm_chardev); + if (ret && ret !=3D -EOPNOTSUPP) { + file->private_data =3D NULL; + atomic_dec(&egm_chardev->open_count); + return ret; + } + return 0; } =20 @@ -85,6 +183,8 @@ static int nvgrace_egm_release(struct inode *inode, stru= ct file *file) struct chardev *egm_chardev =3D container_of(inode->i_cdev, struct chardev, cdev); =20 + unregister_pfn_address_space(&egm_chardev->pfn_address_space); + file->private_data =3D NULL; =20 atomic_dec(&egm_chardev->open_count); --=20 2.34.1