From nobody Mon Jun 8 15:36:52 2026 Received: from CH5PR02CU005.outbound.protection.outlook.com (mail-northcentralusazon11012019.outbound.protection.outlook.com [40.107.200.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7CE733783B5; Thu, 28 May 2026 08:32:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.200.19 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957127; cv=fail; b=UE76reEu9GUCv6JDqi22+whInQo8+LlAOy1Eot2GArBHb89qibQlFCVJ4bHxSnEaKF6zmmPynU/+9D61umAyXiU9/TNj1C1Zi5yVWQLCsQJFYAyIV6OYHHyqUY2S2C1Gpj3KR/G5tfnOVVKTk9LImrbFhHWK/l1KSqiErjA5vjs= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957127; c=relaxed/simple; bh=RXc6h4LSKgyKi45dlbaqkH+beR20QfZkUNCUjK/1zhA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=WyJZHtJeELrNI+v9bGIVHLoIS1X2/IBSioGtKkdQejGlyj2YolQlFJ65iY8PkSJwDcWOKvbovS0vxLO3d5ZEsAKMhrhyPk8rwgST75T8Vz8p6Rwe9CPH0KE0VQi9RlCJ6OhplEZ30p36gpTyvwYYRP1tWyCxaWEuyC7uWZ9B3bw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=aJ9RNqlq; arc=fail smtp.client-ip=40.107.200.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="aJ9RNqlq" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=xOTJhcKtXbWF5aB38bvnSbSIaHO3acDjAUezYK6fF72Eu7qxpKqpu4ZA+yV79g8nLhQhlRqamJRa7epfLseY7ta4iMgMeLkQ/qwFsV2yCWgwKDwRrbKmaVeKhImBOcKimp1ipb5hd4JguXAWT0zkYnV9txKNBc7z6u37KR9/HC/HOlwCbGQjBwsldHGYmd2eThbtDYjUbmdqGEIFM1RnKSKwsdWREvEZhrLpdBFBhBa8fgh7WT4ZUlbARISo5Px3un1dLiPqrhLfM5t5ELppPkbMPGPwypKzw6Ml1X8f1MFuDwHIW2HV5In5amBl6WfF7qg2aOVizHLB2hEFRQr6YA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KMBNqh6jXr4tSZhXYmT4BB8OeZhtSRPQiSpqFyC0NxM=; b=CuAqph9xTSiFNCN2bc4NvhSp37ErRFcUc7FFSJR4dNXnJvvjqpd4Rzl6UpqdcRFxoZazZYsdRuD3WGZMr6GIDx+flJJwANlDNdVG+ZMiK4nTwIvEF5NSuGTOi622gjAkuROtgjHDZlvL+B4sQM8RLKM7ePFcauQm1L6b++gLX91zRaOHERXJTA2g7jN2vyIc+04+kbuQqbylViRmTH6vnnQI8jp09rJy/ldqJVlht2VcLzRffcseC7eshTaxr/qgxrbioI1s0q0U9ogUAvXH275Pg3xzqDbFsM5rx9i4q0wkM8ew/Jb8EpGQWJ8kNS92uwAu69Y+NkLOj1fOmPYm8w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KMBNqh6jXr4tSZhXYmT4BB8OeZhtSRPQiSpqFyC0NxM=; b=aJ9RNqlqYvGSPbzMr9SjwzGWWh5chai7ShD2moSD3nzOCr3as4iKOqRp5spq6y3BGn9QKMoRhx/EGEEVO/nAua9ED/lHFsDy0XgCWmtxDicx2EoD4fRkMKAT3X3d9vBsE+k2RxfYIgerEv1PArKPdeduI79lDPlv2gCm1upY45T3poP69D9NOtErfl2H5grAxJeF1hcMWW2LGdpFG1/EcV0wplstW/UNpyofxJTP/s3fd2xS5/8EV91rmdHHQ8EmGh1Dc+F8eHRknwXnsQFWTrrNaxetqiLC6i8HPsehInDKx4AHd4QGRBIR9JiTT3YFw9KOhVqwwhDtbjDTMJj+4g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) by SA1PR12MB7442.namprd12.prod.outlook.com (2603:10b6:806:2b5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.12; Thu, 28 May 2026 08:32:00 +0000 Received: from DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7]) by DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7%6]) with mapi id 15.21.0071.011; Thu, 28 May 2026 08:32:00 +0000 From: Srirangan Madhavan To: linux-cxl@vger.kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Cc: vsethi@nvidia.com, alwilliamson@nvidia.com, Dan Williams , Sai Yashwanth Reddy Kancherla , Vishal Aslot , Manish Honap , Jiandi An , Richard Cheng , linux-tegra@vger.kernel.org, Srirangan Madhavan Subject: [PATCH v6 1/9] cxl/hdm: Add helpers to restore and commit memdev decoders Date: Thu, 28 May 2026 08:31:46 +0000 Message-ID: <20260528083154.137979-2-smadhavan@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260528083154.137979-1-smadhavan@nvidia.com> References: <20260528083154.137979-1-smadhavan@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SJ0PR03CA0378.namprd03.prod.outlook.com (2603:10b6:a03:3a1::23) To DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9474:EE_|SA1PR12MB7442:EE_ X-MS-Office365-Filtering-Correlation-Id: 57820bb0-ad09-4237-8f82-08debc93964a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|366016|56012099006|11063799006|18002099003|22082099003|6133799003; X-Microsoft-Antispam-Message-Info: B5XWZ8l7fGv1cwEbjVKDYFkF0VMHilBZawOXB33b1mXfP2WXbJgPxzCE4kiyPDuvw7U7IgCmMeHtdWcv0ANgl+9Gn6OGXeX0VHOJwHXAzBU2+oMcRpaiqB5FYHJdspTVlyUT/xpgaj+ISx6tbyXy1+7fXzZZx8q10J4/1zoblDlJbop0Zw37WzTd0KF9eFORbLtbtNomampyHmjzNC8S8PaCuhSh9agsB2/MwjWic17mrSnuat1oWPVJLZr6sSqIP6phsNgqG3/RH3bnKPZiGTAEEkKAN1wgb1ymkgY7qiUOBbL9FOpMn/zkxW6hFngdhAlQoPpCMUJ2MeMNCsHIo1RWCALM979uuxXmmHQ8dWXRlqSGsxPi+gSD1BUxrYF83V+3ZixR/uLrMCdV0zjLaWIVNLKIz2fLuZZI37alCtBWVC0CUBH//E4jp7DJqZRnVbiV79GCbRFhbFYRklveaRTshojeK1NMj3K1azBZHGmjnEji6xcg7S9rfZdwlUnSGgBRPFrq6f7RnUQ3oZdHicl/xk4IfAbHPNezw+JZfG1/bj15HtsIyNRbXhqNou5L1Gz+bMecUppyr4OIFnxShleNiMOy7OPECJ2VGxXPqxXGRb6RV9VPQ8vcbvPswypO65QlJ+G+yzcf/bh7qsKu991Ed97HpoTol0YljIkfv68B8/BaH2da3WDUr35xcq8d X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9474.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(366016)(56012099006)(11063799006)(18002099003)(22082099003)(6133799003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?yeMpGDiaSaR3t6fX6Ak/7nIhUXoA/zyOprlUCPWLp1YfibKCXFyimmO/MM3T?= =?us-ascii?Q?8UgbQZeLkulcwZzOPd/tMUVNICwU8LqYiET+S5D/lKAZ2nMqwUhpGSv7ufHT?= =?us-ascii?Q?S8WrNddVOFzv/N81XbimdIpgUZ4NLMLoXLPUA2EXqPUlzqD1jTB4F+gdgROp?= =?us-ascii?Q?5Y0SWQAsIPWbCwIkkFfE5uOVEeD6eQw3CCBpcg3mFBGwZhRVXQRTMoZMQUMe?= =?us-ascii?Q?z7bFPdzMZEL6GmlggWNzTXfPfcoN3//ZfwYrt4Zee1wwrZwRHBlzfwdHVUU3?= =?us-ascii?Q?hv7LVOLN8Tlvrjt/Jsj7gikabyH9NMGEk5EsNGHk91K59d5BO83r3SOk2jOL?= =?us-ascii?Q?gfcyKWDP/8lpvvl4hC2rHqu+GCWO2hMiDH1zvEp1uRRdYwrujYYYJp74xpbZ?= =?us-ascii?Q?v0AnMo04JkM2z4P+nPQCRVXWek6lln/D3AFVEwun8XJE9fwGcdnfOYk3of7e?= =?us-ascii?Q?MPeqpF1ZjuxkTZ5y5B75Rxeh1A5XWGgzk8e5MSVQUJDsDBsNn/k93ZoojikK?= =?us-ascii?Q?cbhEXWMYuAFN8qZ1pULY2LAsZrXfoAreCp+iXVNIMQWUOtJRK4Pgk+r/OrEj?= =?us-ascii?Q?BLqpiydhhiQRAgBdCi84Q4OeVzUNCIP73oOXGSCg1NeQDJ4/WHMRi+FfGwlT?= =?us-ascii?Q?YyznHpy62DY7umLfcVw48mxUXlQS6eWYmVl9I5e5FB/Q8IGBbuB3JX6xD8Rb?= =?us-ascii?Q?2XM5RXEljvbWyUnkHAtJO+ptpJOYAeu3/gIdSO5amvjcWGEZzbqRfpO/8O+J?= =?us-ascii?Q?kAJTH5urga6ULpHdo0biAxums4D8aH2tCODP6erlNQREgf/Ek2tWPEWqec9a?= =?us-ascii?Q?GCD+O9HJRpnKaI4H1a5JUS+RCd+w0jl4CTWX6T93PbBDyuYQvM/Lnqa5AXtJ?= =?us-ascii?Q?p/Apz4HJbUTUZWaV5T8iti/BSiVLpvSzKsmYFXDMY9wwgvBLFXPINmD+vMNj?= =?us-ascii?Q?spRbDX0g+zC2AfCCiv+mdqgScHfxeFfjC+6rstv2p8H3jI5rmB/w0olM2xiJ?= =?us-ascii?Q?BPeXUfAofCUscyf2MMeiJzb8aD/eji2BUepDOmKXdzvnwt1Yim2SfOnRSg1U?= =?us-ascii?Q?LZkNehSxgm9PFUOY6Vh3hnkhj1DCTHIOvyPIp+tuGXj5dZ7REjfQWRJAYDsp?= =?us-ascii?Q?Ge5xdD4ZrkSG6BrRKKpXP4oyzqse1L6/kVY18v1UlZG+rMjvLsYXLvfDWsX6?= =?us-ascii?Q?nCkLuxMAect8TRsahUqNnMU12JdyLil5mtVLBMty/WzVnFWLpQ6hki/0/DUo?= =?us-ascii?Q?E5hr6LzKPZYe3L4qUPJJJmm1ElEs8SdfrZQ2JvpVr2mEcxn32uPrnZ6Qm3lw?= =?us-ascii?Q?edBSj4osWq/RhgwgYjPJrZp57jRyDKdcMg89viOYCP73cpCXU2hy7LOw3IKI?= =?us-ascii?Q?pAs0vChgVz2XvDxqU6OWsDo+w28wBYaD7HPKIV4HikfAds/CXy2+PlORSxJz?= =?us-ascii?Q?XsH06IQVgckqKkH95XDWBBE/E401TimwMx2IIm5kEH2CfnPkAzwpExhrMYaG?= =?us-ascii?Q?gP2SOp6a2tZsC7xG+8RkoUM17yZbVxs5ygZ9arWQNVchcga1ZnSP4ojqhvBC?= =?us-ascii?Q?O7AXbc92IiNrf9Ql/wS4/FzAcsS5dg7aueXzX4ePtVW7+J4uvVamrGfQI47x?= =?us-ascii?Q?XC9f+7vp8Q33ZvHMNsfHGyRtAzDGZbX0+Zopc1jKLlk1rCnjg5pMdxT2iocX?= =?us-ascii?Q?oForYjMCIiIvStEY4XJWNpjGB5XJDlsu+9igGtPbgu0O7lIvSGNwQ994J0q4?= =?us-ascii?Q?DMoqF4hNEQ=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 57820bb0-ad09-4237-8f82-08debc93964a X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9474.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 May 2026 08:32:00.1854 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: aYwcSL9xT0bsYCrmaAav6bmH433HkRgX6VfsVCDfqqU1KacE9Uq9afYH8ud3SX5+S6PjpTTcr69TEpXIYZV6Uw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7442 Content-Type: text/plain; charset="utf-8" Add helpers to restore endpoint decoder programming for a CXL memdev from CXL core's cached decoder objects, then commit it as a distinct step. Callers are expected to have established reset safety and to hold cxl_rwsem.region for write. cxl_restore_memdev_decoders() restores programmable decoder state while keeping traffic disabled. For HDM-backed endpoints it programs enabled endpoint decoder fields without COMMIT, keeps the HDM Decoder Capability disabled, and mirrors matching endpoint DVSEC ranges where possible. For endpoints without HDM decoder registers, it restores the legacy DVSEC ranges that model endpoint decode. cxl_commit_memdev_decoders() enables the HDM Decoder Capability and commits enabled, unlocked endpoint decoders after safety checks pass. It sets COMMIT only after decoder fields have been restored, does not re-lock decoders, and does not set DVSEC MEM_ENABLE. Signed-off-by: Srirangan Madhavan --- drivers/cxl/core/hdm.c | 318 ++++++++++++++++++++++++++++++++++++++++- drivers/cxl/cxl.h | 2 + 2 files changed, 317 insertions(+), 3 deletions(-) diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c index 0c80b76a5f9b..f7af1041a9fc 100644 --- a/drivers/cxl/core/hdm.c +++ b/drivers/cxl/core/hdm.c @@ -679,7 +679,7 @@ int cxl_dpa_alloc(struct cxl_endpoint_decoder *cxled, u= 64 size) return devm_add_action_or_reset(&port->dev, cxl_dpa_release, cxled); } =20 -static void cxld_set_interleave(struct cxl_decoder *cxld, u32 *ctrl) +static int cxld_set_interleave_fields(struct cxl_decoder *cxld, u32 *ctrl) { u16 eig; u8 eiw; @@ -690,14 +690,22 @@ static void cxld_set_interleave(struct cxl_decoder *c= xld, u32 *ctrl) */ if (WARN_ONCE(ways_to_eiw(cxld->interleave_ways, &eiw), "invalid interleave_ways: %d\n", cxld->interleave_ways)) - return; + return -EINVAL; if (WARN_ONCE(granularity_to_eig(cxld->interleave_granularity, &eig), "invalid interleave_granularity: %d\n", cxld->interleave_granularity)) - return; + return -EINVAL; =20 u32p_replace_bits(ctrl, eig, CXL_HDM_DECODER0_CTRL_IG_MASK); u32p_replace_bits(ctrl, eiw, CXL_HDM_DECODER0_CTRL_IW_MASK); + return 0; +} + +static void cxld_set_interleave(struct cxl_decoder *cxld, u32 *ctrl) +{ + if (cxld_set_interleave_fields(cxld, ctrl)) + return; + *ctrl |=3D CXL_HDM_DECODER0_CTRL_COMMIT; } =20 @@ -927,6 +935,310 @@ static void cxl_decoder_reset(struct cxl_decoder *cxl= d) } } =20 +static int cxl_restore_dvsec_range(struct cxl_memdev *cxlmd, + struct cxl_endpoint_decoder *cxled) +{ + struct cxl_dev_state *cxlds =3D cxlmd->cxlds; + struct cxl_decoder *cxld =3D &cxled->cxld; + struct pci_dev *pdev =3D to_pci_dev(cxlds->dev); + u64 base =3D cxld->hpa_range.start; + u64 size =3D range_len(&cxld->hpa_range); + u32 lo; + int dvsec =3D cxlds->cxl_dvsec; + int id =3D cxld->id; + int rc; + + if (!dvsec) + return 0; + + if (id >=3D CXL_DVSEC_RANGE_MAX) + return 0; + + rc =3D pci_write_config_dword(pdev, dvsec + PCI_DVSEC_CXL_RANGE_BASE_HIGH= (id), + upper_32_bits(base)); + if (rc) + return rc; + + rc =3D pci_read_config_dword(pdev, dvsec + PCI_DVSEC_CXL_RANGE_BASE_LOW(i= d), + &lo); + if (rc) + return rc; + lo &=3D ~PCI_DVSEC_CXL_MEM_BASE_LOW; + lo |=3D lower_32_bits(base) & PCI_DVSEC_CXL_MEM_BASE_LOW; + + rc =3D pci_write_config_dword(pdev, dvsec + PCI_DVSEC_CXL_RANGE_BASE_LOW(= id), + lo); + if (rc) + return rc; + + rc =3D pci_write_config_dword(pdev, dvsec + PCI_DVSEC_CXL_RANGE_SIZE_HIGH= (id), + upper_32_bits(size)); + if (rc) + return rc; + + rc =3D pci_read_config_dword(pdev, dvsec + PCI_DVSEC_CXL_RANGE_SIZE_LOW(i= d), + &lo); + if (rc) + return rc; + + /* + * Preserve MEM_INFO_VALID / MEM_ACTIVE and any reserved bits while + * restoring only the programmable size bits. + */ + lo &=3D ~PCI_DVSEC_CXL_MEM_SIZE_LOW; + lo |=3D lower_32_bits(size) & PCI_DVSEC_CXL_MEM_SIZE_LOW; + + return pci_write_config_dword(pdev, + dvsec + PCI_DVSEC_CXL_RANGE_SIZE_LOW(id), + lo); +} + +static int cxl_restore_hdm_decoder(struct cxl_hdm *cxlhdm, + struct cxl_endpoint_decoder *cxled) +{ + struct cxl_decoder *cxld =3D &cxled->cxld; + void __iomem *hdm; + u64 base, size, skip; + u32 ctrl; + int id; + + id =3D cxld->id; + hdm =3D cxlhdm->regs.hdm_decoder; + ctrl =3D readl(hdm + CXL_HDM_DECODER0_CTRL_OFFSET(id)); + if (ctrl & CXL_HDM_DECODER0_CTRL_LOCK) + return 0; + + base =3D cxld->hpa_range.start; + size =3D range_len(&cxld->hpa_range); + skip =3D cxled->skip; + + ctrl &=3D ~(CXL_HDM_DECODER0_CTRL_LOCK | + CXL_HDM_DECODER0_CTRL_COMMIT | + CXL_HDM_DECODER0_CTRL_COMMITTED | + CXL_HDM_DECODER0_CTRL_COMMIT_ERROR); + if (cxld_set_interleave_fields(cxld, &ctrl)) + return -EINVAL; + cxld_set_type(cxld, &ctrl); + + /* Preserve setup_hw_decoder() programming order, without COMMIT. */ + writel(upper_32_bits(base), hdm + CXL_HDM_DECODER0_BASE_HIGH_OFFSET(id)); + writel(lower_32_bits(base), hdm + CXL_HDM_DECODER0_BASE_LOW_OFFSET(id)); + writel(upper_32_bits(size), hdm + CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(id)); + writel(lower_32_bits(size), hdm + CXL_HDM_DECODER0_SIZE_LOW_OFFSET(id)); + writel(upper_32_bits(skip), hdm + CXL_HDM_DECODER0_SKIP_HIGH(id)); + writel(lower_32_bits(skip), hdm + CXL_HDM_DECODER0_SKIP_LOW(id)); + wmb(); + writel(ctrl, hdm + CXL_HDM_DECODER0_CTRL_OFFSET(id)); + + return 0; +} + +struct cxl_restore_ctx { + struct cxl_memdev *cxlmd; + struct cxl_hdm *cxlhdm; +}; + +static int cxl_restore_decoder(struct device *dev, void *data) +{ + struct cxl_restore_ctx *ctx =3D data; + struct cxl_endpoint_decoder *cxled; + struct cxl_decoder *cxld; + int rc; + + if (!is_endpoint_decoder(dev)) + return 0; + + cxled =3D to_cxl_endpoint_decoder(dev); + cxld =3D &cxled->cxld; + if ((cxld->flags & CXL_DECODER_F_ENABLE) =3D=3D 0) + return 0; + + if (ctx->cxlhdm->regs.hdm_decoder) { + if (cxld->id >=3D ctx->cxlhdm->decoder_count) + return -EINVAL; + + rc =3D cxl_restore_hdm_decoder(ctx->cxlhdm, cxled); + if (rc) + return rc; + } + + return cxl_restore_dvsec_range(ctx->cxlmd, cxled); +} + +static int cxl_restore_decoders(struct cxl_memdev *cxlmd, struct cxl_hdm *= cxlhdm) +{ + struct cxl_port *port =3D cxlhdm->port; + void __iomem *hdm =3D cxlhdm->regs.hdm_decoder; + struct cxl_restore_ctx ctx =3D { + .cxlmd =3D cxlmd, + .cxlhdm =3D cxlhdm, + }; + u32 global_ctrl; + + if (hdm) { + global_ctrl =3D readl(hdm + CXL_HDM_DECODER_CTRL_OFFSET); + writel(global_ctrl & ~CXL_HDM_DECODER_ENABLE, + hdm + CXL_HDM_DECODER_CTRL_OFFSET); + } + + return device_for_each_child(&port->dev, &ctx, cxl_restore_decoder); +} + +/** + * cxl_restore_memdev_decoders - Restore endpoint decoder programming + * @cxlmd: CXL memdev whose endpoint decoders need to be restored + * + * Restore only programmable decoder state from CXL core's cached decoder + * objects. For endpoints with HDM decoder registers, program the HDM deco= der + * fields and mirror decoder ids representable by CXL_DVSEC_RANGE_MAX into= the + * DVSEC range registers when present. For endpoints without HDM decoder + * registers, restore DVSEC range registers only. + * + * This helper leaves CXL.mem disabled: it does not commit HDM decoders, e= nable + * the HDM Decoder Capability, set PCI_DVSEC_CXL_MEM_ENABLE, or restore + * unrelated DVSEC CTRL, CTRL2, LOCK, MEM_ENABLE, or other control state. + * Callers must perform final commit/resume steps only after reset safety = checks + * pass. + * + * Return: 0 on success, negative errno on failure. + */ +int cxl_restore_memdev_decoders(struct cxl_memdev *cxlmd) +{ + struct cxl_port *endpoint =3D cxlmd->endpoint; + struct cxl_hdm *cxlhdm; + int rc; + + lockdep_assert_held_write(&cxl_rwsem.region); + + if (!endpoint) + return -ENODEV; + + cxlhdm =3D dev_get_drvdata(&endpoint->dev); + if (!cxlhdm) + return -ENODEV; + + scoped_guard(rwsem_read, &cxl_rwsem.dpa) + rc =3D cxl_restore_decoders(cxlmd, cxlhdm); + return rc; +} + +static int cxl_commit_restored_hdm_decoder(struct cxl_hdm *cxlhdm, + struct cxl_endpoint_decoder *cxled) +{ + struct cxl_decoder *cxld =3D &cxled->cxld; + void __iomem *hdm =3D cxlhdm->regs.hdm_decoder; + u32 ctrl; + int id; + + if ((cxld->flags & CXL_DECODER_F_ENABLE) =3D=3D 0) + return 0; + + if (!hdm) + return 0; + + id =3D cxld->id; + if (id >=3D cxlhdm->decoder_count) + return -EINVAL; + + /* + * cxl_restore_hdm_decoder() programmed the decoder fields first. This + * control register write sets COMMIT as the final programming step. + */ + ctrl =3D readl(hdm + CXL_HDM_DECODER0_CTRL_OFFSET(id)); + if (ctrl & CXL_HDM_DECODER0_CTRL_LOCK) + return 0; + + if (ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED) + return 0; + + ctrl |=3D CXL_HDM_DECODER0_CTRL_COMMIT; + writel(ctrl, hdm + CXL_HDM_DECODER0_CTRL_OFFSET(id)); + + return cxld_await_commit(hdm, id); +} + +struct cxl_commit_decoder_ctx { + struct cxl_hdm *cxlhdm; + int id; +}; + +static int cxl_commit_restored_decoder_by_id(struct device *dev, void *dat= a) +{ + struct cxl_commit_decoder_ctx *ctx =3D data; + struct cxl_endpoint_decoder *cxled; + int rc; + + if (!is_endpoint_decoder(dev)) + return 0; + + cxled =3D to_cxl_endpoint_decoder(dev); + if (cxled->cxld.id !=3D ctx->id) + return 0; + + rc =3D cxl_commit_restored_hdm_decoder(ctx->cxlhdm, cxled); + return rc ?: 1; +} + +/** + * cxl_commit_memdev_decoders - Commit restored endpoint decoder programmi= ng + * @cxlmd: CXL memdev whose endpoint decoders need to be committed + * + * Resume endpoint decoding after cxl_restore_memdev_decoders() has restor= ed + * programmable decoder fields. For endpoints with HDM decoder registers, = enable + * the HDM Decoder Capability and commit enabled, unlocked endpoint decode= rs. + * Locked decoders are left to their current hardware/firmware-owned state. + * + * This helper does not set PCI_DVSEC_CXL_MEM_ENABLE. Callers must enable + * CXL.mem only after all reset safety checks and decoder restore/commit s= teps + * have completed. + * + * Return: 0 on success, negative errno on failure. + */ +int cxl_commit_memdev_decoders(struct cxl_memdev *cxlmd) +{ + struct cxl_port *endpoint =3D cxlmd->endpoint; + struct cxl_hdm *cxlhdm; + void __iomem *hdm; + u32 global_ctrl; + int i, rc; + + lockdep_assert_held_write(&cxl_rwsem.region); + + if (!endpoint) + return -ENODEV; + + cxlhdm =3D dev_get_drvdata(&endpoint->dev); + if (!cxlhdm) + return -ENODEV; + + hdm =3D cxlhdm->regs.hdm_decoder; + if (!hdm) + return 0; + + global_ctrl =3D readl(hdm + CXL_HDM_DECODER_CTRL_OFFSET); + writel(global_ctrl | CXL_HDM_DECODER_ENABLE, + hdm + CXL_HDM_DECODER_CTRL_OFFSET); + + for (i =3D 0; i < cxlhdm->decoder_count; i++) { + struct cxl_commit_decoder_ctx ctx =3D { + .cxlhdm =3D cxlhdm, + .id =3D i, + }; + + /* + * Per CXL Spec 3.1 8.2.4.20.12 software must commit decoders + * in HPA order. Region setup already enforces that ordering by + * decoder id, so restore commits follow ascending id order. + */ + rc =3D device_for_each_child(&endpoint->dev, &ctx, + cxl_commit_restored_decoder_by_id); + if (rc < 0) + return rc; + } + + return 0; +} + static int cxl_setup_hdm_decoder_from_dvsec( struct cxl_port *port, struct cxl_decoder *cxld, u64 *dpa_base, int which, struct cxl_endpoint_dvsec_info *info) diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 1297594beaec..b51b1e9d6400 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -794,6 +794,8 @@ int cxl_port_setup_regs(struct cxl_port *port, struct cxl_dev_state; int cxl_dvsec_rr_decode(struct cxl_dev_state *cxlds, struct cxl_endpoint_dvsec_info *info); +int cxl_restore_memdev_decoders(struct cxl_memdev *cxlmd); +int cxl_commit_memdev_decoders(struct cxl_memdev *cxlmd); =20 bool is_cxl_region(struct device *dev); =20 --=20 2.43.0 From nobody Mon Jun 8 15:36:52 2026 Received: from CH5PR02CU005.outbound.protection.outlook.com (mail-northcentralusazon11012019.outbound.protection.outlook.com [40.107.200.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C55A379C24; Thu, 28 May 2026 08:32:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.200.19 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957133; cv=fail; b=HJMrZZFG3yzNSigIm21sf/9D7Ax6MWyoXiZQN/E5Vnyve/t3svtQNe+sNIV3sqDRiA7cJX3UKCrEJnlJ2lrdbPm+mOFx0hm37Q9d1pPeACWir5MyBlEVyVDB056yyc3RQ9Z/6IG64M7xtbHcxca7HIHGJ1FN0wzj4oKU+sUaXks= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957133; c=relaxed/simple; bh=2wFcGjWvXKhN5jE/VBC7vR4xwJZp2lXbU/TL7WEVGh8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=T8G8v+cMgrCNVy2mn+Ie5QeFUGZXvvDHlXdrHcOr7Mm/0BPNuPw92Pi6af5HPR0g0wPkRlo5ssSj+ZeI7KfxmE3+8NEYCNM+ich55lLbYzpwVUrW+WTYWbqKVEY0jV9PgPozBDZ1/bpe+UappOhZa2y/5KI1IgL+i4cIVk4Crl8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=OnCLJN6S; arc=fail smtp.client-ip=40.107.200.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="OnCLJN6S" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ZS2FBlta+OCKb4nHKpxm7ZDVGJV87V2STAFK3T/+FBRvSwxH85D63cyxm0ZRrwWkLyr6E5D0JUQ/kFWYD/rTXLDkfWkKtVh9Qda5DPoU7N3SlEN4xx2Qo9abmOwQA9QryULRN58ws5W+NCZfwTaVRfu12dtJVsyl865coGHyx8N2TWDbprTPJYDhSLzCTR2QPt+j6A43EMrXI22MAXb9XAtmvoCEi6BZXXvACqZOdXGoBTIpsG/5jgDd8JXZ4K+d2hf9/2RRVnkNFO8FvN0yHOuHRvEu4UnlX5ElmTokIXHP0cu23gUASuvieDp69gAg+5x1Td1LA4evMglNFoTb5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=YPz81Na0atsrGNpENOLxedbFLl89Y++MtyYvn1ama4Q=; b=Qjpz/Ny1sD+KXdzrSFu3AAqe/jiGbnT6hMtgrEdttmAhXWtEQ+33szzouHC2GfHNvSGV8eRXIbfp2iMLpspdue8328tIl1rDPwypjP4eRX7o7tMX7LK6dbsKP+4uDOlhPM81asPo1qGzRrKpWWD6EdILOzc9P9qwRGs70AcdDH2bHuAOQWITL3rnhc+QbDnDgHg1RLhq2E6R4hGoJIqIphakVGAsJB8ZNkPJCsKHYfuzSrWyPoP0VOhEJeQd0eVysYb1++EJiVMGn9Mj3FNLFxST0xp0nZrvCpmgoNOJPLt9zAF8E4Z5I1BqJWh/Eh5fN0cXwaHMjLF/eTQjIp01Iw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YPz81Na0atsrGNpENOLxedbFLl89Y++MtyYvn1ama4Q=; b=OnCLJN6Ssas19Njj1G+toJ/fxH7UdFmUu5NNv5Vy506bqc8v1e/iWVCj+1bNBf1t/OyG+txjFdxZMnZT3BFY2nHfFddJq/ltukdR8jOMy8OIq9v5P9365coVU3lHyFP1WMGQOqsiCmKhZ2WWT4D669tGm1lDivy8QdvVTpL8uNw7Uz6i3To6hAShv2EpnPHcUll6snDWo0QDd+kvcPqdbY8ubT2bDmE4WfXzuZxeOEFCYR1yxruznTM6z8flBRhZT8gOJghIklN7MyNUTE8KkGMRVCD62MpGi7FJ+ocuHLWT/fiDxBAFc4YeIFkofzq2SiRKH7GfbKNjQMtzxpeWGg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) by SA1PR12MB7442.namprd12.prod.outlook.com (2603:10b6:806:2b5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.12; Thu, 28 May 2026 08:32:02 +0000 Received: from DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7]) by DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7%6]) with mapi id 15.21.0071.011; Thu, 28 May 2026 08:32:02 +0000 From: Srirangan Madhavan To: linux-cxl@vger.kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Cc: vsethi@nvidia.com, alwilliamson@nvidia.com, Dan Williams , Sai Yashwanth Reddy Kancherla , Vishal Aslot , Manish Honap , Jiandi An , Richard Cheng , linux-tegra@vger.kernel.org, Srirangan Madhavan Subject: [PATCH v6 2/9] PCI: Export pci_dev_save_and_disable() and pci_dev_restore() Date: Thu, 28 May 2026 08:31:47 +0000 Message-ID: <20260528083154.137979-3-smadhavan@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260528083154.137979-1-smadhavan@nvidia.com> References: <20260528083154.137979-1-smadhavan@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: BY1P220CA0014.NAMP220.PROD.OUTLOOK.COM (2603:10b6:a03:59d::9) To DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9474:EE_|SA1PR12MB7442:EE_ X-MS-Office365-Filtering-Correlation-Id: 3fbf4212-6206-4c91-ba34-08debc9397d4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|366016|56012099006|11063799006|5023799004|18002099003|22082099003|3023799007|6133799003; X-Microsoft-Antispam-Message-Info: rxKH4TfamZuoqF0m/YVRPim9zf9NQx+xtQA8ZMnXn8984cdQlq2UXZmy6WzgCsRJMPe2g79302fObHRMVOJD2M/8AAvUtdqOSg4tQcqn28UjMDQlWx1VLMW+scXV8JpYMo64gHO65v+IN77McJyEVobJTq3gv/oOTf/u3ZY0+u6CKlOWKbidtfrc2XKLF0aU4HjerP4VQP9j7CaXGum7Uczg5czlXHHnXfM+BQV210Nw0Ndex+QO0SrPKu3iOs5ndIwCRp2PD23GVj13xukR4kV6E6qZY1bA0UVXPRbnJnn5a7GoWrvTyc6xBaaX5sJZSjKGKxDjj+i0FOXbGjsG0siv9ncH2S3TOj8os/ZWxFbiecZdlGh8SlKtZ2AtWXfEB74TXVy3sFEOjYh5p8SQsTo1yNo02I0RSjVohlmgrsZxikMdWuadAODNZ1R7uVddkpyzJquJZDpUoDZZ8jKBCOH/nNtDEfQiKkGjiYPicdHuUEAqHkNvVuN9UfGs082g9dn2XPmLr4NnKeSmHi8RprLYG/zG5ceftgXqGtkwWYNAIFe/5XcPE0bPKv8Y648buTfETmPkWlsdHoopMv8ZPzbRsMlZ5pw+fQfQy9FkOBwm3vKiSUJGih35YRqhAFbnkt4A4QaAPiNMWuPwoJgzJZrWE/mIj06WnSK/EXS1q10rHEX1sHs+Lt38MkrQxOSK X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9474.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(366016)(56012099006)(11063799006)(5023799004)(18002099003)(22082099003)(3023799007)(6133799003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?l2icjN2Yat9E0WaLLOdrO8YmEIUZpgs1FggCyIZPWW/715J+Vk6P0ky+aTuw?= =?us-ascii?Q?WvFt6bkBgZPJYJ1USgkriyEFz5F28y0X8Hoywxae9pr6MV0nz470efHY5CDs?= =?us-ascii?Q?Zp+3TFGcdx7msGJUGIjVAAZD7bK1qqrxnT/NXlOIr2fGAfNKgZmwSparJXOb?= =?us-ascii?Q?TSlCppXEx/57EYbG5apl67QRENT9eMHEe79AilzpABTT9sdEcLItHMjZ/+KX?= =?us-ascii?Q?F+zc+n+Hs7FPM1p33vhk8FKrQmHcd3j+xd+fibECUVcr+Ur3J7mwwttIlrHu?= =?us-ascii?Q?sePn+v0BocdA8O1IfBoOrtjouHsRaPi2TGza5TFdAWvm+Jj1SMIeepRRu0bH?= =?us-ascii?Q?yZFDwy/V8wovogAv9t2jn/aJ3lBTskdV2N6st09cNF1I5Dfsty0BcQP58cXj?= =?us-ascii?Q?HrpVuxiRaFxh1WMR+2qDt4tmyf2bh66/id6CLG/Te5CVsRZ8xM7OvQhCKiOC?= =?us-ascii?Q?GIonA+Snu1jRWAMcX4aS8F9dc34t1xPZtYBFzIpZqKazoXHuBrEjO8aW5WUG?= =?us-ascii?Q?vxt+UXQpFA+qJieVCmWLnCeBH5t4WTSJb2s/HMZDOhO1N6hsjJh3JEUrLwXA?= =?us-ascii?Q?pRmfbYoULwUsrnFCcuCKqLpKuiOCNJLwpDXwqa/GxPqmxPQnkrZCL/aF6n5d?= =?us-ascii?Q?o+aDDTkPbvqO46OWM34S31byUbmhGgKoSG09UV9urx9Bsr7b9+8eNFBn5czn?= =?us-ascii?Q?MOQnLrdnSXyS8TJfykx4JupFTqtJWCjjdFTvm7J0UYx/Ygxe182VKsCZjkBP?= =?us-ascii?Q?LbYXG9OD/SWOM94QhVYgJR94DDXaVeBFwFgg0XeE82y6+Dy9Tnyh8Qxy168M?= =?us-ascii?Q?UcWMB5P3OLqpZxLAHvi5J/r7cszfxu0dWr52CEsHpQ5l7wA1VgQAAH10wc8F?= =?us-ascii?Q?P2lDMDpcIhDC8ZrAidc4IsRNlD4Iw6xpES3AYNHPCoxZruPxSxGqa8zTlFAJ?= =?us-ascii?Q?ZbBC1NI2uCW61f+olHjrODwQWrwRT+xiY3kiGsNmbu5GNtzMgSSyWbBGIbkB?= =?us-ascii?Q?8bfRwWmbaaR8s9cRrwJbz1Qi0HTP+xyBOnXO6CFuREJdPwL7J4qZQ7+NbiTH?= =?us-ascii?Q?vvk8202PzMq/IqmQORwoLXCyuKsTgrmURXO19OqdhEJnhWL6zl2DQ/39R+5L?= =?us-ascii?Q?5FXGEear7pcUop48ubSc4vP5Bsug9JdvGb7+iJBQkDEfP4L6P5ALj5b1vehv?= =?us-ascii?Q?njuqm2rYfgdmZOkSm/RLmNeaiYZEvmXqGUQDtmVVkS8RoqgDedALI2ntVW8v?= =?us-ascii?Q?8Uop+99ikPJhjOSMZ+rkQ6lBicf32tcwPtiYIZ9pINrF1w2yISNUr3HzOL2/?= =?us-ascii?Q?zZPLkE/tceoO2EonRRk46Jo8clugTqe4brPU1JoDL6S2iR2KESmmoQSMQKyN?= =?us-ascii?Q?rwXsHI+Hynm7YPYpD+l0W8zzZcZB4a23QJwoljUi9sWHl72La3ogInWQvxGA?= =?us-ascii?Q?WaGfo4EaBuQ2MEACraBTZg/y5zpFzL34nz5YRFyQqPutfy2ZHhV2+Hj77DBi?= =?us-ascii?Q?2DCPs7CnYGHohvYB7Gazs5WxBTnlL6YfUpgjbPHvPODlQCs+RsRORpxtAd5f?= =?us-ascii?Q?inlKHsav9QQgjetstVDruDB5rUHs58WN6AZhba5ZHufyJTUe7EmysXxk+00M?= =?us-ascii?Q?79JDQFGjZcisNZWuSJi73DaBn8PYPchqLAutbFe5tVdOLJecMt6UjWoeXyMP?= =?us-ascii?Q?D+q+xTk2O4mKPu8PdZD2QE7Udlg85c3Roajz/mTP1tic4LL7aMIojzd6KLDa?= =?us-ascii?Q?7CZD/jDwCg=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3fbf4212-6206-4c91-ba34-08debc9397d4 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9474.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 May 2026 08:32:02.7761 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7uLghgORbQr+XSQvOjwBIcr9S73youhff6H5/J7lkWMpOQjbBlz7nafefquTICYRfMSHIZPdc9AqaeCcRrbGUQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7442 Content-Type: text/plain; charset="utf-8" Export pci_dev_save_and_disable() and pci_dev_restore() so CXL reset orchestration can reuse the PCI core reset lifecycle for non-standard reset flows. These helpers invoke driver reset_prepare/reset_done callbacks, save and restore PCI config state, and disable the device while the caller holds the device lock. Signed-off-by: Srirangan Madhavan --- drivers/pci/pci.c | 22 ++++++++++++++++++++-- include/linux/pci.h | 2 ++ 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index d34266651ad0..75d2f4074750 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -5003,7 +5003,15 @@ void pci_dev_unlock(struct pci_dev *dev) } EXPORT_SYMBOL_GPL(pci_dev_unlock); =20 -static void pci_dev_save_and_disable(struct pci_dev *dev) +/** + * pci_dev_save_and_disable - Save device state and disable it + * @dev: PCI device to save and disable + * + * Save the PCI configuration state, invoke the driver's reset_prepare() + * callback if present, and disable the device by clearing the Command + * register. The device lock must be held by the caller. + */ +void pci_dev_save_and_disable(struct pci_dev *dev) { const struct pci_error_handlers *err_handler =3D dev->driver ? dev->driver->err_handler : NULL; @@ -5036,8 +5044,17 @@ static void pci_dev_save_and_disable(struct pci_dev = *dev) */ pci_write_config_word(dev, PCI_COMMAND, PCI_COMMAND_INTX_DISABLE); } +EXPORT_SYMBOL_GPL(pci_dev_save_and_disable); =20 -static void pci_dev_restore(struct pci_dev *dev) +/** + * pci_dev_restore - Restore device state after reset + * @dev: PCI device to restore + * + * Restore the saved PCI configuration state and invoke the driver's + * reset_done() callback if present. The device lock must be held by the + * caller. + */ +void pci_dev_restore(struct pci_dev *dev) { const struct pci_error_handlers *err_handler =3D dev->driver ? dev->driver->err_handler : NULL; @@ -5054,6 +5071,7 @@ static void pci_dev_restore(struct pci_dev *dev) else if (dev->driver) pci_warn(dev, "reset done"); } +EXPORT_SYMBOL_GPL(pci_dev_restore); =20 /* dev->reset_methods[] is a 0-terminated list of indices into this array = */ const struct pci_reset_fn_method pci_reset_fn_methods[] =3D { diff --git a/include/linux/pci.h b/include/linux/pci.h index 2c4454583c11..d6303e16e11b 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -2012,6 +2012,8 @@ void pci_dev_lock(struct pci_dev *dev); int pci_dev_trylock(struct pci_dev *dev); void pci_dev_unlock(struct pci_dev *dev); DEFINE_GUARD(pci_dev, struct pci_dev *, pci_dev_lock(_T), pci_dev_unlock(_= T)) +void pci_dev_save_and_disable(struct pci_dev *dev); +void pci_dev_restore(struct pci_dev *dev); =20 /* * PCI domain support. Sometimes called PCI segment (eg by ACPI), --=20 2.43.0 From nobody Mon Jun 8 15:36:52 2026 Received: from BN8PR05CU002.outbound.protection.outlook.com (mail-eastus2azon11011017.outbound.protection.outlook.com [52.101.57.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CEA33344D8D; Thu, 28 May 2026 08:32:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.57.17 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957130; cv=fail; b=swjuPwq/gG2uqcsg6och/RoMOt3sKlVlw/C8iR4+/t3X6Bjluh6vkpX921dzETQsKsBZpibGyS8Jcty+G+UJ3B5mKGMmASz31oX2aPM7jecftsVDgzI+9FuEADghnBGBg75eKZHjpaDrxrcOKZFXbXZd3alE/ctEwI2+Kg6W2KI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957130; c=relaxed/simple; bh=cGD3tgsbuR8DV201a8gUp2xQBxtxOuLL345mmL9MbOo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=mwgw6wLCSTI5TIk8Qw4vGs6jMeXA+SNNGFacuGZc3lOHM13WCW/8eGhIiq9gsl0B2bdf5DKqiWLvOoqcF9Zmme1s0fIEilUgeE2+8EKLrdrrk6gDqbBCh0qtnj9PstKi1KbtEYJTS8aC2a/wvdJ6gMjVFmSAdVdClpmgbyzVBgo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=VB86UkmY; arc=fail smtp.client-ip=52.101.57.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="VB86UkmY" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MQaixhF0+pyYmpNg+fy56XHew9pcMzs9F1xIvsZeVDkGahdhXRod5TBUkXUjZh82QYtw+LwYsGs+Yb1YcFToqJrR0sEPQ3oY2PVPcQEhbYZ1XG2HLO/VGcdyJNP9YrYxcgss+ukocmwgxk/Hv+yVwjmxxbv/ohpB1ib0x12bxL1jMaAbo1PUWr1HVqhzifepVdHYi2uMEdKBNb+GLTKuhOqYt9Vwc+v0uP447FolpJmYOy9RoBDH28Mhi3I0xWAcGKPM+L6FVI1PkouIg03Lf2x0bpauuEdMf06eVZqA2KbmyR56nPPju46FzU88GddthLc4iDeQswSdYq484VkVmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9pFmH6gxQ9uKM1KW1MSzqz2UOl3jrEPx2Kn7cxmZEDo=; b=W3XhFWFAALPQ5LA66BxjWOqCo7zAAOCXg9MShNepzaxSxBwOMqY9Vv7FYXTUnnD2fu/OCzMWWkhztVbWqCBYO+mKjPeNFdETO9O3d3qwMNmEb75o+tkQFZJOXO3K0P0fBP1lM2RjOmIbMrrbQ967YfNQ04ttBmbwKUZKM5P7LMG3NVmWlbKGqOzBFH0E1s+1/lvEEr66sR2v2OGW7nC6wwLMALRzGcNao2SzH5jpDURWWw4kK330P2wwAFzWPciadYZ1PEULBU5dQufbYT+WqPhvDkN7ynjDnpxljoaPRp7iMMLIKlPW0ehlCA12zRNlAbZOJ/PDcas2D6wkoGQMfQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9pFmH6gxQ9uKM1KW1MSzqz2UOl3jrEPx2Kn7cxmZEDo=; b=VB86UkmYTVeI5cMjczC86VP7fQSQXsaCqypc+qEXJM+/ZLnzScPG1u27B3adGQALWPB574rTqGsovk3dFvGNaUZDRnjCowz/rLUHfMrq7+j3as7zepn+jcH5SfYQhvP3yr91Y0b0m3KEHFrTaKquJptouJYa1dNMKrvCma6F+x/EOA00Vf29OBGdvg/oJ7hH1z1Z/xuhR+Z4cwte+f47sCyxTOFvKLCdYABf2h9JA97KVA3Nz9NDHgkABewOpy1eoC1WjG5UDIOCNCb6vekiH7MTXuNzE7GfQNaclzAYh5VIN3aoove7GMSb076gnrRN/5Kdb715DXuFEpVP7nuV5Q== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) by CH3PR12MB7617.namprd12.prod.outlook.com (2603:10b6:610:140::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.14; Thu, 28 May 2026 08:32:05 +0000 Received: from DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7]) by DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7%6]) with mapi id 15.21.0071.011; Thu, 28 May 2026 08:32:05 +0000 From: Srirangan Madhavan To: linux-cxl@vger.kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Cc: vsethi@nvidia.com, alwilliamson@nvidia.com, Dan Williams , Sai Yashwanth Reddy Kancherla , Vishal Aslot , Manish Honap , Jiandi An , Richard Cheng , linux-tegra@vger.kernel.org, Srirangan Madhavan Subject: [PATCH v6 3/9] cxl: Add reset-idle and cache flush helpers Date: Thu, 28 May 2026 08:31:48 +0000 Message-ID: <20260528083154.137979-4-smadhavan@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260528083154.137979-1-smadhavan@nvidia.com> References: <20260528083154.137979-1-smadhavan@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: BYAPR21CA0004.namprd21.prod.outlook.com (2603:10b6:a03:114::14) To DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9474:EE_|CH3PR12MB7617:EE_ X-MS-Office365-Filtering-Correlation-Id: 081644ac-2707-4692-205f-08debc939956 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|18002099003|22082099003|3023799007|56012099006|11063799006; X-Microsoft-Antispam-Message-Info: 9b5RiWokK8UiVOJVRzd8uC/OiLqk8nW3s/eckqYztW1ZXftPWbpcdlKYa1rIRt8UsoZ6IIpE9cIuLO1JdQ317oklb4Xx3MXJU/uibf95dFR8XiusqQ/qPi1LwM7oFHYhcDyyocBIJ20SJ3EqNigFSI0xHMN+6qJBgpgJavY6X1LmujucS+5E3Vu9YeHqiscErLhtz1QbMlT9eHFqG18EQdrMlmEy+FvBzBpEVpoXkI414icobgM6d7GBwMXCN6pakaeZoMdzEQkG0NG5NAPR6jh1wryf0dG9Rvpt3Ncv7bGmDjSn3YidsRb3TzRj3KHerBeY0WW7BGgaCr+s22rgHwtXbLraCq7nKOBe4RbSwbXM+M5kYZ8z6cx7X1v82ed0cV7p+k503XpoQTnF8skRIyYBtp0gIubPsEUmdmNSOikHgAiUZNjFcNgkEyE617uB1vZD17Gy9GSxgfD6bVyQJoA0YxfeaAMjtP9z3NPm9MLzdVeD9bG8TQo16TN4d61UQ66toOVVL8SlBeqZuyIkR8lrUDCAXnuD9a9nE1DNjJMu+l9ZNGWSJISzVoFyLT+Sxe6lHNbkWpT7fomyeHftLqGzHtweXAhtnwBCU1rI/P9qTyrkKgU7i6NSUfsvlvsPbZF/HN1iuut4b2gWFP0wuQYPjupZI0Abv0pGHN4wd71B6YiDUulwNnx2sWUDTHkx X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9474.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(18002099003)(22082099003)(3023799007)(56012099006)(11063799006);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?8WAL+ap8hpgFA+BWq0KXNcB68jBuo4Mjo56B8u/ZvEKhx6XbuCeK78ZQXBLZ?= =?us-ascii?Q?vJy+BoggyA6UmRDvxrNJQM56lNOQtemipg/W2GQou7deWSgmUWiOjg+teHJN?= =?us-ascii?Q?1ka/bPJfTMFFs2vawmeASf6rayAqxTAexL9gFszTboKyEYuT0IdZeacwSPwO?= =?us-ascii?Q?hZsTTMFlq5CFG/W16NmHADYL2BofA/TwTs9wdFnF6D6wj9bzbdwwSKkeccPu?= =?us-ascii?Q?mX26Gf5Us8vPEDdoFCGsxtIgpi5Vm5bZ8s48QQHVOH9Zec56qEeKMJR0/MNE?= =?us-ascii?Q?aaR9wUUZ3wy5F3aYZCKdcmcQmXaXfdF+9mGWCPGSg/81cknR1JTA6j03EbK3?= =?us-ascii?Q?Mhh8Iw9ZhbLgnFeOmfS7lVv1O0ygJ7jMfjk0UTsqzcTgEj76v7cTJtp5zZAx?= =?us-ascii?Q?y4q1+Wyatop+/JBKzy3P359e8xRSJ+9NDjNCddz00dFzdEAPBlTYMuBDSjXF?= =?us-ascii?Q?1v0Fbx6tiEUN/OkejAyL/hJBtbkw7SIsOVKmdxis47+ujtMabHwpumALmtIN?= =?us-ascii?Q?38dXBm45FiK2G20L+rDMSmzalmI86Fezcol3dsZabLNWCBk8m+z+oo6tWaoC?= =?us-ascii?Q?VLfj+fa0DnzZ5B2LOyGy1NUhy1iXSJr2SkT/pnx5x0bQo6RLqNFvvJXfwgMC?= =?us-ascii?Q?OHpSpfoadGvN1oUaV0BBE0kKjlKb8POB4J5o4KigxReWrpTqpp+CTI9N39Jj?= =?us-ascii?Q?WH3kEgSLTnd7Btce7z2ymHlbJAHcHa+vu1AL7kxqRSweHfZCrVZ642Yf/iqk?= =?us-ascii?Q?+NwITx8hoqvzZv40HQUsE3cHTuKRNgdISE/xEjW4p4lpVpRpYmeBuqU0n6ap?= =?us-ascii?Q?PTiZTXbibSw1tBPs9Yvex/dmOvnSWXdujI82UzEiclB82XQc/5C4/sFc+5BO?= =?us-ascii?Q?0P9D2yGbaEC8fMm8ONAs9kgzc7n/4CjYrkXek+pb4cJQscCBhLXzGwcyaDU0?= =?us-ascii?Q?l4YVznifwYRziZI8rXgXOLJcPmWEeQrxTASzIOw3z9VT0yMkI5wdpNbyuzDU?= =?us-ascii?Q?uao2JjLlh+hDdq6jvLO0bTfDFR4r3o3jcAvTeMvr6ceANUl5qqYfafmmjcV9?= =?us-ascii?Q?NBc82AmfPp6G3Aw4CRULjSQiKXAUcGNooznbNAnkkQOcj/0LS6fq+0U2jYyN?= =?us-ascii?Q?qQWMwiYHpCZFcJJF6nybeWkNg3IgKuBujNOYZ4GVum5p9oepuGoUGGWM+S8e?= =?us-ascii?Q?FCydVQCQjCfXjZZxRESEedOKhxLOaphhnagQk9MYjWbYGmh+hLe+E1jzWhkm?= =?us-ascii?Q?bVPYon2grp4g5W+LUVaPhSjLGBXv41Ss7D0uQW7eMKm7m/hHI+TdAYKIgjTZ?= =?us-ascii?Q?ZDQvl1WQSEhRxYTEwdx94MUiK6I5Goji17YPm3rW1d0ZgTd0rG6pGU7zV7/I?= =?us-ascii?Q?26eRYwogE7JvRz5ZqjhBAHPi1+F4c9Yo9JZKJOT2ZsZRhm8P8X7cZJcRVvXf?= =?us-ascii?Q?7c+PCp/UvXEtpflldskPu9C56H1lUh128KXEEum8ISog1WRhrC59iSu6227G?= =?us-ascii?Q?CBqxJkhkygTUnOjRe4IUQGRotSg4jpH08osJL8kPyX8r5qPit3FNi66XAyi4?= =?us-ascii?Q?+VfRfvbnqiH3ETIp6fXuPlsorYR4GvIfJ+3ROUvtUdPh2wljuSiF04YMuvwL?= =?us-ascii?Q?ngpIRk9lJjAo5ejwRosCD07wkdXG/FoYrwNxgH8XmuxVNrMyzsCymxjX+ovQ?= =?us-ascii?Q?GLzzkt+d4O5ABTCQgfsstK5WCSefk0BIHHBNbKgNg7MukDBV2hCiDR7J3R/q?= =?us-ascii?Q?PeBmSc/2Ig=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 081644ac-2707-4692-205f-08debc939956 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9474.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 May 2026 08:32:05.2900 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Aa+Dp2ZnJT1WXaYdqgLBbmBFo8BJEdWCnxoNQ13Qdak5YAAsGRHS0rXM1GCFylLs3yO6Sr4G1WmX3v3r5MsTYA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7617 Content-Type: text/plain; charset="utf-8" Add helpers to collect the CXL regions affected by a memdev reset, verify that those regions are idle, and invalidate CPU caches for the affected address ranges before reset. A memdev can participate in an interleaved region through multiple endpoint decoders. Track affected regions in a temporary xarray so each region is checked and cache-invalidated once per reset operation. These helpers prepare the CXL.mem data path for reset. The actual reset orchestration and decoder restore flow are added separately. Signed-off-by: Srirangan Madhavan --- drivers/cxl/core/pci.c | 170 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 170 insertions(+) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index d1f487b3d809..318744695f62 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -4,9 +4,11 @@ #include #include #include +#include #include #include #include +#include #include #include #include @@ -926,3 +928,171 @@ int cxl_port_get_possible_dports(struct cxl_port *por= t) =20 return ctx.count; } + +static int cxl_reset_system_ram_found(struct resource *res, void *data) +{ + return 1; +} + +struct cxl_reset_region_context { + struct xarray regions; +}; + +static void __maybe_unused +cxl_reset_region_context_init(struct cxl_reset_region_context *ctx) +{ + xa_init(&ctx->regions); +} + +static void __maybe_unused +cxl_reset_region_context_destroy(struct cxl_reset_region_context *ctx) +{ + xa_destroy(&ctx->regions); +} + +static int cxl_reset_add_region(struct cxl_reset_region_context *ctx, + struct cxl_region *cxlr) +{ + int rc; + + if (!cxlr || !cxlr->params.res) + return 0; + + rc =3D xa_insert(&ctx->regions, (unsigned long)cxlr, cxlr, GFP_KERNEL); + + /* A region may be referenced by multiple affected endpoint decoders. */ + return rc =3D=3D -EBUSY ? 0 : rc; +} + +static int cxl_reset_collect_region(struct device *dev, void *data) +{ + struct cxl_reset_region_context *ctx =3D data; + struct cxl_endpoint_decoder *cxled; + + if (!is_endpoint_decoder(dev)) + return 0; + + cxled =3D to_cxl_endpoint_decoder(dev); + return cxl_reset_add_region(ctx, cxled->cxld.region); +} + +static int __maybe_unused +cxl_reset_collect_memdev_regions(struct cxl_reset_region_context *ctx, + struct cxl_memdev *cxlmd) +{ + struct cxl_port *endpoint; + + if (!cxlmd || !cxlmd->cxlds) + return -ENODEV; + + endpoint =3D cxlmd->endpoint; + if (!endpoint) + return 0; + + return device_for_each_child(&endpoint->dev, ctx, + cxl_reset_collect_region); +} + +static bool cxl_reset_region_has_system_ram(struct cxl_region *cxlr) +{ + struct cxl_region_params *p =3D &cxlr->params; + int rc; + + if (!p->res) + return false; + + rc =3D walk_iomem_res_desc(IORES_DESC_NONE, + IORESOURCE_SYSTEM_RAM | IORESOURCE_BUSY, + p->res->start, p->res->end, NULL, + cxl_reset_system_ram_found); + + return rc > 0; +} + +static int cxl_reset_validate_region_idle(struct cxl_region *cxlr) +{ + struct resource *res =3D cxlr->params.res; + int rc =3D 0; + + lockdep_assert_held_write(&cxl_rwsem.region); + + if (cxl_reset_region_has_system_ram(cxlr)) { + dev_err(&cxlr->dev, + "Cannot reset while CXL memory is online as System RAM [%pr]\n", + res); + return -EBUSY; + } + + if (!device_trylock(&cxlr->dev)) + return -EAGAIN; + + if (cxlr->dev.driver) { + dev_err(&cxlr->dev, + "Cannot reset while CXL region has an active driver\n"); + rc =3D -EBUSY; + } + + device_unlock(&cxlr->dev); + return rc; +} + +static int __maybe_unused +cxl_reset_validate_regions_idle(struct cxl_reset_region_context *ctx) +{ + struct cxl_region *cxlr; + unsigned long index; + int rc; + + xa_for_each(&ctx->regions, index, cxlr) { + rc =3D cxl_reset_validate_region_idle(cxlr); + if (rc) + return rc; + } + + return 0; +} + +static int cxl_reset_flush_region_cache(struct cxl_region *cxlr) +{ + struct resource *res =3D cxlr->params.res; + int rc; + + if (!res) + return 0; + + rc =3D cpu_cache_invalidate_memregion(res->start, resource_size(res)); + if (rc) + dev_err(&cxlr->dev, "Failed to invalidate CPU cache [%pr]: %d\n", + res, rc); + + return rc; +} + +static int __maybe_unused +cxl_reset_flush_cpu_caches(struct cxl_reset_region_context *ctx) +{ + struct cxl_region *cxlr; + unsigned long index; + int rc; + + if (xa_empty(&ctx->regions)) + return 0; + + if (!cpu_cache_has_invalidate_memregion()) { + if (IS_ENABLED(CONFIG_CXL_REGION_INVALIDATION_TEST)) { + pr_info_once( + "Bypassing cpu_cache_invalidate_memregion() for testing!\n"); + return 0; + } + pr_warn("Failed to synchronize CPU cache state\n"); + return -ENXIO; + } + + xa_for_each(&ctx->regions, index, cxlr) { + rc =3D cxl_reset_flush_region_cache(cxlr); + if (rc) + return rc; + } + + return 0; +} --=20 2.43.0 From nobody Mon Jun 8 15:36:52 2026 Received: from CH5PR02CU005.outbound.protection.outlook.com (mail-northcentralusazon11012019.outbound.protection.outlook.com [40.107.200.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C4AFF3783C7; Thu, 28 May 2026 08:32:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.200.19 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957135; cv=fail; b=PsRlkCVLVNlqvj5aPtzafRbgRCI9Y44xiQMZuM9V19VAC5u3BrsZ0cK220f5L3+Zsu9S3v8xv5cue62hDeM7myx62K5rdKLSN1eGwwcgN3yIsoOwkuqwi28Dq3vTuw4RoNJ+9aK490F7brirU8gJJbeRnDPQM4yllWZqeUvhcoQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957135; c=relaxed/simple; bh=ht0Urj349w7jYSsBVGF9XyGIycCvC48K4QvL0AyghWg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=eQwMyMJHq/Eaa1NTpxnFSIDnpMUGDDYoG2NIqXsy+Pm4WQirHGaG/zY2GRjaMeFRmscJAGZ0R6J2Hla5Hkv5iNGJkyG2QLxIsAelgNBEvLLBmHKNviE7yIaYVdpis7UqCviAHdv22+TdGsl2GWglsxUSfN1p2g6saQesnn2GnNc= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=rCMvOdbr; arc=fail smtp.client-ip=40.107.200.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="rCMvOdbr" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=TkNhd1UfGSdaB75WA3nenRLrrqyiiG/ope58ubf/8TmJbcd25RNDFfFfn0UgoM1bxABzZmNopXzTUHmiZCzCWovgSS102O7mGaHqKbBH7LF5cxu0VaAPGyMPsvpMsPQQ5Wb1D8YRxHXBAOY22M4kNlkOx+SPv0LJiGdMK4r8Rseh82pToi1rvctoFw2Byuaa5mbDEd4mzpgnGF3Old7KQDi3XDTA6oxA6t5V/GNI9XWVTRosiDNE608ITgfjKLHDK67Uwdcgp/DrJLKHVDa2FzxSbCGvEplW+2I7NrXL7YC8ywZ5KoXvNW/PHm8lEY+CBSP7yWRGKigGeTI6+iKkKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=n8O2r3aZrNObPpykVcyVYSuen1iMfLW3OgtWAAWsTk8=; b=TpI1CBB/FBKhcIPp2DV1Pde3DYuYVGzUIJg5SZA97LtuqV+hoYew4GPombHy7jdUnKOkSpBKlZuwg+R5ANMhcn4PtLNZHO9MxwVt1qZVUUBC8RlGH8E3efvNhuW198PLPYWJUnYfcC2iS/6JN4udckkFb//Cqg0vtSwD77vREdfMj0fgbx+b8GMQ6haQA2U7NVE2w6KNHtTN25NO6++LWvi2SBU/V5j7yEouW2YzlDuv7Nfm33gSRwRkkUqLGWpK+0rMQRJ363SiZBI+FW5cSdndvmKtSFZJfVIBb3r/J0XzTgPxNQGzlH6zr1jA0fu3v0qqbM/8SgMPzl8qKv8r3A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=n8O2r3aZrNObPpykVcyVYSuen1iMfLW3OgtWAAWsTk8=; b=rCMvOdbrqZOfEJa5EcNJIivRF2AZcPkUH0PJ15V4zDFW/XG3MxKaBqN/mp4kPmwlo8e3X+ZhG+fAdID9PK9UaOMFMS7IHRFJzamK4n7iSg1sjq0/zOvrHfqmtyOCWqTqkqUt5EZYbP3O3m5PAx7tPI+9qIEHYYy8TRS0lj9xaHL4GghikQ7kflnKyAVRrV0c5iZORfmOg4TCEJi0tEinufqnpWzpp5kJQdLP9gx46qePlgsQKdVfamntRXgxDJ8XC5V3KQoByK68cuYFxZXgseEVipFMbB6RLMwkUHce9ZzgdmPKBfHEm5/7M9TTLTwsqVW1PjhooGPEO7AkAT3cZA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) by SA1PR12MB7442.namprd12.prod.outlook.com (2603:10b6:806:2b5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.12; Thu, 28 May 2026 08:32:08 +0000 Received: from DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7]) by DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7%6]) with mapi id 15.21.0071.011; Thu, 28 May 2026 08:32:08 +0000 From: Srirangan Madhavan To: linux-cxl@vger.kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Cc: vsethi@nvidia.com, alwilliamson@nvidia.com, Dan Williams , Sai Yashwanth Reddy Kancherla , Vishal Aslot , Manish Honap , Jiandi An , Richard Cheng , linux-tegra@vger.kernel.org, Srirangan Madhavan Subject: [PATCH v6 4/9] PCI/CXL: Add sibling function coordination for reset Date: Thu, 28 May 2026 08:31:49 +0000 Message-ID: <20260528083154.137979-5-smadhavan@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260528083154.137979-1-smadhavan@nvidia.com> References: <20260528083154.137979-1-smadhavan@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SJ0PR03CA0385.namprd03.prod.outlook.com (2603:10b6:a03:3a1::30) To DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9474:EE_|SA1PR12MB7442:EE_ X-MS-Office365-Filtering-Correlation-Id: c9d17e24-29b0-49a5-8df3-08debc939af8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|366016|56012099006|11063799006|5023799004|18002099003|22082099003|6133799003; X-Microsoft-Antispam-Message-Info: HGMbX1O68RwRaMEiQJIkMzUw9Y+UQxtuvyqa0LiAtbXKjI6YnsuYgWIDXv1mMMpZhaFREWChPcIJg9BvBdR2A85MJEQfUfri7JZ4QkJyjv48ZhRgvP7oEBkQVc6DlUb3F95oulTak7ryMMR96nDcGCIZszJKbmTFI1I728/6e8WiKtAkGeGBZ93C7IJ5Es3N31FLw29Dl/w+Dahss1vrH+GpGIHrgLq940kqRQFr5zAoxYYUBEWVXsJUJYYeHissl1z4d+TwYvr++1fMNBeETypXweWaNliyDKhtNFlQ4//6bvbiEmnJgFljTKnKPa//Idk8dGS0etcbpGWm0H+2Ph4rSh3R4X0+6y5Y1CY8z6sRX6+iaPe2RMekAsOJ6KOndr8ZAO2Egj6tZh6IwZMFl62FTIcY+a6w2ezutqpR9X7Z8WSyMHcMKi0NigU/kDzO0VY+hoq+fz7DKCM7B9mMkFQu9u6pR8b2MKk7vwYPWBBIp06ToZuf4Ve3pbvrhDyPWczUbRaTtZfwhVI1YkCcR7AX2OQ+um+GaE+Tu5hFT0QKpvbVelbl/+6r7GXXL6iRPW33ox6mqAPaZlLPEKbnPIEh/73qfGS8fEQpvUY42Sg3azltsy6bCCzik1rAJtq2yvUxnGLZ6XE/lLLHQNWWb5FIbIcX++yrQXK+zgwn0GxhAMTGVeqklM2LmznipSch X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9474.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(366016)(56012099006)(11063799006)(5023799004)(18002099003)(22082099003)(6133799003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?okEHK5yQvoY6Qs52g4BcMa10R5IY8KHfsHZRqG+HynzE5YioGP08jnt29v04?= =?us-ascii?Q?gvSk8vhjVBk1pMHwr/EnuV/fDaO7g7wsshKd0HidiFDXacV6LQXG1J+/hjxT?= =?us-ascii?Q?yGbef03A7nSiZhphq3KFw//Qbcv20DKEoE9N/cAsGXr4E4WWm1BRUnhRHKF+?= =?us-ascii?Q?46ngynDksFCWLNa/vcfzRnfVulypjFJcqmjXfM1QWsqdmiaYjTcNjdGwDN5L?= =?us-ascii?Q?UACyymfMWQi9cEz6/Q/hOTj2bAXAUxaVBBw2+1zXW6zYtz0SbacGKnBkvK7F?= =?us-ascii?Q?QrN9VK6gJQNOhhD2tVOROgnBeFeWweGuJNnNsFYGMiyBcE6RDOHsDEiJEo+X?= =?us-ascii?Q?ZsP0jXh/jFA+/bdsCBQVH0wdVJkggdsGZWnhYA6xFNEz1NCOmpALf91IGsnr?= =?us-ascii?Q?q4yyyqs0vg+7WjhcoVDXaNaXtzH7RkIF3/0Es140fNA9oumg6U1W4M0K+O0Z?= =?us-ascii?Q?p7FodSC0yW7T5Aodsv987Kc5/6K/+BsTSBEI1rx5dryShO28c33YCQAj2BGg?= =?us-ascii?Q?mGQguuBBfWh59U3nPpYtsCzdWOB2L3RcQqRYSgti2CEiRuk1PlMtoh3hFyoI?= =?us-ascii?Q?rLJYAn8u08FX88rAeRCF/plBxBZWEmxVwwHr0B+dA1K7P3UreV/FDUcsR/9h?= =?us-ascii?Q?NlW2sKAKMBkUKXk4jEPvMmptiL/vLdnwNYW7RzTwJySlLuz+k0ebsoq8QRNY?= =?us-ascii?Q?c1fjVelfKPhyGbhSEuvhuRCZvl+jNDW9YDtqkzS4DDqWdAnOIL4COF7SwR95?= =?us-ascii?Q?7Tcm2xZRDXKzafMCLiBaM8e5WovKPQFLn+n/p2Zfk+WJ1DjW4nire5XqJI0I?= =?us-ascii?Q?wYBFStWnyybZfJic581Sj+TNtVuPYePrNmGgcbaHVV9zyB5G+owaAIS0G848?= =?us-ascii?Q?1SgV1l/+Gm81jnonwUB1PjsntDqJI+95ny2+m1p2q/2nOikppe+6v7LApH5/?= =?us-ascii?Q?jgPz3MD6gZwzDPXs+d+n6jTWXfYquWfHJkf3Z7XsTTW+NkolSa8pxVf44RhL?= =?us-ascii?Q?MD9axZ9VyvDpkuAvAysg0mZk/M/Yu7vi5zYS2zVTJjmDt7XevCyxpGnu4pWM?= =?us-ascii?Q?YPiUW6U2ib7qou3P5j++um3O71iis+atwI+8k+b9/rt/ycK4o8cmmHyxV03W?= =?us-ascii?Q?ezGlKADN8BcvsRV4NlLXdWbUiP202wK5Km0s6vK+e8V9VdJOFEkHJBCFT0vn?= =?us-ascii?Q?SVfJ91eStgv+FULuhajNFtNauYmUluHy88fUi+KHSJaxCC5oY7NCj3qZAUhC?= =?us-ascii?Q?6lN+k11IqdxReLcnjAsf+23SPyVje7/q7nOD66xbPaThaOvbS/nvhBKK5mAw?= =?us-ascii?Q?1I670BGs4io73Wf8lXHFCISRJC1MbPjcLcujyrfSAVQbMX4UnfexIklTgIqN?= =?us-ascii?Q?OvJNtPW9N7rs4vgWB8rDRCIJIGmNP5/TvFMGaMXv8/JqQWrgR3yQBdg1rxlF?= =?us-ascii?Q?s3IltS5E/m/H/gbfRkmxf/qRkLeoOYOmYVYaX+e82u3C57lW3xWlikOLEuBd?= =?us-ascii?Q?TYgup89FwVztA2qqje3d2BCXnDf7FiN6zpEDbwIVaG7IBXgx1gSHCTMkrK3f?= =?us-ascii?Q?TDqHU2Upx0CRUwDFdJEL6xZchPALBujfk6VUJTqoRkuGTNfDSYh/K6UgCVPV?= =?us-ascii?Q?H9Vyzkzk6gNCpN9BTSp3URioLP+lZIzDy8WATD8mCMdfTUABmRfKWHMBT97W?= =?us-ascii?Q?wX7dqTyAxHoJcLihU+yH4B22S5QyGWZZorXUdnRdMSTEUL2T7fBnTpVPj8uX?= =?us-ascii?Q?PPlKx8hTGQ=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: c9d17e24-29b0-49a5-8df3-08debc939af8 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9474.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 May 2026 08:32:08.0639 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: n75k4bI4Y5LjdR+MEj8JS6Up2FCL4UdQ+9cb6kaU+A50dnws/597yeAVjrodqufc6CvMn1C8bT/zIN4qC3QC4g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7442 Content-Type: text/plain; charset="utf-8" Add helpers to collect CXL sibling PCI functions affected by a CXL reset and prepare them for reset by saving and disabling them. Restore those siblings and drop their references when reset coordination completes. Use the Non-CXL Function Map DVSEC to exclude non-CXL functions, and filter remaining siblings to functions that advertise CXL.cache or CXL.mem capability. Use pci_dev_trylock() for sibling locking and unwind on contention or allocation failure, so competing reset paths fail with an errno. Signed-off-by: Srirangan Madhavan --- drivers/cxl/core/pci.c | 207 ++++++++++++++++++++++++++++++++++ include/uapi/linux/pci_regs.h | 2 + 2 files changed, 209 insertions(+) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 318744695f62..01effbb4e7cd 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -1,9 +1,11 @@ // SPDX-License-Identifier: GPL-2.0-only /* Copyright(c) 2021 Intel Corporation. All rights reserved. */ #include +#include #include #include #include +#include #include #include #include @@ -15,6 +17,10 @@ #include "core.h" #include "trace.h" =20 +#define CXL_RESET_MAX_FUNCTIONS 256 +#define CXL_RESET_FUNCTION_MAP_REGS (CXL_RESET_MAX_FUNCTIONS / 32) +#define CXL_RESET_SIBLINGS_INIT 8 + /** * DOC: cxl core pci * @@ -1096,3 +1102,204 @@ cxl_reset_flush_cpu_caches(struct cxl_reset_region_= context *ctx) =20 return 0; } + +struct cxl_reset_context { + struct pci_dev *target; + struct pci_dev **siblings; + int nr_siblings; + int sibling_capacity; + int nr_siblings_prepared; +}; + +struct cxl_reset_walk_ctx { + struct cxl_reset_context *ctx; + unsigned long *non_cxl_func_map; + int rc; +}; + +static void +cxl_reset_read_non_cxl_func_map(struct pci_dev *pdev, + unsigned long *non_cxl_func_map) +{ + u32 map[CXL_RESET_FUNCTION_MAP_REGS] =3D {}; + u16 dvsec; + int rc, i; + + bitmap_zero(non_cxl_func_map, CXL_RESET_MAX_FUNCTIONS); + + dvsec =3D pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL, + PCI_DVSEC_CXL_FUNCTION_MAP); + if (!dvsec) + return; + + for (i =3D 0; i < CXL_RESET_FUNCTION_MAP_REGS; i++) { + rc =3D pci_read_config_dword(pdev, + dvsec + PCI_DVSEC_CXL_FUNCTION_MAP_REG + + i * sizeof(map[i]), &map[i]); + if (rc) { + pci_warn(pdev, + "failed to read CXL Function Map; treating all siblings as CXL: %d\n", + rc); + bitmap_zero(non_cxl_func_map, CXL_RESET_MAX_FUNCTIONS); + return; + } + } + + bitmap_from_arr32(non_cxl_func_map, map, CXL_RESET_MAX_FUNCTIONS); +} + +static bool cxl_reset_is_cxl_sibling(struct pci_dev *pdev, + struct pci_dev *sibling, + unsigned long *non_cxl_func_map) +{ + if (sibling =3D=3D pdev || sibling->bus !=3D pdev->bus) + return false; + + if (pci_ari_enabled(pdev->bus)) + return !test_bit(sibling->devfn, non_cxl_func_map); + + if (PCI_SLOT(sibling->devfn) !=3D PCI_SLOT(pdev->devfn)) + return false; + + return !test_bit(PCI_FUNC(sibling->devfn) * 32 + + PCI_SLOT(sibling->devfn), non_cxl_func_map); +} + +static bool cxl_reset_has_cache_or_mem(struct pci_dev *pdev) +{ + u16 dvsec, cap; + + dvsec =3D pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL, + PCI_DVSEC_CXL_DEVICE); + if (!dvsec) + return false; + + if (pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap)) + return false; + + return cap & (PCI_DVSEC_CXL_CACHE_CAPABLE | PCI_DVSEC_CXL_MEM_CAPABLE); +} + +static int cxl_reset_add_sibling(struct cxl_reset_context *ctx, + struct pci_dev *sibling) +{ + struct pci_dev **siblings; + int capacity; + + if (ctx->nr_siblings < ctx->sibling_capacity) + goto add; + + capacity =3D ctx->sibling_capacity ? ctx->sibling_capacity * 2 : + CXL_RESET_SIBLINGS_INIT; + siblings =3D krealloc(ctx->siblings, capacity * sizeof(*siblings), + GFP_KERNEL); + if (!siblings) + return -ENOMEM; + + ctx->siblings =3D siblings; + ctx->sibling_capacity =3D capacity; + +add: + pci_dev_get(sibling); + ctx->siblings[ctx->nr_siblings++] =3D sibling; + return 0; +} + +static int cxl_reset_collect_sibling(struct pci_dev *sibling, void *data) +{ + struct cxl_reset_walk_ctx *wctx =3D data; + struct cxl_reset_context *ctx =3D wctx->ctx; + struct pci_dev *pdev =3D ctx->target; + + if (!cxl_reset_is_cxl_sibling(pdev, sibling, wctx->non_cxl_func_map)) + return 0; + + if (!cxl_reset_has_cache_or_mem(sibling)) + return 0; + + wctx->rc =3D cxl_reset_add_sibling(ctx, sibling); + return wctx->rc; +} + +static int cxl_reset_collect_siblings(struct cxl_reset_context *ctx) +{ + DECLARE_BITMAP(non_cxl_func_map, CXL_RESET_MAX_FUNCTIONS); + struct cxl_reset_walk_ctx wctx =3D { + .ctx =3D ctx, + .non_cxl_func_map =3D non_cxl_func_map, + }; + + cxl_reset_read_non_cxl_func_map(ctx->target, non_cxl_func_map); + pci_walk_bus(ctx->target->bus, cxl_reset_collect_sibling, &wctx); + return wctx.rc; +} + +static void cxl_pci_functions_reset_done(struct cxl_reset_context *ctx) +{ + int i; + + for (i =3D ctx->nr_siblings_prepared - 1; i >=3D 0; i--) { + struct pci_dev *sibling =3D ctx->siblings[i]; + + pci_dev_reset_iommu_done(sibling); + pci_dev_restore(sibling); + pci_dev_unlock(sibling); + } + + for (i =3D 0; i < ctx->nr_siblings; i++) + pci_dev_put(ctx->siblings[i]); + + kfree(ctx->siblings); + ctx->siblings =3D NULL; + ctx->nr_siblings =3D 0; + ctx->sibling_capacity =3D 0; + ctx->nr_siblings_prepared =3D 0; +} + +static int __maybe_unused +cxl_pci_functions_reset_prepare(struct cxl_reset_context *ctx) +{ + int rc, i; + + ctx->siblings =3D NULL; + ctx->nr_siblings =3D 0; + ctx->sibling_capacity =3D 0; + ctx->nr_siblings_prepared =3D 0; + + rc =3D cxl_reset_collect_siblings(ctx); + if (rc) + goto err; + + for (i =3D 0; i < ctx->nr_siblings; i++) { + struct pci_dev *sibling =3D ctx->siblings[i]; + + if (!pci_dev_trylock(sibling)) { + rc =3D -EAGAIN; + goto err; + } + + pci_dev_save_and_disable(sibling); + rc =3D pci_dev_reset_iommu_prepare(sibling); + if (rc) { + pci_err(sibling, + "failed to block IOMMU for CXL reset: %d\n", + rc); + /* + * Undo save_and_disable() for this sibling. IOMMU + * prepare failed, so this sibling is not counted in + * nr_siblings_prepared and must not get iommu_done(). + */ + pci_dev_restore(sibling); + pci_dev_unlock(sibling); + goto err; + } + + ctx->nr_siblings_prepared++; + } + + return 0; + +err: + cxl_pci_functions_reset_done(ctx); + return rc; +} diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h index 14f634ab9350..fa1fcd26af01 100644 --- a/include/uapi/linux/pci_regs.h +++ b/include/uapi/linux/pci_regs.h @@ -1349,6 +1349,7 @@ /* CXL r4.0, 8.1.3: PCIe DVSEC for CXL Device */ #define PCI_DVSEC_CXL_DEVICE 0 #define PCI_DVSEC_CXL_CAP 0xA +#define PCI_DVSEC_CXL_CACHE_CAPABLE _BITUL(0) #define PCI_DVSEC_CXL_MEM_CAPABLE _BITUL(2) #define PCI_DVSEC_CXL_HDM_COUNT __GENMASK(5, 4) #define PCI_DVSEC_CXL_CTRL 0xC @@ -1366,6 +1367,7 @@ =20 /* CXL r4.0, 8.1.4: Non-CXL Function Map DVSEC */ #define PCI_DVSEC_CXL_FUNCTION_MAP 2 +#define PCI_DVSEC_CXL_FUNCTION_MAP_REG 0x0C =20 /* CXL r4.0, 8.1.5: Extensions DVSEC for Ports */ #define PCI_DVSEC_CXL_PORT 3 --=20 2.43.0 From nobody Mon Jun 8 15:36:52 2026 Received: from CH5PR02CU005.outbound.protection.outlook.com (mail-northcentralusazon11012019.outbound.protection.outlook.com [40.107.200.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C050537E307; Thu, 28 May 2026 08:32:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.200.19 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957137; cv=fail; b=kkvQXPoAy4OEXGtnNp9lXBR/BAA2eztK5dRMpg/4ktJnynF66e2NCDrSQXR+wiBhdJ0oAaX8DKVk31uMxvntcaCTywO628Vpj433GweY89TTiHWx80+lHzTwxPJ8nTEptD/cYT2LKNytc6fqF6PcPTyWTFbbqvxV5xHG+BfWmpg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957137; c=relaxed/simple; bh=sG5JenU3eBtjxIo7xcLFwqz77ntjv6FQ0bh8RoQcIws=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=TxITJQYF67N4v8zCYp3A0EJVumTon59TeLQAHdkH7gC+Ru5MLESI4dKJiG3+mmYkVCZKdKsF0Gt8IPHTphkNJhJE+kkEtsi2WOywI4xgfUy5IhEp6HqQ8ZqNYCAkAl+dMzgmErv53WMRqoG0SC556XRLLOuLMNxN+Re5v0IvjjM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=J8b18ecP; arc=fail smtp.client-ip=40.107.200.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="J8b18ecP" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=E91BJcGvhIrENathA1qFckY8FuOIdmOQJcd3bkQv0GrzRam3XKIqo3cH1z/pgx6FBL+Nb9SnPRihPKHT/lMWw7Z1SchpgcLWF1paDnPY5jnB2Fti58sapUdxUz2mTYjMHMDpnk3qu3+QwhIwmJ1nUXmjYdIAwD9y5+oWQTYbri3a6Fm8jUZP3pcfCc62kIXlvKnIEEJYAkDH/GAqxPzYI8A9vB+Ol4NvrNJDHr7eWWEjmvZeuf6FUump9DvjzhQUwBDyogOutsMRKmhDNWGFRcWCstkMrIjjitohPZLVJJIa/Z7hpX2ja4pnnY5eych3ksMwBjyYSl2ZjV8F7HpYOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vmYCLcC4yheEDmWm1ZTdlO/W/LFEOhUd+NVj5LxevqE=; b=a1eEhlCKyN775dMgCaCycUKuD0u2upEhMNSowWWoEoa51wXwfSWoxIrcxwcJHUzuWETvO6M6OpLpP9IS80v6YM6m1AtiH6Cxn1TcdFSJTiDKlCYzyNsvyRS4VicWMmDTHtVMzaRaRXkXFbWTLzaeMcFokLEKCWysFby71v/xjl2WTazrxUMBExmaJ9TanyOeUcx7suaxYJHz3/vLgRe2JWz5XUd5TmGQ4ChlUtdw53gdAzQACaJbp710ofWiq/zKYEG/spq4wtrj7ht5vYn/fcYwGoSP+x2jodGOUUvIjGXw4M2kqn0PqUodH8HvQJwBiuMgxRIeF8UmhIjFDq7AUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vmYCLcC4yheEDmWm1ZTdlO/W/LFEOhUd+NVj5LxevqE=; b=J8b18ecPlWrLKg5n9wmsxJPUL/AA9L5oJOnB4Pyvx4PRnK5AUlWkBWJHXpA4/w7b8fRBweY7kyxrW+WJwV7WC1c8qTYdUWU7/S3jcdJ20c73ZR5GizviULxZazAxk/y5qCPFBKSbtzdpjHAlzs7IeDX4Eu3kILNWapXzBEJIr5uesJfHzCd4sM80n5TZdbrv9o6tyz9mA4J/vL4m117H7qjgRCcKRLZZqRaN7Sio0NmtZwSVYnL95Ti/7k6c+Ly++hCmYphGpqO9tZg/Q2tjvsh8HWJa1krtTnNuShEBV8UWdZejImDa143LHoH3aoIdh06NbqRkGv6q4aTzRx8sWQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) by SA1PR12MB7442.namprd12.prod.outlook.com (2603:10b6:806:2b5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.12; Thu, 28 May 2026 08:32:10 +0000 Received: from DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7]) by DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7%6]) with mapi id 15.21.0071.011; Thu, 28 May 2026 08:32:10 +0000 From: Srirangan Madhavan To: linux-cxl@vger.kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Cc: vsethi@nvidia.com, alwilliamson@nvidia.com, Dan Williams , Sai Yashwanth Reddy Kancherla , Vishal Aslot , Manish Honap , Jiandi An , Richard Cheng , linux-tegra@vger.kernel.org, Srirangan Madhavan Subject: [PATCH v6 5/9] cxl/pci: Add CXL DVSEC reset helper Date: Thu, 28 May 2026 08:31:50 +0000 Message-ID: <20260528083154.137979-6-smadhavan@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260528083154.137979-1-smadhavan@nvidia.com> References: <20260528083154.137979-1-smadhavan@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SJ0PR13CA0176.namprd13.prod.outlook.com (2603:10b6:a03:2c7::31) To DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9474:EE_|SA1PR12MB7442:EE_ X-MS-Office365-Filtering-Correlation-Id: ea8615ed-f605-436f-f07f-08debc939c75 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|366016|56012099006|11063799006|5023799004|18002099003|22082099003|6133799003; X-Microsoft-Antispam-Message-Info: VC1HfAY0xTcS/GOIsXhT99nKXVCBCZsTx73P3J807KnwrDzlE3Wg30XXLczDXx6NDdx89/yQuCuS/8u75CTHrKv4fKaD+kubletrTkSnLm4VqEbqk4uJq37cmJVJ+BYkCH9fmu8CIsn1wF6G0+SjIREz0h88jwMo0vok4loNSIy3Tc6Tc9p2y4kSCO4lXz2voaAlUbvisnZiUXg2HvvU7rMk8NFREEE5vxCb4LGG+BJjH2cuhNiDo9qmv/N47jusDfHLZjF/f3GetPGkELoYxfOT1s6D3sWMZ7BuFzkNCrkQarN4/2PmktZZAi6gmZPuboxNvNrG3ulBRiXvESHsNFbA5D0PhiE770T2akyuL773b1AYS/1eII5VzW+7iYSenmaO6c28Mpnx0I19Tqh0v7XKN2cid2uO1ceC3PRSeCOblx81jKe1wMqxQP52ypCJFWpCW2seHzYad0VdFCleUz+O1UC/SnwJG6pPsDPMJP/rLI4WO1lr8lW7o/W5pVrX6+ioHZfIHpgyAxzQGJoJE4pf72syu6CFuWHmxzl//LHom4Hw8VSUF5GO/WwsoFFD7eNK41cn/SESnGFNkiGrjxUYpXT9F9/MFlrxwwo6vE7Br20wOwOG6cgE0gJ5yy1rtL0o60Jyg6caLkKCNpBJQ6JPJEEf+tltqROCXaIMpdm5EDt0hiKP5iMNnsTO+3nu X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9474.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(366016)(56012099006)(11063799006)(5023799004)(18002099003)(22082099003)(6133799003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?nwMXp3WdhPa/XVbyxy0zq22+Xofhd1qhJVdGnT4p9pR02chL33jvKxfOUXEH?= =?us-ascii?Q?iP1GeeVOZqWT5uRr1nNeCmd/rruIe8mbXVEztEk+tMxARSP8KAeX10//bWAO?= =?us-ascii?Q?GBuq9NbcmiPGpfDHgrWctBE+LKzN/6kojJAqtSFuHwzgybIL480Po1ocTjvQ?= =?us-ascii?Q?uptVrP2wOf/Hl9uNgHeEcnGnyhW21Gv/aObS4Q0fDApgCEnZiiqw5Ptb3JtR?= =?us-ascii?Q?L05zd+7SwuWPznZGMoR6xWtQ1gszuA1HmIoCe3zqcneSKDcqcx8viB2AFBkQ?= =?us-ascii?Q?7ZKsSMDNJBGp4OUb8TXMorFRiF9oyi0fispfMzQLE8oo/vtkVWTmAtK+z4Qo?= =?us-ascii?Q?WIEyRYRsbDcNHgSPWA96Ah5CYFogJbER/L8x7bejzgFnmfzz4hAQGm12Zrej?= =?us-ascii?Q?bs2vrEmAocvfxg+bWKwG19gnmAmYP+C7zDgoH5k2WIrB2FjM//f8Gm72Nsq6?= =?us-ascii?Q?2jlKNNQEUOhAsdMKIQ7jpejhkEBS0w/AZ/d87N2P+F/OEnvNaOQkMugvXthK?= =?us-ascii?Q?RgetqJADGzXxmYGeKNHhVWH9JAk31oZY1H0kcpH0XDlxxcuZYYUhmTgp3s79?= =?us-ascii?Q?QHZYOTSqylhvStcYt/c48pz6O1Sui7IdoYu86dgFKUg4r8LsqjxLa7fEzU8y?= =?us-ascii?Q?e3OHrwd1VoQdGoRf7cn+PnW8HwF/o1f4YxMxR1RwCIxg5x4Xslw0G2TD7EaP?= =?us-ascii?Q?81+++1bzU9C5S4BILr6uwQZrqH6dQ3cjufOFcSISH2nZuTcvp/lvzuOaKGcB?= =?us-ascii?Q?ZJaAio5RXqxHOg3e0v2UuutxtWWZs7qiNQ5gnw4aj3nQPFsD2NHvPW4kAZ6c?= =?us-ascii?Q?k8L2Ry6eNT5EEPtkQUwWtg2LU4Ude8uNJD3dJJuM5oZD4/yX0qBJaRh3ap2g?= =?us-ascii?Q?Tck37Z2XzkdIsDQ/apAz+a+f2C5Z0ydinmEPCnXzpDOSHsuL1oZ+m688ITx+?= =?us-ascii?Q?U9WW8JNxVsGE+aMMcU2CX6kwQ4d5uj5AwUQaEjS7eUCXW5sGATWKwVB3ecXZ?= =?us-ascii?Q?qGLykpYEfBIyM8/6GM/gf8OKQ//XjKRs0Fu8ciAeAw35c+szgH4fBvwj5taQ?= =?us-ascii?Q?wp0T+5DSxQZ3+CQeCn2zRq0NoUEQUEYDTVsNMdcKQbE2MaG3GGJkMHhjNkkg?= =?us-ascii?Q?Pi5wm1HPEzihYIFIAhso5gGvxZOM79lCT6CKt7xncM2Bc+ZpYYoUiwjJT7N8?= =?us-ascii?Q?VgiQPBNwZq6BeGEL0+B4t9JxtqCfHNILTuKv0vHLK4Z0rIKOS1hEySYJuFnO?= =?us-ascii?Q?lJe8zgunA6mpseaVTFRqefHnBSoVYyrWwZanjOSXYZ4aUJkPBjnPlyH1LyDu?= =?us-ascii?Q?lK/fYMxTPYLm/dZqfuxfxoKi+9UtzT88fIPcAeIWBGcWRcjuQk6Et3Mtqrgu?= =?us-ascii?Q?1DvECBope+5POe6ZOkGh5Ma5NBgnAhnXCOtsWJElgEThJePM1nPc+VhzkJA5?= =?us-ascii?Q?fEwGiRUWoL5N5ZTQOShaHQA+3BMgPVLV+Ec1NDqRZADWMxqEZEb1aeSMYlq2?= =?us-ascii?Q?D4WVyDVgEOp0UWysAd3nO69PyPo8UWAzMN2GbyLdJRMULC7vg2URvXtGrapg?= =?us-ascii?Q?hdaewWF/B5+IYZLIM7EgK9HGSiWfYtJ9qrESrqFij4WqDbaOA0YUtkneuy6V?= =?us-ascii?Q?WOwBxDmtYQbSiAwwCHBsxzRBbeb3sJZPBh4EtTxyVQQkRZTdwsI1aGRQSOpf?= =?us-ascii?Q?QWZMAzSAdQRXT+X20b+fE/nbQD/TdtvdauXvIUv7FKRJK+i9ntzY80bjpGTo?= =?us-ascii?Q?g0wxFdyhWA=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: ea8615ed-f605-436f-f07f-08debc939c75 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9474.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 May 2026 08:32:10.5400 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8QUz2hKHRX+cHsBDD1yLBS6/uXW53rIVxxabi4AOw6P1sVMjq0hUpJ1Z4skARVI/XLCB1gC8I008wKhz9nn++w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7442 Content-Type: text/plain; charset="utf-8" Add a helper to execute CXL Reset through the CXL Device DVSEC. The helper verifies reset capability, waits for pending PCIe transactions, disables CXL.cache, optionally initiates cache writeback and invalidation, and then starts CXL Reset through the DVSEC Control2 register. Block IOMMU traffic while reset is active, then restore IOMMU translations after reset completes. Wait for the DVSEC reset timeout before checking reset completion, and report reset error or timeout status from the DVSEC Status2 register. Add the CXL Device DVSEC reset and cache control definitions needed by the helper. Signed-off-by: Srirangan Madhavan --- drivers/cxl/core/pci.c | 185 ++++++++++++++++++++++++++++++++++ include/uapi/linux/pci_regs.h | 13 +++ 2 files changed, 198 insertions(+) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 01effbb4e7cd..1dd880f5a333 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -20,6 +20,9 @@ #define CXL_RESET_MAX_FUNCTIONS 256 #define CXL_RESET_FUNCTION_MAP_REGS (CXL_RESET_MAX_FUNCTIONS / 32) #define CXL_RESET_SIBLINGS_INIT 8 +#define CXL_RESET_CACHE_WBI_POLL_US 100 +#define CXL_RESET_CACHE_WBI_TIMEOUT_US (100 * USEC_PER_MSEC) +#define CXL_RESET_MIN_QUIET_MS 100 =20 /** * DOC: cxl core pci @@ -1303,3 +1306,185 @@ cxl_pci_functions_reset_prepare(struct cxl_reset_co= ntext *ctx) cxl_pci_functions_reset_done(ctx); return rc; } + +static int cxl_reset_update_ctrl2(struct pci_dev *pdev, int dvsec, u16 set, + u16 clear) +{ + u16 ctrl2; + int rc; + + rc =3D pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, &ctrl2); + if (rc) + return rc; + + ctrl2 &=3D ~clear; + ctrl2 |=3D set; + + return pci_write_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, ctrl2); +} + +static int cxl_reset_wait_cache_inv(struct pci_dev *pdev, int dvsec) +{ + int remaining_us =3D CXL_RESET_CACHE_WBI_TIMEOUT_US; + u16 status2; + int rc; + + do { + usleep_range(CXL_RESET_CACHE_WBI_POLL_US, + CXL_RESET_CACHE_WBI_POLL_US + 1); + remaining_us -=3D CXL_RESET_CACHE_WBI_POLL_US; + + rc =3D pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_STATUS2, + &status2); + if (rc) + return rc; + + if (status2 & PCI_DVSEC_CXL_CACHE_INV) + return 0; + } while (remaining_us > 0); + + pci_err(pdev, "CXL cache WB+I timed out\n"); + return -ETIMEDOUT; +} + +static int cxl_reset_enable_cache(struct pci_dev *pdev, int dvsec, u16 cap) +{ + if (!(cap & PCI_DVSEC_CXL_CACHE_CAPABLE)) + return 0; + + return cxl_reset_update_ctrl2(pdev, dvsec, 0, + PCI_DVSEC_CXL_DISABLE_CACHING); +} + +static int cxl_reset_disable_cache(struct pci_dev *pdev, int dvsec, u16 ca= p) +{ + int rc; + + if (!(cap & PCI_DVSEC_CXL_CACHE_CAPABLE)) + return 0; + + rc =3D cxl_reset_update_ctrl2(pdev, dvsec, + PCI_DVSEC_CXL_DISABLE_CACHING, 0); + if (rc) + return rc; + + if (!(cap & PCI_DVSEC_CXL_CACHE_WBI_CAPABLE)) + return 0; + + rc =3D cxl_reset_update_ctrl2(pdev, dvsec, + PCI_DVSEC_CXL_INIT_CACHE_WBI, 0); + if (rc) + goto err_enable_cache; + + rc =3D cxl_reset_wait_cache_inv(pdev, dvsec); + if (rc) + goto err_enable_cache; + + return 0; + +err_enable_cache: + /* + * Best effort rollback: preserve the original WB+I failure even if + * re-enabling CXL.cache also fails. + */ + cxl_reset_enable_cache(pdev, dvsec, cap); + return rc; +} + +static int cxl_reset_wait_done(struct pci_dev *pdev, int dvsec, u16 cap) +{ + static const u32 reset_timeout_ms[] =3D { 10, 100, 1000, 10000, 100000 }; + u32 timeout_ms; + u16 status2; + int rc, idx; + + idx =3D FIELD_GET(PCI_DVSEC_CXL_RST_TIMEOUT, cap); + if (idx >=3D ARRAY_SIZE(reset_timeout_ms)) + idx =3D ARRAY_SIZE(reset_timeout_ms) - 1; + timeout_ms =3D reset_timeout_ms[idx]; + + msleep(max_t(u32, timeout_ms, CXL_RESET_MIN_QUIET_MS)); + + rc =3D pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_STATUS2, + &status2); + if (rc) + return rc; + + if (status2 & PCI_DVSEC_CXL_RST_ERR) { + pci_err(pdev, "CXL reset error\n"); + return -EIO; + } + + if (!(status2 & PCI_DVSEC_CXL_RST_DONE)) { + pci_err(pdev, "CXL reset timed out\n"); + return -ETIMEDOUT; + } + + return 0; +} + +static int __maybe_unused cxl_dev_reset(struct pci_dev *pdev, bool mem_cle= ar) +{ + int dvsec, rc; + u16 ctrl2_clear =3D 0; + u16 cap; + + dvsec =3D pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL, + PCI_DVSEC_CXL_DEVICE); + if (!dvsec) + return -ENODEV; + + rc =3D pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap); + if (rc) + return rc; + + if (!(cap & PCI_DVSEC_CXL_RST_CAPABLE)) + return -EOPNOTSUPP; + + if (mem_clear && !(cap & PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE)) + return -EOPNOTSUPP; + + if (!pci_wait_for_pending_transaction(pdev)) + pci_err(pdev, "timed out waiting for pending transactions\n"); + + rc =3D pci_dev_reset_iommu_prepare(pdev); + if (rc) { + pci_err(pdev, "failed to block IOMMU for CXL reset: %d\n", + rc); + return rc; + } + + rc =3D cxl_reset_disable_cache(pdev, dvsec, cap); + if (rc) + goto out_iommu; + if (cap & PCI_DVSEC_CXL_CACHE_CAPABLE) + ctrl2_clear |=3D PCI_DVSEC_CXL_DISABLE_CACHING; + + if (mem_clear) { + rc =3D cxl_reset_update_ctrl2(pdev, dvsec, + PCI_DVSEC_CXL_RST_MEM_CLR_EN, 0); + if (rc) + goto out_ctrl2; + ctrl2_clear |=3D PCI_DVSEC_CXL_RST_MEM_CLR_EN; + } + + rc =3D cxl_reset_update_ctrl2(pdev, dvsec, + PCI_DVSEC_CXL_INIT_CXL_RST, 0); + if (rc) + goto out_ctrl2; + + rc =3D cxl_reset_wait_done(pdev, dvsec, cap); + if (rc) + goto out_iommu; + + rc =3D cxl_reset_update_ctrl2(pdev, dvsec, 0, + PCI_DVSEC_CXL_DISABLE_CACHING); + +out_ctrl2: + if (rc && ctrl2_clear) + cxl_reset_update_ctrl2(pdev, dvsec, 0, ctrl2_clear); + +out_iommu: + pci_dev_reset_iommu_done(pdev); + return rc; +} diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h index fa1fcd26af01..7fc1d34fcce7 100644 --- a/include/uapi/linux/pci_regs.h +++ b/include/uapi/linux/pci_regs.h @@ -1352,8 +1352,21 @@ #define PCI_DVSEC_CXL_CACHE_CAPABLE _BITUL(0) #define PCI_DVSEC_CXL_MEM_CAPABLE _BITUL(2) #define PCI_DVSEC_CXL_HDM_COUNT __GENMASK(5, 4) +#define PCI_DVSEC_CXL_CACHE_WBI_CAPABLE _BITUL(6) +#define PCI_DVSEC_CXL_RST_CAPABLE _BITUL(7) +#define PCI_DVSEC_CXL_RST_TIMEOUT __GENMASK(10, 8) +#define PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE _BITUL(11) #define PCI_DVSEC_CXL_CTRL 0xC #define PCI_DVSEC_CXL_MEM_ENABLE _BITUL(2) +#define PCI_DVSEC_CXL_CTRL2 0x10 +#define PCI_DVSEC_CXL_DISABLE_CACHING _BITUL(0) +#define PCI_DVSEC_CXL_INIT_CACHE_WBI _BITUL(1) +#define PCI_DVSEC_CXL_INIT_CXL_RST _BITUL(2) +#define PCI_DVSEC_CXL_RST_MEM_CLR_EN _BITUL(3) +#define PCI_DVSEC_CXL_STATUS2 0x12 +#define PCI_DVSEC_CXL_CACHE_INV _BITUL(0) +#define PCI_DVSEC_CXL_RST_DONE _BITUL(1) +#define PCI_DVSEC_CXL_RST_ERR _BITUL(2) #define PCI_DVSEC_CXL_RANGE_SIZE_HIGH(i) (0x18 + (i * 0x10)) #define PCI_DVSEC_CXL_RANGE_SIZE_LOW(i) (0x1C + (i * 0x10)) #define PCI_DVSEC_CXL_MEM_INFO_VALID _BITUL(0) --=20 2.43.0 From nobody Mon Jun 8 15:36:52 2026 Received: from CH5PR02CU005.outbound.protection.outlook.com (mail-northcentralusazon11012066.outbound.protection.outlook.com [40.107.200.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C2FF438239B; Thu, 28 May 2026 08:32:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.200.66 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957141; cv=fail; b=IipAslf+KzKzAkg3eKyQp5ZF92STfMfi+9/r422mw9khVvnAQ6x2FQVYIcRBNYCFdZvy3vQ6N+A8BxdlzQ7CgFOmHJUkFwvXdtS4dpNUXfzZlSdiWYgUzQkCJ2sY0p/PzbaX3kNpVNIxMbKhY4CNy/kVOh73M/Zxyo18PuuL8jc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957141; c=relaxed/simple; bh=3xrFeNZmScWmec77Ta8ZH22G1FjuSm4HD1cXmtuZvGk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=aSQEMBuFJz+Taqi+zctJmU0Dsohrf1MgzkCAb2CjJgbgPiR6F3mUiCJFJ+jtDeHZvo0il7VOoUnrE9jZWxbt8QT35YiGOnhYvQnzgTuqnuqyljRSGxL+OI5oTym35LSW7uzuP/FUE3adovehwvOhwaSg29ava0iEBMHsrO38ET8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=N29v6cad; arc=fail smtp.client-ip=40.107.200.66 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="N29v6cad" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=FJcrRS+zFtzlOwgKuXhkMrDTYjL2PVB+PbAAiMxUj9chzJaRMmmcmxaDFWFOP/CJOvw5VHQ9r90j+RWRNpgmg3BUDpzR47CiuV+rJl/+3FYvFh7yYwkmgKocQQJuV+vf6OjZmf0Pyqz0A10MRGTVpsWrTkBkwZgoJTp6D/gxOFy7OvnPJnkw3H/8zAQAByf/CnW/3AGe8Iq1zP6BmTxoA3LPkddwTbpYy3IasXSt+hd6IIkQhuRbnNKKL6t5+o21Su8b8xt7Lmtm2x3hQSfCPh5r/lMq+I/Wp5+I9go9be5N4/Ild0hv5UhwI5VwmF5mPVJU3l7OED+wNLCo3zjemQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=A2/8jP6kCUEPzTgBRpyKMJowgy8lnUzSlIQiWx3Payg=; b=sMGuZw5U3pp7E3zNzejyz5KTsSfc8TrMQ9jPJMy1nMl3tfAYSJYPiBQE5VHbFkRWyochAluU6ZoGBw6KV9Evhd+u+EDGSaZdWpOS0v1+BM962wlokW/rBmxPDI0Nb9oTanmIDm0LVrtu8AKVSLaGWzcM2swZPl2zQAX7N0WvkvYDaP+o0aSuglK0O4uJSh2yji9tm5zpisQgPGmueyHVUYpxWrCtpShFEDr255xvF7rkyZu5nJZu28aIIla00fPXWAdXziHLX6QNcxv8aoWaCf+ULWvlTzuCg1c0h/lWPjqR3FZGLIL6JtanORFHoXmjnWZyYnyGBQ3Np6iAqxnEAA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=A2/8jP6kCUEPzTgBRpyKMJowgy8lnUzSlIQiWx3Payg=; b=N29v6cad4CWFXyLkGfPYEMnrSpYCDIW2omXlTcj91T/+791hJ/DJ/l8e2UpwTck/xVmIQxUNbeH8jFnst5mfHAJLWsedcvhhfOnpoO1Gy8t3WMT5qWqkXTr9Z5Zy2NR4vMX0TrBUHv6JJiXx9MerUq9s+L8IPKqVo60OKSgXt8+d5WfumPwih5xUeAF8FGF/woQKqQdiB0kdG+56WPfVRrKx78x+XieVmuWn/VNkJviJhXbo60UbdXUM5lIvMeXuZzK2uGPpvg61t1vOJhzA76Rs40wob9QdVnCTSSY33M3P4Qci30JzDRORe88awxqQALZ6K3NOu/Q73aHmUxsVig== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) by SA1PR12MB7442.namprd12.prod.outlook.com (2603:10b6:806:2b5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.12; Thu, 28 May 2026 08:32:13 +0000 Received: from DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7]) by DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7%6]) with mapi id 15.21.0071.011; Thu, 28 May 2026 08:32:12 +0000 From: Srirangan Madhavan To: linux-cxl@vger.kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Cc: vsethi@nvidia.com, alwilliamson@nvidia.com, Dan Williams , Sai Yashwanth Reddy Kancherla , Vishal Aslot , Manish Honap , Jiandi An , Richard Cheng , linux-tegra@vger.kernel.org, Srirangan Madhavan Subject: [PATCH v6 6/9] cxl/pci: Track memdevs affected by CXL reset Date: Thu, 28 May 2026 08:31:51 +0000 Message-ID: <20260528083154.137979-7-smadhavan@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260528083154.137979-1-smadhavan@nvidia.com> References: <20260528083154.137979-1-smadhavan@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SJ0PR13CA0178.namprd13.prod.outlook.com (2603:10b6:a03:2c7::33) To DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9474:EE_|SA1PR12MB7442:EE_ X-MS-Office365-Filtering-Correlation-Id: e6776fba-6a9e-4f7a-75ce-08debc939dd7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|366016|56012099006|11063799006|18002099003|22082099003|3023799007|6133799003; X-Microsoft-Antispam-Message-Info: xz+Ct3DDKNScGvcwDzKhX50/zzcgUGkbShgB+KPz4RrHyYx9lSfte446npCZb+587nVV2rpDjbk0phqXATWEwfw7j1VqbnnfxYKhjKYyv/Vtw+iJMnyQC/iwW6pNANjIkwdSWs5JLgKVrDpcoAWFzOsTNvA1FIj/K8CWdjuYwX3sR+uHAxqX05qXmRtSUrV7SEzOI+qK+fI7Ve76glJ7sIMhGWAEJhHnKhh8vxzbHXeLYO3AqbholY50xRsP5P/yckPXDUCm2ggk1NVdPoHTwrnt7wAREmZQ3DE7vKJxBRJi8OdSEhhWggSXm4uzyHJvN6DqypYgx0qgAPv5xoRwDmRvkuReD4P64YUBhbO6P7TDz3Y1VnOaI0bL335RSqD0sN5dFhTrYpUKmqmpZX4lNAtT9TZWUqCzorpnMiSyTSra5N2aL99xdpxZRk1deEzPTGomtWIfMC0e79TTjl2BrmaYXK6OuJ6H54GP1LJ93CYqszCEKAFLphhcJcIbuEP4dnyu/knV+TUJfclZIFW6uiK9zfT8H7MSq9EANrkEv7NN0LjX8ELstj4iyrNCf5sP0QjodZWqPM37e8y0bCGjHlPG6AWfESrjCvwY62vPJsTguf91CPX70QzT1DongE4uXKKP+1kEpUmeR55ztnkJxXpvQG58j2m6+qb4K87wBj6lqX5XcQ24ZQi/r2sBaVqhfF70U6GkYLVLX2pmn5WgOg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9474.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(366016)(56012099006)(11063799006)(18002099003)(22082099003)(3023799007)(6133799003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?K5tngl4YFE2P0ZXTy8sVYS4+8liU/cMeHx1ATy/VFp5wpk6TLAyzQwRX6Z9x?= =?us-ascii?Q?k5hvqMUEuSEVfzV7kfT/uRO/BhIyc2/PwrcLBkiyy6D4CFF5djVh8Ar3Rul6?= =?us-ascii?Q?jqHEyB4g6Ju5F5C+PW8/0S7n9FG10bNxoXEFGWPVuZWarEd9JW7KYHjooIuU?= =?us-ascii?Q?lWo4fxMBVEsv+ro9gdxcwfADPtRrWRpygOg0WykhVTLPsBATk6zCTrIwy3dB?= =?us-ascii?Q?B59HIafv6eWr3b58lqNS7HCgNTxpIx4gyq0QUujFyvsfygyEKPXxJH9I4L+m?= =?us-ascii?Q?HPmHd3vq8Q59cf32H69FAzIAxm4yxfeg2ieWfPIb3sgho7M+tVCNSV1dCB9X?= =?us-ascii?Q?SPloE9Pj33rtmDBlTpRe6XEYMRdoppr8pomHkcpDzSop2sO6CRmPOwMT32v5?= =?us-ascii?Q?nI1ibnomAKELp27ch15/72RfD/uVVULewPK3Umvs/MkaJrwONiYOe3Hx8+8U?= =?us-ascii?Q?hZXiR38+TBkLI+0CIc+liIElZvyEPquhR9ZeMBNTHBV8PqCxY0PJwULImHAP?= =?us-ascii?Q?PpHB30+l2vjGxO2qXYa99BBJezr6PwuFfYnioy66bJWdrPOCj9GXC7Wj5dzR?= =?us-ascii?Q?dsSsK68OwLyg3kfMlHa3P1yvC4fn0CDnqZnSBBkrOyY8s04h8lm4j5bzKqba?= =?us-ascii?Q?WkWf9poyda3orvHoX05xv8h5DAJT4punBYYraV0wgjbcS8vkzZU35HAqleqA?= =?us-ascii?Q?FjXPD+HKo6QT6TmDxx9u3fkMZ0pBzryd/bPyY2/rHHV0zueyrelnJbr2Wo2H?= =?us-ascii?Q?UFntNarLDLSMDCicp8jtCZBr6xjpAAAEhDvPa2Kiozqyszx0H3EQNjOlSwv1?= =?us-ascii?Q?8DbdtNRXDJQ5Xtr1Hf9TpkSmbxDGxhOwMo8xXPYcNfXpbHsqq952nBNk6pXa?= =?us-ascii?Q?hUgiKIWc5M4Sb4yTH+YCZIQufiydCPl7cez6CyN3/mWpHbpyNuvPr5fA5OHD?= =?us-ascii?Q?Vb2PExrgy6HD77/Aqs61aFuZ1OUxyhSzqLUCJ7J7vTuLjLXbl4aZQ+4PlLlI?= =?us-ascii?Q?+xJF4N2bLBPH9PLJ+88qTV4O0uS3G6oKz0dxbPuvEQXm8OLsKO8rGs4jahN9?= =?us-ascii?Q?LMX6nLHeAtneoRydIWy8dta3yCh8jK4JXEVqp87/l3fEEDtgGDAIVgO6/liS?= =?us-ascii?Q?XOCZWid9jKt7TMpSnqgNl5/O/X/yTjjcG3HA+hWOBVHz1okI+tylmH+mYRW/?= =?us-ascii?Q?d0l1PcLniZkkYl7AJHY3G681esr44FC6XkbYq5pR4/Hwl8AHTkZ6nYnGPIza?= =?us-ascii?Q?1O6yw5u2/nR5eMJuPU4EWHp+KjDSbKoxtWiDMc7BuQnYm+kwFDr74oXhGiVz?= =?us-ascii?Q?5s0si3yoK2cWD4ZH0X+ZL+03tOt65cZeQAkOoDCrFOaKN0eBSgjMhVSVhPDA?= =?us-ascii?Q?ZbPWg6DA8yfbn8qTnAltwcIFHy0fFT3tbAeJtYrbaKMKtZviFrqn1gsbZTVo?= =?us-ascii?Q?Uw157NT22qy1DdA+GrM5JNrxAIplIwACT7oghVviqTkTSMKy0vlMgzJ1HHLR?= =?us-ascii?Q?jHSnnhp9zG9P0oAiE8NJJ2Q4HV58nZ/l+TMCULGfG6DsUoG0efuXyGYSSMvr?= =?us-ascii?Q?Aeoy6FN+/Ze4uFvnwrWvxmW+9k6BveYHOz1nR+sOY/pzIdTUqd595mTWBSgN?= =?us-ascii?Q?eycHng6RWW+2n921z06LdXHCsznXNRaktN5zZ3nqyQRIJ5zOfrC46ZObJLAD?= =?us-ascii?Q?UpXcq1/oAAseF72hjBXzEZt38nBn1xI13H74jNSlsdRwBWU4+sMdXgMkb+sI?= =?us-ascii?Q?XD1VKxfvng=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: e6776fba-6a9e-4f7a-75ce-08debc939dd7 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9474.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 May 2026 08:32:12.8653 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: xi46sLx1iQIHvMF1+TKLL6Tw/vepzMndGa9+d9/O2W8D5lDBgywV2EBUo5H7WQUMU3wbGoPscP8Lc5hmKZgAsg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7442 Content-Type: text/plain; charset="utf-8" CXL reset is scoped to the CXL.cache/mem function set, so reset orchestration needs to account for the target memdev and any affected sibling-function memdevs. Add reset context tracking for affected memdevs. Collect the memdevs associated with the target and sibling PCI functions, track which ones are active, collect their regions, and provide helpers to lock and revalidate the active memdevs before reset proceeds. The reset orchestration and CXL.mem restore flow are added separately. Signed-off-by: Srirangan Madhavan --- drivers/cxl/core/pci.c | 176 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 176 insertions(+) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 1dd880f5a333..c755c18c8d84 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -1106,8 +1106,17 @@ cxl_reset_flush_cpu_caches(struct cxl_reset_region_c= ontext *ctx) return 0; } =20 +struct cxl_reset_memdev { + struct cxl_memdev *cxlmd; + bool active; + bool locked; +}; + struct cxl_reset_context { struct pci_dev *target; + struct cxl_reset_memdev *memdevs; + int nr_memdevs; + int memdev_capacity; struct pci_dev **siblings; int nr_siblings; int sibling_capacity; @@ -1237,6 +1246,173 @@ static int cxl_reset_collect_siblings(struct cxl_re= set_context *ctx) return wctx.rc; } =20 +static int cxl_reset_match_memdev_by_parent(struct device *dev, + const void *parent) +{ + return is_cxl_memdev(dev) && dev->parent =3D=3D parent; +} + +static bool cxl_reset_memdev_active(struct cxl_memdev *cxlmd) +{ + return cxlmd->dev.driver && cxlmd->endpoint && + !IS_ERR(cxlmd->endpoint); +} + +static int cxl_reset_collect_pci_memdev(struct cxl_reset_context *ctx, + struct pci_dev *pdev) +{ + struct cxl_reset_memdev *memdevs; + struct cxl_memdev *cxlmd; + struct device *dev; + int capacity, i; + + dev =3D bus_find_device(&cxl_bus_type, NULL, &pdev->dev, + cxl_reset_match_memdev_by_parent); + if (!dev) + return 0; + + cxlmd =3D to_cxl_memdev(dev); + for (i =3D 0; i < ctx->nr_memdevs; i++) { + if (ctx->memdevs[i].cxlmd =3D=3D cxlmd) { + put_device(dev); + return 0; + } + } + + if (ctx->nr_memdevs < ctx->memdev_capacity) + goto add; + + capacity =3D ctx->memdev_capacity ? ctx->memdev_capacity * 2 : + CXL_RESET_SIBLINGS_INIT; + memdevs =3D krealloc(ctx->memdevs, capacity * sizeof(*memdevs), + GFP_KERNEL); + if (!memdevs) { + put_device(dev); + return -ENOMEM; + } + + ctx->memdevs =3D memdevs; + ctx->memdev_capacity =3D capacity; + +add: + ctx->memdevs[ctx->nr_memdevs++] =3D (struct cxl_reset_memdev) { + .cxlmd =3D cxlmd, + }; + return 0; +} + +/* + * CXL Reset is device scoped for CXL.cache/mem. Use the affected PCI + * function set to find memdevs whose regions and endpoint decoder state m= ust + * be handled around the reset. + */ +static int __maybe_unused cxl_reset_collect_memdevs(struct cxl_reset_conte= xt *ctx) +{ + int rc, i; + + rc =3D cxl_reset_collect_pci_memdev(ctx, ctx->target); + if (rc) + return rc; + + for (i =3D 0; i < ctx->nr_siblings; i++) { + rc =3D cxl_reset_collect_pci_memdev(ctx, ctx->siblings[i]); + if (rc) + return rc; + } + + return 0; +} + +static int __maybe_unused +cxl_reset_collect_regions(struct cxl_reset_context *ctx, + struct cxl_reset_region_context *region_ctx) +{ + int rc, i; + + lockdep_assert_held_write(&cxl_rwsem.region); + + for (i =3D 0; i < ctx->nr_memdevs; i++) { + struct cxl_reset_memdev *rmd =3D &ctx->memdevs[i]; + struct cxl_memdev *cxlmd =3D rmd->cxlmd; + + if (!device_trylock(&cxlmd->dev)) + return -EAGAIN; + + if (cxl_reset_memdev_active(cxlmd)) { + rc =3D cxl_reset_collect_memdev_regions(region_ctx, + cxlmd); + if (!rc) + rmd->active =3D true; + } else { + rc =3D 0; + } + + device_unlock(&cxlmd->dev); + if (rc) + return rc; + } + + return 0; +} + +static void cxl_reset_unlock_memdevs(struct cxl_reset_context *ctx) +{ + int i; + + for (i =3D ctx->nr_memdevs - 1; i >=3D 0; i--) { + struct cxl_reset_memdev *rmd =3D &ctx->memdevs[i]; + + if (!rmd->locked) + continue; + + device_unlock(&rmd->cxlmd->dev); + rmd->locked =3D false; + } +} + +static int __maybe_unused cxl_reset_lock_memdevs(struct cxl_reset_context = *ctx) +{ + int i; + + lockdep_assert_held_write(&cxl_rwsem.region); + + for (i =3D 0; i < ctx->nr_memdevs; i++) { + struct cxl_reset_memdev *rmd =3D &ctx->memdevs[i]; + struct cxl_memdev *cxlmd =3D rmd->cxlmd; + + if (!rmd->active) + continue; + + if (!device_trylock(&cxlmd->dev)) + goto err; + + rmd->locked =3D true; + if (!cxl_reset_memdev_active(cxlmd)) { + cxl_reset_unlock_memdevs(ctx); + return -ENODEV; + } + } + + return 0; + +err: + cxl_reset_unlock_memdevs(ctx); + return -EAGAIN; +} + +static void __maybe_unused cxl_reset_put_memdevs(struct cxl_reset_context = *ctx) +{ + int i; + + for (i =3D 0; i < ctx->nr_memdevs; i++) + put_device(&ctx->memdevs[i].cxlmd->dev); + + kfree(ctx->memdevs); + ctx->memdevs =3D NULL; + ctx->nr_memdevs =3D 0; + ctx->memdev_capacity =3D 0; +} + static void cxl_pci_functions_reset_done(struct cxl_reset_context *ctx) { int i; --=20 2.43.0 From nobody Mon Jun 8 15:36:52 2026 Received: from CH5PR02CU005.outbound.protection.outlook.com (mail-northcentralusazon11012066.outbound.protection.outlook.com [40.107.200.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 05DF3383C77; Thu, 28 May 2026 08:32:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.200.66 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957144; cv=fail; b=SFVbsybEwX+YV7vNLv8o94l0Uu9x5UPEYqvspMIaEhC1IycZaXS1k5WbHjD5088XDPh/NbeT66+jbcF2JPuC2WPNz45KipjLjeWyRGdWHIJdgTui20KtPPhCQFqrlavSEZ8V58JVhtAzO87c1AKK5zNldaQHhSaKVpAK9V8VDnY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957144; c=relaxed/simple; bh=MnEdg7lTPb26NnY+50InBTmIZJgJUX0dUiBVe9qS4Ng=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=NlOK/0l42Q8dcVWOfUauGJlNPc02wZ0v7Meq3/qotCE2h1R4gYpzjLQTRtEPa3DlPSa7eM8CbggE7ct9KP8Z4yFLIOjJlTH0cVRhcApfBg4wble5ZCnqqoY0S/TtYHph+q74HAFpUnGjRef57fjdf3ldY5NlGXqa+TAJFgd4MiQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=LPXEnrKh; arc=fail smtp.client-ip=40.107.200.66 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="LPXEnrKh" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=oBvHi5a0+1kYiaTOnSAMEty4Tlf8FMB+Q0+cL4rOcnb3/80o5GcWz5XfxibLndYBpsM6+N8yndPpOL6jKz37F3h/4ryg+8olDn0NCMb9HgZ/GihuzWrRhlMefwkRNo97jqqNnRd36qOzL4buXLiya90j2eUtI8HetlHTzenRRErQT0lnOuu5VmTU4HY7El751rAXkWgqKpqyFBShI5iBAAijY5gEOfcGFBNIHpH6JfEN6VoeWoKBYRharNZTdbwXl4n4FEP15356nfPiKZZSMnixdtIAYq45TUadZdDd/mas8QQ1Wq7mGwvrhJTBUzTh4I482zrq+fDbaDVJ+5e3HA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XVL47WseP4as7+dPo20tW24c8KlwY28+06G2yt9/ezw=; b=o3p8Mfm3YhEgxKpW/I02R566jyjlzbhWXeDc4PF4cnc/qF/1B0EdYq0KIqjNHhLHj3k3PtwSsNjtEDPA1f68+LYm/SG9QUj5Q3FAR4M5F09erKj4rItvCZacIeOjyb2P3kKCgaewD3pEsjE1wF1587BiXg+o0SLqV+a4SniT/XiudPGUqxv4WJKbukvwSSRu5KhapkeO03bOhryQnTHLxADao96eW0F2+yaqGpex40hwQr68G0fK/2EB1O7BGu610Qie8bRe0GA1ttBmi3hPpvmpl6ShqzvoLZr4oRVW05AY312OB0bZEnNDBmnfSQi1db2W1TOAGJrzZbZ+OhUI1A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XVL47WseP4as7+dPo20tW24c8KlwY28+06G2yt9/ezw=; b=LPXEnrKhEVV8ymLgw+IcEovCA+q8x3X+b1wZU617so/k2gT79w7QzpyZ+UUpdZ+YgTjq45/B1pXh6c7t5uGMDuvj6S+eZDdVoZmPx/HfoGTHAbZTXc2T9OBhojfkF54l8krbBeLKoCEYc96pCTmWA/xuT0K3++L1wtHtfGSDP4nHnfrWiYraD9/wiWALcH4Lb1ZDSUEunWTcreXK/f2Dk/aGSjFYUq4JK5ng4Gigg/FhngB/dfCYmr6BvOawZhiQ9oxX9B6GBKuk+/3YYp6ZjcW1667Le00Nfq6ImbjXP2yBmo/S5HOJNKriOXNIW67vupwzmrVMlQ6Ty2BGxJvHNA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) by SA1PR12MB7442.namprd12.prod.outlook.com (2603:10b6:806:2b5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.12; Thu, 28 May 2026 08:32:15 +0000 Received: from DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7]) by DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7%6]) with mapi id 15.21.0071.011; Thu, 28 May 2026 08:32:15 +0000 From: Srirangan Madhavan To: linux-cxl@vger.kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Cc: vsethi@nvidia.com, alwilliamson@nvidia.com, Dan Williams , Sai Yashwanth Reddy Kancherla , Vishal Aslot , Manish Honap , Jiandi An , Richard Cheng , linux-tegra@vger.kernel.org, Srirangan Madhavan Subject: [PATCH v6 7/9] cxl/pci: Orchestrate CXL reset for affected memdevs Date: Thu, 28 May 2026 08:31:52 +0000 Message-ID: <20260528083154.137979-8-smadhavan@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260528083154.137979-1-smadhavan@nvidia.com> References: <20260528083154.137979-1-smadhavan@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SJ0PR13CA0162.namprd13.prod.outlook.com (2603:10b6:a03:2c7::17) To DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9474:EE_|SA1PR12MB7442:EE_ X-MS-Office365-Filtering-Correlation-Id: 496b2d83-ad87-44cd-8701-08debc939f55 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|366016|56012099006|11063799006|5023799004|18002099003|22082099003|6133799003; X-Microsoft-Antispam-Message-Info: oyrF7x8cSTGPQIKpCWYfOEc29txbANXkuTztWd5YacJECuswyDpEjlHC93O+DG/AjPmgbsu0Es2lUXvBQtpAtpClvhDDk8T7I8vFp8r4gZy3WwFLgbrjtn7dwfW84QQ0C/0dmFc7Wfz8eD5lbh2kZCfkkQ0WoQuiGLhqtzHEZkFfI07mmRp1MqFOIgdDRvnVlmAJFDb42amSX7B0tjoSekXNk5+dwSO/nM3pJCeyJ17ss82tVS0Dkci//TUIDvFl/ByIYWQn+CAm3idhSlT4gkpqAXfJHBLTGsJJhsMvCelVO0hKcRNYNrrlMXs/O0j/tDQkqerXjqLjwNYXWCAOhc5FlJt7LNSpVUQjHKPvu2xUBiOvMph8/eaGgcsRnb9uSm9surZ6/FKh7cgLTKuwaY1hCBzDjiL7rXXkRHGOTqLEFyfGtcu6STCBECXR+oKOgEAvcsZLcnsI/oAWHZ/7mpTURlvbuVzHzVB/PlF+OU8ShzjbISyU5QBXqrJ1hZANZFwITgaKjhJ8GyTRVjYAq1i1wH2XSQAAM0c2a8t8cB+afCk2S9Du6aqKGjHSTjQik6NsOnJ4RKBOL0MNOjloa3O3hkx/m5bfCKjgEODpynDulEa21vqhlfplY0MDJLjMGmxkBsrX0Gv8Pq4MSlok+8eNfHN3/5b2pRGmL8brdIIAzRLFu5ZCwqwabcRTkTasxDuoZCLQGj86tEWtOXUhEA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9474.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(366016)(56012099006)(11063799006)(5023799004)(18002099003)(22082099003)(6133799003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?vWBhTS1tVp716KbYMslIP8+WZEorafaL3eFVn4/pLHNyacoiVCi5hfYf2+CR?= =?us-ascii?Q?Asy0657Yw5jASLeL4PMgiwiDB9GC6w/q0fDIia5eopRy/q5kHL0BD5KON3Sz?= =?us-ascii?Q?sbJHyu+mByg3q+oF79xB6Ful12oKjDwWUDYe6bkhFHt/BX6DLcHXg1hL3hwY?= =?us-ascii?Q?dSlLnnW/OJsTSFzjzVcBk5Mu00jnaAjSYFhRlJQCQP7rgT4EQqm0VyeSdeYf?= =?us-ascii?Q?Z+q1JYMmTOoiQd7UZ9NhJ+wdFleq1jm3StSEmjb+GQjhsSWR+f1cMmtrfEIS?= =?us-ascii?Q?779251Ny963ySnT2K4cNEaXUqV1EfpsMzt8QJhcj+JElQNCLy20+sHkFFNwt?= =?us-ascii?Q?mijJ6KcM8GHxUB8bTS1l+H+doO/249cRB/PJK3M92vJlTORQ8szy9GHB5nZ/?= =?us-ascii?Q?tdcoRWvErkKmhaGDnStFMIl1jsIxPoInkrBrVI8Ner0qyYvdT9IzWKV3clWB?= =?us-ascii?Q?czawYznnNjnG9F+tqiLLx7T7X6Cdwk2CbN3e7xMlJ45K0Iq7wjdyfYJL9n5Y?= =?us-ascii?Q?R1Pl1FqTH6oIKii8qjj3gv8BcJwG+iAF7Ao7FdksuZvxlzq5eKwHd5J1Dp+D?= =?us-ascii?Q?QgKztQF1csewjj2Ht5A94YHk5Om2kHFNmaYh3BT9ffZ43LjsWgmLsj6htowA?= =?us-ascii?Q?sAdUIRS3Wj5i7rHdYD44T20jj/DpofIcK6QsNVNV/iSegugu+m90EqIn9Sqc?= =?us-ascii?Q?ShXfT2yvSEpYTMFPQylDlf8eWCvI/z9nGcbvTZg3T0OlZGYYa4lCPs+eydF7?= =?us-ascii?Q?mKiLNwhxRQorpeJ6fCrgGk2IbwfxNN7kKOKEqL1g7f0ZWwz9mDt3amJ1k8NQ?= =?us-ascii?Q?aACNZs4FjRBBrnT55EkjA58F+XnYK2tIbfQ5Mj5eXT41EyIbTeb1mzHqzUsD?= =?us-ascii?Q?jeJ/nYYFH9Ibb9nL/FIIxL8/QStmev09mBYyYBe28zmpqgaSM9Zx54rka+ic?= =?us-ascii?Q?+SM+JFXDxHdi+TEpXtt/35AvRtCRRu+Be4ZmlVTQNGRevYNeAjiI/DL5J/VV?= =?us-ascii?Q?KG4+mTMto2WwKmDyu3ebxNkZe7taphDhFolW/59O3Xbb2thISKfTqlZQp6Zi?= =?us-ascii?Q?caUXpkIw7oaEjWH043UMvcJWCdn/cOhRVHNxzcgYp2YL7j0Hyi2mWNiB1sVx?= =?us-ascii?Q?2WhP+kL5jy6Ok86pwc0BV9qG1vd4spMlVdxYV8sq6M7zsaX8o8FQpEaPhQfP?= =?us-ascii?Q?W7pEV+t+L3fjwyOR3Hmf1/9tCo5Jnf5pxoWIS4778XJcqhJ/U+RkU1KXEjTM?= =?us-ascii?Q?ua8qtbm2UNCsey7IqV0V+07blMnXp21GjlEn3+LcX2FEvMdNOvzxcXjveTeU?= =?us-ascii?Q?vPwOlEk2eEBgPe5zniuk+e1kVUwu2vOXeF2OxJ1DvqmHGLFVkJwkJfwjl89w?= =?us-ascii?Q?aQaFLqjqqgUmZSPJW0/GWCg4k+oVBv0YN8lTmZ++pif93bROWW5dg/jvG0HW?= =?us-ascii?Q?cTc7iXqtDBL1AvU6AOJSWVqVGRTS0pJNfXwemf+zAjPwiMbr65YZifmrstIv?= =?us-ascii?Q?3HMqVZ/Fd59DFndNpzbji1kEuqiaBslZhN3xjNSqLxPE7nbQcyp7du4jth4w?= =?us-ascii?Q?DqqZbSNlYm3eoZQiwjSJPISc2HucwklZjkKOAJeKk5+1O0PnT+3KJMN++5UR?= =?us-ascii?Q?BoyFmbUtahbbunl/JEadzoD+O9QWhFuwCco6n/fz6dYKjmZOchhBV3VeT6F4?= =?us-ascii?Q?fl4MwFgvFs5A72vwGTf1liHTSp7/o+5c+lMBkfKQY7ZT4uTpCgX7GpU0MaPw?= =?us-ascii?Q?6AU6IzE3cw=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 496b2d83-ad87-44cd-8701-08debc939f55 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9474.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 May 2026 08:32:15.3529 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: hzO1VcDuelWqXxV1P4/4Npr/YLAar0wvYLa/zB1HvNZ4ynk5uGiRJbJjciOblbVLEZ3LezZf7a9Sea6LLqdDbQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7442 Content-Type: text/plain; charset="utf-8" Add the reset flow that coordinates the target function, affected CXL sibling functions, and any active memdevs in the CXL.cache/mem reset scope. The flow collects regions for the affected memdevs under cxl_rwsem.region, verifies that those regions are idle, flushes CPU caches for the affected ranges, saves and disables the target and sibling PCI functions, and locks active memdevs to revalidate that their endpoints are still present before reset. After the CXL DVSEC reset completes, restore PCI config space so CXL MMIO is accessible, restore decoder programming for all active affected memdevs, commit their restored decoders, and only then re-enable CXL.mem for the affected set. Signed-off-by: Srirangan Madhavan --- drivers/cxl/core/pci.c | 414 +++++++++++++++++++++++++++++++++++------ 1 file changed, 358 insertions(+), 56 deletions(-) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index c755c18c8d84..486c447e98f3 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -947,14 +947,12 @@ struct cxl_reset_region_context { struct xarray regions; }; =20 -static void __maybe_unused -cxl_reset_region_context_init(struct cxl_reset_region_context *ctx) +static void cxl_reset_region_context_init(struct cxl_reset_region_context = *ctx) { xa_init(&ctx->regions); } =20 -static void __maybe_unused -cxl_reset_region_context_destroy(struct cxl_reset_region_context *ctx) +static void cxl_reset_region_context_destroy(struct cxl_reset_region_conte= xt *ctx) { xa_destroy(&ctx->regions); } @@ -985,9 +983,8 @@ static int cxl_reset_collect_region(struct device *dev,= void *data) return cxl_reset_add_region(ctx, cxled->cxld.region); } =20 -static int __maybe_unused -cxl_reset_collect_memdev_regions(struct cxl_reset_region_context *ctx, - struct cxl_memdev *cxlmd) +static int cxl_reset_collect_memdev_regions(struct cxl_reset_region_contex= t *ctx, + struct cxl_memdev *cxlmd) { struct cxl_port *endpoint; =20 @@ -1045,8 +1042,7 @@ static int cxl_reset_validate_region_idle(struct cxl_= region *cxlr) return rc; } =20 -static int __maybe_unused -cxl_reset_validate_regions_idle(struct cxl_reset_region_context *ctx) +static int cxl_reset_validate_regions_idle(struct cxl_reset_region_context= *ctx) { struct cxl_region *cxlr; unsigned long index; @@ -1077,26 +1073,41 @@ static int cxl_reset_flush_region_cache(struct cxl_= region *cxlr) return rc; } =20 -static int __maybe_unused -cxl_reset_flush_cpu_caches(struct cxl_reset_region_context *ctx) +static int cxl_reset_cpu_cache_flush_preflight(struct cxl_reset_region_con= text *ctx, + bool *skip) { - struct cxl_region *cxlr; - unsigned long index; - int rc; + if (skip) + *skip =3D false; =20 if (xa_empty(&ctx->regions)) return 0; =20 - if (!cpu_cache_has_invalidate_memregion()) { - if (IS_ENABLED(CONFIG_CXL_REGION_INVALIDATION_TEST)) { - pr_info_once( - "Bypassing cpu_cache_invalidate_memregion() for testing!\n"); - return 0; - } - pr_warn("Failed to synchronize CPU cache state\n"); - return -ENXIO; + if (cpu_cache_has_invalidate_memregion()) + return 0; + + if (IS_ENABLED(CONFIG_CXL_REGION_INVALIDATION_TEST)) { + pr_info_once( + "Bypassing cpu_cache_invalidate_memregion() for testing!\n"); + if (skip) + *skip =3D true; + return 0; } =20 + pr_warn("Failed to synchronize CPU cache state\n"); + return -ENXIO; +} + +static int cxl_reset_flush_cpu_caches(struct cxl_reset_region_context *ctx) +{ + struct cxl_region *cxlr; + unsigned long index; + bool skip; + int rc; + + rc =3D cxl_reset_cpu_cache_flush_preflight(ctx, &skip); + if (rc || skip) + return rc; + xa_for_each(&ctx->regions, index, cxlr) { rc =3D cxl_reset_flush_region_cache(cxlr); if (rc) @@ -1120,7 +1131,11 @@ struct cxl_reset_context { struct pci_dev **siblings; int nr_siblings; int sibling_capacity; + int nr_siblings_locked; int nr_siblings_prepared; + bool target_locked; + bool target_saved; + bool target_iommu_prepared; }; =20 struct cxl_reset_walk_ctx { @@ -1306,7 +1321,7 @@ static int cxl_reset_collect_pci_memdev(struct cxl_re= set_context *ctx, * function set to find memdevs whose regions and endpoint decoder state m= ust * be handled around the reset. */ -static int __maybe_unused cxl_reset_collect_memdevs(struct cxl_reset_conte= xt *ctx) +static int cxl_reset_collect_memdevs(struct cxl_reset_context *ctx) { int rc, i; =20 @@ -1323,7 +1338,7 @@ static int __maybe_unused cxl_reset_collect_memdevs(s= truct cxl_reset_context *ct return 0; } =20 -static int __maybe_unused +static int cxl_reset_collect_regions(struct cxl_reset_context *ctx, struct cxl_reset_region_context *region_ctx) { @@ -1370,7 +1385,7 @@ static void cxl_reset_unlock_memdevs(struct cxl_reset= _context *ctx) } } =20 -static int __maybe_unused cxl_reset_lock_memdevs(struct cxl_reset_context = *ctx) +static int cxl_reset_lock_memdevs(struct cxl_reset_context *ctx) { int i; =20 @@ -1400,7 +1415,7 @@ static int __maybe_unused cxl_reset_lock_memdevs(stru= ct cxl_reset_context *ctx) return -EAGAIN; } =20 -static void __maybe_unused cxl_reset_put_memdevs(struct cxl_reset_context = *ctx) +static void cxl_reset_put_memdevs(struct cxl_reset_context *ctx) { int i; =20 @@ -1417,14 +1432,20 @@ static void cxl_pci_functions_reset_done(struct cxl= _reset_context *ctx) { int i; =20 + /* + * Config state was restored early for CXL MMIO access. Complete PCI + * reset recovery here by unblocking IOMMU and running reset_done(). + */ for (i =3D ctx->nr_siblings_prepared - 1; i >=3D 0; i--) { struct pci_dev *sibling =3D ctx->siblings[i]; =20 pci_dev_reset_iommu_done(sibling); pci_dev_restore(sibling); - pci_dev_unlock(sibling); } =20 + for (i =3D ctx->nr_siblings_locked - 1; i >=3D 0; i--) + pci_dev_unlock(ctx->siblings[i]); + for (i =3D 0; i < ctx->nr_siblings; i++) pci_dev_put(ctx->siblings[i]); =20 @@ -1432,31 +1453,39 @@ static void cxl_pci_functions_reset_done(struct cxl= _reset_context *ctx) ctx->siblings =3D NULL; ctx->nr_siblings =3D 0; ctx->sibling_capacity =3D 0; + ctx->nr_siblings_locked =3D 0; ctx->nr_siblings_prepared =3D 0; } =20 -static int __maybe_unused -cxl_pci_functions_reset_prepare(struct cxl_reset_context *ctx) +static int cxl_pci_functions_lock(struct cxl_reset_context *ctx) { - int rc, i; - - ctx->siblings =3D NULL; - ctx->nr_siblings =3D 0; - ctx->sibling_capacity =3D 0; - ctx->nr_siblings_prepared =3D 0; + int i; =20 - rc =3D cxl_reset_collect_siblings(ctx); - if (rc) - goto err; + ctx->nr_siblings_locked =3D 0; =20 for (i =3D 0; i < ctx->nr_siblings; i++) { struct pci_dev *sibling =3D ctx->siblings[i]; =20 if (!pci_dev_trylock(sibling)) { - rc =3D -EAGAIN; - goto err; + cxl_pci_functions_reset_done(ctx); + return -EAGAIN; } =20 + ctx->nr_siblings_locked++; + } + + return 0; +} + +static int cxl_pci_functions_reset_prepare(struct cxl_reset_context *ctx) +{ + int rc, i; + + ctx->nr_siblings_prepared =3D 0; + + for (i =3D 0; i < ctx->nr_siblings_locked; i++) { + struct pci_dev *sibling =3D ctx->siblings[i]; + pci_dev_save_and_disable(sibling); rc =3D pci_dev_reset_iommu_prepare(sibling); if (rc) { @@ -1469,7 +1498,6 @@ cxl_pci_functions_reset_prepare(struct cxl_reset_cont= ext *ctx) * nr_siblings_prepared and must not get iommu_done(). */ pci_dev_restore(sibling); - pci_dev_unlock(sibling); goto err; } =20 @@ -1483,6 +1511,79 @@ cxl_pci_functions_reset_prepare(struct cxl_reset_con= text *ctx) return rc; } =20 +/* + * Restore PCI config space after reset so CXL MMIO is accessible for memd= ev + * restore. Driver reset_done callbacks remain deferred to final cleanup. + */ +static void cxl_pci_functions_restore_state(struct cxl_reset_context *ctx) +{ + int i; + + for (i =3D ctx->nr_siblings_prepared - 1; i >=3D 0; i--) + pci_restore_state(ctx->siblings[i]); +} + +static int cxl_pci_target_lock(struct cxl_reset_context *ctx) +{ + struct pci_dev *pdev =3D ctx->target; + + if (!pci_dev_trylock(pdev)) + return -EAGAIN; + + ctx->target_locked =3D true; + return 0; +} + +static int cxl_pci_target_reset_prepare(struct cxl_reset_context *ctx) +{ + struct pci_dev *pdev =3D ctx->target; + int rc; + + /* Disable first to stop new transactions, then drain in-flight ones. */ + pci_dev_save_and_disable(pdev); + ctx->target_saved =3D true; + + if (!pci_wait_for_pending_transaction(pdev)) + pci_err(pdev, "timed out waiting for pending transactions\n"); + + rc =3D pci_dev_reset_iommu_prepare(pdev); + if (rc) { + pci_err(pdev, "failed to block IOMMU for CXL reset: %d\n", rc); + return rc; + } + + ctx->target_iommu_prepared =3D true; + return 0; +} + +static void cxl_pci_target_restore_state(struct cxl_reset_context *ctx) +{ + if (ctx->target_saved) + pci_restore_state(ctx->target); +} + +static void cxl_pci_target_reset_done(struct cxl_reset_context *ctx) +{ + if (ctx->target_iommu_prepared) { + pci_dev_reset_iommu_done(ctx->target); + ctx->target_iommu_prepared =3D false; + } + + /* + * cxl_pci_target_restore_state() restores config space before memdev + * restore. Complete PCI reset recovery here with reset_done(). + */ + if (ctx->target_saved) { + pci_dev_restore(ctx->target); + ctx->target_saved =3D false; + } + + if (ctx->target_locked) { + pci_dev_unlock(ctx->target); + ctx->target_locked =3D false; + } +} + static int cxl_reset_update_ctrl2(struct pci_dev *pdev, int dvsec, u16 set, u16 clear) { @@ -1599,7 +1700,7 @@ static int cxl_reset_wait_done(struct pci_dev *pdev, = int dvsec, u16 cap) return 0; } =20 -static int __maybe_unused cxl_dev_reset(struct pci_dev *pdev, bool mem_cle= ar) +static int cxl_dev_reset(struct pci_dev *pdev, bool mem_clear) { int dvsec, rc; u16 ctrl2_clear =3D 0; @@ -1620,19 +1721,9 @@ static int __maybe_unused cxl_dev_reset(struct pci_d= ev *pdev, bool mem_clear) if (mem_clear && !(cap & PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE)) return -EOPNOTSUPP; =20 - if (!pci_wait_for_pending_transaction(pdev)) - pci_err(pdev, "timed out waiting for pending transactions\n"); - - rc =3D pci_dev_reset_iommu_prepare(pdev); - if (rc) { - pci_err(pdev, "failed to block IOMMU for CXL reset: %d\n", - rc); - return rc; - } - rc =3D cxl_reset_disable_cache(pdev, dvsec, cap); if (rc) - goto out_iommu; + return rc; if (cap & PCI_DVSEC_CXL_CACHE_CAPABLE) ctrl2_clear |=3D PCI_DVSEC_CXL_DISABLE_CACHING; =20 @@ -1651,7 +1742,7 @@ static int __maybe_unused cxl_dev_reset(struct pci_de= v *pdev, bool mem_clear) =20 rc =3D cxl_reset_wait_done(pdev, dvsec, cap); if (rc) - goto out_iommu; + return rc; =20 rc =3D cxl_reset_update_ctrl2(pdev, dvsec, 0, PCI_DVSEC_CXL_DISABLE_CACHING); @@ -1660,7 +1751,218 @@ static int __maybe_unused cxl_dev_reset(struct pci_= dev *pdev, bool mem_clear) if (rc && ctrl2_clear) cxl_reset_update_ctrl2(pdev, dvsec, 0, ctrl2_clear); =20 -out_iommu: - pci_dev_reset_iommu_done(pdev); + return rc; +} + +static int cxl_reset_restore_memdev(struct cxl_reset_memdev *rmd) +{ + struct cxl_memdev *cxlmd =3D rmd->cxlmd; + int rc; + + if (!rmd->active) + return 0; + + rc =3D cxl_restore_memdev_decoders(cxlmd); + if (rc) + dev_err(&cxlmd->dev, + "Failed to restore CXL.mem decoders after reset: %d\n", + rc); + + return rc; +} + +static int cxl_reset_commit_memdev(struct cxl_reset_memdev *rmd) +{ + struct cxl_memdev *cxlmd =3D rmd->cxlmd; + int rc; + + if (!rmd->active) + return 0; + + rc =3D cxl_commit_memdev_decoders(cxlmd); + if (rc) + dev_err(&cxlmd->dev, + "Failed to commit CXL.mem decoders after reset: %d\n", + rc); + + return rc; +} + +static int cxl_reset_enable_memdev(struct cxl_reset_memdev *rmd) +{ + struct cxl_memdev *cxlmd =3D rmd->cxlmd; + struct cxl_dev_state *cxlds =3D cxlmd->cxlds; + int rc; + + if (!rmd->active) + return 0; + + cxlds->media_ready =3D false; + + rc =3D cxl_set_mem_enable(cxlds, PCI_DVSEC_CXL_MEM_ENABLE); + if (rc < 0) { + dev_err(&cxlmd->dev, + "Failed to enable CXL.mem after reset: %d\n", rc); + return rc; + } + + rc =3D cxl_await_media_ready(cxlds); + if (rc) { + dev_err(&cxlmd->dev, + "Media not active after CXL reset: %d\n", rc); + return rc; + } + cxlds->media_ready =3D true; + + return 0; +} + +static void cxl_reset_disable_memdevs(struct cxl_reset_context *ctx) +{ + int rc, i; + + for (i =3D ctx->nr_memdevs - 1; i >=3D 0; i--) { + struct cxl_memdev *cxlmd =3D ctx->memdevs[i].cxlmd; + + if (!ctx->memdevs[i].active) + continue; + + rc =3D cxl_set_mem_enable(cxlmd->cxlds, 0); + if (rc < 0) + dev_err(&cxlmd->dev, + "Failed to disable CXL.mem after reset restore failure; device state m= ay be inconsistent: %d\n", + rc); + } +} + +static int cxl_reset_restore_memdevs(struct cxl_reset_context *ctx) +{ + int rc; + int i; + + lockdep_assert_held_write(&cxl_rwsem.region); + + for (i =3D 0; i < ctx->nr_memdevs; i++) { + rc =3D cxl_reset_restore_memdev(&ctx->memdevs[i]); + if (rc) + return rc; + } + + for (i =3D 0; i < ctx->nr_memdevs; i++) { + rc =3D cxl_reset_commit_memdev(&ctx->memdevs[i]); + if (rc) + return rc; + } + + for (i =3D 0; i < ctx->nr_memdevs; i++) { + rc =3D cxl_reset_enable_memdev(&ctx->memdevs[i]); + if (rc) { + cxl_reset_disable_memdevs(ctx); + return rc; + } + } + + return 0; +} + +static void cxl_reset_context_destroy(struct cxl_reset_context *ctx) +{ + /* + * LIFO unwind for regular completion and partial initialization: + * memdevs, sibling functions, target function, then references. + * Each cleanup helper tolerates being called after its state was + * already released on an earlier error path. + */ + cxl_reset_unlock_memdevs(ctx); + cxl_pci_functions_reset_done(ctx); + cxl_pci_target_reset_done(ctx); + cxl_reset_put_memdevs(ctx); +} + +static int cxl_do_reset_locked(struct cxl_reset_context *ctx, bool mem_cle= ar) +{ + struct cxl_reset_region_context region_ctx; + int rc; + + lockdep_assert_held_write(&cxl_rwsem.region); + + cxl_reset_region_context_init(®ion_ctx); + + rc =3D cxl_reset_collect_regions(ctx, ®ion_ctx); + if (rc) + goto out; + + rc =3D cxl_pci_target_lock(ctx); + if (rc) + goto out; + + rc =3D cxl_pci_functions_lock(ctx); + if (rc) + goto out; + + rc =3D cxl_reset_lock_memdevs(ctx); + if (rc) + goto out; + + rc =3D cxl_reset_cpu_cache_flush_preflight(®ion_ctx, NULL); + if (rc) + goto out; + + rc =3D cxl_reset_validate_regions_idle(®ion_ctx); + if (rc) + goto out; + + rc =3D cxl_reset_flush_cpu_caches(®ion_ctx); + if (rc) + goto out; + + rc =3D cxl_pci_target_reset_prepare(ctx); + if (rc) + goto out; + + rc =3D cxl_pci_functions_reset_prepare(ctx); + if (rc) + goto out; + + rc =3D cxl_dev_reset(ctx->target, mem_clear); + + cxl_pci_target_restore_state(ctx); + cxl_pci_functions_restore_state(ctx); + + if (!rc) + rc =3D cxl_reset_restore_memdevs(ctx); + + cxl_reset_unlock_memdevs(ctx); + +out: + cxl_reset_region_context_destroy(®ion_ctx); + return rc; +} + +static int __maybe_unused cxl_do_reset(struct pci_dev *pdev, bool mem_clea= r) +{ + struct cxl_reset_context ctx =3D { + .target =3D pdev, + }; + int rc; + + /* + * Snapshot the CXL r3.2 9.7 device reset scope before taking + * cxl_rwsem.region. Hot-added functions after this point are not + * coordinated by this reset operation. + */ + rc =3D cxl_reset_collect_siblings(&ctx); + if (rc) + goto out; + + rc =3D cxl_reset_collect_memdevs(&ctx); + if (rc) + goto out; + + scoped_guard(rwsem_write, &cxl_rwsem.region) + rc =3D cxl_do_reset_locked(&ctx, mem_clear); + +out: + cxl_reset_context_destroy(&ctx); return rc; } --=20 2.43.0 From nobody Mon Jun 8 15:36:52 2026 Received: from CH5PR02CU005.outbound.protection.outlook.com (mail-northcentralusazon11012066.outbound.protection.outlook.com [40.107.200.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24B8B385D66; Thu, 28 May 2026 08:32:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.200.66 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957147; cv=fail; b=c4CAtnX8P5Lb0ETryf6haC9YZVaSQcW/zdHfzFr1vmRU3WjPwD7GYedh7ZCvYQZ4wLiHB6rLyabiocv4OFYsUxxM9gWIsFYsieoCVyxiwsQfvxHYc298sIIfB7v7zng9y0v7bsFPrr9UwtfPkP++kPlrUmacDSVkpzrD52Mav0M= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957147; c=relaxed/simple; bh=r0oP8HspGixxaRVdgvaVdfubPlpl7drX92P8NRov32U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=FILVCFQ1l4GC8JmbmZfm17y1dAY/7RgY9aSPpRa74qIkvR2m0oGugBd9ayISbYn141EbsbaZaUMTbQEAwf/sqlS1dxghgKiBgoz97qW+8lAxULTB+qSzaovxjQ8/gbtudUaadjjpQ27ZYBZJyEat4D869+HJxD4CcHcYp/BY9FU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=qhp3btRG; arc=fail smtp.client-ip=40.107.200.66 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="qhp3btRG" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=uPPXiYeaoHSdlFPRVY0YBb6I7I6krz0dcUfzpc00Uq4cJa6Jd1nMHgP1Jgo8hCrz++Pftwt5CaaHTmgpbsEna7qz+FFMf62g2vlveJzEBwmuMTuSP3oli1Lzv74tocWey99wwkD5tnAAfwQeCxfTkculDJW216mZoKC+EwF8iySlTCooxgGf3Yh4tHSYFFcqYOORRo6nuwzYbleZ8LxnRoYIDuO07R4ZWC9zBZMTFEFyKIbPSJkN26GIJZnDU80HUOhko/nJyFU0G2KBVhHaZ17JEhtHfT9n81HONsTKZFVRN5kuLgD5sj9PsllOa80S1xqlz8rxZ7XxsrQqhmTv+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1v9MgZ1BuMBnzAarKA2NXV7LfL4EkMn5RcuUJrPrn34=; b=Xs6MvEzZ6dvfaRV1diWe4U/hfoKU4AzrHD32t1ymbLnDfH2CS4u2dWu276RGKp94QB14VBoTGMvlkWG3qYUw9babvZJL9tTdmhA6eOvJ/uVP8hTrsQswpD2cvhklf3g4LkzA/wIfedNKaI28UfSWBlM+T+IgQ7Xp4iZ0rFHQxNPoEGVTefzJUTzWLIhElOJITEgjtdRwS+8KG8Yx0iYXiA8Vn8GzTcjNf6tQjnIafLyB+GS60ahr/ei5UtjRlSA75vOPHhKaJSXYbChSLCRfwhKvBd8xiyel/EhZJPzvp6JbbCbd1sHPkoB2cZA2v/J5DrOtdct43aN5QroX7iKbkA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1v9MgZ1BuMBnzAarKA2NXV7LfL4EkMn5RcuUJrPrn34=; b=qhp3btRGt3/0uiu8Oov8UkO4P59vrDDz0z9T07Wl5qqg1NwyrpFfsZvfCEcOrXmL/mrbcIDxR1U0JxMHZs5IVWSkgMiOkL+CDPn8WkKCfgT7OG18RJyoPWsAaqT5lf3DH927zgrPJG5VI7U5IhpvYFs4YtZYi2akMMY4Gqr2b1HO8WJPK0b5EXSuUxoI/bQyGcc3T7E9MPmvARWIryD84esgGd3hgSVrDF+WXbeaBRTAlibRFYtOlgiuGbXwvB4rO3L0+s/KOl/koBnWn51TqXFf4atGN3RHhz3r7IVQFX2SXM0l6VitC8l6afRuE4tTexHGACLItbflfUdxSI4Paw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) by SA1PR12MB7442.namprd12.prod.outlook.com (2603:10b6:806:2b5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.12; Thu, 28 May 2026 08:32:17 +0000 Received: from DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7]) by DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7%6]) with mapi id 15.21.0071.011; Thu, 28 May 2026 08:32:17 +0000 From: Srirangan Madhavan To: linux-cxl@vger.kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Cc: vsethi@nvidia.com, alwilliamson@nvidia.com, Dan Williams , Sai Yashwanth Reddy Kancherla , Vishal Aslot , Manish Honap , Jiandi An , Richard Cheng , linux-tegra@vger.kernel.org, Srirangan Madhavan Subject: [PATCH v6 8/9] cxl/memdev: Add cxl_reset sysfs attribute Date: Thu, 28 May 2026 08:31:53 +0000 Message-ID: <20260528083154.137979-9-smadhavan@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260528083154.137979-1-smadhavan@nvidia.com> References: <20260528083154.137979-1-smadhavan@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SJ0PR13CA0160.namprd13.prod.outlook.com (2603:10b6:a03:2c7::15) To DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9474:EE_|SA1PR12MB7442:EE_ X-MS-Office365-Filtering-Correlation-Id: 3af920c7-52cf-42f9-74ef-08debc93a0b1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|366016|56012099006|11063799006|5023799004|18002099003|22082099003|3023799007|6133799003; X-Microsoft-Antispam-Message-Info: qdww30NM5l/xTvr2xs2fh6L0QYRewLU+PXQ9dZ3vWQ4ivKlJkSSzK9rcRC6ixnBfLD4hRaJEAeV4oORmH5QXYsPxVrxN5laoRNgWnW64NiZ73yKjTOki0Qs2Zy3Nzlg5PqbHuHh+cSJ5vQ2wA7rd4apan9HRsQaMuwVxWJv7b9eHqlIbx4983QjcATREI0fJruTFHXxU/d0xiWnuk2HV2cfW3Uhvgx/Qh9zrgb/iqaXomkar/nRAggv3dXZ/xL0FnPxJD6U6ccBKbmGkFSz3Qqj6lsuhF5/U+JOs8G8Nq30mjX4tifM4xWlADGVmGOwLg+DI/v/IzA4BRe3JRqaHQnCPlqjfFpv+QNnjmOLiwGFET/A7O3k0RKHfiTi3zc/vBp/Pk+mCG05BEVkgr/+W3CZXc6ys7S6kEELVKIGJ97tE1UZE4/aQV6B/asPDFo434Ire/oJQhevb/So1sMv842ah6AOnXI38VEOuNsPIRTL6t+fbpZ2DppQZ//N0S7gbL7j/ZRiCv3E0v+UZk9D/UB++gYdzpPpxVwEbGxsTVvovKortdpnzxCAIEX5T4QRmskFtLfjCwsN/m7IaC0PBPHhqrv/1AiLme4ox2ll9Iu7gB0vZgqBvBnkyfE3Pv8XY31wVp9srROU5ehP59H0sipU3vQMWSKVFCt9mbOB5Bofo5FKSQ+z3VcCzLTZ/d6hGuIPa9oQV8Z+fE+VQnBGMwg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9474.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(366016)(56012099006)(11063799006)(5023799004)(18002099003)(22082099003)(3023799007)(6133799003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?WvT9JPo71IktQdPjtSV+4YZiBp0TA9aTKwr/rFQWTV6UHDWxLNjhTsB193T2?= =?us-ascii?Q?Ne0+9BOzfpfFWitoocwuQEEVB3V30i3lU+B6/+v5RX8vE4x7BdYJs0U1i4Mk?= =?us-ascii?Q?QUTdULeXDpA+FfUISm4z9x/32CqX3GfvUBuN2uFOKgn58yPpTwO1Q/EjTGUG?= =?us-ascii?Q?mHPWEbAk8zPWQFgl6BMFuOTOUIFYCSgQ0PR0UAyTo6bTZ3vsKvpVNqi389tj?= =?us-ascii?Q?cSxiseR84YPskXtX84mUsSX5m8hmBXtE7Co93IwVfED4VRFoNtvruvBTrQFv?= =?us-ascii?Q?EYsidkdLrrDR3FhB63mQDi+QtbsMyyaqlDyJgNBpkFDZS+5LnWKTzH7bORZp?= =?us-ascii?Q?Su53+9WsesORNRoJvC7k9cCVftlFpexgy0IyUrkHt8LFaiqHYZocd+GaFY8m?= =?us-ascii?Q?Er2PsBAWU+a2m19lSsWIkPOqfKWzg0NzQAkDN9RWBlADN+vSLDmW82HVKJ4G?= =?us-ascii?Q?8BLOO9XZOJigzU1glGYjdRlPaqKNgFYYnE+okHEhEiOn85CNaJ1B5ArrP5Td?= =?us-ascii?Q?zRZkv2TnMR9yvhNHG8l9kS4sCPZjyBwL7aPHW3dYT2pbVRqxVldLp8NAfHqt?= =?us-ascii?Q?IKpWxJq7P+XSnp/+eg9tJeTeNkpQRTDE7WQJLwKWP6Ns6DHRfApTac6a6AO4?= =?us-ascii?Q?q1VgB7hfWG3qMZmE8klWGhgeA9QmuJPu2J6a9FWudEKUHAcnW6OGqJ8l5EEk?= =?us-ascii?Q?IndbTc83MFqJwZwU7I+A9xpLIOC2ST8sdnXih4PkfpxV/KG6LuUXgInUGVrs?= =?us-ascii?Q?EOvIgtEcLoAl/jTxPUBp5CbI1bJu+gIJR6cexj0tXBTXMoYnG8mlCvhy4G2A?= =?us-ascii?Q?75IwuHzjC88M8vOEWRtd5jasI7u8lVT0tulWty97DgXolGMWXzEI3+d6RoFP?= =?us-ascii?Q?MVODBM18L2aqAcaYRkkrZU/KYTE+7evfWQGdmWNwAKd6uxYDmFFWZYHDXuX2?= =?us-ascii?Q?2FpViEAIUYFAahT3AtFYX683xPBQWxH5l78YaFCfctmLMjZT8u6ZQ04t5rlx?= =?us-ascii?Q?S+YwltPzr9ccKxOX2KcJXno+Fu6U2oBoXX9FQKOO0K9J0HDpHFW+OrmY4QKj?= =?us-ascii?Q?Km28ORncGaHzBOb90JEh6G12fh67bQiGwIf0eyXvZv6iPxETGAR8WvQdtgWl?= =?us-ascii?Q?ewJU+SCCLk/Q9JkPZjLNplZfp38Yxbv8uAL+QeABKdqkAU2g5GXEB0pwqywQ?= =?us-ascii?Q?1l8mZzwa3RGLo1jUExAqObMTxWFt98v6My4zxrRsArQ4E845MWW7yFJUipUf?= =?us-ascii?Q?1qbNFAJkOrUlJmpfnOGGLYtDNpwA5pTjC5i/eaiQzhwLk0UpOTreN9w0FgFT?= =?us-ascii?Q?9+YU//PeSfwaHosnEUL+08rlGDfn7JJZ1gqUkIEA6YcRbRXEcspybirUFV+Y?= =?us-ascii?Q?xU+gxeCp1uGzrpS7HYE/53hq0+oyveJUJ21ZG/djCM204C+pOuIBmp+5pYMy?= =?us-ascii?Q?v+NgUu5XVvJasOmqKG6H3lAzp0foxY+A+LkuI8ofCJJ+aMg7btTbOUEN6KSe?= =?us-ascii?Q?1G3mV4MHfvxwoid33k4D4bQVWjBrG7Yof6weIq3itshNQZWnF29icbAFGt8H?= =?us-ascii?Q?FM4m4e7aGqrepdsTbZyS2rv87AFmJW4JaiZjLutRQZZZo/ix3QJz9qq6X1Zn?= =?us-ascii?Q?PZpMFIHXw9dg0vipCXRKqz586pwysAyMt+YajTtg+s7r7wYvXrT2QVZ9wF0d?= =?us-ascii?Q?em9FqK28F8i/zGFyTAzPgbe7Fdglb7DdlznsNaxp4Dg0bQFf8Sgh3T2O1M5J?= =?us-ascii?Q?AWwbRHCrZg=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3af920c7-52cf-42f9-74ef-08debc93a0b1 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9474.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 May 2026 08:32:17.6731 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Na2rtTu6ln2rnb4x/AkE5lQpLZROsR7IR/K8V/8naisv/RyzbasUn8KDVbcIaL62piBJUMzx8oHg1qiBNzhz5w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7442 Content-Type: text/plain; charset="utf-8" Expose CXL reset through the CXL memdev device. The reset flow depends on CXL memdev state to identify affected regions, coordinate decoder restore, and keep CXL-specific policy out of the PCI sysfs ABI. Add a write-only cxl_reset attribute under memX. The attribute is visible only when the memdev's PCI parent advertises CXL Reset capability. Writing a true boolean value invokes the CXL reset orchestration. Signed-off-by: Srirangan Madhavan --- drivers/cxl/core/memdev.c | 30 +++++++++++ drivers/cxl/core/pci.c | 102 +++++++++++++++++++++++++++++++++++++- drivers/cxl/cxl.h | 3 ++ drivers/cxl/cxlmem.h | 2 + 4 files changed, 136 insertions(+), 1 deletion(-) diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 80e65690eb77..af67fa3d11b8 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -199,6 +199,26 @@ static ssize_t security_erase_store(struct device *dev, static struct device_attribute dev_attr_security_erase =3D __ATTR(erase, 0200, NULL, security_erase_store); =20 +static ssize_t cxl_reset_store(struct device *dev, + struct device_attribute *attr, const char *buf, + size_t len) +{ + struct cxl_memdev *cxlmd =3D to_cxl_memdev(dev); + bool reset; + int rc; + + rc =3D kstrtobool(buf, &reset); + if (rc) + return rc; + + if (!reset) + return -EINVAL; + + rc =3D cxl_memdev_reset(cxlmd); + return rc ? rc : len; +} +static DEVICE_ATTR_WO(cxl_reset); + bool cxl_memdev_has_poison_cmd(struct cxl_memdev *cxlmd, enum poison_cmd_enabled_bits cmd) { @@ -421,6 +441,7 @@ static struct attribute *cxl_memdev_attributes[] =3D { &dev_attr_payload_max.attr, &dev_attr_label_storage_size.attr, &dev_attr_numa_node.attr, + &dev_attr_cxl_reset.attr, NULL, }; =20 @@ -485,8 +506,16 @@ static struct attribute *cxl_memdev_security_attribute= s[] =3D { static umode_t cxl_memdev_visible(struct kobject *kobj, struct attribute *= a, int n) { + struct device *dev =3D kobj_to_dev(kobj); + struct cxl_memdev *cxlmd =3D to_cxl_memdev(dev); + if (!IS_ENABLED(CONFIG_NUMA) && a =3D=3D &dev_attr_numa_node.attr) return 0; + + if (a =3D=3D &dev_attr_cxl_reset.attr && + !cxl_memdev_reset_capable(cxlmd)) + return 0; + return a->mode; } =20 @@ -1099,6 +1128,7 @@ static int cxlmd_add(struct cxl_memdev *cxlmd, struct= cxl_dev_state *cxlds) =20 cxlmd->cxlds =3D cxlds; cxlds->cxlmd =3D cxlmd; + cxl_memdev_init_reset(cxlmd); =20 rc =3D cdev_device_add(&cxlmd->cdev, &cxlmd->dev); if (rc) { diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 486c447e98f3..09f016544d24 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -1207,6 +1207,22 @@ static bool cxl_reset_has_cache_or_mem(struct pci_de= v *pdev) return cap & (PCI_DVSEC_CXL_CACHE_CAPABLE | PCI_DVSEC_CXL_MEM_CAPABLE); } =20 +static bool cxl_reset_is_type2(struct pci_dev *pdev) +{ + u16 dvsec, cap; + + dvsec =3D pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL, + PCI_DVSEC_CXL_DEVICE); + if (!dvsec) + return false; + + if (pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap)) + return false; + + return (cap & PCI_DVSEC_CXL_CACHE_CAPABLE) && + (cap & PCI_DVSEC_CXL_MEM_CAPABLE); +} + static int cxl_reset_add_sibling(struct cxl_reset_context *ctx, struct pci_dev *sibling) { @@ -1939,7 +1955,7 @@ static int cxl_do_reset_locked(struct cxl_reset_conte= xt *ctx, bool mem_clear) return rc; } =20 -static int __maybe_unused cxl_do_reset(struct pci_dev *pdev, bool mem_clea= r) +static int cxl_do_reset(struct pci_dev *pdev, bool mem_clear) { struct cxl_reset_context ctx =3D { .target =3D pdev, @@ -1966,3 +1982,87 @@ static int __maybe_unused cxl_do_reset(struct pci_de= v *pdev, bool mem_clear) cxl_reset_context_destroy(&ctx); return rc; } + +static struct pci_dev *cxl_reset_get_fn0(struct pci_dev *pdev) +{ + unsigned int devfn; + + /* + * CXL Reset control/status is exposed in Function 0 and affects all + * CXL.cache/mem functions in the device. + */ + if (pci_ari_enabled(pdev->bus)) + devfn =3D 0; + else + devfn =3D PCI_DEVFN(PCI_SLOT(pdev->devfn), 0); + + if (pdev->devfn =3D=3D devfn) + return pci_dev_get(pdev); + + return pci_get_slot(pdev->bus, devfn); +} + +static bool cxl_memdev_probe_reset_capable(struct cxl_memdev *cxlmd) +{ + struct device *dev =3D cxlmd->dev.parent; + struct pci_dev *pdev, *fn0; + int dvsec; + u16 cap; + + if (!dev || !dev_is_pci(dev)) + return false; + + pdev =3D to_pci_dev(dev); + if (!cxl_reset_is_type2(pdev)) + return false; + + fn0 =3D cxl_reset_get_fn0(pdev); + if (!fn0) + return false; + + dvsec =3D pci_find_dvsec_capability(fn0, PCI_VENDOR_ID_CXL, + PCI_DVSEC_CXL_DEVICE); + if (!dvsec) + goto out; + + if (pci_read_config_word(fn0, dvsec + PCI_DVSEC_CXL_CAP, &cap)) + goto out; + + pci_dev_put(fn0); + return cap & PCI_DVSEC_CXL_RST_CAPABLE; + +out: + pci_dev_put(fn0); + return false; +} + +void cxl_memdev_init_reset(struct cxl_memdev *cxlmd) +{ + cxlmd->reset_capable =3D cxl_memdev_probe_reset_capable(cxlmd); +} + +bool cxl_memdev_reset_capable(struct cxl_memdev *cxlmd) +{ + return cxlmd->reset_capable; +} + +int cxl_memdev_reset(struct cxl_memdev *cxlmd) +{ + struct device *dev =3D cxlmd->dev.parent; + struct pci_dev *fn0; + int rc; + + if (!cxl_memdev_reset_capable(cxlmd)) + return -EOPNOTSUPP; + + if (!dev || !dev_is_pci(dev)) + return -ENODEV; + + fn0 =3D cxl_reset_get_fn0(to_pci_dev(dev)); + if (!fn0) + return -ENODEV; + + rc =3D cxl_do_reset(fn0, false); + pci_dev_put(fn0); + return rc; +} diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index b51b1e9d6400..bf65996e24dc 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -796,6 +796,9 @@ int cxl_dvsec_rr_decode(struct cxl_dev_state *cxlds, struct cxl_endpoint_dvsec_info *info); int cxl_restore_memdev_decoders(struct cxl_memdev *cxlmd); int cxl_commit_memdev_decoders(struct cxl_memdev *cxlmd); +void cxl_memdev_init_reset(struct cxl_memdev *cxlmd); +bool cxl_memdev_reset_capable(struct cxl_memdev *cxlmd); +int cxl_memdev_reset(struct cxl_memdev *cxlmd); =20 bool is_cxl_region(struct device *dev); =20 diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index 776c50d1db51..c8e7349fb130 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -48,6 +48,7 @@ struct cxl_memdev_attach { * @cxl_nvd: optional bridge to an nvdimm if the device supports pmem * @endpoint: connection to the CXL port topology for this memory device * @attach: creator of this memdev depends on CXL link attach to operate + * @reset_capable: cached CXL Reset support * @id: id number of this memdev instance. * @depth: endpoint port depth * @scrub_cycle: current scrub cycle set for this device @@ -65,6 +66,7 @@ struct cxl_memdev { struct cxl_nvdimm *cxl_nvd; struct cxl_port *endpoint; const struct cxl_memdev_attach *attach; + bool reset_capable; int id; int depth; u8 scrub_cycle; --=20 2.43.0 From nobody Mon Jun 8 15:36:52 2026 Received: from CH5PR02CU005.outbound.protection.outlook.com (mail-northcentralusazon11012066.outbound.protection.outlook.com [40.107.200.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8DD1B386C36; Thu, 28 May 2026 08:32:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.200.66 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957149; cv=fail; b=F9i3gciamoMf5kBcXjEcF7S545tnGagiqgDhjc3oNwP4vXGx5k3ikWMkltGYe40m87Kgc++nM6tt/hpM8dUqpnR2kU6Rr71TokLT5cexX0PvYcGH/AYMwlOELKSZiUsL4s5MQi4vahU4KoUoqlCaOlc7Jlq9gYgc9cVixoplZ3k= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779957149; c=relaxed/simple; bh=k7dnzyqnbjb4EhmF9QDZ1LYbH35pP5JQWKYSDDQKhrs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=czaS2lmXSfI44BqOpsp4g9561WtUk2TUEijg7ttlGohZrQrzRh625VsWUPiuRFwERhO2JT/Tx5H942iAm6qz3HyEoY1ZTa6Fuu7Tu+bwvHfEHT+spDxndSh3+AYXt5cpVvBdyS/InvWNw02TPxJM5I90SwTC8Ts0BLTqTNxAOg0= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=L0/Vd6i9; arc=fail smtp.client-ip=40.107.200.66 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="L0/Vd6i9" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=i3qxYsFmj3KJDaMFZmPJ8bnYGirwyHELK5fLSvSTIQclPW8xjHP4Kglx4qcf9u92OKv9Xgw2EQ41+vG372Txxo8YIyxxEF7HuB2BwPRr+KH+WRxzdAAqpcfbROzqvhxMgGufNMFa1feG2QntOEOVGvBshzcCC3xmyVcuS1ejum5l3p2MlZCZymDr3Fvjd/XEYw333AjjlHLlHg0Bcp7JYYBs5BYsZ8yDn7efdv8P3vZ2kEDGbzq1hxLcY8tF4DBXgkajMnL4YIpytWzTImaCZhRueSJd+X0qm5Ezh+UfOrlntk0u2upoWfKIjBD7lkKJecTv0OX7phVpcQFBatn+dA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=RdJMEiXQh+7Mm9hjyY4lPhzVWxuKJV9CJ2m0WkCq/ys=; b=pzECV/Pjgg/IrMtAztDaNcEr9ZOPGmqf+Luvu16MBSWfor3+LMIxwc42SehnPLnvisliR5huBZRL+3upcR81yxrK6W1k3qYUCvzKSwuhq6yLnUZIThz9dxUtzVpJzBZUvqNPgLGLyj3twRZGQwW0CszRTDBkI+jYYUssYBcd3Mqe99KdNNfcwPHjarng445L/Dq1GGCI4v3Vje3z3J+0Uea7udh4HeN92Z+aKPq9VqFfxGUDnMD9Yf2HAv0ZhDOA1L+bezF5nQlbm7UhCJmBddVz0C4ocbp9OxmLhJ14jHe/Js0MDrIxlASbuYIu4S+8ZoxT6tn11JXeZ92T8a+O5g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=RdJMEiXQh+7Mm9hjyY4lPhzVWxuKJV9CJ2m0WkCq/ys=; b=L0/Vd6i9DJIXQ+sC223YCgt6cUiumtZu9BFpIIUBZx8+vGD6bxQm8ubxrkbqBsX1BYQhmXIl/OivTGTnmWW4sBIYF1p4cThpIxhpRjjOc5gCi1vFeAoEFi5Y/mOZIfgLUrAb6c/wHvhqnseecCLe/tfxESXjQDu7s9Gk7nSgKoMRJWIy/OKVbjv3jpvmSMt7wLp2AdhwNIumCXTJpLb2jpAoORsYvAy8aZXFRWw4WlJDWoG5lf8giWB8auZJHNXkdBwnD0iZiUNSMWHCaH/3VJktZo3ldabR3yEhUk9Ex9XfjOACFNcHdWC+ZrgA4LB3nZa8UVnIs7/3eq5C9+sjzg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) by SA1PR12MB7442.namprd12.prod.outlook.com (2603:10b6:806:2b5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.12; Thu, 28 May 2026 08:32:20 +0000 Received: from DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7]) by DS7PR12MB9474.namprd12.prod.outlook.com ([fe80::31ad:931:ef07:8ad7%6]) with mapi id 15.21.0071.011; Thu, 28 May 2026 08:32:20 +0000 From: Srirangan Madhavan To: linux-cxl@vger.kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Cc: vsethi@nvidia.com, alwilliamson@nvidia.com, Dan Williams , Sai Yashwanth Reddy Kancherla , Vishal Aslot , Manish Honap , Jiandi An , Richard Cheng , linux-tegra@vger.kernel.org, Srirangan Madhavan Subject: [PATCH v6 9/9] Documentation/ABI: Document CXL memdev cxl_reset Date: Thu, 28 May 2026 08:31:54 +0000 Message-ID: <20260528083154.137979-10-smadhavan@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260528083154.137979-1-smadhavan@nvidia.com> References: <20260528083154.137979-1-smadhavan@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: BY1P220CA0014.NAMP220.PROD.OUTLOOK.COM (2603:10b6:a03:59d::9) To DS7PR12MB9474.namprd12.prod.outlook.com (2603:10b6:8:252::17) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9474:EE_|SA1PR12MB7442:EE_ X-MS-Office365-Filtering-Correlation-Id: 18628973-fe0c-4728-9269-08debc93a28c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|366016|56012099006|11063799006|18002099003|22082099003|6133799003; X-Microsoft-Antispam-Message-Info: R9CleqvWbTUwTgi+8OuYcri8xaWD+gCrqAT21ZGPLz8MpzS7mjrLpmxb0KFx88ZkNwXDEHZL+1hYSH1egc0kQYPszOOl/lFaQ0VEemJ4yNwzMl1CaOUMqJfue5IDTLNgYL46i2gXQ1EnTHgnc7ZJiBMul/wU7JTNI/r43lhDnIsfzeEp+4oilxBIHeQVRk3Gv4GpDjKtywYL3+RlCBqFuof+kuwzlwfBp2jSsSAPj4qQoPmbBDE+rH/2ekAcBkt5+cYUKW7q8pxoQk+j4b+rFL/D6fiMkFmyXtnbJGQjloquqrntBEC7hZiHzZeHMm3fmXgVvPLm9SKXKnC9W2pDldZMjPLnllYAF43shWFCw7HZdZfd58J9vsGDUhL9dABmRZp6O/l2mGt231nrLyE5P20mKraiJHXc5Wc9M1fJA82/VpnZ7k+Mg//NOmTONEvIvaEfi0cZQhfvRNigr2RL+9YtRV2oqBdf7iuwxZDZr2k5N56TNrxt7FP3wpf6k2IoO4kPEGbtoE19tU4DZF4hYVvJjqERxoBh6T4BGPVRbczQHvENrpQb3XDS3v94zuxJVoQ4GjpHO+YSdUGvyf/qKIuYoeAjy+Opw1VuyRYx4ZlzEnB1atT8tfg1f8J8Wo+xj5X9K+/uSLJ+PpSuegUS+UcDIxGJsqOZQwqUjojOZOIanHr2tZCTz2EEyOmssGyw X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9474.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(366016)(56012099006)(11063799006)(18002099003)(22082099003)(6133799003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?BjF8oR6ARjtXPS7iJkUIv2TirYNyEV8Qke2iVN1r2S2TBJXS79tiXXq1nIcN?= =?us-ascii?Q?hUNS1iWGgFjz8kCBX0mNn/pF5vyy5RxmOnySEvdX2geRD90aPXb/QmWTMONR?= =?us-ascii?Q?apKH0JOKSCla33/FBNFkQlZg812l8Pda/MvvwwP9ySj+EVSgNhzLFcHBtIwM?= =?us-ascii?Q?StaFQh0jAXLogZX1VYrO3UyRiZ8MUr6PIAk0+dzQJuNCG+/RpmSWOOIdWM7k?= =?us-ascii?Q?EF80WrZ4kzgL5lNZIrs821djRvyKrgxx9kLFcGr5LE6MduVDW9doIgaoQ0Sk?= =?us-ascii?Q?jnrb838I6fi9F+kjqPsWcXIP1aCme7w7djp50Gy+PHRaxzQNXx/EDTHQ9EZN?= =?us-ascii?Q?+MjmjJg8kYDDZ1Ko5KH60L23ntCRLbii+fyGqj8ewOedPYY+2TiRITvCelgU?= =?us-ascii?Q?Vh0U0vD+ZQ8vfqBgsy59hCnh7ZQzOGMXWfGU16Kx77x25BJ7HUN0aBqgLmMH?= =?us-ascii?Q?FM68+8zrSZT6vO9qyY2gzS4uh6UWEQofo8Wcn6IYKywQ9wPrCXzWFjOy4w7I?= =?us-ascii?Q?XsZziB2lIny4TqOMrlFb4qJP741K3B+oe4LQw17dKnvAEyLaOKtOEl7trlVQ?= =?us-ascii?Q?ghR0FgcjzmxEVHOTHBASZBq5i9dvsLzvOcQw2u1RV3KaAoYbbufpVvLQlRay?= =?us-ascii?Q?+LB9ZFr1qPbPW67qnbxV5mHk4TiY1TT8ATo+r5TB4MGc8n/OjOJs1QkTSGmn?= =?us-ascii?Q?UjeRocO94h+EAAs8NBJRWRg8QIU/p+xB7nLiiPW19/2PkRukymdHSRrI58Ie?= =?us-ascii?Q?jjIeWk5Fq/X7UzRp8ZWvbscc4vmwe1ncFdIAdkQDvG+oRmz+pHcn6KNOjSRY?= =?us-ascii?Q?JEGJGCfoWcJ6SVrh99bzJYW1sMYvS0M3DJJXcLvO5BabHcrbcfa6Jr9mzXm7?= =?us-ascii?Q?s/2AuM+X9KrGUQzFAx6NWUKrjUPBTBCt+qOnKII5m/ZLmQu2GVlR1BSx1XOK?= =?us-ascii?Q?Zjm9Ymg7LNfZGTMlBSE/kOdOpYwjuJ2oKBF5FQQ+0JMjVL037A7WQmjNimBh?= =?us-ascii?Q?XwJ6DEtWbUb+4JE3geRJ0SFtx3F14W/9OSXUO+Ce16d1NEl4dj8zJK4y0E7W?= =?us-ascii?Q?1MDmqJPG0xBQLerUPpWj34pvK397fSkqecyO6s8c90NfkNHxAyfDd7aO9IbC?= =?us-ascii?Q?dzbmvuW8/RqbMWHY6vNTshX3yWLYtEcVXsdsmc4wsUQJzxeCMW9XAFJSJwpQ?= =?us-ascii?Q?b6WN73/MobY7X683vxo3IE4D8DOnbHH1oB1aZmSjt0ZgV0IzV7wSHVz7aSY8?= =?us-ascii?Q?g0p+/EJ7c7pngIcVOW5H5XmBUtXKLxrz9TrU12QQRwkUAPa1bBrCq5odyQFD?= =?us-ascii?Q?UEABDRnWPb2UShYxs2kiSibEEr9Kfmg8ezWqnSyG3p9nFLliywrGgEsh388X?= =?us-ascii?Q?U9JLv6CRiEYdwxBEqqEBMw/wAXxk6BibnXKh2jj7NNd6TENlqC7BxGZbUneV?= =?us-ascii?Q?Z6V46kPuI2/CW/YG0++CnXQ4p8fYGPtOLr3PK4UFIbm/EZWC98nu3lvIoEgG?= =?us-ascii?Q?ObVD/D5WKa3UjvbG4JiAzhWysWVnqRqb/DZcEJFTz07TmJp9YUgzV4eLjDL1?= =?us-ascii?Q?5cJw5QZt8OHtfeC4qhlruDwbW7QUcHaH8eviI3FuyaxrKC+vDdUuvIaByDC+?= =?us-ascii?Q?yEma3lbrBH+4Go4Z+uDEx+HpYIMIuoaU4fbQN/SYfctVIV0aD8tSqSk78eHu?= =?us-ascii?Q?2rOuaQ7lSKe5TB1q7mmhrV/d/m166N7TtrNbRPt7jyN8KjSWMG8eGf96iV2d?= =?us-ascii?Q?SH++MC5/mg=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 18628973-fe0c-4728-9269-08debc93a28c X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9474.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 May 2026 08:32:20.7732 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4mOCmDIn68MC2YBztnS2MotpmJwaEJS/TN3UpgCHEMg9Rd2UAgpqMINZGNNpYKXdZ1zFv8P6zt/vT/QF+zYp9w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7442 Content-Type: text/plain; charset="utf-8" Document the write-only cxl_reset attribute under CXL memdev devices. The attribute is visible only when the memdev's PCI parent advertises CXL Reset capability, and writing a true boolean value requests the CXL reset flow. Signed-off-by: Srirangan Madhavan --- Documentation/ABI/testing/sysfs-bus-cxl | 28 +++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/Documentation/ABI/testing/sysfs-bus-cxl b/Documentation/ABI/te= sting/sysfs-bus-cxl index 16a9b3d2e2c0..d5d055e7a756 100644 --- a/Documentation/ABI/testing/sysfs-bus-cxl +++ b/Documentation/ABI/testing/sysfs-bus-cxl @@ -110,6 +110,34 @@ Description: affinity for this device. =20 =20 +What: /sys/bus/cxl/devices/memX/cxl_reset +Date: May, 2026 +KernelVersion: v7.1 +Contact: linux-cxl@vger.kernel.org +Description: + (WO) Write a boolean true value, for example "1" or "true", to + request CXL Reset for this memory device. The driver performs + CXL-specific reset coordination for the target memdev before + issuing reset, including any required preparation for affected + CXL memory regions and related CXL memory devices. + + CXL Reset control is Function 0 scoped. A write to this + attribute resets the CXL.cache and CXL.mem state for all + CXL.cache or CXL.mem functions in the same CXL device reset + scope, not only the memX device associated with this file. + + The optional CXL Reset Memory Clear operation is not exposed by + this attribute. + + A reset fails with -EBUSY if any affected CXL region is + online as System RAM or has an active region driver bound. + Userspace must first quiesce and release affected CXL memory + mappings. + + If this file is not present, then CXL Reset is not supported + for the device. + + What: /sys/bus/cxl/devices/memX/security/state Date: June, 2023 KernelVersion: v6.5 --=20 2.43.0