From nobody Sat Feb 7 17:09:28 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 40C3325F99B; Tue, 16 Dec 2025 00:55:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765846535; cv=none; b=S3Z85mIFFu2N2SSRw8BE225r2KSfHapr23oQ+TeLuwqQqf0Eb94toWS9Zyr/5bFzCi0FyRSGZk0RO/wIZO54yhvRouaHvJAesg4iFWlpI5mKEXa7yFCqZHL4uJRtwnED0OcA/fIZZjTmfOEDw4PkO+oDy/+OqUgKuj5CkL9SaK4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765846535; c=relaxed/simple; bh=n8gUZPw7fFS/ouO5litSACm7nPNUbvHr1RQeqbjFMuA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=S/eD1eflFiWsJGRImgEDYzCMO5LK0Nc6l3N6LmrrSFTow2RncfqYGb0BjIZ3pO4knIAC748tvrGx3mnlTBISU4ruevL9uNGhjYftChcQj2zK4XUq9FSNZC+r78sliVuEYDyTzOO/4Jt5J06nMF+Pe23IDxZWFJBsSb2SNmG53ws= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=SiVvK+Px; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="SiVvK+Px" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1765846533; x=1797382533; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=n8gUZPw7fFS/ouO5litSACm7nPNUbvHr1RQeqbjFMuA=; b=SiVvK+Px+JTBnnGMJhwKjO7bj8HLZYvVhrsjoM7TlVK786s71xtKx9Ng cih0v66tkYhNO1GKqPydQfdgkQ/aGBUg8dVJtGqvb6u/ezNiGq0uYvqC4 nun88weLjwrIAX3Le/Q2xdsEGQhlNcewf8Vgei/6/4vtaGxCCiX+5MCDk tcW3e2N9gGHblGItKQEq6HeXlYxMktFSACckjHUY0JRskvnMZH/uX1jde 7hTBUIuQzep/zZ5Sb/acC2fD6XqCg6p832wWxr9nuCAq+qsr57BFW0glD 6ch8Df3LK/l6XNvzvSGaExF9D1au1Ceg8E4oB1I8d31srgIbvxooLu31u Q==; X-CSE-ConnectionGUID: Mdu94FASTJmWb70aI8omFA== X-CSE-MsgGUID: Yln9hdzZRaW8VBWGkZdPbQ== X-IronPort-AV: E=McAfee;i="6800,10657,11643"; a="79215471" X-IronPort-AV: E=Sophos;i="6.21,152,1763452800"; d="scan'208";a="79215471" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Dec 2025 16:55:32 -0800 X-CSE-ConnectionGUID: BvKWkTH8Q2SOHl5p3LZegQ== X-CSE-MsgGUID: tf1rDnl2QXWU+I6CAvHJMA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.21,152,1763452800"; d="scan'208";a="198131522" Received: from dwillia2-desk.jf.intel.com ([10.88.27.145]) by fmviesa008.fm.intel.com with ESMTP; 15 Dec 2025 16:55:31 -0800 From: Dan Williams To: dave.jiang@intel.com Cc: linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org, Smita.KoralahalliChannabasappa@amd.com, alison.schofield@intel.com, terry.bowman@amd.com, alejandro.lucero-palau@amd.com, linux-pci@vger.kernel.org, Jonathan.Cameron@huawei.com, Shiju Jose , Ben Cheatham , Jonathan Cameron , Alejandro Lucero Subject: [PATCH v2 1/6] cxl/mem: Fix devm_cxl_memdev_edac_release() confusion Date: Mon, 15 Dec 2025 16:56:11 -0800 Message-ID: <20251216005616.3090129-2-dan.j.williams@intel.com> X-Mailer: git-send-email 2.51.1 In-Reply-To: <20251216005616.3090129-1-dan.j.williams@intel.com> References: <20251216005616.3090129-1-dan.j.williams@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A device release method is only for undoing allocations on the path to preparing the device for device_add(). In contrast, devm allocations are post device_add(), are acquired during / after ->probe() and are released synchronous with ->remove(). So, a "devm" helper in a "release" method is a clear anti-pattern. Move this devm release action where it belongs, an action created at edac object creation time. Otherwise, this leaks resources until cxl_memdev_release() time which may be long after these xarray and error record caches have gone idle. Note, this also fixes up the type of @cxlmd->err_rec_array which needlessly dropped type-safety. Fixes: 0b5ccb0de1e2 ("cxl/edac: Support for finding memory operation attrib= utes from the current boot") Cc: Dave Jiang Cc: Jonathan Cameron Cc: Shiju Jose Cc: Alison Schofield Reviewed-by: Alison Schofield Reviewed-by: Ben Cheatham Reviewed-by: Dave Jiang Reviewed-by: Jonathan Cameron Tested-by: Shiju Jose Reviewed-by: Shiju Jose Tested-by: Alejandro Lucero Signed-off-by: Dan Williams --- drivers/cxl/cxlmem.h | 5 +-- drivers/cxl/core/edac.c | 64 ++++++++++++++++++++++----------------- drivers/cxl/core/memdev.c | 1 - 3 files changed, 38 insertions(+), 32 deletions(-) diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index 434031a0c1f7..c12ab4fc9512 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -63,7 +63,7 @@ struct cxl_memdev { int depth; u8 scrub_cycle; int scrub_region_id; - void *err_rec_array; + struct cxl_mem_err_rec *err_rec_array; }; =20 static inline struct cxl_memdev *to_cxl_memdev(struct device *dev) @@ -877,7 +877,6 @@ int devm_cxl_memdev_edac_register(struct cxl_memdev *cx= lmd); int devm_cxl_region_edac_register(struct cxl_region *cxlr); int cxl_store_rec_gen_media(struct cxl_memdev *cxlmd, union cxl_event *evt= ); int cxl_store_rec_dram(struct cxl_memdev *cxlmd, union cxl_event *evt); -void devm_cxl_memdev_edac_release(struct cxl_memdev *cxlmd); #else static inline int devm_cxl_memdev_edac_register(struct cxl_memdev *cxlmd) { return 0; } @@ -889,8 +888,6 @@ static inline int cxl_store_rec_gen_media(struct cxl_me= mdev *cxlmd, static inline int cxl_store_rec_dram(struct cxl_memdev *cxlmd, union cxl_event *evt) { return 0; } -static inline void devm_cxl_memdev_edac_release(struct cxl_memdev *cxlmd) -{ return; } #endif =20 #ifdef CONFIG_CXL_SUSPEND diff --git a/drivers/cxl/core/edac.c b/drivers/cxl/core/edac.c index 79994ca9bc9f..81160260e26b 100644 --- a/drivers/cxl/core/edac.c +++ b/drivers/cxl/core/edac.c @@ -1988,6 +1988,40 @@ static int cxl_memdev_soft_ppr_init(struct cxl_memde= v *cxlmd, return 0; } =20 +static void err_rec_free(void *_cxlmd) +{ + struct cxl_memdev *cxlmd =3D _cxlmd; + struct cxl_mem_err_rec *array_rec =3D cxlmd->err_rec_array; + struct cxl_event_gen_media *rec_gen_media; + struct cxl_event_dram *rec_dram; + unsigned long index; + + cxlmd->err_rec_array =3D NULL; + xa_for_each(&array_rec->rec_dram, index, rec_dram) + kfree(rec_dram); + xa_destroy(&array_rec->rec_dram); + + xa_for_each(&array_rec->rec_gen_media, index, rec_gen_media) + kfree(rec_gen_media); + xa_destroy(&array_rec->rec_gen_media); + kfree(array_rec); +} + +static int devm_cxl_memdev_setup_err_rec(struct cxl_memdev *cxlmd) +{ + struct cxl_mem_err_rec *array_rec =3D + kzalloc(sizeof(*array_rec), GFP_KERNEL); + + if (!array_rec) + return -ENOMEM; + + xa_init(&array_rec->rec_gen_media); + xa_init(&array_rec->rec_dram); + cxlmd->err_rec_array =3D array_rec; + + return devm_add_action_or_reset(&cxlmd->dev, err_rec_free, cxlmd); +} + int devm_cxl_memdev_edac_register(struct cxl_memdev *cxlmd) { struct edac_dev_feature ras_features[CXL_NR_EDAC_DEV_FEATURES]; @@ -2038,15 +2072,9 @@ int devm_cxl_memdev_edac_register(struct cxl_memdev = *cxlmd) } =20 if (repair_inst) { - struct cxl_mem_err_rec *array_rec =3D - devm_kzalloc(&cxlmd->dev, sizeof(*array_rec), - GFP_KERNEL); - if (!array_rec) - return -ENOMEM; - - xa_init(&array_rec->rec_gen_media); - xa_init(&array_rec->rec_dram); - cxlmd->err_rec_array =3D array_rec; + rc =3D devm_cxl_memdev_setup_err_rec(cxlmd); + if (rc) + return rc; } } =20 @@ -2088,22 +2116,4 @@ int devm_cxl_region_edac_register(struct cxl_region = *cxlr) } EXPORT_SYMBOL_NS_GPL(devm_cxl_region_edac_register, "CXL"); =20 -void devm_cxl_memdev_edac_release(struct cxl_memdev *cxlmd) -{ - struct cxl_mem_err_rec *array_rec =3D cxlmd->err_rec_array; - struct cxl_event_gen_media *rec_gen_media; - struct cxl_event_dram *rec_dram; - unsigned long index; - - if (!IS_ENABLED(CONFIG_CXL_EDAC_MEM_REPAIR) || !array_rec) - return; - - xa_for_each(&array_rec->rec_dram, index, rec_dram) - kfree(rec_dram); - xa_destroy(&array_rec->rec_dram); =20 - xa_for_each(&array_rec->rec_gen_media, index, rec_gen_media) - kfree(rec_gen_media); - xa_destroy(&array_rec->rec_gen_media); -} -EXPORT_SYMBOL_NS_GPL(devm_cxl_memdev_edac_release, "CXL"); diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index e370d733e440..4dff7f44d908 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -27,7 +27,6 @@ static void cxl_memdev_release(struct device *dev) struct cxl_memdev *cxlmd =3D to_cxl_memdev(dev); =20 ida_free(&cxl_memdev_ida, cxlmd->id); - devm_cxl_memdev_edac_release(cxlmd); kfree(cxlmd); } =20 --=20 2.51.1 From nobody Sat Feb 7 17:09:28 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 321C526B777; Tue, 16 Dec 2025 00:55:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765846536; cv=none; b=Y/+dG1kY6uFUyfysAdtRfyMv2vTqCKiTWtW+5lkst1iiEIIvDfrLdjUq5rMU4amZ5M1yYWxrDBnI1Vy4r/6aitAhaV9A52BrgnY3cD6mEYVensbwWAYUIiSZsSKDj6L29rCBZzSTydSDgWM/63i/DpOJcxAWVUwGNtcyQbZhq5U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765846536; c=relaxed/simple; bh=JyMV8IPoO7I2zY1Yvyj9dIsbkueYAbqKbFHBG5CwT0Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SmDnpP5fAmyRl3vVSRm07aZ5YtGUxBX5cNPYm9D16OlOvKapayDdw7/buG4Il7+f2XzN/QDBOANdjCsZIbfLh84tDI1BYO9EBGpCNoKilFAH/NGK8TIxzEMfZbyzUy2H0Z2wPH3qXwq49s1cDLQUJTUvLn2oXTIZJO6YuIr4+Lg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=CrzMhdPe; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="CrzMhdPe" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1765846534; x=1797382534; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=JyMV8IPoO7I2zY1Yvyj9dIsbkueYAbqKbFHBG5CwT0Q=; b=CrzMhdPe3j/WU6M/TgOcuU04x1Nvq/J4G+jfhrTc868pktDdQ1P8Fq09 81HN4p56V/zIYGWZof2KyqIqZp13T/IkabRqLK/Kr42ozn9spOvtcdnhA u86b62+PQG4mI8rbSexkY+GMZESeAtSmxHLZPetZE5b5koyG7VuKi5b+N NDL75f9HXgetqq//sG4xfZXlICNKJxPh8cqW4rz/4DNbAhrRQ6wUDeZrA x7zIs62D0GlVxjpjv1oomjSYg2+ot15Gzg4T0hu2uo8ghC3tFtmqIZsKh E0CFOXIrWMXrfFpJpq6Kehh0yeH3YxbQXkkrLqzEORsLEv+b31uAD8xIV w==; X-CSE-ConnectionGUID: RMsjDk8nTCyZWGZQ0IeGbA== X-CSE-MsgGUID: 6iAADDCZTvSJ9SE0Pgn/MQ== X-IronPort-AV: E=McAfee;i="6800,10657,11643"; a="79215477" X-IronPort-AV: E=Sophos;i="6.21,152,1763452800"; d="scan'208";a="79215477" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Dec 2025 16:55:32 -0800 X-CSE-ConnectionGUID: sFLu+6bLTealsoz5rNy18A== X-CSE-MsgGUID: e3fTtkFYTCaIOEP6PY86Dg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.21,152,1763452800"; d="scan'208";a="198131528" Received: from dwillia2-desk.jf.intel.com ([10.88.27.145]) by fmviesa008.fm.intel.com with ESMTP; 15 Dec 2025 16:55:32 -0800 From: Dan Williams To: dave.jiang@intel.com Cc: linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org, Smita.KoralahalliChannabasappa@amd.com, alison.schofield@intel.com, terry.bowman@amd.com, alejandro.lucero-palau@amd.com, linux-pci@vger.kernel.org, Jonathan.Cameron@huawei.com, Alejandro Lucero , Jonathan Cameron , Ben Cheatham Subject: [PATCH v2 2/6] cxl/mem: Arrange for always-synchronous memdev attach Date: Mon, 15 Dec 2025 16:56:12 -0800 Message-ID: <20251216005616.3090129-3-dan.j.williams@intel.com> X-Mailer: git-send-email 2.51.1 In-Reply-To: <20251216005616.3090129-1-dan.j.williams@intel.com> References: <20251216005616.3090129-1-dan.j.williams@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preparation for CXL accelerator drivers that have a hard dependency on CXL capability initialization, arrange for cxl_mem_probe() to always run synchronous with the device_add() of cxl_memdev instances. I.e. cxl_mem_driver registration is always complete before the first memdev creation event. At present, cxl_pci does not care about the attach state of the cxl_memdev because all generic memory expansion functionality can be handled by the cxl_core. For accelerators, however, that driver needs to perform driver specific initialization if CXL is available, or execute a fallback to PCIe only operation. This synchronous attach guarantee is also needed for Soft Reserve Recovery, which is an effort that needs to assert that devices have had a chance to attach before making a go / no-go decision on proceeding with CXL subsystem initialization. By moving devm_cxl_add_memdev() to cxl_mem.ko it removes async module loading as one reason that a memdev may not be attached upon return from devm_cxl_add_memdev(). Cc: Smita Koralahalli Cc: Alejandro Lucero Reviewed-by: Jonathan Cameron Tested-by: Alison Schofield Reviewed-by: Alison Schofield Reviewed-by: Ben Cheatham Reviewed-by: Dave Jiang Tested-by: Alejandro Lucero Signed-off-by: Dan Williams --- drivers/cxl/Kconfig | 2 +- drivers/cxl/cxlmem.h | 2 ++ drivers/cxl/core/memdev.c | 10 +++++++--- drivers/cxl/mem.c | 17 +++++++++++++++++ 4 files changed, 27 insertions(+), 4 deletions(-) diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig index 48b7314afdb8..f1361ed6a0d4 100644 --- a/drivers/cxl/Kconfig +++ b/drivers/cxl/Kconfig @@ -22,6 +22,7 @@ if CXL_BUS config CXL_PCI tristate "PCI manageability" default CXL_BUS + select CXL_MEM help The CXL specification defines a "CXL memory device" sub-class in the PCI "memory controller" base class of devices. Device's identified by @@ -89,7 +90,6 @@ config CXL_PMEM =20 config CXL_MEM tristate "CXL: Memory Expansion" - depends on CXL_PCI default CXL_BUS help The CXL.mem protocol allows a device to act as a provider of "System diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index c12ab4fc9512..012e68acad34 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -95,6 +95,8 @@ static inline bool is_cxl_endpoint(struct cxl_port *port) return is_cxl_memdev(port->uport_dev); } =20 +struct cxl_memdev *__devm_cxl_add_memdev(struct device *host, + struct cxl_dev_state *cxlds); struct cxl_memdev *devm_cxl_add_memdev(struct device *host, struct cxl_dev_state *cxlds); int devm_cxl_sanitize_setup_notifier(struct device *host, diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 4dff7f44d908..7a4153e1c6a7 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -1050,8 +1050,12 @@ static const struct file_operations cxl_memdev_fops = =3D { .llseek =3D noop_llseek, }; =20 -struct cxl_memdev *devm_cxl_add_memdev(struct device *host, - struct cxl_dev_state *cxlds) +/* + * Core helper for devm_cxl_add_memdev() that wants to both create a devic= e and + * assert to the caller that upon return cxl_mem::probe() has been invoked. + */ +struct cxl_memdev *__devm_cxl_add_memdev(struct device *host, + struct cxl_dev_state *cxlds) { struct cxl_memdev *cxlmd; struct device *dev; @@ -1093,7 +1097,7 @@ struct cxl_memdev *devm_cxl_add_memdev(struct device = *host, put_device(dev); return ERR_PTR(rc); } -EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL"); +EXPORT_SYMBOL_FOR_MODULES(__devm_cxl_add_memdev, "cxl_mem"); =20 static void sanitize_teardown_notifier(void *data) { diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c index 6e6777b7bafb..55883797ab2d 100644 --- a/drivers/cxl/mem.c +++ b/drivers/cxl/mem.c @@ -201,6 +201,22 @@ static int cxl_mem_probe(struct device *dev) return devm_add_action_or_reset(dev, enable_suspend, NULL); } =20 +/** + * devm_cxl_add_memdev - Add a CXL memory device + * @host: devres alloc/release context and parent for the memdev + * @cxlds: CXL device state to associate with the memdev + * + * Upon return the device will have had a chance to attach to the + * cxl_mem driver, but may fail if the CXL topology is not ready + * (hardware CXL link down, or software platform CXL root not attached) + */ +struct cxl_memdev *devm_cxl_add_memdev(struct device *host, + struct cxl_dev_state *cxlds) +{ + return __devm_cxl_add_memdev(host, cxlds); +} +EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL"); + static ssize_t trigger_poison_list_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) @@ -248,6 +264,7 @@ static struct cxl_driver cxl_mem_driver =3D { .probe =3D cxl_mem_probe, .id =3D CXL_DEVICE_MEMORY_EXPANDER, .drv =3D { + .probe_type =3D PROBE_FORCE_SYNCHRONOUS, .dev_groups =3D cxl_mem_groups, }, }; --=20 2.51.1 From nobody Sat Feb 7 17:09:28 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6EE522701B8; Tue, 16 Dec 2025 00:55:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765846536; cv=none; b=QqlXl2H3k4AbC5ZalAuSxeD3VXQPVK0Sf69XQ/zaIfS16o34VQiPe1ZZIGKQMq3GyFOgID0EnrUW78IJx82CSH8L/Fg8inUZKhgVHXEtsrbIFhN/c5m6iBQSTncfDfYRVSLReR12wFQmLzWrsVf/ZWRLXRI6a7K9/OoLUxrq7pk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765846536; c=relaxed/simple; bh=wHO7V2oELIaRdDJ1ZRbnM7/nWHFm2hlak1HFVK1ad+c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aDLXF2krwv65h+nJitMGdeJc+p7Fd9jHwznu4DckHR63fo0iPmq4kfU9MEExBlk/6NePua13wMpJDtPdwqzvcan7vG4Qvta0+Wsd56lKzQV6KSLRIQL7tHrx7JoQriIxcclAGRgq5QEfObe+ohBlDMzJ96oCmdg+K7fIUgxgorQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=j1CKDVO/; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="j1CKDVO/" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1765846535; x=1797382535; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=wHO7V2oELIaRdDJ1ZRbnM7/nWHFm2hlak1HFVK1ad+c=; b=j1CKDVO/6+7SrlUlMzoO4ZhlO45SOwJopbru+OeR06z9R5uUol5nunVa cD5XW+NQ1JnQRs+Lwz2Xgc0z39SQxgu1Boa8aXPn5KZl+CJLgS8veafUV nYuKvuZzAtT6Sd9ZK/5Z3+nNShKrcYUJC44EhMmeVsRVvKQtIFiON0M8I cjnLktF6WU9wkPWv0S5VUw0uourJ7hX1P3SeA0bBxeRiZZovRz/faab+p iHOozjTCDfxQW+YF+eFUyVHt2jjKRHLxZ4H1/mj8R9JckdVm9sUpfj+xx TWWSmb/gXYocGBSSshxFmy9RQY0BQf7ql6dYJsViE150D6P/C2QWjuvP1 g==; X-CSE-ConnectionGUID: s2ea93RwTVi0AfxMicDp6Q== X-CSE-MsgGUID: 19zFchiTSgmL4GHTUzTM/g== X-IronPort-AV: E=McAfee;i="6800,10657,11643"; a="79215480" X-IronPort-AV: E=Sophos;i="6.21,152,1763452800"; d="scan'208";a="79215480" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Dec 2025 16:55:33 -0800 X-CSE-ConnectionGUID: kmT8rWGRQEyf/1becGAl0A== X-CSE-MsgGUID: Yw7GjjbDQuSMs3mrqwNkUA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.21,152,1763452800"; d="scan'208";a="198131531" Received: from dwillia2-desk.jf.intel.com ([10.88.27.145]) by fmviesa008.fm.intel.com with ESMTP; 15 Dec 2025 16:55:32 -0800 From: Dan Williams To: dave.jiang@intel.com Cc: linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org, Smita.KoralahalliChannabasappa@amd.com, alison.schofield@intel.com, terry.bowman@amd.com, alejandro.lucero-palau@amd.com, linux-pci@vger.kernel.org, Jonathan.Cameron@huawei.com, Alejandro Lucero , Jonathan Cameron , Ben Cheatham Subject: [PATCH v2 3/6] cxl/port: Arrange for always synchronous endpoint attach Date: Mon, 15 Dec 2025 16:56:13 -0800 Message-ID: <20251216005616.3090129-4-dan.j.williams@intel.com> X-Mailer: git-send-email 2.51.1 In-Reply-To: <20251216005616.3090129-1-dan.j.williams@intel.com> References: <20251216005616.3090129-1-dan.j.williams@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make it so that upon return from devm_cxl_add_endpoint() that cxl_mem_probe() can assume that the endpoint has had a chance to complete cxl_port_probe(). I.e. cxl_port module loading has completed prior to device registration. Delete the MODULE_SOFTDEP() as it is not sufficient for this purpose, but a hard link-time dependency is reliable. Specifically MODULE_SOFTDEP() does not guarantee that the module loading has completed prior to the completion of the current module's init. Cc: Smita Koralahalli Cc: Alejandro Lucero Reviewed-by: Jonathan Cameron Tested-by: Alison Schofield Reviewed-by: Alison Schofield Reviewed-by: Ben Cheatham Reviewed-by: Dave Jiang Tested-by: Alejandro Lucero Signed-off-by: Dan Williams --- drivers/cxl/cxl.h | 2 ++ drivers/cxl/mem.c | 43 ------------------------------------------- drivers/cxl/port.c | 40 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 42 insertions(+), 43 deletions(-) diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index ba17fa86d249..c796c3db36e0 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -780,6 +780,8 @@ struct cxl_port *devm_cxl_add_port(struct device *host, struct cxl_dport *parent_dport); struct cxl_root *devm_cxl_add_root(struct device *host, const struct cxl_root_ops *ops); +int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd, + struct cxl_dport *parent_dport); struct cxl_root *find_cxl_root(struct cxl_port *port); =20 DEFINE_FREE(put_cxl_root, struct cxl_root *, if (_T) put_device(&_T->port.= dev)) diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c index 55883797ab2d..d62931526fd4 100644 --- a/drivers/cxl/mem.c +++ b/drivers/cxl/mem.c @@ -45,44 +45,6 @@ static int cxl_mem_dpa_show(struct seq_file *file, void = *data) return 0; } =20 -static int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *c= xlmd, - struct cxl_dport *parent_dport) -{ - struct cxl_port *parent_port =3D parent_dport->port; - struct cxl_port *endpoint, *iter, *down; - int rc; - - /* - * Now that the path to the root is established record all the - * intervening ports in the chain. - */ - for (iter =3D parent_port, down =3D NULL; !is_cxl_root(iter); - down =3D iter, iter =3D to_cxl_port(iter->dev.parent)) { - struct cxl_ep *ep; - - ep =3D cxl_ep_load(iter, cxlmd); - ep->next =3D down; - } - - /* Note: endpoint port component registers are derived from @cxlds */ - endpoint =3D devm_cxl_add_port(host, &cxlmd->dev, CXL_RESOURCE_NONE, - parent_dport); - if (IS_ERR(endpoint)) - return PTR_ERR(endpoint); - - rc =3D cxl_endpoint_autoremove(cxlmd, endpoint); - if (rc) - return rc; - - if (!endpoint->dev.driver) { - dev_err(&cxlmd->dev, "%s failed probe\n", - dev_name(&endpoint->dev)); - return -ENXIO; - } - - return 0; -} - static int cxl_debugfs_poison_inject(void *data, u64 dpa) { struct cxl_memdev *cxlmd =3D data; @@ -275,8 +237,3 @@ MODULE_DESCRIPTION("CXL: Memory Expansion"); MODULE_LICENSE("GPL v2"); MODULE_IMPORT_NS("CXL"); MODULE_ALIAS_CXL(CXL_DEVICE_MEMORY_EXPANDER); -/* - * create_endpoint() wants to validate port driver attach immediately after - * endpoint registration. - */ -MODULE_SOFTDEP("pre: cxl_port"); diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index 51c8f2f84717..7937e7e53797 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -156,10 +156,50 @@ static struct cxl_driver cxl_port_driver =3D { .probe =3D cxl_port_probe, .id =3D CXL_DEVICE_PORT, .drv =3D { + .probe_type =3D PROBE_FORCE_SYNCHRONOUS, .dev_groups =3D cxl_port_attribute_groups, }, }; =20 +int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd, + struct cxl_dport *parent_dport) +{ + struct cxl_port *parent_port =3D parent_dport->port; + struct cxl_port *endpoint, *iter, *down; + int rc; + + /* + * Now that the path to the root is established record all the + * intervening ports in the chain. + */ + for (iter =3D parent_port, down =3D NULL; !is_cxl_root(iter); + down =3D iter, iter =3D to_cxl_port(iter->dev.parent)) { + struct cxl_ep *ep; + + ep =3D cxl_ep_load(iter, cxlmd); + ep->next =3D down; + } + + /* Note: endpoint port component registers are derived from @cxlds */ + endpoint =3D devm_cxl_add_port(host, &cxlmd->dev, CXL_RESOURCE_NONE, + parent_dport); + if (IS_ERR(endpoint)) + return PTR_ERR(endpoint); + + rc =3D cxl_endpoint_autoremove(cxlmd, endpoint); + if (rc) + return rc; + + if (!endpoint->dev.driver) { + dev_err(&cxlmd->dev, "%s failed probe\n", + dev_name(&endpoint->dev)); + return -ENXIO; + } + + return 0; +} +EXPORT_SYMBOL_FOR_MODULES(devm_cxl_add_endpoint, "cxl_mem"); + static int __init cxl_port_init(void) { return cxl_driver_register(&cxl_port_driver); --=20 2.51.1 From nobody Sat Feb 7 17:09:28 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 328EC224891; Tue, 16 Dec 2025 00:55:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765846537; cv=none; b=fBEELipG8Ms0RKoioZHgR72EOFlUC/wiiMgnlCIGMsAJE6Ed1IorR1sJ8OkbcpnyZCMI51IQl+qdEdTf6TWbipn0HfzrDtb5B+If0xHZflcaZiDdiMgTr7PQFQLS2spSTMs36ZqU+J2Yn3vl0d1WN+Ko9291SX34NN/U9wyzcEY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765846537; c=relaxed/simple; bh=227I6CJV9ocBGsFP/LsCYQMuN5o0I8rj+s7ym38P2+0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tFIuZTBV5ITkjOxJkarO1Q9cd2924U9GLYhWt9vxKuzMNl2ekeH+bUJh6zurV8CHXolR7+3Tsov3uE/F/bIfhbPgRBrWdyK0JQmTAyAG+eK7myJZdUZ2i2gPBAzfNrFRJ7/zDphm8WE6wTHhFgdfOzbf4RcMVDRCLfUpAaH6AZI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=OiLnKeg3; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="OiLnKeg3" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1765846536; x=1797382536; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=227I6CJV9ocBGsFP/LsCYQMuN5o0I8rj+s7ym38P2+0=; b=OiLnKeg34BIm/aXMvm42pu6ST2bD86BMMNX83YvIDgNJZ7H4TiMCcMrl g/jogDUfhDSG2YzkwoObbdm6Ei59lh5u7yMCkpLk03sb7XFIVdZLIftTC 8dO3StYyVlgShyD+Wids3uCT/OwC54uiMe31eKlXztibRwy/yw5nDJiXH haCWOxgsNbxY9uIZ7gd1Ini0wADC6bOSvLXt2R8/MxsPNiofdThcaNG8B k483bRVTcTNAKwx03DxdSS3PkjblUf7PAJcFDQ1rapU+APMFIKZHm1WZ1 RUL6T0PtHfYO5OJGHfKwnI8nwzno/4NgEOWiPir9C4TrHLAnLTVzxLTBt A==; X-CSE-ConnectionGUID: 4mz2pVzPTouVWTFzTGoidw== X-CSE-MsgGUID: 2ZZ3QqDQSL2siLmAzDNUQA== X-IronPort-AV: E=McAfee;i="6800,10657,11643"; a="79215484" X-IronPort-AV: E=Sophos;i="6.21,152,1763452800"; d="scan'208";a="79215484" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Dec 2025 16:55:33 -0800 X-CSE-ConnectionGUID: 6ZXeJJmnQSa8D2fkNgB7QQ== X-CSE-MsgGUID: V8hmgQ9FSUq7ZjI+9ccEPQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.21,152,1763452800"; d="scan'208";a="198131535" Received: from dwillia2-desk.jf.intel.com ([10.88.27.145]) by fmviesa008.fm.intel.com with ESMTP; 15 Dec 2025 16:55:32 -0800 From: Dan Williams To: dave.jiang@intel.com Cc: linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org, Smita.KoralahalliChannabasappa@amd.com, alison.schofield@intel.com, terry.bowman@amd.com, alejandro.lucero-palau@amd.com, linux-pci@vger.kernel.org, Jonathan.Cameron@huawei.com, Ben Cheatham , Alejandro Lucero Subject: [PATCH v2 4/6] cxl/mem: Convert devm_cxl_add_memdev() to scope-based-cleanup Date: Mon, 15 Dec 2025 16:56:14 -0800 Message-ID: <20251216005616.3090129-5-dan.j.williams@intel.com> X-Mailer: git-send-email 2.51.1 In-Reply-To: <20251216005616.3090129-1-dan.j.williams@intel.com> References: <20251216005616.3090129-1-dan.j.williams@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preparation for adding more setup steps, convert the current implementation to scope-based cleanup. The cxl_memdev_shutdown() is only required after cdev_device_add(). With that moved to a helper function it precludes the need to add scope-based-handler for that cleanup if devm_add_action_or_reset() fails. Cc: Smita Koralahalli Reviewed-by: Alison Schofield Reviewed-by: Dave Jiang Reviewed-by: Ben Cheatham Tested-by: Alejandro Lucero Signed-off-by: Dan Williams Reviewed-by: Jonathan Cameron --- drivers/cxl/core/memdev.c | 70 ++++++++++++++++++++++++--------------- 1 file changed, 44 insertions(+), 26 deletions(-) diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 7a4153e1c6a7..18efbf294db5 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -1050,6 +1050,45 @@ static const struct file_operations cxl_memdev_fops = =3D { .llseek =3D noop_llseek, }; =20 +/* + * Activate ioctl operations, no cxl_memdev_rwsem manipulation needed as t= his is + * ordered with cdev_add() publishing the device. + */ +static int cxlmd_add(struct cxl_memdev *cxlmd, struct cxl_dev_state *cxlds) +{ + int rc; + + cxlmd->cxlds =3D cxlds; + cxlds->cxlmd =3D cxlmd; + + rc =3D cdev_device_add(&cxlmd->cdev, &cxlmd->dev); + if (rc) { + /* + * The cdev was briefly live, shutdown any ioctl operations that + * saw that state. + */ + cxl_memdev_shutdown(&cxlmd->dev); + return rc; + } + + return 0; +} + +DEFINE_FREE(put_cxlmd, struct cxl_memdev *, + if (!IS_ERR_OR_NULL(_T)) put_device(&_T->dev);) + +static struct cxl_memdev *cxl_memdev_autoremove(struct cxl_memdev *cxlmd) +{ + int rc; + + rc =3D devm_add_action_or_reset(cxlmd->cxlds->dev, cxl_memdev_unregister, + cxlmd); + if (rc) + return ERR_PTR(rc); + + return cxlmd; +} + /* * Core helper for devm_cxl_add_memdev() that wants to both create a devic= e and * assert to the caller that upon return cxl_mem::probe() has been invoked. @@ -1057,45 +1096,24 @@ static const struct file_operations cxl_memdev_fops= =3D { struct cxl_memdev *__devm_cxl_add_memdev(struct device *host, struct cxl_dev_state *cxlds) { - struct cxl_memdev *cxlmd; struct device *dev; - struct cdev *cdev; int rc; =20 - cxlmd =3D cxl_memdev_alloc(cxlds, &cxl_memdev_fops); + struct cxl_memdev *cxlmd __free(put_cxlmd) =3D + cxl_memdev_alloc(cxlds, &cxl_memdev_fops); if (IS_ERR(cxlmd)) return cxlmd; =20 dev =3D &cxlmd->dev; rc =3D dev_set_name(dev, "mem%d", cxlmd->id); if (rc) - goto err; - - /* - * Activate ioctl operations, no cxl_memdev_rwsem manipulation - * needed as this is ordered with cdev_add() publishing the device. - */ - cxlmd->cxlds =3D cxlds; - cxlds->cxlmd =3D cxlmd; - - cdev =3D &cxlmd->cdev; - rc =3D cdev_device_add(cdev, dev); - if (rc) - goto err; + return ERR_PTR(rc); =20 - rc =3D devm_add_action_or_reset(host, cxl_memdev_unregister, cxlmd); + rc =3D cxlmd_add(cxlmd, cxlds); if (rc) return ERR_PTR(rc); - return cxlmd; =20 -err: - /* - * The cdev was briefly live, shutdown any ioctl operations that - * saw that state. - */ - cxl_memdev_shutdown(dev); - put_device(dev); - return ERR_PTR(rc); + return cxl_memdev_autoremove(no_free_ptr(cxlmd)); } EXPORT_SYMBOL_FOR_MODULES(__devm_cxl_add_memdev, "cxl_mem"); =20 --=20 2.51.1 From nobody Sat Feb 7 17:09:28 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 381B527CB35; Tue, 16 Dec 2025 00:55:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765846538; cv=none; b=CNnwpdxu+hXPSP+DHFP/rqGCMURQO2HRZ8cTu62XvBqozf/MjTyPDf0/bTyGfsbL7ISOuufiavP9sKRgSxIU9lg7Huv+FpS8QQxYKHwlPvsE/iXkoPBc+lO0zyRUgCeaN9Xc9AqdGEppQVWQFbEYIU8c4RcHcc6FAryBUp8hWHQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765846538; c=relaxed/simple; bh=3im/o3r75eE2iLJ7BlqrvuXoMC/tLGq0o0JFtlK+2JI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=kTjIltuDio71jpw2c0+XIE0BK6z3EbvMPJ+OFMbG173mc7IWcz8Loq3hTOSvZ1y4x5aTx/x+r1Lxx/povZdbc10Lx4tR21VVUldLFH/itO9XDztT7/bzH7SZjDCvtUqWsWytfrQ8+PpJ5xUAkL5eim397IunB+QbtAdN2A0ChYg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=TzotR7ti; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="TzotR7ti" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1765846536; x=1797382536; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=3im/o3r75eE2iLJ7BlqrvuXoMC/tLGq0o0JFtlK+2JI=; b=TzotR7tinZiC/dRKyESxCS7cI5hXw1ezq/ALxJbHekq55yYI3g7cN+2G VwbRLOk0Owao2dY+yiQjXv6xnmahw4stsiwZYsl7c+4sU90ky4ynCOnpb RYA3FkWh2nw/HjGiMY10vlUHwbc66yZzS5dqjJE9FoljRwMpOVUEUB650 h/0/j8Bt2DSNzofPQq+Z5A9+MkgJKkRV40o23DrOUZzG87haBuLEKogbr qg6Ln2o5KLpW/El/BEC32YnsWXRDJGgCnZHP0vgSEey9PHk26VyXhwDON 8PhrVhurR24NG3lgPHh+lxioVDpIr/ZTHy8lhbqh98snKhC000KUjBbNC Q==; X-CSE-ConnectionGUID: bqHrtSvNSXmGu6ZtKW/s5Q== X-CSE-MsgGUID: O/TqbFUfQgaNnQimwaShrA== X-IronPort-AV: E=McAfee;i="6800,10657,11643"; a="79215487" X-IronPort-AV: E=Sophos;i="6.21,152,1763452800"; d="scan'208";a="79215487" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Dec 2025 16:55:34 -0800 X-CSE-ConnectionGUID: YSXU9NG1R2Wfz2dKRXidKA== X-CSE-MsgGUID: GnaKlRQgRlClJtiGOCWaLA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.21,152,1763452800"; d="scan'208";a="198131542" Received: from dwillia2-desk.jf.intel.com ([10.88.27.145]) by fmviesa008.fm.intel.com with ESMTP; 15 Dec 2025 16:55:33 -0800 From: Dan Williams To: dave.jiang@intel.com Cc: linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org, Smita.KoralahalliChannabasappa@amd.com, alison.schofield@intel.com, terry.bowman@amd.com, alejandro.lucero-palau@amd.com, linux-pci@vger.kernel.org, Jonathan.Cameron@huawei.com, Jonathan Cameron , Ben Cheatham , Alejandro Lucero Subject: [PATCH v2 5/6] cxl/mem: Drop @host argument to devm_cxl_add_memdev() Date: Mon, 15 Dec 2025 16:56:15 -0800 Message-ID: <20251216005616.3090129-6-dan.j.williams@intel.com> X-Mailer: git-send-email 2.51.1 In-Reply-To: <20251216005616.3090129-1-dan.j.williams@intel.com> References: <20251216005616.3090129-1-dan.j.williams@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable In all cases the device that created the 'struct cxl_dev_state' instance is also the device to host the devm cleanup of devm_cxl_add_memdev(). This simplifies the function prototype, and limits a degree of freedom of the API. Cc: Smita Koralahalli Reviewed-by: Jonathan Cameron Reviewed-by: Alison Schofield (=E2=9C=93 DKIM/= intel.com) Reviewed-by: Dave Jiang (=E2=9C=93 DKIM/intel.com) Reviewed-by: Ben Cheatham (=E2=9C=93 DKIM/amd.c= om) Tested-by: Alejandro Lucero (=E2=9C=93 DKIM/amd.com) Signed-off-by: Dan Williams --- drivers/cxl/cxlmem.h | 6 ++---- drivers/cxl/core/memdev.c | 3 +-- drivers/cxl/mem.c | 9 +++++---- drivers/cxl/pci.c | 2 +- tools/testing/cxl/test/mem.c | 2 +- 5 files changed, 10 insertions(+), 12 deletions(-) diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index 012e68acad34..9db31c7993c4 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -95,10 +95,8 @@ static inline bool is_cxl_endpoint(struct cxl_port *port) return is_cxl_memdev(port->uport_dev); } =20 -struct cxl_memdev *__devm_cxl_add_memdev(struct device *host, - struct cxl_dev_state *cxlds); -struct cxl_memdev *devm_cxl_add_memdev(struct device *host, - struct cxl_dev_state *cxlds); +struct cxl_memdev *__devm_cxl_add_memdev(struct cxl_dev_state *cxlds); +struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds); int devm_cxl_sanitize_setup_notifier(struct device *host, struct cxl_memdev *cxlmd); struct cxl_memdev_state; diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 18efbf294db5..63da2bd4436e 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -1093,8 +1093,7 @@ static struct cxl_memdev *cxl_memdev_autoremove(struc= t cxl_memdev *cxlmd) * Core helper for devm_cxl_add_memdev() that wants to both create a devic= e and * assert to the caller that upon return cxl_mem::probe() has been invoked. */ -struct cxl_memdev *__devm_cxl_add_memdev(struct device *host, - struct cxl_dev_state *cxlds) +struct cxl_memdev *__devm_cxl_add_memdev(struct cxl_dev_state *cxlds) { struct device *dev; int rc; diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c index d62931526fd4..677996c65272 100644 --- a/drivers/cxl/mem.c +++ b/drivers/cxl/mem.c @@ -165,17 +165,18 @@ static int cxl_mem_probe(struct device *dev) =20 /** * devm_cxl_add_memdev - Add a CXL memory device - * @host: devres alloc/release context and parent for the memdev * @cxlds: CXL device state to associate with the memdev * * Upon return the device will have had a chance to attach to the * cxl_mem driver, but may fail if the CXL topology is not ready * (hardware CXL link down, or software platform CXL root not attached) + * + * The parent of the resulting device and the devm context for allocations= is + * @cxlds->dev. */ -struct cxl_memdev *devm_cxl_add_memdev(struct device *host, - struct cxl_dev_state *cxlds) +struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds) { - return __devm_cxl_add_memdev(host, cxlds); + return __devm_cxl_add_memdev(cxlds); } EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL"); =20 diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 0be4e508affe..1c6fc5334806 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -1006,7 +1006,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const = struct pci_device_id *id) if (rc) dev_dbg(&pdev->dev, "No CXL Features discovered\n"); =20 - cxlmd =3D devm_cxl_add_memdev(&pdev->dev, cxlds); + cxlmd =3D devm_cxl_add_memdev(cxlds); if (IS_ERR(cxlmd)) return PTR_ERR(cxlmd); =20 diff --git a/tools/testing/cxl/test/mem.c b/tools/testing/cxl/test/mem.c index 176dcde570cd..8a22b7601627 100644 --- a/tools/testing/cxl/test/mem.c +++ b/tools/testing/cxl/test/mem.c @@ -1767,7 +1767,7 @@ static int cxl_mock_mem_probe(struct platform_device = *pdev) =20 cxl_mock_add_event_logs(&mdata->mes); =20 - cxlmd =3D devm_cxl_add_memdev(&pdev->dev, cxlds); + cxlmd =3D devm_cxl_add_memdev(cxlds); if (IS_ERR(cxlmd)) return PTR_ERR(cxlmd); =20 --=20 2.51.1 From nobody Sat Feb 7 17:09:28 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7139B2248B8; Tue, 16 Dec 2025 00:55:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765846539; cv=none; b=aeDeX6RnR/yK1m8Tpn3U77ehcqhiCm7Qf4iFUkTWoW85v/IoDdl5fynZl7sP+tUvsi/H4ry0XFrNFmrzMJNlQuPukYHJLsl6huONzK/oRUmOnlTi9WNeKRicpcHnsp+V8Wipq3RksdItVoi/XSbR73otma5oyQpWaBnGxspu4Rs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765846539; c=relaxed/simple; bh=mQWplLegDxIK0Nc6SI+GBREhDsJR//DTXIHztZu4EWw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=tMlOu7grNoRHRqyVABi17rvdoL6vcVmxy2fXFC/ZBo7xj8tVImfk05R8AWPolPlpz3rV0SR3en0H9WypvjfVXnpC/7CA5k+loihJR9zfeh4cNH+WHj8Lx6b6f3O0B0SIfcxYJno0Vs8xwce3BfZR5KjWcO+SApEEfeC7AUnwk3Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=meZ9zK3B; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="meZ9zK3B" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1765846537; x=1797382537; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=mQWplLegDxIK0Nc6SI+GBREhDsJR//DTXIHztZu4EWw=; b=meZ9zK3BPvEmYN4s4IbD4Z3XMXc1zPXf25xGpEqcm1Yi4aMpmE1S0uvs BWJ28Uc2eAfvIoSQcgXB++r5fbK04WJaZEtxUSZr50tXCCMreFGpqy7BD XdxRKvmN2O3pCkV3pseWNsAvGIX4NuXQDyfkoNiDAXuWYqAEosZXYvFsN GobocOsVRwtF5ozCLKFHhN4MPNgxy5mSFkSUBp0b9Zw8KFp4qUaHcQmPd idzErAvpL6ZaEwQMmXHYTaxNwYcg0AK+z7rdUGU4dS/kByrsJEZ55b3gS EsZT7f2bhcP6mDfizevj1B1t9hhoEcoiVoAeSnGcfEf4tWziYFfsV3Qnh Q==; X-CSE-ConnectionGUID: Bja5M+AbTwGj854FlvAMaQ== X-CSE-MsgGUID: Mjx/sKrwQWOrKg8/JiqruQ== X-IronPort-AV: E=McAfee;i="6800,10657,11643"; a="79215490" X-IronPort-AV: E=Sophos;i="6.21,152,1763452800"; d="scan'208";a="79215490" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Dec 2025 16:55:34 -0800 X-CSE-ConnectionGUID: Ha5UA8z2Sf2UsVN8uACKYg== X-CSE-MsgGUID: +IdhYi7bT72lAjXOAAuCYw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.21,152,1763452800"; d="scan'208";a="198131546" Received: from dwillia2-desk.jf.intel.com ([10.88.27.145]) by fmviesa008.fm.intel.com with ESMTP; 15 Dec 2025 16:55:33 -0800 From: Dan Williams To: dave.jiang@intel.com Cc: linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org, Smita.KoralahalliChannabasappa@amd.com, alison.schofield@intel.com, terry.bowman@amd.com, alejandro.lucero-palau@amd.com, linux-pci@vger.kernel.org, Jonathan.Cameron@huawei.com, Ben Cheatham , Alejandro Lucero Subject: [PATCH v2 6/6] cxl/mem: Introduce cxl_memdev_attach for CXL-dependent operation Date: Mon, 15 Dec 2025 16:56:16 -0800 Message-ID: <20251216005616.3090129-7-dan.j.williams@intel.com> X-Mailer: git-send-email 2.51.1 In-Reply-To: <20251216005616.3090129-1-dan.j.williams@intel.com> References: <20251216005616.3090129-1-dan.j.williams@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Unlike the cxl_pci class driver that opportunistically enables memory expansion with no other dependent functionality, CXL accelerator drivers have distinct PCIe-only and CXL-enhanced operation states. If CXL is available some additional coherent memory/cache operations can be enabled, otherwise traditional DMA+MMIO over PCIe/CXL.io is a fallback. Allow for a driver to pass a routine to be called in cxl_mem_probe() context. This ability is inspired by and mirrors the semantics of faux_device_create(). It allows for the caller to run CXL-topology attach-dependent logic on behalf of the caller. The probe callback runs after the port topology is successfully attached for the given memdev. Additionally the presence of @cxlmd->attach indicates that the accelerator driver be detached when CXL operation ends. This conceptually makes a CXL link loss event mirror a PCIe link loss event which results in triggering the ->remove() callback of affected devices+drivers. A driver can re-attach to recover back to PCIe-only operation. Live recovery, i.e. without a ->remove()/->probe() cycle, is left as a future consideration. Cc: Smita Koralahalli Reviewed-by: Ben Cheatham Reviewed-by: Dave Jiang (=E2=9C=93 DKIM/intel.com) Tested-by: Alejandro Lucero (=E2=9C=93 DKIM/amd.com) Signed-off-by: Dan Williams Reviewed-by: Jonathan Cameron --- drivers/cxl/cxlmem.h | 12 ++++++++++-- drivers/cxl/core/memdev.c | 33 +++++++++++++++++++++++++++++---- drivers/cxl/mem.c | 20 ++++++++++++++++---- drivers/cxl/pci.c | 2 +- tools/testing/cxl/test/mem.c | 2 +- 5 files changed, 57 insertions(+), 12 deletions(-) diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index 9db31c7993c4..ef202b34e5ea 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -34,6 +34,10 @@ (FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) !=3D \ CXLMDEV_RESET_NEEDED_NOT) =20 +struct cxl_memdev_attach { + int (*probe)(struct cxl_memdev *cxlmd); +}; + /** * struct cxl_memdev - CXL bus object representing a Type-3 Memory Device * @dev: driver core device object @@ -43,6 +47,7 @@ * @cxl_nvb: coordinate removal of @cxl_nvd if present * @cxl_nvd: optional bridge to an nvdimm if the device supports pmem * @endpoint: connection to the CXL port topology for this memory device + * @attach: creator of this memdev depends on CXL link attach to operate * @id: id number of this memdev instance. * @depth: endpoint port depth * @scrub_cycle: current scrub cycle set for this device @@ -59,6 +64,7 @@ struct cxl_memdev { struct cxl_nvdimm_bridge *cxl_nvb; struct cxl_nvdimm *cxl_nvd; struct cxl_port *endpoint; + const struct cxl_memdev_attach *attach; int id; int depth; u8 scrub_cycle; @@ -95,8 +101,10 @@ static inline bool is_cxl_endpoint(struct cxl_port *por= t) return is_cxl_memdev(port->uport_dev); } =20 -struct cxl_memdev *__devm_cxl_add_memdev(struct cxl_dev_state *cxlds); -struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds); +struct cxl_memdev *__devm_cxl_add_memdev(struct cxl_dev_state *cxlds, + const struct cxl_memdev_attach *attach); +struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds, + const struct cxl_memdev_attach *attach); int devm_cxl_sanitize_setup_notifier(struct device *host, struct cxl_memdev *cxlmd); struct cxl_memdev_state; diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 63da2bd4436e..3ab4cd8f19ed 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -641,14 +641,24 @@ static void detach_memdev(struct work_struct *work) struct cxl_memdev *cxlmd; =20 cxlmd =3D container_of(work, typeof(*cxlmd), detach_work); - device_release_driver(&cxlmd->dev); + + /* + * When the creator of @cxlmd sets ->attach it indicates CXL operation + * is required. In that case, @cxlmd detach escalates to parent device + * detach. + */ + if (cxlmd->attach) + device_release_driver(cxlmd->dev.parent); + else + device_release_driver(&cxlmd->dev); put_device(&cxlmd->dev); } =20 static struct lock_class_key cxl_memdev_key; =20 static struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds, - const struct file_operations *fops) + const struct file_operations *fops, + const struct cxl_memdev_attach *attach) { struct cxl_memdev *cxlmd; struct device *dev; @@ -664,6 +674,8 @@ static struct cxl_memdev *cxl_memdev_alloc(struct cxl_d= ev_state *cxlds, goto err; cxlmd->id =3D rc; cxlmd->depth =3D -1; + cxlmd->attach =3D attach; + cxlmd->endpoint =3D ERR_PTR(-ENXIO); =20 dev =3D &cxlmd->dev; device_initialize(dev); @@ -1081,6 +1093,18 @@ static struct cxl_memdev *cxl_memdev_autoremove(stru= ct cxl_memdev *cxlmd) { int rc; =20 + /* + * If @attach is provided fail if the driver is not attached upon + * return. Note that failure here could be the result of a race to + * teardown the CXL port topology. I.e. cxl_mem_probe() could have + * succeeded and then cxl_mem unbound before the lock is acquired. + */ + guard(device)(&cxlmd->dev); + if (cxlmd->attach && !cxlmd->dev.driver) { + cxl_memdev_unregister(cxlmd); + return ERR_PTR(-ENXIO); + } + rc =3D devm_add_action_or_reset(cxlmd->cxlds->dev, cxl_memdev_unregister, cxlmd); if (rc) @@ -1093,13 +1117,14 @@ static struct cxl_memdev *cxl_memdev_autoremove(str= uct cxl_memdev *cxlmd) * Core helper for devm_cxl_add_memdev() that wants to both create a devic= e and * assert to the caller that upon return cxl_mem::probe() has been invoked. */ -struct cxl_memdev *__devm_cxl_add_memdev(struct cxl_dev_state *cxlds) +struct cxl_memdev *__devm_cxl_add_memdev(struct cxl_dev_state *cxlds, + const struct cxl_memdev_attach *attach) { struct device *dev; int rc; =20 struct cxl_memdev *cxlmd __free(put_cxlmd) =3D - cxl_memdev_alloc(cxlds, &cxl_memdev_fops); + cxl_memdev_alloc(cxlds, &cxl_memdev_fops, attach); if (IS_ERR(cxlmd)) return cxlmd; =20 diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c index 677996c65272..333c366b69e7 100644 --- a/drivers/cxl/mem.c +++ b/drivers/cxl/mem.c @@ -142,6 +142,12 @@ static int cxl_mem_probe(struct device *dev) return rc; } =20 + if (cxlmd->attach) { + rc =3D cxlmd->attach->probe(cxlmd); + if (rc) + return rc; + } + rc =3D devm_cxl_memdev_edac_register(cxlmd); if (rc) dev_dbg(dev, "CXL memdev EDAC registration failed rc=3D%d\n", rc); @@ -166,17 +172,23 @@ static int cxl_mem_probe(struct device *dev) /** * devm_cxl_add_memdev - Add a CXL memory device * @cxlds: CXL device state to associate with the memdev + * @attach: Caller depends on CXL topology attachment * * Upon return the device will have had a chance to attach to the - * cxl_mem driver, but may fail if the CXL topology is not ready - * (hardware CXL link down, or software platform CXL root not attached) + * cxl_mem driver, but may fail to attach if the CXL topology is not ready + * (hardware CXL link down, or software platform CXL root not attached). + * + * When @attach is NULL it indicates the caller wants the memdev to remain + * registered even if it does not immediately attach to the CXL hierarchy.= When + * @attach is provided a cxl_mem_probe() failure leads to failure of this = routine. * * The parent of the resulting device and the devm context for allocations= is * @cxlds->dev. */ -struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds) +struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds, + const struct cxl_memdev_attach *attach) { - return __devm_cxl_add_memdev(cxlds); + return __devm_cxl_add_memdev(cxlds, attach); } EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL"); =20 diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 1c6fc5334806..549368a9c868 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -1006,7 +1006,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const = struct pci_device_id *id) if (rc) dev_dbg(&pdev->dev, "No CXL Features discovered\n"); =20 - cxlmd =3D devm_cxl_add_memdev(cxlds); + cxlmd =3D devm_cxl_add_memdev(cxlds, NULL); if (IS_ERR(cxlmd)) return PTR_ERR(cxlmd); =20 diff --git a/tools/testing/cxl/test/mem.c b/tools/testing/cxl/test/mem.c index 8a22b7601627..cb87e8c0e63c 100644 --- a/tools/testing/cxl/test/mem.c +++ b/tools/testing/cxl/test/mem.c @@ -1767,7 +1767,7 @@ static int cxl_mock_mem_probe(struct platform_device = *pdev) =20 cxl_mock_add_event_logs(&mdata->mes); =20 - cxlmd =3D devm_cxl_add_memdev(cxlds); + cxlmd =3D devm_cxl_add_memdev(cxlds, NULL); if (IS_ERR(cxlmd)) return PTR_ERR(cxlmd); =20 --=20 2.51.1