From nobody Tue Feb 10 05:17:06 2026 Received: from OS0P286CU011.outbound.protection.outlook.com (mail-japanwestazon11010020.outbound.protection.outlook.com [52.101.228.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C9162369987; Wed, 17 Dec 2025 15:17:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.228.20 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765984681; cv=fail; b=EtyuEJ7p5VIBAfDk+Z5pohqtdboOcj2OTTsxHx0fl+/qSSLt4k5gvslUN1PMnnW1Wiv0ZO1CWNUad91s7+vi3gZp9hFb9y6yB9upmkTFiPGGN/4781hAqObBSPjT1XEcOXZJK22bsq9upjOiF6z468vvfQYLmJFV+gGgII+Mf94= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765984681; c=relaxed/simple; bh=g9w1J7/ZEjbSLI0H9IYjPXrn9NLNX10s8CgJG6cB384=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=KM/8PBDLSAY4lRYpJ4m+/4SS1HxAovsM7V1gWVqnDleen/Gq/1jNSJqE/DugrmUw9bE+zLozw6FFJtPNaAhsjG0XDXyegDJ6t6aDMtKHcSBnUXVvS8KIUKuQ101Vl1pMNdsgTvu0Eh0+3bfUdb4YYjcPxwmrLxrySIa3c3zqa2Q= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=valinux.co.jp; spf=pass smtp.mailfrom=valinux.co.jp; dkim=pass (1024-bit key) header.d=valinux.co.jp header.i=@valinux.co.jp header.b=CidfA6cW; arc=fail smtp.client-ip=52.101.228.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=valinux.co.jp Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=valinux.co.jp Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=valinux.co.jp header.i=@valinux.co.jp header.b="CidfA6cW" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=LO1bWkoFS811cf712X/m/FWTRZVfv+qgpbhc7cz4QPeeMds2tY/Z0XRJy8/rI5I1nW/WeKR1gcEjEBWiNsOYR6BxUGWGyUX6cJUQcoB+nGlUcv8liCH8vpZvW5CggWgNpVVzY3E/lV4Oza0CYToz39L8ySMIe27KfBqSYEJKZtSDORUvwrCXvoMRIPdmwymZZjHBLAqmykriqpnyLcV7HMOhCdS5GZh7Bly807fJ6zZN24SdaV1ei6A6M/cBYmm0izHi/Di1McncTUdRs9pWJro3uZ3jtnsdekrqmLPiS6VwrVIpzH6rstNq/LwVihgQlcPXF6jWDDI7qfCANtuecQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=upbR1KCMeIs2DM48PtwJNwJBzPLA5l4dOeMx6TDWdPE=; b=Y0T6AITeLRLeLd4HTWYowNxVDgiJXOitjmqSq5sgwE7v3H/JA77YY9kSASEUgakGJoaN6D7SBAzJHQsuZw/hndj1e9rasaVxixdRAQtBVXGMZ6Sscu3+f+K6YGqAEZ3rmV+S1ZiRndFCWCNoHjdRgndKWOJVyKnMJmn6ZR3HBdAgh5iVQ/DTleZtPYCJaK7uY9Cm4m+n6oMdJi0yHpcVd0iNLlfhyuk+c9QJVH1xPMjKHFF5hRtzlF47zVAWWA4AAJVNcv8v9K9NktYcfiLDzu8xZfVi6R3HcI/fshkSXF+isbYyOuc+TJxQKS+NsDcnRmfLFMPM/UcgTMG2kmdn/w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=valinux.co.jp; dmarc=pass action=none header.from=valinux.co.jp; dkim=pass header.d=valinux.co.jp; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=valinux.co.jp; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=upbR1KCMeIs2DM48PtwJNwJBzPLA5l4dOeMx6TDWdPE=; b=CidfA6cWGQYxiGWh5KiHiqk1Qck3rb21OmSV8USh+laNfN5PYqFwlvFDhchgJKJms1lRevsBgqwunXTx2PTvntfOuSWYWoWPp+sMn5c52hpmbMqfSjw+AAP04EpassyQ9QuinT3xwH+qRPRDiY4EHDWVAGz4z1F6YSzAgD/sA1w= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=valinux.co.jp; Received: from TYWP286MB2697.JPNP286.PROD.OUTLOOK.COM (2603:1096:400:24c::11) by TYCP286MB2863.JPNP286.PROD.OUTLOOK.COM (2603:1096:400:306::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9434.6; Wed, 17 Dec 2025 15:16:37 +0000 Received: from TYWP286MB2697.JPNP286.PROD.OUTLOOK.COM ([fe80::fb7e:f4ed:a580:9d03]) by TYWP286MB2697.JPNP286.PROD.OUTLOOK.COM ([fe80::fb7e:f4ed:a580:9d03%5]) with mapi id 15.20.9434.001; Wed, 17 Dec 2025 15:16:37 +0000 From: Koichiro Den To: Frank.Li@nxp.com, dave.jiang@intel.com, ntb@lists.linux.dev, linux-pci@vger.kernel.org, dmaengine@vger.kernel.org, linux-renesas-soc@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: mani@kernel.org, kwilczynski@kernel.org, kishon@kernel.org, bhelgaas@google.com, corbet@lwn.net, geert+renesas@glider.be, magnus.damm@gmail.com, robh@kernel.org, krzk+dt@kernel.org, conor+dt@kernel.org, vkoul@kernel.org, joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, jdmason@kudzu.us, allenbh@gmail.com, andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, Basavaraj.Natikar@amd.com, Shyam-sundar.S-k@amd.com, kurt.schwemmer@microsemi.com, logang@deltatee.com, jingoohan1@gmail.com, lpieralisi@kernel.org, utkarsh02t@gmail.com, jbrunet@baylibre.com, dlemoal@kernel.org, arnd@arndb.de, elfring@users.sourceforge.net, den@valinux.co.jp Subject: [RFC PATCH v3 26/35] NTB: ntb_transport: Introduce DW eDMA backed transport mode Date: Thu, 18 Dec 2025 00:16:00 +0900 Message-ID: <20251217151609.3162665-27-den@valinux.co.jp> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20251217151609.3162665-1-den@valinux.co.jp> References: <20251217151609.3162665-1-den@valinux.co.jp> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: TYWP286CA0030.JPNP286.PROD.OUTLOOK.COM (2603:1096:400:262::20) To TYWP286MB2697.JPNP286.PROD.OUTLOOK.COM (2603:1096:400:24c::11) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: TYWP286MB2697:EE_|TYCP286MB2863:EE_ X-MS-Office365-Filtering-Correlation-Id: aaf6cf86-2274-4725-ce24-08de3d7f459e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|10070799003|1800799024|366016; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?S3V7jzYr6nRDnjL6JW1+NGOSLeBVX1AVUkZR7JAn5soFdtTzhABH6/SzWWEG?= =?us-ascii?Q?csxVQrKuYKaSeTmpJfGxf0K6QTAg7gas45QT7RjK4JeZv2g+h9AdTZNxOWVW?= =?us-ascii?Q?nc/R1YlrjHhph5Awth7qi4eFs5aCBpj7fdSMU3sGcfzd8JxjHeW/G7WsnV+n?= =?us-ascii?Q?ft2K8n4q92BJxDQOlwFYw+40q5Zxe5JefYW2S0Cddi6XHfXq9BTQAZ0d4Djl?= =?us-ascii?Q?Lb+08pl+S+3byHq60tNWaYApNrMLFaQ8nID/lrltM0NSiDQAcfwjwa0PO7VD?= =?us-ascii?Q?LylCUVwuIedQ/99ua+bQ+aiJLgIZFn1l0StnIC0hxnuW96ypOz1C3cDu30nD?= =?us-ascii?Q?PJd9/30ag7AnLid8R5qExH2xOBJ8yoCfNZKKVeWyzUUP7zMsrle80s74zRQC?= =?us-ascii?Q?lNyay2c8sGzt50W0tDYZ4bHwhvz/tz9IVBrkhZz56R2wu4fimjQTtdTXoAb2?= =?us-ascii?Q?d+Pkqcg5WEc7+q1pPPzeo+8aztIqvHDK94YOz8JgRWX6HpkSmy1czjQqFCV+?= =?us-ascii?Q?1xtvFheOv1cUtoORT6MvWw1jZZQqWZMBmMqXKa5LvLuyJPqRqaT/6eAOp3pd?= =?us-ascii?Q?iVAXD76zIvd/TdkHacDCEtWOwev1onEmeb0lCUq/Gz4YOWXP/7cp3PrmszAK?= =?us-ascii?Q?exrGSPMOR5KEj+QyI3ABCJ9m2UzmvDRy57tfbnfW5+TgzWL79QGHq6RL00Sh?= =?us-ascii?Q?dAQb+oEqXqlzvdCYLLAUpofbla/N+2dG0Nckb27aD2ldxLcixaJmowZJq6aW?= =?us-ascii?Q?EfY2mVy0KAhLfpmhUpU6pHsVAtpsgjly148Iuua3nDYjNeaEbB9q2Ex9Ij3q?= =?us-ascii?Q?rZugC9VYzeszWnKAdvBQaF4S16Z3lVSLfTWghwRbkNkIuWjd1U+cKN8cqwBn?= =?us-ascii?Q?BvYZHrs9xNUhACWXi0jEL00+mmVHO8LjS/ORFlrua+Q87KyQcGKfRhS9SVM3?= =?us-ascii?Q?t9x++6iTn8lhV/f/GG0AbvT/h1V3JusYNGhCsR+nrikj6/x0mND5WTcMvhdn?= =?us-ascii?Q?Wcj3w+qLkdkipd5X7Zsbtea3Ukyme4xogAeleOV6X+bbF8m+ogeS8SVNQA4o?= =?us-ascii?Q?qYjKchb5xsE6GrhpWKWXiSknc+tdRDmPGkFuHZFM6hDOi8KYvWongAIYBs5f?= =?us-ascii?Q?13cd8hORVvVDog5oIdiqJTYobXO2xNgaswKLRUoubH7J6XdeQB7F19scyDTj?= =?us-ascii?Q?VN9uISnzIywKr0Z85W1SYzcstDD7ODwAI0Lkqe9BiU1kL7N3i+JMSKVTmstk?= =?us-ascii?Q?la8l7mreJ5YX80vkZp5w+0RgZRLesJADphja2kmaXY2Ju0bHlRZucKl0dz5i?= =?us-ascii?Q?xi8Qv0sPDaukvCrAVCrbcXBT4SaD2U7vJts38A+NtmfpZBBA1CmrdAKRAS3w?= =?us-ascii?Q?3oJ9s87Tv6cT5vgh3ODgJ+NKvwNZiwOiqgk47vi1F698ZjLJTlTOK60emW8Q?= =?us-ascii?Q?xcrd0yz8IIJA+A5S4HMZk/lbojyYp5MK?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:TYWP286MB2697.JPNP286.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(7416014)(376014)(10070799003)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?7ncvDeaMNkcp7cjLbFD/BI3+BLmAwzTijZ3Tb4cyGvH4B3b4hdVBLfvJPB8m?= =?us-ascii?Q?9rNXM+yezT5y6j17mPh3cfVZQ3AqMptSjlE2gF/LO+J7BtkfGr1XppLHBjq0?= =?us-ascii?Q?yfmIUCojW4M1freKEa08gkdA1E1kX5K+NIjewLHahNi9LLdZ8r6+Ua+em6bn?= =?us-ascii?Q?lIX/07JRQGryGCdWH3KozZUDm0Rb9hUrilqOkLhUlieCA9pxvpJGjqpppUtA?= =?us-ascii?Q?gW9jwMHhZGw1XIxnqNQt+q6AsKu0ZIt5wRvnIjXvnLLU+XqZW8kPDFm3NlGP?= =?us-ascii?Q?2EXL5e6U9fjjjIhgBI5PmMKuygNc9mOm+vwdY9BVMcCUgSQNJOCfRc6k2CMg?= =?us-ascii?Q?qX6j6I/RxkxnSsEf+AJG60zbDIc0DHJFRZA0LjNlS+YpiUc+YQ9vWYAScMhY?= =?us-ascii?Q?Hm8/dNIqZWGHlS3L0pm18GDCKT5Tg23yd2DytzNE8lp0sMmUDH4ZLi3mMnt6?= =?us-ascii?Q?RJTLtnJTf06jVsnSuAxuRXTLnZ7dLXSGVuS0amWxs0vxuWATDsUx/sCSAsqf?= =?us-ascii?Q?Ywzb+OtIXssqUyrlTTVqA3vBRqaZZEnT63ci/kJfyQW9zhmIzHwm4/eKSiNF?= =?us-ascii?Q?SQIWmXxhIDD0YazGfWN3hnatvb635mFuhRljVwHVYhQpPepiGxH8v5l9tJHC?= =?us-ascii?Q?dHml7zmxG+UySuRpZCGk9anFSuXijSnsZhqJDpjNwyNLQCSVJzB6bgIDqdH1?= =?us-ascii?Q?1IdCdGeX9HpV5wlt73X3BDaQ6/0I1XlwruHXaoN26gzPfS4AQB9PK/Nm5duD?= =?us-ascii?Q?FyCpvLCyxKAE4z0aP/72YgHcO+IjkN8Vn9ydH3OYI7kqAGdwAzWmiZx1G0kO?= =?us-ascii?Q?7hG1daAymBCR0myvhjotXjfK0hm9NQyGLhCj8ar6jagWUH9Yeh787OU1YWvB?= =?us-ascii?Q?v2kgT8N2netqpvLt90mG7vGIQdr1Dl4abN+0nAJ6J+OyTxJsPjdOQ4fspt7H?= =?us-ascii?Q?LtEuWBnMDbjWX7RyJ21gLmeQ5tuMwxLQyZoZLEWP7kVtnmvUj4rTam8vhk/G?= =?us-ascii?Q?B2evcE+qwsusJdDJZpAFDmAIL2vzOzXhSJXKC/hEBb+RSBqv4v3zGRFOsLb7?= =?us-ascii?Q?0RDWg0a0XAeopvjvYLr8kx7SsqKz4L1T8AVBtpaeWsgCH1lVtrNtcdaQ3nBl?= =?us-ascii?Q?pxtN/VOWs9ysK7QYkbyKTjlr8HEFVngdt+/GKZbtT+EftFbcclMOlBV0xbfc?= =?us-ascii?Q?KOWZRoc/H5hQJ8dOU8zQT3OCHtl5gaDOFW8ROnyUFQ7fluWasqpPlMV5sdTX?= =?us-ascii?Q?331jRBtofoLknQ4MoIepzIgW75Eo7HrAszY4tqjZ1P+hBcn9J0WYLPJ2v40e?= =?us-ascii?Q?9GLwFA8wSbhY2siCKRsYugKw/ayuoAM6W3fBcCf0fxCC2wVgHNxXITlEJ+uK?= =?us-ascii?Q?r0ChlTbzoVZ9PngsrOo2TM7MGUPzAshRv4NaUYhUa8OvwZFQRoa9HaKyVaSx?= =?us-ascii?Q?Cb6FXhskyom/e8FOxh/Q55BXc3MWwBqmRymYhLJarED5frBf7C+os1s5PeXh?= =?us-ascii?Q?I0ZOLKGi+fpTJHiS4MeUOaTkMUqytDqY5TtlCmv7KncyNluFiWhJjD1ugO0p?= =?us-ascii?Q?8Q45OOJtGSMC/3IEu8QU0Qo3klx+s9qUMQUVRRtjgZwiKfdxFsdms/BuI2Jv?= =?us-ascii?Q?5+Zsx3jJdtiwK7iJnGjqCsc=3D?= X-OriginatorOrg: valinux.co.jp X-MS-Exchange-CrossTenant-Network-Message-Id: aaf6cf86-2274-4725-ce24-08de3d7f459e X-MS-Exchange-CrossTenant-AuthSource: TYWP286MB2697.JPNP286.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Dec 2025 15:16:37.2474 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 7a57bee8-f73d-4c5f-a4f7-d72c91c8c111 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8c8/bQ09WLncXZW6yLHE4ZTCm6nT5JvpYl3H4Ijrd8RmG8FIcAg/Yu4mEkB21/57dhKgyEywVZ3xv+T+1Lca8g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYCP286MB2863 Content-Type: text/plain; charset="utf-8" Add a new ntb_transport backend that uses a DesignWare eDMA engine located on the endpoint, to be driven by both host and endpoint. The endpoint exposes a dedicated memory window which contains the eDMA register block, a small control structure (struct ntb_edma_info) and per-channel linked-list (LL) rings for read channels. Endpoint drives its local eDMA write channels for its transmission, while host side uses the remote eDMA read channels for its transmission. A key benefit of this backend is that the memory window no longer needs to carry data-plane payload. This makes the design less sensitive to limited memory window space and allows scaling to multiple queue pairs. The memory window layout is specific to the eDMA-backed backend, so there is no automatic fallback to the memcpy-based default transport that requires the different layout. Signed-off-by: Koichiro Den --- drivers/ntb/Kconfig | 12 + drivers/ntb/Makefile | 2 + drivers/ntb/ntb_transport_core.c | 15 +- drivers/ntb/ntb_transport_edma.c | 987 +++++++++++++++++++++++++++ drivers/ntb/ntb_transport_internal.h | 15 + 5 files changed, 1029 insertions(+), 2 deletions(-) create mode 100644 drivers/ntb/ntb_transport_edma.c diff --git a/drivers/ntb/Kconfig b/drivers/ntb/Kconfig index df16c755b4da..5ba6d0b7f5ba 100644 --- a/drivers/ntb/Kconfig +++ b/drivers/ntb/Kconfig @@ -37,4 +37,16 @@ config NTB_TRANSPORT =20 If unsure, say N. =20 +config NTB_TRANSPORT_EDMA + bool "NTB Transport backed by remote eDMA" + depends on NTB_TRANSPORT + depends on PCI + select DMA_ENGINE + select NTB_EDMA + help + Enable a transport backend that uses a remote DesignWare eDMA engine + exposed through a dedicated NTB memory window. The host uses the + endpoint's eDMA engine to move data in both directions. + Say Y here if you intend to use the 'use_remote_edma' module parameter. + endif # NTB diff --git a/drivers/ntb/Makefile b/drivers/ntb/Makefile index 9b66e5fafbc0..b9086b32ecde 100644 --- a/drivers/ntb/Makefile +++ b/drivers/ntb/Makefile @@ -6,3 +6,5 @@ ntb-y :=3D core.o ntb-$(CONFIG_NTB_MSI) +=3D msi.o =20 ntb_transport-y :=3D ntb_transport_core.o +ntb_transport-$(CONFIG_NTB_TRANSPORT_EDMA) +=3D ntb_transport_edma.o +ntb_transport-$(CONFIG_NTB_TRANSPORT_EDMA) +=3D hw/edma/ntb_hw_edma.o diff --git a/drivers/ntb/ntb_transport_core.c b/drivers/ntb/ntb_transport_c= ore.c index 40c2548f5930..bd21232f26fe 100644 --- a/drivers/ntb/ntb_transport_core.c +++ b/drivers/ntb/ntb_transport_core.c @@ -104,6 +104,12 @@ module_param(use_msi, bool, 0644); MODULE_PARM_DESC(use_msi, "Use MSI interrupts instead of doorbells"); #endif =20 +bool use_remote_edma; +#ifdef CONFIG_NTB_TRANSPORT_EDMA +module_param(use_remote_edma, bool, 0644); +MODULE_PARM_DESC(use_remote_edma, "Use remote eDMA mode (when enabled, use= _msi is ignored)"); +#endif + static struct dentry *nt_debugfs_dir; =20 /* Only two-ports NTB devices are supported */ @@ -156,7 +162,7 @@ enum { #define drv_client(__drv) \ container_of((__drv), struct ntb_transport_client, driver) =20 -#define NTB_QP_DEF_NUM_ENTRIES 100 +#define NTB_QP_DEF_NUM_ENTRIES 128 #define NTB_LINK_DOWN_TIMEOUT 10 =20 static void ntb_transport_rxc_db(unsigned long data); @@ -1189,7 +1195,11 @@ static int ntb_transport_probe(struct ntb_client *se= lf, struct ntb_dev *ndev) =20 nt->ndev =3D ndev; =20 - rc =3D ntb_transport_default_init(nt); + if (use_remote_edma) + rc =3D ntb_transport_edma_init(nt); + else + rc =3D ntb_transport_default_init(nt); + if (rc) return rc; =20 @@ -1950,6 +1960,7 @@ ntb_transport_create_queue(void *data, struct device = *client_dev, =20 nt->qp_bitmap_free &=3D ~qp_bit; =20 + qp->qp_bit =3D qp_bit; qp->cb_data =3D data; qp->rx_handler =3D handlers->rx_handler; qp->tx_handler =3D handlers->tx_handler; diff --git a/drivers/ntb/ntb_transport_edma.c b/drivers/ntb/ntb_transport_e= dma.c new file mode 100644 index 000000000000..6ae5da0a1367 --- /dev/null +++ b/drivers/ntb/ntb_transport_edma.c @@ -0,0 +1,987 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * NTB transport backend for remote DesignWare eDMA. + * + * This implements the backend_ops used when use_remote_edma=3D1 and + * relies on drivers/ntb/hw/edma/ for low-level eDMA/MW programming. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "hw/edma/ntb_hw_edma.h" +#include "ntb_transport_internal.h" + +#define NTB_EDMA_RING_ORDER 7 +#define NTB_EDMA_RING_ENTRIES (1U << NTB_EDMA_RING_ORDER) +#define NTB_EDMA_RING_MASK (NTB_EDMA_RING_ENTRIES - 1) + +#define NTB_EDMA_MAX_POLL 32 + +/* + * Remote eDMA mode implementation + */ +struct ntb_transport_ctx_edma { + remote_edma_mode_t remote_edma_mode; + struct device *dma_dev; + struct workqueue_struct *wq; + struct ntb_edma_chans chans; +}; + +struct ntb_transport_qp_edma { + struct ntb_transport_qp *qp; + + /* + * For ensuring peer notification in non-atomic context. + * ntb_peer_db_set might sleep or schedule. + */ + struct work_struct db_work; + + u32 rx_prod; + u32 rx_cons; + u32 tx_cons; + u32 tx_issue; + + spinlock_t rx_lock; + spinlock_t tx_lock; + + struct work_struct rx_work; + struct work_struct tx_work; +}; + +struct ntb_edma_desc { + u32 len; + u32 flags; + u64 addr; /* DMA address */ + u64 data; +}; + +struct ntb_edma_ring { + struct ntb_edma_desc desc[NTB_EDMA_RING_ENTRIES]; + u32 head; + u32 tail; +}; + +static inline bool ntb_qp_edma_is_rc(struct ntb_transport_qp *qp) +{ + struct ntb_transport_ctx_edma *ctx =3D qp->transport->priv; + + return ctx->remote_edma_mode =3D=3D REMOTE_EDMA_RC; +} + +static inline bool ntb_qp_edma_is_ep(struct ntb_transport_qp *qp) +{ + struct ntb_transport_ctx_edma *ctx =3D qp->transport->priv; + + return ctx->remote_edma_mode =3D=3D REMOTE_EDMA_EP; +} + +static inline bool ntb_qp_edma_enabled(struct ntb_transport_qp *qp) +{ + return ntb_qp_edma_is_rc(qp) || ntb_qp_edma_is_ep(qp); +} + +static inline unsigned int ntb_edma_ring_sel(struct ntb_transport_qp *qp, + unsigned int n) +{ + return n ^ !!ntb_qp_edma_is_ep(qp); +} + +static inline struct ntb_edma_ring * +ntb_edma_ring_local(struct ntb_transport_qp *qp, unsigned int n) +{ + unsigned int r =3D ntb_edma_ring_sel(qp, n); + + return &((struct ntb_edma_ring *)qp->rx_buff)[r]; +} + +static inline struct ntb_edma_ring __iomem * +ntb_edma_ring_remote(struct ntb_transport_qp *qp, unsigned int n) +{ + unsigned int r =3D ntb_edma_ring_sel(qp, n); + + return &((struct ntb_edma_ring __iomem *)qp->tx_mw)[r]; +} + +static inline struct ntb_edma_desc * +ntb_edma_desc_local(struct ntb_transport_qp *qp, unsigned int n, unsigned = int i) +{ + return &ntb_edma_ring_local(qp, n)->desc[i]; +} + +static inline struct ntb_edma_desc __iomem * +ntb_edma_desc_remote(struct ntb_transport_qp *qp, unsigned int n, + unsigned int i) +{ + return &ntb_edma_ring_remote(qp, n)->desc[i]; +} + +static inline u32 *ntb_edma_head_local(struct ntb_transport_qp *qp, + unsigned int n) +{ + return &ntb_edma_ring_local(qp, n)->head; +} + +static inline u32 __iomem *ntb_edma_head_remote(struct ntb_transport_qp *q= p, + unsigned int n) +{ + return &ntb_edma_ring_remote(qp, n)->head; +} + +static inline u32 *ntb_edma_tail_local(struct ntb_transport_qp *qp, + unsigned int n) +{ + return &ntb_edma_ring_local(qp, n)->tail; +} + +static inline u32 __iomem *ntb_edma_tail_remote(struct ntb_transport_qp *q= p, + unsigned int n) +{ + return &ntb_edma_ring_remote(qp, n)->tail; +} + +/* The 'i' must be generated by ntb_edma_ring_idx() */ +#define NTB_DESC_TX_O(qp, i) ntb_edma_desc_remote(qp, 0, i) +#define NTB_DESC_TX_I(qp, i) ntb_edma_desc_local(qp, 0, i) +#define NTB_DESC_RX_O(qp, i) ntb_edma_desc_remote(qp, 1, i) +#define NTB_DESC_RX_I(qp, i) ntb_edma_desc_local(qp, 1, i) + +#define NTB_HEAD_TX_I(qp) ntb_edma_head_local(qp, 0) +#define NTB_HEAD_RX_O(qp) ntb_edma_head_remote(qp, 1) + +#define NTB_TAIL_TX_O(qp) ntb_edma_tail_remote(qp, 0) +#define NTB_TAIL_RX_I(qp) ntb_edma_tail_local(qp, 1) + +/* ntb_edma_ring helpers */ +static __always_inline u32 ntb_edma_ring_idx(u32 v) +{ + return v & NTB_EDMA_RING_MASK; +} + +static __always_inline u32 ntb_edma_ring_used_entry(u32 head, u32 tail) +{ + if (head >=3D tail) { + WARN_ON_ONCE((head - tail) > (NTB_EDMA_RING_ENTRIES - 1)); + return head - tail; + } + + WARN_ON_ONCE((U32_MAX - tail + head + 1) > (NTB_EDMA_RING_ENTRIES - 1)); + return U32_MAX - tail + head + 1; +} + +static __always_inline u32 ntb_edma_ring_free_entry(u32 head, u32 tail) +{ + return NTB_EDMA_RING_ENTRIES - ntb_edma_ring_used_entry(head, tail) - 1; +} + +static __always_inline bool ntb_edma_ring_full(u32 head, u32 tail) +{ + return ntb_edma_ring_free_entry(head, tail) =3D=3D 0; +} + +static unsigned int ntb_transport_edma_tx_free_entry(struct ntb_transport_= qp *qp) +{ + struct ntb_transport_qp_edma *edma =3D qp->priv; + unsigned int head, tail; + + scoped_guard(spinlock_irqsave, &edma->tx_lock) { + /* In this scope, only 'head' might proceed */ + tail =3D READ_ONCE(edma->tx_issue); + head =3D READ_ONCE(*NTB_HEAD_TX_I(qp)); + } + /* + * 'used' amount indicates how much the other end has refilled, + * which are available for us to use for TX. + */ + return ntb_edma_ring_used_entry(head, tail); +} + +static void ntb_transport_edma_debugfs_stats_show(struct seq_file *s, + struct ntb_transport_qp *qp) +{ + seq_printf(s, "rx_bytes - \t%llu\n", qp->rx_bytes); + seq_printf(s, "rx_pkts - \t%llu\n", qp->rx_pkts); + seq_printf(s, "rx_err_no_buf - %llu\n", qp->rx_err_no_buf); + seq_printf(s, "rx_buff - \t0x%p\n", qp->rx_buff); + seq_printf(s, "rx_max_entry - \t%u\n", qp->rx_max_entry); + seq_printf(s, "rx_alloc_entry - \t%u\n\n", qp->rx_alloc_entry); + + seq_printf(s, "tx_bytes - \t%llu\n", qp->tx_bytes); + seq_printf(s, "tx_pkts - \t%llu\n", qp->tx_pkts); + seq_printf(s, "tx_ring_full - \t%llu\n", qp->tx_ring_full); + seq_printf(s, "tx_err_no_buf - %llu\n", qp->tx_err_no_buf); + seq_printf(s, "tx_mw - \t0x%p\n", qp->tx_mw); + seq_printf(s, "tx_max_entry - \t%u\n", qp->tx_max_entry); + seq_printf(s, "free tx - \t%u\n", ntb_transport_tx_free_entry(qp)); + seq_putc(s, '\n'); + + seq_puts(s, "Using Remote eDMA - Yes\n"); + seq_printf(s, "QP Link - \t%s\n", qp->link_is_up ? "Up" : "Down"); +} + +static void ntb_transport_edma_uninit(struct ntb_transport_ctx *nt) +{ + struct ntb_transport_ctx_edma *ctx =3D nt->priv; + + if (ctx->wq) + destroy_workqueue(ctx->wq); + ctx->wq =3D NULL; + + ntb_edma_teardown_chans(&ctx->chans); + + switch (ctx->remote_edma_mode) { + case REMOTE_EDMA_EP: + ntb_edma_teardown_mws(nt->ndev); + break; + case REMOTE_EDMA_RC: + ntb_edma_teardown_peer(nt->ndev); + break; + case REMOTE_EDMA_UNKNOWN: + default: + break; + } + + ctx->remote_edma_mode =3D REMOTE_EDMA_UNKNOWN; +} + +static void ntb_transport_edma_db_work(struct work_struct *work) +{ + struct ntb_transport_qp_edma *edma =3D + container_of(work, struct ntb_transport_qp_edma, db_work); + struct ntb_transport_qp *qp =3D edma->qp; + + ntb_peer_db_set(qp->ndev, qp->qp_bit); +} + +static void ntb_transport_edma_notify_peer(struct ntb_transport_qp_edma *e= dma) +{ + struct ntb_transport_qp *qp =3D edma->qp; + struct ntb_transport_ctx_edma *ctx =3D qp->transport->priv; + + if (!ntb_edma_notify_peer(&ctx->chans, qp->qp_num)) + return; + + /* + * Called from contexts that may be atomic. Since ntb_peer_db_set() + * may sleep, delegate the actual doorbell write to a workqueue. + */ + queue_work(system_highpri_wq, &edma->db_work); +} + +static void ntb_transport_edma_isr(void *data, int qp_num) +{ + struct ntb_transport_ctx *nt =3D data; + struct ntb_transport_qp_edma *edma; + struct ntb_transport_ctx_edma *ctx; + struct ntb_transport_qp *qp; + + if (qp_num < 0 || qp_num >=3D nt->qp_count) + return; + + qp =3D &nt->qp_vec[qp_num]; + if (WARN_ON(!qp)) + return; + + ctx =3D (struct ntb_transport_ctx_edma *)qp->transport->priv; + edma =3D qp->priv; + + queue_work(ctx->wq, &edma->rx_work); + queue_work(ctx->wq, &edma->tx_work); +} + +static int ntb_transport_edma_rc_init(struct ntb_transport_ctx *nt) +{ + struct ntb_transport_ctx_edma *ctx =3D nt->priv; + struct ntb_dev *ndev =3D nt->ndev; + struct pci_dev *pdev =3D ndev->pdev; + int peer_mw; + int rc; + + if (!use_remote_edma || ctx->remote_edma_mode !=3D REMOTE_EDMA_UNKNOWN) + return 0; + + peer_mw =3D ntb_peer_mw_count(ndev); + if (peer_mw <=3D 0) + return -ENODEV; + + rc =3D ntb_edma_setup_peer(ndev, peer_mw - 1, nt->qp_count); + if (rc) { + dev_err(&pdev->dev, "Failed to enable remote eDMA: %d\n", rc); + return rc; + } + + rc =3D ntb_edma_setup_chans(get_dma_dev(ndev), &ctx->chans, true); + if (rc) { + dev_err(&pdev->dev, "Failed to setup eDMA channels: %d\n", rc); + goto err_teardown_peer; + } + + rc =3D ntb_edma_setup_intr_chan(get_dma_dev(ndev), &ctx->chans); + if (rc) { + dev_err(&pdev->dev, "Failed to setup eDMA notify channel: %d\n", + rc); + goto err_teardown_chans; + } + + ctx->remote_edma_mode =3D REMOTE_EDMA_RC; + return 0; + +err_teardown_chans: + ntb_edma_teardown_chans(&ctx->chans); +err_teardown_peer: + ntb_edma_teardown_peer(ndev); + return rc; +} + + +static int ntb_transport_edma_ep_init(struct ntb_transport_ctx *nt) +{ + struct ntb_transport_ctx_edma *ctx =3D nt->priv; + struct ntb_dev *ndev =3D nt->ndev; + struct pci_dev *pdev =3D ndev->pdev; + int peer_mw; + int rc; + + if (!use_remote_edma || ctx->remote_edma_mode =3D=3D REMOTE_EDMA_EP) + return 0; + + /** + * This check assumes that the endpoint (pci-epf-vntb.c) + * ntb_dev_ops implements .get_private_data() while the host side + * (ntb_hw_epf.c) does not. + */ + if (!ntb_get_private_data(ndev)) + return 0; + + peer_mw =3D ntb_peer_mw_count(ndev); + if (peer_mw <=3D 0) + return -ENODEV; + + rc =3D ntb_edma_setup_mws(ndev, peer_mw - 1, nt->qp_count, + ntb_transport_edma_isr, nt); + if (rc) { + dev_err(&pdev->dev, + "Failed to set up memory window for eDMA: %d\n", rc); + return rc; + } + + rc =3D ntb_edma_setup_chans(get_dma_dev(ndev), &ctx->chans, false); + if (rc) { + dev_err(&pdev->dev, "Failed to setup eDMA channels: %d\n", rc); + ntb_edma_teardown_mws(ndev); + return rc; + } + + ctx->remote_edma_mode =3D REMOTE_EDMA_EP; + return 0; +} + + +static int ntb_transport_edma_setup_qp_mw(struct ntb_transport_ctx *nt, + unsigned int qp_num) +{ + struct ntb_transport_qp *qp =3D &nt->qp_vec[qp_num]; + struct ntb_dev *ndev =3D nt->ndev; + struct ntb_queue_entry *entry; + struct ntb_transport_mw *mw; + unsigned int mw_num, mw_count, qp_count; + unsigned int qp_offset, rx_info_offset; + unsigned int mw_size, mw_size_per_qp; + unsigned int num_qps_mw; + size_t edma_total; + unsigned int i; + int node; + + mw_count =3D nt->mw_count; + qp_count =3D nt->qp_count; + + mw_num =3D QP_TO_MW(nt, qp_num); + mw =3D &nt->mw_vec[mw_num]; + + if (!mw->virt_addr) + return -ENOMEM; + + if (mw_num < qp_count % mw_count) + num_qps_mw =3D qp_count / mw_count + 1; + else + num_qps_mw =3D qp_count / mw_count; + + mw_size =3D min(nt->mw_vec[mw_num].phys_size, mw->xlat_size); + if (max_mw_size && mw_size > max_mw_size) + mw_size =3D max_mw_size; + + mw_size_per_qp =3D round_down((unsigned int)mw_size / num_qps_mw, SZ_64); + qp_offset =3D mw_size_per_qp * (qp_num / mw_count); + rx_info_offset =3D mw_size_per_qp - sizeof(struct ntb_rx_info); + + qp->tx_mw_size =3D mw_size_per_qp; + qp->tx_mw =3D nt->mw_vec[mw_num].vbase + qp_offset; + if (!qp->tx_mw) + return -EINVAL; + qp->tx_mw_phys =3D nt->mw_vec[mw_num].phys_addr + qp_offset; + if (!qp->tx_mw_phys) + return -EINVAL; + qp->rx_info =3D qp->tx_mw + rx_info_offset; + qp->rx_buff =3D mw->virt_addr + qp_offset; + qp->remote_rx_info =3D qp->rx_buff + rx_info_offset; + + /* Due to housekeeping, there must be at least 2 buffs */ + qp->tx_max_frame =3D min(transport_mtu, mw_size_per_qp / 2); + qp->rx_max_frame =3D min(transport_mtu, mw_size_per_qp / 2); + + /* In eDMA mode, decouple from MW sizing and force ring-sized entries */ + edma_total =3D 2 * sizeof(struct ntb_edma_ring); + if (rx_info_offset < edma_total) { + dev_err(&ndev->dev, "Ring space requires %zuB (>=3D%uB)\n", + edma_total, rx_info_offset); + return -EINVAL; + } + qp->tx_max_entry =3D NTB_EDMA_RING_ENTRIES; + qp->rx_max_entry =3D NTB_EDMA_RING_ENTRIES; + + /* + * Checking to see if we have more entries than the default. + * We should add additional entries if that is the case so we + * can be in sync with the transport frames. + */ + node =3D dev_to_node(&ndev->dev); + for (i =3D qp->rx_alloc_entry; i < qp->rx_max_entry; i++) { + entry =3D kzalloc_node(sizeof(*entry), GFP_KERNEL, node); + if (!entry) + return -ENOMEM; + + entry->qp =3D qp; + ntb_list_add(&qp->ntb_rx_q_lock, &entry->entry, + &qp->rx_free_q); + qp->rx_alloc_entry++; + } + + memset(qp->rx_buff, 0, edma_total); + + qp->rx_pkts =3D 0; + qp->tx_pkts =3D 0; + + return 0; +} + +static int ntb_transport_edma_rx_complete(struct ntb_transport_qp *qp) +{ + struct device *dma_dev =3D get_dma_dev(qp->ndev); + struct ntb_transport_qp_edma *edma =3D qp->priv; + struct ntb_queue_entry *entry; + struct ntb_edma_desc *in; + unsigned int len; + bool link_down; + u32 idx; + + if (ntb_edma_ring_used_entry(READ_ONCE(*NTB_TAIL_RX_I(qp)), + edma->rx_cons) =3D=3D 0) + return 0; + + idx =3D ntb_edma_ring_idx(edma->rx_cons); + in =3D NTB_DESC_RX_I(qp, idx); + if (!(in->flags & DESC_DONE_FLAG)) + return 0; + + link_down =3D in->flags & LINK_DOWN_FLAG; + in->flags =3D 0; + len =3D in->len; /* might be smaller than entry->len */ + + entry =3D (struct ntb_queue_entry *)(uintptr_t)in->data; + if (WARN_ON(!entry)) + return 0; + + if (link_down) { + ntb_qp_link_down(qp); + edma->rx_cons++; + ntb_list_add(&qp->ntb_rx_q_lock, &entry->entry, &qp->rx_free_q); + return 1; + } + + dma_unmap_single(dma_dev, entry->addr, entry->len, DMA_FROM_DEVICE); + + qp->rx_bytes +=3D len; + qp->rx_pkts++; + edma->rx_cons++; + + if (qp->rx_handler && qp->client_ready) + qp->rx_handler(qp, qp->cb_data, entry->cb_data, len); + + ntb_list_add(&qp->ntb_rx_q_lock, &entry->entry, &qp->rx_free_q); + return 1; +} + +static void ntb_transport_edma_rx_work(struct work_struct *work) +{ + struct ntb_transport_qp_edma *edma =3D container_of( + work, struct ntb_transport_qp_edma, rx_work); + struct ntb_transport_qp *qp =3D edma->qp; + struct ntb_transport_ctx_edma *ctx =3D qp->transport->priv; + unsigned int i; + + for (i =3D 0; i < NTB_EDMA_MAX_POLL; i++) { + if (!ntb_transport_edma_rx_complete(qp)) + break; + } + + if (ntb_transport_edma_rx_complete(qp)) + queue_work(ctx->wq, &edma->rx_work); +} + +static void ntb_transport_edma_tx_work(struct work_struct *work) +{ + struct ntb_transport_qp_edma *edma =3D container_of( + work, struct ntb_transport_qp_edma, tx_work); + struct ntb_transport_qp *qp =3D edma->qp; + struct ntb_edma_desc *in, __iomem *out; + struct ntb_queue_entry *entry; + unsigned int len; + void *cb_data; + u32 idx; + + while (ntb_edma_ring_used_entry(READ_ONCE(edma->tx_issue), + edma->tx_cons) !=3D 0) { + /* Paired with smp_wmb() in ntb_transport_edma_tx_enqueue_inner() */ + smp_rmb(); + + idx =3D ntb_edma_ring_idx(edma->tx_cons); + in =3D NTB_DESC_TX_I(qp, idx); + entry =3D (struct ntb_queue_entry *)(uintptr_t)in->data; + if (!entry || !(entry->flags & DESC_DONE_FLAG)) + break; + + in->data =3D 0; + + cb_data =3D entry->cb_data; + len =3D entry->len; + + out =3D NTB_DESC_TX_O(qp, idx); + + WRITE_ONCE(edma->tx_cons, edma->tx_cons + 1); + + /* + * No need to add barrier in-between to enforce ordering here. + * The other side proceeds only after both flags and tail are + * updated. + */ + iowrite32(entry->flags, &out->flags); + iowrite32(edma->tx_cons, NTB_TAIL_TX_O(qp)); + + ntb_transport_edma_notify_peer(edma); + + ntb_list_add(&qp->ntb_tx_free_q_lock, &entry->entry, + &qp->tx_free_q); + + if (qp->tx_handler) + qp->tx_handler(qp, qp->cb_data, cb_data, len); + + /* stat updates */ + qp->tx_bytes +=3D len; + qp->tx_pkts++; + } +} + +static void ntb_transport_edma_tx_cb(void *data, + const struct dmaengine_result *res) +{ + struct ntb_queue_entry *entry =3D data; + struct ntb_transport_qp *qp =3D entry->qp; + struct ntb_transport_ctx *nt =3D qp->transport; + struct device *dma_dev =3D get_dma_dev(qp->ndev); + enum dmaengine_tx_result dma_err =3D res->result; + struct ntb_transport_ctx_edma *ctx =3D nt->priv; + struct ntb_transport_qp_edma *edma =3D qp->priv; + + switch (dma_err) { + case DMA_TRANS_READ_FAILED: + case DMA_TRANS_WRITE_FAILED: + case DMA_TRANS_ABORTED: + entry->errors++; + entry->len =3D -EIO; + break; + case DMA_TRANS_NOERROR: + default: + break; + } + dma_unmap_sg(dma_dev, &entry->sgl, 1, DMA_TO_DEVICE); + sg_dma_address(&entry->sgl) =3D 0; + + entry->flags |=3D DESC_DONE_FLAG; + + queue_work(ctx->wq, &edma->tx_work); +} + +static int ntb_transport_edma_submit(struct device *d, struct dma_chan *ch= an, + size_t len, void *rc_src, dma_addr_t dst, + struct ntb_queue_entry *entry) +{ + struct scatterlist *sgl =3D &entry->sgl; + struct dma_async_tx_descriptor *txd; + struct dma_slave_config cfg; + dma_cookie_t cookie; + int nents, rc; + + if (!d) + return -ENODEV; + + if (!chan) + return -ENXIO; + + if (WARN_ON(!rc_src || !dst)) + return -EINVAL; + + if (WARN_ON(sg_dma_address(sgl))) + return -EINVAL; + + sg_init_one(sgl, rc_src, len); + nents =3D dma_map_sg(d, sgl, 1, DMA_TO_DEVICE); + if (nents <=3D 0) + return -EIO; + + memset(&cfg, 0, sizeof(cfg)); + cfg.dst_addr =3D dst; + cfg.src_addr_width =3D DMA_SLAVE_BUSWIDTH_4_BYTES; + cfg.dst_addr_width =3D DMA_SLAVE_BUSWIDTH_4_BYTES; + cfg.direction =3D DMA_MEM_TO_DEV; + + txd =3D dmaengine_prep_slave_sg_config(chan, sgl, 1, DMA_MEM_TO_DEV, + DMA_CTRL_ACK | DMA_PREP_INTERRUPT, &cfg); + if (!txd) { + rc =3D -EIO; + goto out_unmap; + } + + txd->callback_result =3D ntb_transport_edma_tx_cb; + txd->callback_param =3D entry; + + cookie =3D dmaengine_submit(txd); + if (dma_submit_error(cookie)) { + rc =3D -EIO; + goto out_unmap; + } + dma_async_issue_pending(chan); + return 0; +out_unmap: + dma_unmap_sg(d, sgl, 1, DMA_TO_DEVICE); + return rc; +} + +static int ntb_transport_edma_tx_enqueue_inner(struct ntb_transport_qp *qp, + struct ntb_queue_entry *entry) +{ + struct device *dma_dev =3D get_dma_dev(qp->ndev); + struct ntb_transport_qp_edma *edma =3D qp->priv; + struct ntb_transport_ctx *nt =3D qp->transport; + struct ntb_edma_desc *in, __iomem *out; + struct ntb_transport_ctx_edma *ctx =3D nt->priv; + unsigned int len =3D entry->len; + struct dma_chan *chan; + u32 issue, idx, head; + dma_addr_t dst; + int rc; + + WARN_ON_ONCE(entry->flags & DESC_DONE_FLAG); + + scoped_guard(spinlock_irqsave, &edma->tx_lock) { + head =3D READ_ONCE(*NTB_HEAD_TX_I(qp)); + issue =3D edma->tx_issue; + if (ntb_edma_ring_used_entry(head, issue) =3D=3D 0) { + qp->tx_ring_full++; + return -ENOSPC; + } + + /* + * ntb_transport_edma_tx_work() checks entry->flags + * so it needs to be set before tx_issue++. + */ + idx =3D ntb_edma_ring_idx(issue); + in =3D NTB_DESC_TX_I(qp, idx); + in->data =3D (uintptr_t)entry; + + /* Make in->data visible before tx_issue++ */ + smp_wmb(); + + WRITE_ONCE(edma->tx_issue, edma->tx_issue + 1); + } + + /* Publish the final transfer length to the other end */ + out =3D NTB_DESC_TX_O(qp, idx); + iowrite32(len, &out->len); + ioread32(&out->len); + + if (unlikely(!len)) { + entry->flags |=3D DESC_DONE_FLAG; + queue_work(ctx->wq, &edma->tx_work); + return 0; + } + + /* Paired with dma_wmb() in ntb_transport_edma_rx_enqueue_inner() */ + dma_rmb(); + + /* kick remote eDMA read transfer */ + dst =3D (dma_addr_t)in->addr; + chan =3D ntb_edma_pick_chan(&ctx->chans, qp->qp_num); + rc =3D ntb_transport_edma_submit(dma_dev, chan, len, + entry->buf, dst, entry); + if (rc) { + entry->errors++; + entry->len =3D -EIO; + entry->flags |=3D DESC_DONE_FLAG; + queue_work(ctx->wq, &edma->tx_work); + } + return 0; +} + +static int ntb_transport_edma_tx_enqueue(struct ntb_transport_qp *qp, + struct ntb_queue_entry *entry, + void *cb, void *data, unsigned int len, + unsigned int flags) +{ + struct device *dma_dev; + + if (entry->addr) { + /* Deferred unmap */ + dma_dev =3D get_dma_dev(qp->ndev); + dma_unmap_single(dma_dev, entry->addr, entry->len, + DMA_TO_DEVICE); + } + + entry->cb_data =3D cb; + entry->buf =3D data; + entry->len =3D len; + entry->flags =3D flags; + entry->errors =3D 0; + entry->addr =3D 0; + + WARN_ON_ONCE(!ntb_qp_edma_enabled(qp)); + + return ntb_transport_edma_tx_enqueue_inner(qp, entry); +} + +static int ntb_transport_edma_rx_enqueue_inner(struct ntb_transport_qp *qp, + struct ntb_queue_entry *entry) +{ + struct device *dma_dev =3D get_dma_dev(qp->ndev); + struct ntb_transport_qp_edma *edma =3D qp->priv; + struct ntb_edma_desc *in, __iomem *out; + unsigned int len =3D entry->len; + void *data =3D entry->buf; + dma_addr_t dst; + u32 idx; + int rc; + + dst =3D dma_map_single(dma_dev, data, len, DMA_FROM_DEVICE); + rc =3D dma_mapping_error(dma_dev, dst); + if (rc) + return rc; + + guard(spinlock_bh)(&edma->rx_lock); + + if (ntb_edma_ring_full(READ_ONCE(edma->rx_prod), + READ_ONCE(edma->rx_cons))) { + rc =3D -ENOSPC; + goto out_unmap; + } + + idx =3D ntb_edma_ring_idx(edma->rx_prod); + in =3D NTB_DESC_RX_I(qp, idx); + out =3D NTB_DESC_RX_O(qp, idx); + + iowrite32(len, &out->len); + iowrite64(dst, &out->addr); + + WARN_ON(in->flags & DESC_DONE_FLAG); + in->data =3D (uintptr_t)entry; + entry->addr =3D dst; + + /* Ensure len/addr are visible before the head update */ + dma_wmb(); + + WRITE_ONCE(edma->rx_prod, edma->rx_prod + 1); + iowrite32(edma->rx_prod, NTB_HEAD_RX_O(qp)); + + return 0; +out_unmap: + dma_unmap_single(dma_dev, dst, len, DMA_FROM_DEVICE); + return rc; +} + +static int ntb_transport_edma_rx_enqueue(struct ntb_transport_qp *qp, + struct ntb_queue_entry *entry) +{ + int rc; + + rc =3D ntb_transport_edma_rx_enqueue_inner(qp, entry); + if (rc) { + ntb_list_add(&qp->ntb_rx_q_lock, &entry->entry, + &qp->rx_free_q); + return rc; + } + + ntb_list_add(&qp->ntb_rx_q_lock, &entry->entry, &qp->rx_pend_q); + + if (qp->active) + tasklet_schedule(&qp->rxc_db_work); + + return 0; +} + +static void ntb_transport_edma_rx_poll(struct ntb_transport_qp *qp) +{ + struct ntb_transport_ctx *nt =3D qp->transport; + struct ntb_transport_ctx_edma *ctx =3D nt->priv; + struct ntb_transport_qp_edma *edma =3D qp->priv; + + queue_work(ctx->wq, &edma->rx_work); + queue_work(ctx->wq, &edma->tx_work); +} + +static int ntb_transport_edma_qp_init(struct ntb_transport_ctx *nt, + unsigned int qp_num) +{ + struct ntb_transport_qp *qp =3D &nt->qp_vec[qp_num]; + struct ntb_transport_qp_edma *edma; + struct ntb_dev *ndev =3D nt->ndev; + int node; + + node =3D dev_to_node(&ndev->dev); + + qp->priv =3D kzalloc_node(sizeof(*edma), GFP_KERNEL, node); + if (!qp->priv) + return -ENOMEM; + + edma =3D (struct ntb_transport_qp_edma *)qp->priv; + edma->qp =3D qp; + edma->rx_prod =3D 0; + edma->rx_cons =3D 0; + edma->tx_cons =3D 0; + edma->tx_issue =3D 0; + + spin_lock_init(&edma->rx_lock); + spin_lock_init(&edma->tx_lock); + + INIT_WORK(&edma->db_work, ntb_transport_edma_db_work); + INIT_WORK(&edma->rx_work, ntb_transport_edma_rx_work); + INIT_WORK(&edma->tx_work, ntb_transport_edma_tx_work); + + return 0; +} + +static void ntb_transport_edma_qp_free(struct ntb_transport_qp *qp) +{ + struct ntb_transport_qp_edma *edma =3D qp->priv; + + cancel_work_sync(&edma->db_work); + cancel_work_sync(&edma->rx_work); + cancel_work_sync(&edma->tx_work); + + kfree(qp->priv); +} + +static int ntb_transport_edma_pre_link_up(struct ntb_transport_ctx *nt) +{ + struct ntb_dev *ndev =3D nt->ndev; + struct pci_dev *pdev =3D ndev->pdev; + int rc; + + rc =3D ntb_transport_edma_ep_init(nt); + if (rc) + dev_err(&pdev->dev, "Failed to init EP: %d\n", rc); + + return rc; +} + +static int ntb_transport_edma_post_link_up(struct ntb_transport_ctx *nt) +{ + struct ntb_dev *ndev =3D nt->ndev; + struct pci_dev *pdev =3D ndev->pdev; + int rc; + + rc =3D ntb_transport_edma_rc_init(nt); + if (rc) + dev_err(&pdev->dev, "Failed to init RC: %d\n", rc); + + return rc; +} + +static int ntb_transport_edma_enable(struct ntb_transport_ctx *nt, + unsigned int *mw_count) +{ + struct ntb_dev *ndev =3D nt->ndev; + struct ntb_transport_ctx_edma *ctx =3D nt->priv; + + if (!use_remote_edma) + return 0; + + /* + * We need at least one MW for the transport plus one MW reserved + * for the remote eDMA window (see ntb_edma_setup_mws/peer). + */ + if (*mw_count <=3D 1) { + dev_err(&ndev->dev, + "remote eDMA requires at least two MWS (have %u)\n", + *mw_count); + return -ENODEV; + } + + ctx->wq =3D alloc_workqueue("ntb-edma-wq", WQ_UNBOUND | WQ_SYSFS, 0); + if (!ctx->wq) { + ntb_transport_edma_uninit(nt); + return -ENOMEM; + } + + /* Reserve the last peer MW exclusively for the eDMA window. */ + *mw_count -=3D 1; + + return 0; +} + +static void ntb_transport_edma_disable(struct ntb_transport_ctx *nt) +{ + ntb_transport_edma_uninit(nt); +} + +static const struct ntb_transport_backend_ops edma_backend_ops =3D { + .enable =3D ntb_transport_edma_enable, + .disable =3D ntb_transport_edma_disable, + .qp_init =3D ntb_transport_edma_qp_init, + .qp_free =3D ntb_transport_edma_qp_free, + .pre_link_up =3D ntb_transport_edma_pre_link_up, + .post_link_up =3D ntb_transport_edma_post_link_up, + .setup_qp_mw =3D ntb_transport_edma_setup_qp_mw, + .tx_free_entry =3D ntb_transport_edma_tx_free_entry, + .tx_enqueue =3D ntb_transport_edma_tx_enqueue, + .rx_enqueue =3D ntb_transport_edma_rx_enqueue, + .rx_poll =3D ntb_transport_edma_rx_poll, + .debugfs_stats_show =3D ntb_transport_edma_debugfs_stats_show, +}; + +int ntb_transport_edma_init(struct ntb_transport_ctx *nt) +{ + struct ntb_dev *ndev =3D nt->ndev; + int node; + + node =3D dev_to_node(&ndev->dev); + nt->priv =3D kzalloc_node(sizeof(struct ntb_transport_ctx_edma), GFP_KERN= EL, + node); + if (!nt->priv) + return -ENOMEM; + + nt->backend_ops =3D edma_backend_ops; + /* + * On remote eDMA mode, one DMA read channel is used for Host side + * to interrupt EP. + */ + use_msi =3D false; + return 0; +} diff --git a/drivers/ntb/ntb_transport_internal.h b/drivers/ntb/ntb_transpo= rt_internal.h index 51ff08062d73..9fff65980d3d 100644 --- a/drivers/ntb/ntb_transport_internal.h +++ b/drivers/ntb/ntb_transport_internal.h @@ -8,6 +8,7 @@ extern unsigned long max_mw_size; extern unsigned int transport_mtu; extern bool use_msi; +extern bool use_remote_edma; =20 #define QP_TO_MW(nt, qp) ((qp) % nt->mw_count) =20 @@ -29,6 +30,11 @@ struct ntb_queue_entry { struct ntb_payload_header __iomem *tx_hdr; struct ntb_payload_header *rx_hdr; }; + +#ifdef CONFIG_NTB_TRANSPORT_EDMA + dma_addr_t addr; + struct scatterlist sgl; +#endif }; =20 struct ntb_rx_info { @@ -202,4 +208,13 @@ int ntb_transport_init_queue(struct ntb_transport_ctx = *nt, unsigned int qp_num); struct device *get_dma_dev(struct ntb_dev *ndev); =20 +#ifdef CONFIG_NTB_TRANSPORT_EDMA +int ntb_transport_edma_init(struct ntb_transport_ctx *nt); +#else +static inline int ntb_transport_edma_init(struct ntb_transport_ctx *nt) +{ + return -EOPNOTSUPP; +} +#endif /* CONFIG_NTB_TRANSPORT_EDMA */ + #endif /* _NTB_TRANSPORT_INTERNAL_H_ */ --=20 2.51.0