From nobody Fri Jul 24 21:52:58 2026 Received: from SJ2PR03CU001.outbound.protection.outlook.com (mail-westusazon11012042.outbound.protection.outlook.com [52.101.43.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 88B863DB633; Thu, 23 Jul 2026 07:23:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.43.42 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784791403; cv=fail; b=kV8CDthzoA2VxqQVw7uOSsec9FWC2iodRNKCqQrQKKNGKhEQ4hv17EWnuY/5NDvDFKxHNRyTUqkTr//oD9rwaGSr6XUAiz0OfEpgVfTMM1+Da9jDfNV5UzA0k7rOZzHnMKQ8OTt2SHs8rinqZS4rczxyorUPaTQEV3FdbYpvIqU= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784791403; c=relaxed/simple; bh=un+WLtelwryIBQD5MJBt+TGs30fQIcCABwn7kYcQHR4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=uYLWv1c+NxI6RL4d6wFHlJql7W8/0k6ejB1NbW/S0iMwIYgfBJrzP6fV68fjicKMQ2BrPG55HiQS4M7CCrsniV5muP5IDZ1o+jqNz6VzUvX2BScHdYI3SvsgItdqM3REQr2wga5ew+H8rxw0qoB2nnO/A9autCDKUYOhdIkBaoQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=DhTZsoJu; arc=fail smtp.client-ip=52.101.43.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="DhTZsoJu" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=TE+O+rjzcxWFX1WrMXz3WlAPFBDYxSX0TF/6AUTIWdwrMbYpAuoxqTikdT3dbTIHplTOryNu0rXw7JnuHuomCxdqVaLlDRdvfCt1TQ7tQKqJo2fc+L4Irr1SqKuJM0F4AhBFAH+B97KmynL2KM9SDMurIPSf4d9YeI9Lvis32/vyyuXJKbXjtUBtmydKAqLudO99Uq+cKiE60hln5f+qk0H0gLY9e8M4m1/9gPeJRZkC7H+w15vOsoQvjF0Hg9sV2aRk/3aIatLvCJMgwe1WousCB8tq7UZMrEMAO+sroQzfmd0ODiv015PR6dor3zjjMypTwjXOrPqQy/4mJ9EsOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pbFwkqqVv0uqCxKjxsa4wg4lsKdOvR5SNZm3cVy3/oc=; b=b+hpwFPeXNX/plAknvL9oeub0y7Sb5+sT/kA9sxLo1qaDtrJqaVnFumOTJSA3owItaQRDzKxyDhnHsUSHpqaVdq4V2LfPxl+fXJtKtbIMPJDE/Bh2qMfMUUdq+HglxYE2T9iqx9DlrvU1mo40pphvMMU+nVcbxxcvUySXNbt8O6xw6WFEPqYXzCcBWp+L5KwIKGjOu/ir0mCesPm/JbJNFMRdfuRRPROkBPIFqHkupYRR3fjwpxITfIkzfPWz/JLBOn+tEG34cxhAQAMTOCbM5mxAQcQQ6ARkMDJFz9OQ56hNjU01CNcaIS/IAAqDk5L0EL6BqHmpIE9ni+c24MUFA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=lunn.ch smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pbFwkqqVv0uqCxKjxsa4wg4lsKdOvR5SNZm3cVy3/oc=; b=DhTZsoJunmU4xK0vEUYwCvzJ82f074QXhZE4tCVIkIcH94W2+NMxF+aLwR/gRHs+r4evrAzAo3EDQ9RkJJekzvDW59qMPNcrmaXWUmZvr50ETS39fTmC62bAyjfZTtP8LqY618Tcem8qznxso6A+1fDyUXERzQgZvvl3hwoqdNCeBd9Pkb1gft4mElcS5z4gk/2lXg6ptoaLsrnQJ1tIyJN0wKTNmjNdO0xd7M7X1ZaExtZglQKh4b1nMi7fQa66IW2lyxvJrFtUlXaLkU0IvFlun3KbpdE/buVFvr2GD5DacT1lKXcOW6/R2QnHZ+ZJkHJW7ZRnBhZcR2Hc8Nmoww== Received: from DS1P221CA0023.NAMP221.PROD.OUTLOOK.COM (2603:10b6:8:242::6) by CH3PR12MB7571.namprd12.prod.outlook.com (2603:10b6:610:147::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.245.11; Thu, 23 Jul 2026 07:23:12 +0000 Received: from DS2PEPF000061C4.namprd02.prod.outlook.com (2603:10b6:8:242:cafe::24) by DS1P221CA0023.outlook.office365.com (2603:10b6:8:242::6) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.245.11 via Frontend Transport; Thu, 23 Jul 2026 07:23:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DS2PEPF000061C4.mail.protection.outlook.com (10.167.23.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.270.5 via Frontend Transport; Thu, 23 Jul 2026 07:23:12 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Thu, 23 Jul 2026 00:22:59 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Thu, 23 Jul 2026 00:22:58 -0700 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.2562.20 via Frontend Transport; Thu, 23 Jul 2026 00:22:54 -0700 From: Tariq Toukan To: Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , , Paolo Abeni CC: Dragos Tatulea , Gal Pressman , "Leon Romanovsky" , , , Mark Bloch , Nimrod Oren , Saeed Mahameed , Tariq Toukan Subject: [PATCH net-next 1/3] net/mlx5: initialize doorbell dma pools Date: Thu, 23 Jul 2026 10:22:20 +0300 Message-ID: <20260723072222.1863558-2-tariqt@nvidia.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20260723072222.1863558-1-tariqt@nvidia.com> References: <20260723072222.1863558-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PEPF000061C4:EE_|CH3PR12MB7571:EE_ X-MS-Office365-Filtering-Correlation-Id: 0536a0e7-670c-4594-ced7-08dee88b4103 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|1800799024|36860700016|82310400026|376014|56012099006|10067099003|22082099003|18002099003|11063799006; X-Microsoft-Antispam-Message-Info: LRfQ7Bd4kJ9BrIyXEda0Qrn1pXwx0pA/2pfguzRKWYuZTea4FErGsIc8DSwjIdzuWVthcPm3bvNRKLT1dVFZN9HTrz4mydFB3nFP3LNNw1hXOAx8wuPe8oAGrMWB4euMFg4Ydu4ynrjnLdREha3D2eVVPzPLgbB/12zxmZU0LIwaFEjXQNcqym1jJU1+t1oSlrhyUFV4gqC2ZF5Gfu65o6TzSqduX8mJgNvUscaq8WcP0BMO3+WVaGoIf72Gv/NcQmdO32KqfSg00vwq13d3buwW7Xz62HKATc8VmO1sKD6vUMt7P2HjtodUtBP8NHjaqc86uVlIWwn1YigJbMtbXiKMbjXRyQayI+TaEJnbw2GYpfK0RscrQktWlNEno/jWXzCOPUtktSAxLPo0/vHnhrk1pqYSrxhYwpzJmzmf34iMSmgbBFo4RBuN3b+Fu6MTWeJcp5F+5X+LDxoqhbuHA/AZCh80Ihd6BiQ5+wLJ6MXV3aFIW5wRfvwWcoH0A97Zg0nhoVwO0pBArgg1pjN8QJPVyTZ3Y4SEMeq7VhneF4eBIh5hhQ6/gulkpKfeAmEFmMnGzlYP+RsLabp2D/at6gbsYGEM1robCNM33dNn5o1Kj5LB6upSip6/Pr9fBijcLlmNJj2fF/MftwVfDX8bIgCugiNO/jOs8yprVtY8f2fAth9cbROPrPeAidxHe0t4iTJNTjMcIfYPzIhiwpUXsw== X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(23010399003)(1800799024)(36860700016)(82310400026)(376014)(56012099006)(10067099003)(22082099003)(18002099003)(11063799006);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: mWNk+DFyjWbTBheBHlhL2WIX7wmF3tzokwNwcFqIaIzcz0c85FiuQVx1S3Dimw9GMcXoFu7cayWJH4V0YRsHfFqaKFG73d0EJSO+4d/CdP6uhnOTa08OW8dinvYU1DodnRp632+cg3k8lQ+h7kGbj3bHpvoxNQwFlvtZXc8xXheH/CtdvMunat4toLYptjlf21O24qO6z/GEg7Qx5w8dsMKCxHh3QdHDIxtG11bmMw6E+a/ms3l4J20o5Ovf0ocpvII6CTUsVzbKV/tVuzG+czgAVEBA+721/06tD9k0tfKynikOVwmFVkToI/jHaBsKqpPUL0SBUAFvP7Ln8/w4/PkXuroZEKtUSq/Rl+m8c30WDR/E7qqRG19yx6InXWVfpAR2UYiMsMazbZqfgOZfDPVD4vNtImKO+FaUy0yYSWLBYiTWkCBAx0z/7XCqDsif X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Jul 2026 07:23:12.0513 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0536a0e7-670c-4594-ced7-08dee88b4103 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS2PEPF000061C4.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7571 Content-Type: text/plain; charset="utf-8" From: Nimrod Oren Add per-node doorbell dma pool creation and cleanup to mdev lifecycle. Signed-off-by: Nimrod Oren Reviewed-by: Dragos Tatulea Signed-off-by: Tariq Toukan --- .../net/ethernet/mellanox/mlx5/core/alloc.c | 37 +++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/main.c | 7 ++++ .../ethernet/mellanox/mlx5/core/mlx5_core.h | 2 + include/linux/mlx5/driver.h | 2 + 4 files changed, 48 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/alloc.c b/drivers/net/= ethernet/mellanox/mlx5/core/alloc.c index 4fe9d7d4f143..3c9938068c56 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/alloc.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/alloc.c @@ -444,6 +444,43 @@ void mlx5_frag_buf_free(struct mlx5_core_dev *dev, str= uct mlx5_frag_buf *buf) } EXPORT_SYMBOL_GPL(mlx5_frag_buf_free); =20 +void mlx5_db_pools_cleanup(struct mlx5_core_dev *dev) +{ + struct mlx5_priv *priv =3D &dev->priv; + int node; + + for_each_node_state(node, N_POSSIBLE) + if (priv->db_node_pools[node]) + mlx5_dma_pool_destroy(priv->db_node_pools[node]); + + kfree(priv->db_node_pools); + priv->db_node_pools =3D NULL; +} + +int mlx5_db_pools_init(struct mlx5_core_dev *dev) +{ + struct mlx5_priv *priv =3D &dev->priv; + int node; + + priv->db_node_pools =3D kzalloc_objs(*priv->db_node_pools, nr_node_ids); + if (!priv->db_node_pools) + return -ENOMEM; + + for_each_node_state(node, N_POSSIBLE) { + struct mlx5_dma_pool *pool; + + pool =3D mlx5_dma_pool_create(dev, node, + order_base_2(cache_line_size())); + if (!pool) { + mlx5_db_pools_cleanup(dev); + return -ENOMEM; + } + priv->db_node_pools[node] =3D pool; + } + + return 0; +} + static struct mlx5_db_pgdir *mlx5_alloc_db_pgdir(struct mlx5_core_dev *dev, int node) { diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/e= thernet/mellanox/mlx5/core/main.c index 643b4aac2033..b3cb090b5677 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -1830,6 +1830,10 @@ int mlx5_mdev_init(struct mlx5_core_dev *dev, int pr= ofile_idx) if (err) goto err_frag_buf_pools_init; =20 + err =3D mlx5_db_pools_init(dev); + if (err) + goto err_db_pools_init; + INIT_LIST_HEAD(&priv->traps); =20 err =3D mlx5_cmd_init(dev); @@ -1891,6 +1895,8 @@ int mlx5_mdev_init(struct mlx5_core_dev *dev, int pro= file_idx) err_timeout_init: mlx5_cmd_cleanup(dev); err_cmd_init: + mlx5_db_pools_cleanup(dev); +err_db_pools_init: mlx5_frag_buf_pools_cleanup(dev); err_frag_buf_pools_init: debugfs_remove(dev->priv.dbg.dbg_root); @@ -1917,6 +1923,7 @@ void mlx5_mdev_uninit(struct mlx5_core_dev *dev) mlx5_health_cleanup(dev); mlx5_tout_cleanup(dev); mlx5_cmd_cleanup(dev); + mlx5_db_pools_cleanup(dev); mlx5_frag_buf_pools_cleanup(dev); debugfs_remove_recursive(dev->priv.dbg.dbg_root); mutex_destroy(&priv->pgdir_mutex); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h b/drivers/= net/ethernet/mellanox/mlx5/core/mlx5_core.h index 09e669f83dba..d6713a2ce676 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h @@ -438,6 +438,8 @@ int mlx5_mdev_init(struct mlx5_core_dev *dev, int profi= le_idx); void mlx5_mdev_uninit(struct mlx5_core_dev *dev); int mlx5_frag_buf_pools_init(struct mlx5_core_dev *dev); void mlx5_frag_buf_pools_cleanup(struct mlx5_core_dev *dev); +int mlx5_db_pools_init(struct mlx5_core_dev *dev); +void mlx5_db_pools_cleanup(struct mlx5_core_dev *dev); int mlx5_init_one(struct mlx5_core_dev *dev); int mlx5_init_one_devl_locked(struct mlx5_core_dev *dev); void mlx5_uninit_one(struct mlx5_core_dev *dev); diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index b1871c0821d0..4246d6d904ba 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -569,6 +569,7 @@ enum mlx5_page_mgt_mode { }; =20 struct mlx5_frag_buf_node_pools; +struct mlx5_dma_pool; struct mlx5_ft_pool; struct mlx5_priv { /* IRQ table valid only for real pci devices PF or VF */ @@ -602,6 +603,7 @@ struct mlx5_priv { struct list_head pgdir_list; =20 struct mlx5_frag_buf_node_pools **frag_buf_node_pools; + struct mlx5_dma_pool **db_node_pools; /* end: alloc stuff */ =20 struct mlx5_adev **adev; --=20 2.44.0 From nobody Fri Jul 24 21:52:58 2026 Received: from PH7PR06CU001.outbound.protection.outlook.com (mail-westus3azon11010032.outbound.protection.outlook.com [52.101.201.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A98F3DAAAD; Thu, 23 Jul 2026 07:23:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.201.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784791412; cv=fail; b=GDJ3gMEOdd98gC71ZgoLdSVHyr5XuigTHwXnwE5D/9PO3naZN98vKDU0rYodEnIh7ilbkV8cSAMcXshUXOsD7kkMUhBcpLP2bGGAdi6pOaPip0kFloexu8hpuPuOc3j7D2/vTQ3fsv9USfBuGCrrKUqL4zYX+snwR5j1GuU5m68= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784791412; c=relaxed/simple; bh=xmLYMWL/23p0MWjCjmxUytMgG9Fg5ADWwTixMO3Bl+w=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=GsbOxueZ41egmxNlkZSFwOGTBjchI761vmrM9dG3axOdLaddHX6YGXdOiYZHTp8Tx418Jxz3794ijPV52uYQg40iNdWr8CDEE2Oh8bP7qnpyu1p32BV1AlMvY+VmFDkypCKbrGrDPQDaVr/lX88wDx8meor0pzPbc6+t8NOQ/ew= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=M6sjl7KP; arc=fail smtp.client-ip=52.101.201.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="M6sjl7KP" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Oij0Fp11YFmlxWsHoowEi8eZlkC+a7Z/o+gdkVnTRX5vTc3Blfozx2X8a/u2CWM7KLdEs5i3QKelxkNPvlSEv1zj4pF69v5xQlLzkTU1QN3c7cigmf3WsaxkHVPdwT+kXMMFvXTR7/j45yaauxrGsAxK4ZX03eAVxmRK/Cvu2QUTaKg72zZPPy74B/gAbnCSe0pAxm6t5ts1MIb9Luf8SDMEODSEoRepmmMeBNntr7LiG59hU/aezdt0ifpPouXcNBwNLERfSGzqAU+taxfJRaMrColn/7VMlF7EJyfagtUuuThHbKxriQzhBM/67kMcmIRqYbgkvRagha+ZD+mGFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0zuYFDvOaoaTOldoH5nRlVia8qUnP3j0a0jj1I2Axg4=; b=ttPVf65eCt3Of1JXl/4gptIwm2D/cBLj9WWqVB6/Nwnn3tmgQJGn1oSRZq4btoLE/ABBh2Dt8PLFQFG9gAJn/0Fy20VuacVtrVkAM8CnP6JwkHHNhgpO9i0VZUJGkHOz4d/rq1Nxyx1VHEXZ49mR2CG6giKPAg2hQzWAS0Y9cDWHY1jzJS9A4LYU0WxCguJu6g5mZA/dq1hIJJVHMsuRjeCT6KwNVfSD8Xe2deZddK0YEo7vHQCOqov5jKuaPTRtwlqhVHTuHFJ0Au8VCMxAAcKh1vJMH5r5soUNrG/yd+kLs8rg1QddTZnVg8Ndtdmp+YNu4IeXe7d77elpRZncTg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=lunn.ch smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0zuYFDvOaoaTOldoH5nRlVia8qUnP3j0a0jj1I2Axg4=; b=M6sjl7KPBXLSkSlwiiNp4TlYv+TuvXgLYBx4KyyEbpx198p+Rq2xb9jWsdM0uYFd0wKnjcy/sYA8q5FkJgLfoqCcX6jL5VRcSdMh2WBJVPMRQAvGZdt1iRy9mPnrGGJYnPGtWes4tKnI4UCKlRlLN1zIPzedDFflB45H9kUb7+c/pn8aLUH4f0PeTpWro0r0NeFMiZXr1W2y/s5A+BaGcrSh/sVSYpGH3DTq29cpMzVteMgDOJy3krwrPo/IogHLKZ/uQJvMV6+VhE8cWKvytduWjGM6lbo08G+vEDQM5VXt4HHFlPESSXpelbT20OZWtEBLvAUS03XfeSPsQA/p7A== Received: from CH2PR11CA0012.namprd11.prod.outlook.com (2603:10b6:610:54::22) by CH2PR12MB4263.namprd12.prod.outlook.com (2603:10b6:610:a6::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.245.11; Thu, 23 Jul 2026 07:23:19 +0000 Received: from DS2PEPF000061C8.namprd02.prod.outlook.com (2603:10b6:610:54:cafe::3f) by CH2PR11CA0012.outlook.office365.com (2603:10b6:610:54::22) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.245.10 via Frontend Transport; Thu, 23 Jul 2026 07:23:19 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DS2PEPF000061C8.mail.protection.outlook.com (10.167.23.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.270.5 via Frontend Transport; Thu, 23 Jul 2026 07:23:19 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Thu, 23 Jul 2026 00:23:03 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Thu, 23 Jul 2026 00:23:02 -0700 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.2562.20 via Frontend Transport; Thu, 23 Jul 2026 00:22:59 -0700 From: Tariq Toukan To: Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , , Paolo Abeni CC: Dragos Tatulea , Gal Pressman , "Leon Romanovsky" , , , Mark Bloch , Nimrod Oren , Saeed Mahameed , Tariq Toukan Subject: [PATCH net-next 2/3] net/mlx5: allocate doorbells from dma pools Date: Thu, 23 Jul 2026 10:22:21 +0300 Message-ID: <20260723072222.1863558-3-tariqt@nvidia.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20260723072222.1863558-1-tariqt@nvidia.com> References: <20260723072222.1863558-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PEPF000061C8:EE_|CH2PR12MB4263:EE_ X-MS-Office365-Filtering-Correlation-Id: 378dadbb-037e-4793-78e7-08dee88b4531 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|1800799024|36860700016|82310400026|376014|56012099006|10067099003|22082099003|18002099003|11063799006; X-Microsoft-Antispam-Message-Info: 75ITjaRPwPsUpYP9ikwIa3N2Qcrem4VSH0YYQMh2p0HCV9QEViLOZfFEGXGKHVy7YppFq3phOXP36Zix8hWs48+JPjormtyus3y+lWmkHWOdXhl+O9ihf/zjuTaXiA/My9rvPcRsBx98C4IRyMN8F3Rv2jVZtAs5nDcBPvdW4OD4gdxCeU+UQdWRbvn4Dqqq+SBZDk2dIF7fzKbcU9SFUQxc6PinPeBpe5lbVD1Yh9QpAMOn+izAz6xUPeZjGXjNKaU4qHR0MS9JsueWEocpN1kkbU09un9GDKbg1wYj3xvJsqh9BYM5D8OKuQjvQ9zOh5PY3HFIV7w/jq0mvz75g+3fd3+Q6c4kZP93c7x8P+cq3ZHGMgjm6JTdbZLJigHzlDNEt+fRrE4ueaU+NL9ALmWe9XOqHJEadDcRVuX+Tvg1NvRclksFA6v/uz+2ePPpj4UmsHTSmwbxf+fwdoendYeyVwwW6pY7amXfE4rUkFnK8acu7/ruos1le7TRIjiyBw+hyW6GhgCRyaWzBV176vLaF/ytGuxjG96XzT2EnvteJhKV0ZCaK3emKA9gXo7VoRaiILeVVwH7oGrvVq7N2GEnk1nkwcPOllwPQWiULfcnZ5XPGSK7xU53V47p14uD7SvSDVTcMOv9DwpsdPBgHzfL6CavDoM9jq3vQGhCDyni2RZcD4DNFfK8CALJPcGTHAGYrrulO4oKONcHJPpclQ== X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(23010399003)(1800799024)(36860700016)(82310400026)(376014)(56012099006)(10067099003)(22082099003)(18002099003)(11063799006);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: qi97DQ/X6919C0GbbQv3ow7v/mU6BbURnc2agcD1+NxjUUotEBbhzVu2+6okel4WF3eZ4rQJaXHY1MZZx9lbPlwXGUuKf3AocP4RN5BXK5AmphenE2QdXcA22/Tb2hFTGl3UGtPxJbfiaonqjSuYNCzADmi4fyXG5aRb0ezHOCpwJCCATO0vD4RrAtlKDIJqMGneGgQO9/U4aY5h3/IKH6ldlPgdhthIRsZqHt+FZT38rldIPbG4T9m7CeFaxsCguDgiu3NRpRigmS1p9OpHx0fRsiHw0FTdV9RSBcojtO7ER6E48WKgSRyCkhCjHmM1iB3YcFEHJaKjqmxCrPU6wOF2wXA/g3Z/S03PUL+Pa0IF2ny/pTcsIqRQj5KroGSd/42CVIHlgZz9FAYT+keKCIGtD3Xmr83Qw9OwcEAU434NFly1Yxtrp03xw0MMwQHQ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Jul 2026 07:23:19.0673 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 378dadbb-037e-4793-78e7-08dee88b4531 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS2PEPF000061C8.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4263 Content-Type: text/plain; charset="utf-8" From: Nimrod Oren Allocate doorbells from dma pools instead of the pgdir allocator. Doorbell records remain cache-line sized coherent DMA allocations, but their sub-allocation is now handled by the common mlx5 DMA pool infrastructure. This also makes doorbell allocation honor the requested NUMA node when reusing existing backing pages. The old pgdir allocator used the requested node only when allocating a new pgdir page; later allocations scanned one global pgdir list and could take any pgdir with a free entry, even if that page had been allocated for a different NUMA node. Selecting the per-node DMA pool before sub-allocation keeps reused doorbell records on pages allocated for the requested node. Signed-off-by: Nimrod Oren Reviewed-by: Dragos Tatulea Signed-off-by: Tariq Toukan --- .../net/ethernet/mellanox/mlx5/core/alloc.c | 111 +++--------------- .../net/ethernet/mellanox/mlx5/core/main.c | 4 - include/linux/mlx5/driver.h | 5 +- 3 files changed, 18 insertions(+), 102 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/alloc.c b/drivers/net/= ethernet/mellanox/mlx5/core/alloc.c index 3c9938068c56..3975c726d35c 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/alloc.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/alloc.c @@ -48,13 +48,6 @@ #define MLX5_FRAG_BUF_POOLS_NUM \ (PAGE_SHIFT - MLX5_FRAG_BUF_POOL_MIN_BLOCK_SHIFT + 1) =20 -struct mlx5_db_pgdir { - struct list_head list; - unsigned long *bitmap; - __be32 *db_page; - dma_addr_t db_dma; -}; - struct mlx5_dma_pool { /* Protects page_list and per-page allocation bitmaps. */ struct mutex lock; @@ -481,106 +474,36 @@ int mlx5_db_pools_init(struct mlx5_core_dev *dev) return 0; } =20 -static struct mlx5_db_pgdir *mlx5_alloc_db_pgdir(struct mlx5_core_dev *dev, - int node) -{ - u32 db_per_page =3D PAGE_SIZE / cache_line_size(); - struct mlx5_db_pgdir *pgdir; - - pgdir =3D kzalloc_node(sizeof(*pgdir), GFP_KERNEL, node); - if (!pgdir) - return NULL; - - pgdir->bitmap =3D bitmap_zalloc_node(db_per_page, GFP_KERNEL, node); - if (!pgdir->bitmap) { - kfree(pgdir); - return NULL; - } - - bitmap_fill(pgdir->bitmap, db_per_page); - - pgdir->db_page =3D mlx5_dma_zalloc_coherent_node(dev, PAGE_SIZE, - &pgdir->db_dma, node); - if (!pgdir->db_page) { - bitmap_free(pgdir->bitmap); - kfree(pgdir); - return NULL; - } - - return pgdir; -} - -static int mlx5_alloc_db_from_pgdir(struct mlx5_db_pgdir *pgdir, - struct mlx5_db *db) +int mlx5_db_alloc_node(struct mlx5_core_dev *dev, struct mlx5_db *db, int = node) { - u32 db_per_page =3D PAGE_SIZE / cache_line_size(); + struct mlx5_dma_pool_page *page; + struct mlx5_dma_pool *pool; + unsigned long idx; int offset; - int i; - - i =3D find_first_bit(pgdir->bitmap, db_per_page); - if (i >=3D db_per_page) - return -ENOMEM; =20 - __clear_bit(i, pgdir->bitmap); + node =3D node =3D=3D NUMA_NO_NODE ? numa_mem_id() : node; =20 - db->u.pgdir =3D pgdir; - db->index =3D i; - offset =3D db->index * cache_line_size(); - db->db =3D pgdir->db_page + offset / sizeof(*pgdir->db_page); - db->dma =3D pgdir->db_dma + offset; + pool =3D dev->priv.db_node_pools[node]; + page =3D mlx5_dma_pool_alloc(pool, &idx); + if (!page) + return -ENOMEM; =20 - db->db[0] =3D 0; - db->db[1] =3D 0; + offset =3D idx << pool->block_shift; + db->u.pool_page =3D page; + db->index =3D idx; + db->db =3D (__be32 *)((u8 *)page->buf + offset); + db->dma =3D page->dma + offset; =20 return 0; } - -int mlx5_db_alloc_node(struct mlx5_core_dev *dev, struct mlx5_db *db, int = node) -{ - struct mlx5_db_pgdir *pgdir; - int ret =3D 0; - - mutex_lock(&dev->priv.pgdir_mutex); - - list_for_each_entry(pgdir, &dev->priv.pgdir_list, list) - if (!mlx5_alloc_db_from_pgdir(pgdir, db)) - goto out; - - pgdir =3D mlx5_alloc_db_pgdir(dev, node); - if (!pgdir) { - ret =3D -ENOMEM; - goto out; - } - - list_add(&pgdir->list, &dev->priv.pgdir_list); - - /* This should never fail -- we just allocated an empty page: */ - WARN_ON(mlx5_alloc_db_from_pgdir(pgdir, db)); - -out: - mutex_unlock(&dev->priv.pgdir_mutex); - - return ret; -} EXPORT_SYMBOL_GPL(mlx5_db_alloc_node); =20 void mlx5_db_free(struct mlx5_core_dev *dev, struct mlx5_db *db) { - u32 db_per_page =3D PAGE_SIZE / cache_line_size(); - - mutex_lock(&dev->priv.pgdir_mutex); - - __set_bit(db->index, db->u.pgdir->bitmap); - - if (bitmap_full(db->u.pgdir->bitmap, db_per_page)) { - dma_free_coherent(mlx5_core_dma_dev(dev), PAGE_SIZE, - db->u.pgdir->db_page, db->u.pgdir->db_dma); - list_del(&db->u.pgdir->list); - bitmap_free(db->u.pgdir->bitmap); - kfree(db->u.pgdir); - } + struct mlx5_dma_pool_page *page =3D db->u.pool_page; + struct mlx5_dma_pool *pool =3D page->pool; =20 - mutex_unlock(&dev->priv.pgdir_mutex); + mlx5_dma_pool_free(pool, page, db->index); } EXPORT_SYMBOL_GPL(mlx5_db_free); =20 diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/e= thernet/mellanox/mlx5/core/main.c index b3cb090b5677..5f28d906c35b 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -1819,8 +1819,6 @@ int mlx5_mdev_init(struct mlx5_core_dev *dev, int pro= file_idx) INIT_LIST_HEAD(&priv->bfregs.wc_head.list); =20 mutex_init(&priv->alloc_mutex); - mutex_init(&priv->pgdir_mutex); - INIT_LIST_HEAD(&priv->pgdir_list); =20 priv->numa_node =3D dev_to_node(mlx5_core_dma_dev(dev)); priv->dbg.dbg_root =3D debugfs_create_dir(dev_name(dev->device), @@ -1900,7 +1898,6 @@ int mlx5_mdev_init(struct mlx5_core_dev *dev, int pro= file_idx) mlx5_frag_buf_pools_cleanup(dev); err_frag_buf_pools_init: debugfs_remove(dev->priv.dbg.dbg_root); - mutex_destroy(&priv->pgdir_mutex); mutex_destroy(&priv->alloc_mutex); mutex_destroy(&priv->bfregs.wc_head.lock); mutex_destroy(&priv->bfregs.reg_head.lock); @@ -1926,7 +1923,6 @@ void mlx5_mdev_uninit(struct mlx5_core_dev *dev) mlx5_db_pools_cleanup(dev); mlx5_frag_buf_pools_cleanup(dev); debugfs_remove_recursive(dev->priv.dbg.dbg_root); - mutex_destroy(&priv->pgdir_mutex); mutex_destroy(&priv->alloc_mutex); mutex_destroy(&priv->bfregs.wc_head.lock); mutex_destroy(&priv->bfregs.reg_head.lock); diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index 4246d6d904ba..2c56bb1d676d 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -599,9 +599,6 @@ struct mlx5_priv { struct mutex alloc_mutex; int numa_node; =20 - struct mutex pgdir_mutex; - struct list_head pgdir_list; - struct mlx5_frag_buf_node_pools **frag_buf_node_pools; struct mlx5_dma_pool **db_node_pools; /* end: alloc stuff */ @@ -808,7 +805,7 @@ struct mlx5_core_dev { struct mlx5_db { __be32 *db; union { - struct mlx5_db_pgdir *pgdir; + struct mlx5_dma_pool_page *pool_page; struct mlx5_ib_user_db_page *user_page; } u; dma_addr_t dma; --=20 2.44.0 From nobody Fri Jul 24 21:52:58 2026 Received: from DM5PR21CU001.outbound.protection.outlook.com (mail-centralusazon11011037.outbound.protection.outlook.com [52.101.62.37]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 566AB3E8326; Thu, 23 Jul 2026 07:23:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.62.37 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784791413; cv=fail; b=Slvk0L1X1phqccAfqDoJBJvSXhNk7jorKYDbOARcbMP9ymtXErpD2Zhkdz7xsovTjswnbpiSpcgSXxllq6cOYJnRVD7qdKcRVkvjmovGZepEJITDt7EASbyyLDIiT6JW0te3if7ijPdQwfDpzlq5frno+HvNww48QHKBTWsqqZ4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784791413; c=relaxed/simple; bh=IwFfheX/GPd4PE+lr4+iRyyEqpGDfloGCvvnJYHukD4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=jVEJ1Mdoc1Off50AkvhuhOPnTj+sh2OsywmA5+xcZYScR8Vsnvg8x8BJcJQUKyt+p+cTN5sgvCH2jfmvDEvsJ6X4U+hXvESlswYDOkI1q3rJFzaqJgCjBUWARFvLp8pJHX461WNsT/6TXM1pwuWTHZHcOJd/nm+PmlUkIIwmXMg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=SHwKqvj+; arc=fail smtp.client-ip=52.101.62.37 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="SHwKqvj+" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=EeCHHcMZtlEZRQ0V8+dmoZMezB7f3CRV19hZkVhqQeV1aCbTmAaafXovPITaaTGyZBE97rzDPvza4YOQBuVOxkH0HzXEyB7BgqHpApDpQTYCLoxfd7hIRi5Fy/CBF07RkheN/Y7Oo68NBur1x8Vl9M5u0tCROwTYxT+hEjm8+twK1hb5xhDSqbUHwTr6uYuyAF8U3w4LNTL5fZBB9iKl/ny7SQFAMvfP/x2PMqFrDYt61v9LVK9SiOGDCILDKz7CNzx12NbPMezmx8f9Oht4E6nsncPH9zWIy57TMC/Mi5Y8Grbznu8q589DLUGMAbzV+3JAy+KSmvFq3XEJG8x/kA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=GA2EIwc90au0Y+OmQ2WLN0q1d3KI3+BMm+axC6TVZEM=; b=rHGKxhAFDFJLLeGCy/mMg5P3npV7tiIVqorCSnt8Y8E9n7VhmUENliGnJlQWggNWOdKHNPrnPUiAhuOLB7TjgG0J4TORRVm1oZZEpXO25bzu9MToDPDz18G4NhrJbIVMtVYj4zi62y9YFd4q2dL6jARc6bWERVTBp3W8UdpckzQ6d24e2cCOjYmHCJUlCUtv/d7+Pp+722t+0y7xr0Z/ysV2t5ZzaYBJJqp4qP04gVJ6b4agmWOR8yfyrgwvn/u/KjR8WxrZn2dsprTexKWn47ht0gunSgGMvji79CCvCpbY/MRJTfI+mLjP5Eq8IrGLfOikw+o/p8kPUcLvCuCJDw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=lunn.ch smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GA2EIwc90au0Y+OmQ2WLN0q1d3KI3+BMm+axC6TVZEM=; b=SHwKqvj+RBOcyGLc08KzHtnfIdKVQF6N4p3pS6uS9NfjBoyX0P0+ksvcY2cBrYHBxlmk7JhQG5mwDxG3eOYj8SOwVmN0CQ4ahW+Gmn896FA1hDsLfZBKTT8pDHgTcml3csmdXqds02zDnoBlz0ljZ/ZznFR5rdOL8AogNML8u4F1tqTjcvFc3sKsOD3FDNN4nL4xLT/Q0DLL2mzp8oQvmgXmYlB0gWSgvIB6GRF3Ki4TcoSt3Up/nBUQdVTIqkME/C80ia12IXvlbFGcE+wAlFcFXu+wkRwVyaUhStr2VNLIpxkAzu9e6vGQNNm1MkYDbZHZWFhb0reD5SXcKfCZEA== Received: from PH7P221CA0045.NAMP221.PROD.OUTLOOK.COM (2603:10b6:510:33c::17) by SA0PR12MB7002.namprd12.prod.outlook.com (2603:10b6:806:2c0::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.245.10; Thu, 23 Jul 2026 07:23:23 +0000 Received: from CY4PEPF0000EE3B.namprd03.prod.outlook.com (2603:10b6:510:33c:cafe::8e) by PH7P221CA0045.outlook.office365.com (2603:10b6:510:33c::17) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.245.10 via Frontend Transport; Thu, 23 Jul 2026 07:23:22 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CY4PEPF0000EE3B.mail.protection.outlook.com (10.167.242.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.270.5 via Frontend Transport; Thu, 23 Jul 2026 07:23:22 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 23 Jul 2026 00:23:07 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Thu, 23 Jul 2026 00:23:06 -0700 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.2562.20 via Frontend Transport; Thu, 23 Jul 2026 00:23:03 -0700 From: Tariq Toukan To: Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , , Paolo Abeni CC: Dragos Tatulea , Gal Pressman , "Leon Romanovsky" , , , Mark Bloch , Nimrod Oren , Saeed Mahameed , Tariq Toukan Subject: [PATCH net-next 3/3] net/mlx5: add debugfs stats for doorbell dma pools Date: Thu, 23 Jul 2026 10:22:22 +0300 Message-ID: <20260723072222.1863558-4-tariqt@nvidia.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20260723072222.1863558-1-tariqt@nvidia.com> References: <20260723072222.1863558-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EE3B:EE_|SA0PR12MB7002:EE_ X-MS-Office365-Filtering-Correlation-Id: b4e9bedf-5e76-4e7f-b3b7-08dee88b4707 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|23010399003|376014|82310400026|1800799024|56012099006|10067099003|11063799006|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: t9YnTQCtgqb/HN85fnmNYQ2RjzP7mxqmpm1jiURhR/Cli53/tcNUVjdzYWKz3DVfl5/WC9R308a3REL5QWWm9Nc/iX1yWpHuSGB0IgWEoiwzhYFId+Btw8E//CQWpBiJLgrdLcmuM4kaWaMaeUbMWCQX25nlA5uhIRM4UlsSC9ZxHUHzmWyk9C3YaE4LnMuSRRBtzmEL3zNC/He3wy4HF2T5wXGhEWyXhwDL4hn2AgzQXfYJLw6OQeXEydYTrHPx1J/dbCbGXHuq09/Ma146SYTZQd3vujTRwGuAnOGDKPBInTY0m/eUUX6PNjyzSIB2IW2XOQC2NfVBHGRxy6Gqj2yVicS9p6hwHBvA4bpLoCzX9qezp0cY11ZsPL08Rf2ffe3j5Hxq/kHluL1UAfg1eHPsrUbRBTOCZhBl/nkbMOwFPlOyhhKCuaUMcnwvgsWqflTwCOXGbvlb9JqaQ+CHN7f1IXRKWxnhxymhnoGlDK9nVBsBGahXI1PNca9oFHUuExuTsiAZl0WvP6nv8Ld5AIjq9COwwelNs60qytpV1mT3kNTQJefhrLe+/Zmc87niWU3+XYgZ4OkrUFtsK+SONlOH5uNTASLlvRO7ptrqaTSA4v20Ny08AbRfVnD7/YLACVDVg8wLSy89bX4sXzxJ8z3ChWD7YFXDfjAi78QVMmlWEKDFs5UUfYsJQ7ARYEdBQu5Pxk94eZ2ffzF1mxEgnw== X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230040)(36860700016)(23010399003)(376014)(82310400026)(1800799024)(56012099006)(10067099003)(11063799006)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: IgxfdyKj+jVyUkMHdgCOrauXvPJz5xWrU2DQFVMZUDF6JyoJSd34bqJZlkLrDujy2As9X52Pltd1pfZrmPYe1SE8V0y4ydXSoJMD8WRAY5iH9lRDTFRbaIT8NSCtUQvlcy3UtkwQdpAk3by+FpE5vccYpdNDPSFHc6wp0jzL+eo8oMWc5kKAAauIZaycAqDX9xDyXx5DdUbbIOjktFTY6sePw6Y/GtubOGY5s47l++Df9c1O75SHmja+c4yo22q/b5p5Ir3bWefoUsTq4jPVN0vFIBz1/0RXc+Rixxl6isEakSzXlYJbC3kM5anvtkeO8jxpwSdwTZkMhkR1IJxWf9No93cNi12guYeYH+/UhxRCjmykswygj5WoNZoiVtiXdvoypLJhHLsyznpKqtJElr3CgECy7vBb8sRxlXNbEu46Jq3tQK4LwqRvSAt0sZo8 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Jul 2026 07:23:22.1493 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b4e9bedf-5e76-4e7f-b3b7-08dee88b4707 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE3B.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB7002 Content-Type: text/plain; charset="utf-8" From: Nimrod Oren Add a debugfs file exposing per-node DMA pool usage for doorbell allocations. # cat /sys/kernel/debug/mlx5//db_dma_pools node block_size used_blocks allocated_blocks 0 64 0 0 1 64 0 0 Signed-off-by: Nimrod Oren Signed-off-by: Tariq Toukan --- .../net/ethernet/mellanox/mlx5/core/alloc.c | 30 +++++++++++++++++++ include/linux/mlx5/driver.h | 1 + 2 files changed, 31 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/alloc.c b/drivers/net/= ethernet/mellanox/mlx5/core/alloc.c index 3975c726d35c..60ff413999f7 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/alloc.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/alloc.c @@ -437,11 +437,37 @@ void mlx5_frag_buf_free(struct mlx5_core_dev *dev, st= ruct mlx5_frag_buf *buf) } EXPORT_SYMBOL_GPL(mlx5_frag_buf_free); =20 +static int mlx5_db_dma_pools_debugfs_show(struct seq_file *file, void *pri= v) +{ + struct mlx5_core_dev *dev =3D file->private; + int node; + + mlx5_dma_pools_debugfs_print_header(file); + + if (!dev->priv.db_node_pools) + return 0; + + for_each_node_state(node, N_POSSIBLE) { + struct mlx5_dma_pool *pool =3D dev->priv.db_node_pools[node]; + + if (!pool) + continue; + + mlx5_dma_pool_debugfs_stats_print(file, pool); + } + + return 0; +} +DEFINE_SHOW_ATTRIBUTE(mlx5_db_dma_pools_debugfs); + void mlx5_db_pools_cleanup(struct mlx5_core_dev *dev) { struct mlx5_priv *priv =3D &dev->priv; int node; =20 + debugfs_remove(priv->dbg.db_dma_pools_debugfs); + priv->dbg.db_dma_pools_debugfs =3D NULL; + for_each_node_state(node, N_POSSIBLE) if (priv->db_node_pools[node]) mlx5_dma_pool_destroy(priv->db_node_pools[node]); @@ -471,6 +497,10 @@ int mlx5_db_pools_init(struct mlx5_core_dev *dev) priv->db_node_pools[node] =3D pool; } =20 + priv->dbg.db_dma_pools_debugfs =3D + debugfs_create_file("db_dma_pools", 0444, priv->dbg.dbg_root, + dev, &mlx5_db_dma_pools_debugfs_fops); + return 0; } =20 diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index 2c56bb1d676d..83d0a83bbfbc 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -548,6 +548,7 @@ struct mlx5_debugfs_entries { struct dentry *cq_debugfs; struct dentry *cmdif_debugfs; struct dentry *frag_buf_dma_pools_debugfs; + struct dentry *db_dma_pools_debugfs; struct dentry *pages_debugfs; struct dentry *lag_debugfs; }; --=20 2.44.0