From nobody Wed Sep 30 11:25:46 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A83D644A708; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790754372; cv=none; b=IOSQIwaGTF/O7WjGXHGhHYJYJ5Noij2c0O8oNRkqU/oYZnX3ZcPpnIA8hFXxthoS5qwJrNR6e/W+y5u2UwJKgJf4IBSx1/AXZZHyoOHsIaReemogFB3xInKaM8ZZw+Ka3odDhA7WxI6lOzkUcSTTaKpXV34KD7s62q0DOS6A0Lk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790754372; c=relaxed/simple; bh=4xGUGprrK9rTmY2Wp5Yujo+nW+a+yMaA5Hizw5sUHCg=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=m0P1XhufxtnLMYHYO++JfwOHBJ4bs31xih3cnjjWK1fS2bjEPLI5vUINnR8lxcPwqHaO/Pr0o3Ao0mAAGlAdbdHGduWrKdGDWNzM5iVM0AlgU4tSPmQjYoX4Bsa2XY2NkmyMYJ+OMtbgpvewuztytHSbweP8Vk26+ttQ9tVc0Zg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=AlpY7GhD; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="AlpY7GhD" Received: by smtp.kernel.org (Postfix) with ESMTPS id 449DDC4AF14; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1790754372; bh=4xGUGprrK9rTmY2Wp5Yujo+nW+a+yMaA5Hizw5sUHCg=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=AlpY7GhDBIjS201ObtoR2HxXr87LJ9fTyPHxMXX9iG/Jc6s8sokIi6qlQbIyR3VXM uIUk1jN+tdTNdYGbca5BX5XKtfttvlUJH9tSBwVNKnLOBIZ3wYDLnuecS15j+7lYiy OwCRFHUlh7uVxosK3KTzGxUgcVIx6z5OjOw3OKmgejCdc1XfzhNdos3BNQBIWRjeWb URqTz7B52FfjAynhH1CvvYfK+hjRr5YTH9Dic0RNqnwB3L8Nlj16CCNe+kzabi0XcW Q1tzgAPh8/ujBWxEhulhl9jg5qpo6YpmrlbCkWev8Lselz/GEo4vtOUN3e1xJ8YF/A qvqdjAaAk8LaQ== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FDA5CA5FB1; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) From: Jack Wu via B4 Relay Date: Wed, 30 Sep 2026 15:46:08 +0800 Subject: [PATCH v9 1/6] net: wwan: t9xx: Add PCIe core Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260930-t9xx_driver_v1-v9-1-3248b9256956@compal.com> References: <20260930-t9xx_driver_v1-v9-0-3248b9256956@compal.com> In-Reply-To: <20260930-t9xx_driver_v1-v9-0-3248b9256956@compal.com> To: Loic Poulain , Sergey Ryazanov , Johannes Berg , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jack Wu , Wen-Zhi Huang , Shi-Wei Yeh , Minano Tseng , Matthias Brugger , AngeloGioacchino Del Regno , Simon Horman , Jonathan Corbet , Shuah Khan , Robert Yu , Jeff Chang Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, linux-doc@vger.kernel.org X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1790754369; l=40783; i=jackbb_wu@compal.com; s=20260526; h=from:subject:message-id; bh=4oCj7d1TynlJtX7D03k6BT+0+aJ1heP6RLotdgmgxsc=; b=cUzg0n/BffhUkrYpdNFNI0WOdTB4qwuvcLjtSODbtbVyyiMqHo+UCxLA9YlxYiN2gpWO4Exkg z5NwZ2dVe21CshLbEWE5FVCGol2u0iDgjDri83dU1driaJdLjrF9Z9s X-Developer-Key: i=jackbb_wu@compal.com; a=ed25519; pk=VH1prTWixNl8OEUPPSfII3p46MzJpQN8J3+ecE1tZXg= X-Endpoint-Received: by B4 Relay for jackbb_wu@compal.com/20260526 with auth_id=793 X-Original-From: Jack Wu Reply-To: jackbb_wu@compal.com From: Jack Wu Registers the T900 device driver with the kernel. Set up all the fundamental configurations for the device: PCIe layer, Modem Host Cross Core Interface (MHCCIF), modem common control operations, build infrastructure and MAINTAINERS entry. * PCIe layer code implements driver probe and removal, MSI-X interrupt initialization and de-initialization, and the way of resetting the device. * MHCCIF provides the interrupt channels used for the boot handshake and for the host-to-device reset doorbell. * Modem common control operations provide the basic read/write functions of the device's hardware registers, mask/unmask/get/clear functions of the device's interrupt registers and inquiry functions of the device's status. * Add MAINTAINERS entry for the MediaTek T9XX 5G WWAN modem device driver. Removal resets the modem. There is no software reset that returns the device from running firmware to a state the next probe can boot, and the next driver load has to redo the device handshake and status sync, so mtk_pci_remove() evaluates MRST._RST on the device's own ACPI node. Signed-off-by: Jack Wu --- MAINTAINERS | 11 + drivers/net/wwan/Kconfig | 11 + drivers/net/wwan/Makefile | 1 + drivers/net/wwan/t9xx/Makefile | 9 + drivers/net/wwan/t9xx/mtk_dev.h | 42 ++ drivers/net/wwan/t9xx/pcie/mtk_pci.c | 1033 ++++++++++++++++++++++++++= ++++ drivers/net/wwan/t9xx/pcie/mtk_pci.h | 143 +++++ drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h | 34 + 8 files changed, 1284 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 461a3eed6129..8f13ef7440ce 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -16494,6 +16494,17 @@ L: netdev@vger.kernel.org S: Supported F: drivers/net/wwan/t7xx/ =20 +MEDIATEK T9XX 5G WWAN MODEM DRIVER +M: Jack Wu +M: Robert Yu +M: Jeff Chang +R: Wen-Zhi Huang +R: Shi-Wei Yeh +R: Minano Tseng +L: netdev@vger.kernel.org +S: Supported +F: drivers/net/wwan/t9xx/ + MEDIATEK USB3 DRD IP DRIVER M: Chunfeng Yun L: linux-usb@vger.kernel.org diff --git a/drivers/net/wwan/Kconfig b/drivers/net/wwan/Kconfig index 88df55d78d90..55c45af410ee 100644 --- a/drivers/net/wwan/Kconfig +++ b/drivers/net/wwan/Kconfig @@ -121,6 +121,17 @@ config MTK_T7XX =20 If unsure, say N. =20 +config MTK_T9XX + tristate "MediaTek PCIe 5G WWAN modem T9xx device" + depends on PCI && ACPI + help + Enables MediaTek PCIe based 5G WWAN modem (T9xx series) device. + + To compile this driver as a module, choose M here: the module will be + called mtk_t9xx. + + If unsure, say N. + endif # WWAN =20 endmenu diff --git a/drivers/net/wwan/Makefile b/drivers/net/wwan/Makefile index 3960c0ae2445..7361eef4c472 100644 --- a/drivers/net/wwan/Makefile +++ b/drivers/net/wwan/Makefile @@ -14,3 +14,4 @@ obj-$(CONFIG_QCOM_BAM_DMUX) +=3D qcom_bam_dmux.o obj-$(CONFIG_RPMSG_WWAN_CTRL) +=3D rpmsg_wwan_ctrl.o obj-$(CONFIG_IOSM) +=3D iosm/ obj-$(CONFIG_MTK_T7XX) +=3D t7xx/ +obj-$(CONFIG_MTK_T9XX) +=3D t9xx/ diff --git a/drivers/net/wwan/t9xx/Makefile b/drivers/net/wwan/t9xx/Makefile new file mode 100644 index 000000000000..bd6063c85c2b --- /dev/null +++ b/drivers/net/wwan/t9xx/Makefile @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: GPL-2.0-only + +ccflags-y +=3D -I$(src)/pcie +ccflags-y +=3D -I$(src) + +obj-$(CONFIG_MTK_T9XX) +=3D mtk_t9xx.o + +mtk_t9xx-y :=3D \ + pcie/mtk_pci.o diff --git a/drivers/net/wwan/t9xx/mtk_dev.h b/drivers/net/wwan/t9xx/mtk_de= v.h new file mode 100644 index 000000000000..329e5e6695df --- /dev/null +++ b/drivers/net/wwan/t9xx/mtk_dev.h @@ -0,0 +1,42 @@ +/* SPDX-License-Identifier: GPL-2.0-only + * + * Copyright (c) 2022, MediaTek Inc. + */ + +#ifndef __MTK_DEV_H__ +#define __MTK_DEV_H__ + +#include +#include +#include +#include +#include +#include +#include + +#define MTK_DEV_STR_LEN 16 + +enum mtk_user_id { + MTK_USER_CTRL, + MTK_USER_MAX +}; + +enum mtk_dev_evt_h2d { + DEV_EVT_H2D_DEVICE_RESET =3D BIT(2), +}; + +enum mtk_dev_evt_d2h { + DEV_EVT_D2H_BOOT_FLOW_SYNC =3D BIT(4), + DEV_EVT_D2H_ASYNC_HS_NOTIFY_SAP =3D BIT(5), + DEV_EVT_D2H_ASYNC_HS_NOTIFY_MD =3D BIT(6), +}; + +/* mtk_md_dev defines the structure of MTK modem device */ +struct mtk_md_dev { + struct device *dev; + void *hw_priv; + u32 hw_ver; + char dev_str[MTK_DEV_STR_LEN]; +}; + +#endif /* __MTK_DEV_H__ */ diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.c b/drivers/net/wwan/t9xx/p= cie/mtk_pci.c new file mode 100644 index 000000000000..34ee823119fc --- /dev/null +++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.c @@ -0,0 +1,1033 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022, MediaTek Inc. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "mtk_dev.h" +#include "mtk_pci.h" +#include "mtk_pci_reg.h" + +#define MTK_MHCCIF_RC_BASE_ADDR 0x1000A000 + +static const int mtk_pci_irq_tbl[MTK_IRQ_SRC_MAX] =3D { + [MTK_IRQ_SRC_MHCCIF] =3D 28, + [MTK_IRQ_SRC_CLDMA0] =3D 27, + [MTK_IRQ_SRC_CLDMA1] =3D 26, +}; + +#define MTK_PCI_TRANSPARENT_ATR_SIZE (0x3F) +#define MTK_PCI_MINIMUM_ATR_SIZE (0x1000) +#define ATR_SIZE_LO32_MASK GENMASK_ULL(31, 0) +#define ATR_SIZE_HI32_MASK GENMASK_ULL(63, 32) +#define ATR_SIZE_BIAS_FROM_LO32 2 +#define ATR_ADDR_ALIGN_MASK 0xFFFFF000 +#define ATR_EN BIT(0) +#define ATR_PARAM_OFFSET 16 +#define SET_HW_BITS(dest, chs, mhccif, dev) \ + ({ \ + if ((chs) & (dev)) \ + (dest) |=3D FIELD_PREP(mhccif, 1); \ + }) + +struct mtk_mhccif_cb { + struct list_head entry; + int (*evt_cb)(u32 status, void *data); + void *data; + u32 chs; +}; + +/** + * mtk_pci_setup_atr() - Configure a PCIe address translation rule + * @mdev: MTK MD device + * @cfg: ATR configuration parameters + * + * Return: 0 on success, negative error code on failure. + */ +int mtk_pci_setup_atr(struct mtk_md_dev *mdev, struct mtk_atr_cfg *cfg) +{ + struct mtk_pci_priv *priv =3D mdev->hw_priv; + u32 addr, val, size_h, size_l; + int atr_size, pos, offset; + + if (cfg->transparent) { + /* No address conversion is performed */ + atr_size =3D MTK_PCI_TRANSPARENT_ATR_SIZE; + } else { + if (cfg->size < MTK_PCI_MINIMUM_ATR_SIZE) + cfg->size =3D MTK_PCI_MINIMUM_ATR_SIZE; + + if (cfg->src_addr & (cfg->size - 1)) { + dev_err(mdev->dev, "Invalid atr src addr is not aligned to size\n"); + return -EFAULT; + } + + if (cfg->trsl_addr & (cfg->size - 1)) { + dev_err(mdev->dev, + "Invalid atr trsl addr is not aligned to size, %llx, %llx\n", + cfg->trsl_addr, cfg->size - 1); + return -EFAULT; + } + + size_l =3D FIELD_GET(ATR_SIZE_LO32_MASK, cfg->size); + size_h =3D FIELD_GET(ATR_SIZE_HI32_MASK, cfg->size); + pos =3D ffs(size_l); + if (pos) { + atr_size =3D pos - ATR_SIZE_BIAS_FROM_LO32; + } else { + pos =3D ffs(size_h); + atr_size =3D pos + 32 - ATR_SIZE_BIAS_FROM_LO32; + } + } + + /* Calculate table offset */ + offset =3D ATR_PORT_OFFSET * cfg->port + ATR_TABLE_OFFSET * cfg->table; + + addr =3D REG_ATR_PCIE_WIN0_T0_SRC_ADDR_MSB + offset; + val =3D (u32)(cfg->src_addr >> 32); + mtk_pci_mac_write32(priv, addr, val); + + addr =3D REG_ATR_PCIE_WIN0_T0_TRSL_ADDR_MSB + offset; + val =3D (u32)(cfg->trsl_addr >> 32); + mtk_pci_mac_write32(priv, addr, val); + + addr =3D REG_ATR_PCIE_WIN0_T0_TRSL_ADDR_LSB + offset; + val =3D (u32)(cfg->trsl_addr & ATR_ADDR_ALIGN_MASK); + mtk_pci_mac_write32(priv, addr, val); + + /* TRSL_PARAM */ + addr =3D REG_ATR_PCIE_WIN0_T0_TRSL_PARAM + offset; + val =3D (cfg->trsl_param << ATR_PARAM_OFFSET) | cfg->trsl_id; + mtk_pci_mac_write32(priv, addr, val); + + /* Enable ATR last, after translation target is fully programmed */ + addr =3D REG_ATR_PCIE_WIN0_T0_SRC_ADDR_LSB + offset; + val =3D (u32)(cfg->src_addr & ATR_ADDR_ALIGN_MASK) | (atr_size << 1) | AT= R_EN; + mtk_pci_mac_write32(priv, addr, val); + + /* Ensure ATR is set */ + mtk_pci_mac_read32(priv, addr); + + return 0; +} + +/** + * mtk_pci_atr_disable() - Disable all PCIe address translation rules + * @priv: MTK PCI private data + */ +void mtk_pci_atr_disable(struct mtk_pci_priv *priv) +{ + int port, tbl, offset; + u32 val; + + /* Disable all ATR table for all ports */ + for (port =3D ATR_SRC_PCI_WIN0; port <=3D ATR_SRC_AXIS_3; port++) + for (tbl =3D 0; tbl < ATR_TABLE_NUM_PER_ATR; tbl++) { + /* Calculate table offset */ + offset =3D ATR_PORT_OFFSET * port + ATR_TABLE_OFFSET * tbl; + val =3D mtk_pci_mac_read32(priv, REG_ATR_PCIE_WIN0_T0_SRC_ADDR_LSB + of= fset); + val =3D val & (~BIT(0)); + /* Disable table by SRC_ADDR_L */ + mtk_pci_mac_write32(priv, REG_ATR_PCIE_WIN0_T0_SRC_ADDR_LSB + offset, v= al); + } +} + +/** + * mtk_pci_get_dev_state() - Read the device state from the modem + * @mdev: MTK MD device + * + * Return: Device state value. + */ +u32 mtk_pci_get_dev_state(struct mtk_md_dev *mdev) +{ + return mtk_pci_mac_read32(mdev->hw_priv, REG_PCIE_DEBUG_DUMMY_7); +} + +/** + * mtk_pci_ack_dev_state() - Acknowledge the device state to the modem + * @mdev: MTK MD device + * @state: State value to acknowledge + */ +void mtk_pci_ack_dev_state(struct mtk_md_dev *mdev, u32 state) +{ + mtk_pci_mac_write32(mdev->hw_priv, REG_PCIE_DEBUG_DUMMY_7, state); +} + +/** + * mtk_pci_get_irq_id() - Map an IRQ source to its hardware IRQ ID + * @mdev: MTK MD device + * @irq_src: IRQ source enum + * + * Return: IRQ ID on success, -EINVAL on failure. + */ +int mtk_pci_get_irq_id(struct mtk_md_dev *mdev, enum mtk_irq_src irq_src) +{ + int irq_id =3D -EINVAL; + + if (irq_src > MTK_IRQ_SRC_MIN && irq_src < MTK_IRQ_SRC_MAX) { + irq_id =3D mtk_pci_irq_tbl[irq_src]; + if (irq_id < 0 || irq_id >=3D MTK_IRQ_CNT_MAX) + irq_id =3D -EINVAL; + } + + return irq_id; +} + +/** + * mtk_pci_get_virq_id() - Get the Linux virtual IRQ for a hardware IRQ ID + * @mdev: MTK MD device + * @irq_id: Hardware IRQ ID + * + * Return: Virtual IRQ number on success, negative error code on failure. + */ +int mtk_pci_get_virq_id(struct mtk_md_dev *mdev, int irq_id) +{ + struct pci_dev *pdev =3D to_pci_dev(mdev->dev); + struct mtk_pci_priv *priv =3D mdev->hw_priv; + + if (irq_id < 0 || irq_id >=3D priv->irq_cnt) + return -EINVAL; + + return pci_irq_vector(pdev, irq_id); +} + +/** + * mtk_pci_register_irq() - Register a callback for a hardware IRQ + * @mdev: MTK MD device + * @irq_id: Hardware IRQ ID + * @irq_cb: Callback function + * @data: Private data passed to callback + * + * Return: 0 on success, negative error code on failure. + */ +int mtk_pci_register_irq(struct mtk_md_dev *mdev, int irq_id, + int (*irq_cb)(int irq_id, void *data), void *data) +{ + struct mtk_pci_priv *priv =3D mdev->hw_priv; + + if ((irq_id < 0 || irq_id >=3D MTK_IRQ_CNT_MAX) || !irq_cb) + return -EINVAL; + + spin_lock(&priv->irq_cb_lock); + if (priv->irq_cb_list[irq_id]) { + spin_unlock(&priv->irq_cb_lock); + dev_err(mdev->dev, + "Unable to register irq, irq_id=3D%d, it's already been register by %ps= .\n", + irq_id, priv->irq_cb_list[irq_id]); + return -EFAULT; + } + priv->irq_cb_data[irq_id] =3D data; + smp_wmb(); /* Ensure data is visible before callback */ + WRITE_ONCE(priv->irq_cb_list[irq_id], irq_cb); + spin_unlock(&priv->irq_cb_lock); + + return 0; +} + +/** + * mtk_pci_unregister_irq() - Unregister a hardware IRQ callback + * @mdev: MTK MD device + * @irq_id: Hardware IRQ ID + * + * Return: 0 on success, negative error code on failure. + */ +int mtk_pci_unregister_irq(struct mtk_md_dev *mdev, int irq_id) +{ + struct mtk_pci_priv *priv =3D mdev->hw_priv; + int virq_id; + + if (irq_id < 0 || irq_id >=3D MTK_IRQ_CNT_MAX) + return -EINVAL; + + if (!READ_ONCE(priv->irq_cb_list[irq_id])) { + dev_err(mdev->dev, "irq_id=3D%d has not been registered\n", irq_id); + return -EFAULT; + } + + /* Stop the source and wait for in-flight handlers + * before the callback or its data can disappear. + */ + mtk_pci_mask_irq(mdev, irq_id); + virq_id =3D mtk_pci_get_virq_id(mdev, irq_id); + if (virq_id >=3D 0) + synchronize_irq(virq_id); + + spin_lock(&priv->irq_cb_lock); + WRITE_ONCE(priv->irq_cb_list[irq_id], NULL); + priv->irq_cb_data[irq_id] =3D NULL; + spin_unlock(&priv->irq_cb_lock); + + return 0; +} + +/** + * mtk_pci_mask_irq() - Mask (disable) a hardware IRQ + * @mdev: MTK MD device + * @irq_id: Hardware IRQ ID + * + * Return: 0 on success, negative error code on failure. + */ +int mtk_pci_mask_irq(struct mtk_md_dev *mdev, int irq_id) +{ + struct mtk_pci_priv *priv =3D mdev->hw_priv; + + if (irq_id < 0 || irq_id >=3D MTK_IRQ_CNT_MAX) { + dev_err(mdev->dev, "Failed to mask irq: input irq_id=3D%d\n", irq_id); + return -EINVAL; + } + + mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_CLR_GRP0_0, BIT(irq_id)); + + return 0; +} + +/** + * mtk_pci_unmask_irq() - Unmask (enable) a hardware IRQ + * @mdev: MTK MD device + * @irq_id: Hardware IRQ ID + * + * Return: 0 on success, negative error code on failure. + */ +int mtk_pci_unmask_irq(struct mtk_md_dev *mdev, int irq_id) +{ + struct mtk_pci_priv *priv =3D mdev->hw_priv; + + if (irq_id < 0 || irq_id >=3D MTK_IRQ_CNT_MAX) { + dev_err(mdev->dev, "Failed to unmask irq: input irq_id=3D%d\n", irq_id); + return -EINVAL; + } + + mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_SET_GRP0_0, BIT(irq_id)); + + return 0; +} + +/** + * mtk_pci_clear_irq() - Clear (acknowledge) a hardware IRQ + * @mdev: MTK MD device + * @irq_id: Hardware IRQ ID + * + * Return: 0 on success, negative error code on failure. + */ +int mtk_pci_clear_irq(struct mtk_md_dev *mdev, int irq_id) +{ + struct mtk_pci_priv *priv =3D mdev->hw_priv; + + if (irq_id < 0 || irq_id >=3D MTK_IRQ_CNT_MAX) { + dev_err(mdev->dev, "Failed to clear irq: input irq_id=3D%d\n", irq_id); + return -EINVAL; + } + + mtk_pci_mac_write32(priv, REG_MSIX_ISTATUS_HOST_GRP0_0, BIT(irq_id)); + + return 0; +} + +static u32 mtk_pci_ext_d2h_evt_hw_bits(u32 chs) +{ + u32 hw_bits =3D 0; + + SET_HW_BITS(hw_bits, chs, MHCCIF_EP2RC_EVT_BOOT_FLOW_SYNC, + DEV_EVT_D2H_BOOT_FLOW_SYNC); + SET_HW_BITS(hw_bits, chs, MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_SAP, + DEV_EVT_D2H_ASYNC_HS_NOTIFY_SAP); + SET_HW_BITS(hw_bits, chs, MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_MD, + DEV_EVT_D2H_ASYNC_HS_NOTIFY_MD); + + return hw_bits; +} + +static u32 mtk_pci_ext_d2h_evt_chs(u32 hw_bits) +{ + u32 chs =3D 0; + + if (!hw_bits) + return chs; + + chs =3D FIELD_PREP(DEV_EVT_D2H_BOOT_FLOW_SYNC, + FIELD_GET(MHCCIF_EP2RC_EVT_BOOT_FLOW_SYNC, hw_bits)) | + FIELD_PREP(DEV_EVT_D2H_ASYNC_HS_NOTIFY_SAP, + FIELD_GET(MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_SAP, hw_bits)) | + FIELD_PREP(DEV_EVT_D2H_ASYNC_HS_NOTIFY_MD, + FIELD_GET(MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_MD, hw_bits)); + + return chs; +} + +/** + * mtk_pci_register_ext_evt() - Register a callback for MHCCIF device even= ts + * @mdev: MTK MD device + * @chs: Bitmask of event channels to register + * @evt_cb: Callback function + * @data: Private data passed to callback + * + * Return: 0 on success, negative error code on failure. + */ +int mtk_pci_register_ext_evt(struct mtk_md_dev *mdev, u32 chs, + int (*evt_cb)(u32 status, void *data), void *data) +{ + struct mtk_pci_priv *priv =3D mdev->hw_priv; + struct mtk_mhccif_cb *cb; + int ret =3D 0; + + if (!chs || !evt_cb) + return -EINVAL; + + spin_lock_bh(&priv->mhccif_lock); + list_for_each_entry(cb, &priv->mhccif_cb_list, entry) { + if (cb->chs & chs) { + ret =3D -EFAULT; + dev_err(mdev->dev, + "Unable to register evt, intersection: chs=3D0x%08x&0x%08x cb=3D%ps\n", + chs, cb->chs, cb->evt_cb); + goto err_spin_unlock; + } + } + cb =3D kzalloc(sizeof(*cb), GFP_ATOMIC); + if (!cb) { + ret =3D -ENOMEM; + goto err_spin_unlock; + } + cb->evt_cb =3D evt_cb; + cb->data =3D data; + cb->chs =3D chs; + list_add_tail(&cb->entry, &priv->mhccif_cb_list); +err_spin_unlock: + spin_unlock_bh(&priv->mhccif_lock); + + return ret; +} + +/** + * mtk_pci_unregister_ext_evt() - Unregister an MHCCIF device event callba= ck + * @mdev: MTK MD device + * @chs: Bitmask of event channels to unregister + */ +void mtk_pci_unregister_ext_evt(struct mtk_md_dev *mdev, u32 chs) +{ + struct mtk_pci_priv *priv =3D mdev->hw_priv; + struct mtk_mhccif_cb *cb, *next; + + if (!chs) + return; + + spin_lock_bh(&priv->mhccif_lock); + list_for_each_entry_safe(cb, next, &priv->mhccif_cb_list, entry) { + if (cb->chs =3D=3D chs) { + list_del(&cb->entry); + kfree(cb); + goto out; + } + } + dev_warn(mdev->dev, + "Unable to unregister evt, no chs=3D0x%08x has been registered.\n", chs= ); +out: + spin_unlock_bh(&priv->mhccif_lock); +} + +/** + * mtk_pci_mask_ext_evt() - Mask (disable) MHCCIF device events + * @mdev: MTK MD device + * @chs: Bitmask of event channels to mask + */ +void mtk_pci_mask_ext_evt(struct mtk_md_dev *mdev, u32 chs) +{ + u32 hw_bits =3D mtk_pci_ext_d2h_evt_hw_bits(chs); + + mtk_pci_write32(mdev, MTK_MHCCIF_RC_BASE_ADDR + + MHCCIF_EP2RC_SW_INT_EAP_MASK_SET, hw_bits); +} + +/** + * mtk_pci_unmask_ext_evt() - Unmask (enable) MHCCIF device events + * @mdev: MTK MD device + * @chs: Bitmask of event channels to unmask + */ +void mtk_pci_unmask_ext_evt(struct mtk_md_dev *mdev, u32 chs) +{ + u32 hw_bits =3D mtk_pci_ext_d2h_evt_hw_bits(chs); + + mtk_pci_write32(mdev, MTK_MHCCIF_RC_BASE_ADDR + + MHCCIF_EP2RC_SW_INT_EAP_MASK_CLR, hw_bits); +} + +/** + * mtk_pci_clear_ext_evt() - Clear (acknowledge) MHCCIF device events + * @mdev: MTK MD device + * @chs: Bitmask of event channels to clear + */ +void mtk_pci_clear_ext_evt(struct mtk_md_dev *mdev, u32 chs) +{ + u32 hw_bits =3D mtk_pci_ext_d2h_evt_hw_bits(chs); + + mtk_pci_write32(mdev, MTK_MHCCIF_RC_BASE_ADDR + + MHCCIF_EP2RC_SW_INT_ACK, hw_bits); +} + +static u32 mtk_pci_ext_h2d_evt_hw_bits(u32 chs) +{ + u32 hw_bits =3D 0; + + SET_HW_BITS(hw_bits, chs, MHCCIF_RC2EP_EVT_DEVICE_RESET, + DEV_EVT_H2D_DEVICE_RESET); + return hw_bits; +} + +/** + * mtk_pci_send_ext_evt() - Send an MHCCIF event to the modem + * @mdev: MTK MD device + * @ch: Event channel to trigger (must be a single bit) + * + * Return: 0 on success, negative error code on failure. + */ +int mtk_pci_send_ext_evt(struct mtk_md_dev *mdev, u32 ch) +{ + u32 rc_base =3D MTK_MHCCIF_RC_BASE_ADDR; + u32 hw_bits; + + /* Only allow one ch to be triggered at a time */ + if (!is_power_of_2(ch)) { + dev_err(mdev->dev, "Unsupported ext evt ch=3D0x%08x\n", ch); + return -EINVAL; + } + + hw_bits =3D mtk_pci_ext_h2d_evt_hw_bits(ch); + if (!hw_bits) { + dev_err(mdev->dev, "Unmapped ext evt ch=3D0x%08x\n", ch); + return -EINVAL; + } + + mtk_pci_write32(mdev, rc_base + MHCCIF_RC2EP_SW_BSY, hw_bits); + mtk_pci_write32(mdev, rc_base + MHCCIF_RC2EP_SW_TCHNUM, ffs(hw_bits) - 1); + return 0; +} + +static u32 mtk_pci_get_ext_evt_hw_status(struct mtk_md_dev *mdev) +{ + return mtk_pci_read32(mdev, MTK_MHCCIF_RC_BASE_ADDR + + MHCCIF_EP2RC_SW_INT_STS); +} + +static void mtk_pci_ack_ext_evt_hw(struct mtk_md_dev *mdev, u32 hw_bits) +{ + mtk_pci_write32(mdev, MTK_MHCCIF_RC_BASE_ADDR + + MHCCIF_EP2RC_SW_INT_ACK, hw_bits); + /* Ensure the ack lands before level 1 is cleared */ + mtk_pci_read32(mdev, MTK_MHCCIF_RC_BASE_ADDR + + MHCCIF_EP2RC_SW_INT_STS); +} + +/** + * mtk_pci_pldr() - Reset the modem via its ACPI MRST._RST method + * @mdev: MTK MD device + * + * Return: + * * 0 - Success, device was reset. + * * -ENODEV - No ACPI, no handle or no MRST._RST method. Device + * untouched. + * * -EIO - MRST._RST failed. Device untouched, still running; + * a different reset may be attempted. + */ +int mtk_pci_pldr(struct mtk_md_dev *mdev) +{ + acpi_status acpi_ret; + acpi_handle handle; + + if (acpi_disabled) { + dev_err(mdev->dev, "Unsupported, acpi function isn't enable\n"); + return -ENODEV; + } + + handle =3D ACPI_HANDLE(mdev->dev); + if (!handle) { + dev_err(mdev->dev, "Unsupported, acpi handle isn't found\n"); + return -ENODEV; + } + if (!acpi_has_method(handle, "MRST._RST")) { + dev_err(mdev->dev, "Unsupported, pldr method isn't supported\n"); + return -ENODEV; + } + acpi_ret =3D acpi_evaluate_object(handle, "MRST._RST", NULL, NULL); + if (ACPI_FAILURE(acpi_ret)) { + dev_err(mdev->dev, "Failed to execute MRST._RST method: %s\n", + acpi_format_exception(acpi_ret)); + return -EIO; + } + + return 0; +} + +/** + * mtk_pci_link_check() - Check if the PCIe link to the modem is active + * @mdev: MTK MD device + * + * Return: true if the device is present, false otherwise. + */ +bool mtk_pci_link_check(struct mtk_md_dev *mdev) +{ + return pci_device_is_present(to_pci_dev(mdev->dev)); +} + +static void mtk_mhccif_isr_work(struct work_struct *work) +{ + struct mtk_pci_priv *priv =3D + container_of(work, struct mtk_pci_priv, mhccif_work); + struct mtk_md_dev *mdev =3D priv->irq_desc->mdev; + struct mtk_mhccif_cb *cb; + u32 stat, mask, chs; + + stat =3D mtk_pci_get_ext_evt_hw_status(mdev); + mask =3D mtk_pci_read32(mdev, MTK_MHCCIF_RC_BASE_ADDR + + MHCCIF_EP2RC_SW_INT_EAP_MASK); + if (unlikely(stat =3D=3D U32_MAX && !(mtk_pci_link_check(mdev)))) { + /* When link failed, we don't need to unmask/clear. */ + dev_err(mdev->dev, "Failed to check link in MHCCIF handler.\n"); + return; + } + + stat &=3D ~mask; + /* Acknowledge level 2 before dispatch: an event that re-asserts + * while a callback runs must survive as a new status bit. + */ + if (stat) + mtk_pci_ack_ext_evt_hw(mdev, stat); + + chs =3D mtk_pci_ext_d2h_evt_chs(stat); + /* Callbacks must not sleep or modify mhccif_cb_list */ + spin_lock_bh(&priv->mhccif_lock); + list_for_each_entry(cb, &priv->mhccif_cb_list, entry) { + if (cb->chs & chs) + cb->evt_cb(cb->chs & chs, cb->data); + } + spin_unlock_bh(&priv->mhccif_lock); + + mtk_pci_clear_irq(mdev, priv->mhccif_irq_id); + mtk_pci_unmask_irq(mdev, priv->mhccif_irq_id); +} + +static const struct pci_device_id t9xx_pci_table[] =3D { + { PCI_DEVICE(MTK_PCI_VENDOR_ID, 0x0900), MTK_PCI_CLASS, PCI_ANY_ID }, + { PCI_DEVICE(CEI_PCI_VENDOR_ID, 0x01CA), MTK_PCI_CLASS, PCI_ANY_ID }, + {/* end: all zeroes */} +}; + +MODULE_DEVICE_TABLE(pci, t9xx_pci_table); + +static int mtk_pci_atr_init(struct mtk_md_dev *mdev) +{ + struct pci_dev *pdev =3D to_pci_dev(mdev->dev); + struct mtk_pci_priv *priv =3D mdev->hw_priv; + struct mtk_atr_cfg cfg; + int port, ret; + + mtk_pci_atr_disable(priv); + + /* Config ATR for RC to access device's register */ + cfg.src_addr =3D pci_resource_start(pdev, MTK_BAR_2_3_IDX); + cfg.size =3D ATR_PCIE_REG_SIZE; + cfg.trsl_addr =3D ATR_PCIE_REG_TRSL_ADDR; + cfg.port =3D ATR_PCIE_REG_PORT; + cfg.table =3D ATR_PCIE_REG_TABLE_NUM; + cfg.trsl_id =3D ATR_PCIE_REG_TRSL_PORT; + cfg.trsl_param =3D 0x0; + cfg.transparent =3D 0x0; + ret =3D mtk_pci_setup_atr(mdev, &cfg); + if (ret) + return ret; + + /* Config ATR for EP to access RC's memory */ + for (port =3D ATR_SRC_AXIS_0; port <=3D ATR_SRC_AXIS_3; port++) { + cfg.src_addr =3D ATR_PCIE_DEV_DMA_SRC_ADDR; + cfg.size =3D ATR_PCIE_DEV_DMA_SIZE; + cfg.trsl_addr =3D ATR_PCIE_DEV_DMA_TRSL_ADDR; + cfg.port =3D port; + cfg.table =3D ATR_PCIE_DEV_DMA_TABLE_NUM; + cfg.trsl_id =3D ATR_DST_PCI_TRX; + cfg.trsl_param =3D 0x0; + /* Enable transparent translation */ + cfg.transparent =3D ATR_PCIE_DEV_DMA_TRANSPARENT; + ret =3D mtk_pci_setup_atr(mdev, &cfg); + if (ret) + return ret; + } + + return 0; +} + +static int mtk_pci_bar_init(struct mtk_md_dev *mdev) +{ + struct pci_dev *pdev =3D to_pci_dev(mdev->dev); + struct mtk_pci_priv *priv =3D mdev->hw_priv; + + /* Fixed offsets used on these mappings (MSI-X registers up to + * 0x3080 on BAR0/1, the ATR-biased MHCCIF window on BAR2/3) are + * guaranteed by the only two devices this driver binds (0x0900, + * 0x01CA), whose BAR sizes are fixed by hardware; an undersized + * BAR here is impossible, so no pci_resource_len() checks. + */ + priv->mac_reg_base =3D pcim_iomap_region(pdev, MTK_BAR_0_1_IDX, + mdev->dev_str); + if (IS_ERR(priv->mac_reg_base)) { + dev_err(mdev->dev, "Failed to map BAR0/1\n"); + return PTR_ERR(priv->mac_reg_base); + } + + priv->bar23_addr =3D pcim_iomap_region(pdev, MTK_BAR_2_3_IDX, + mdev->dev_str); + if (IS_ERR(priv->bar23_addr)) { + dev_err(mdev->dev, "Failed to map BAR2/3\n"); + return PTR_ERR(priv->bar23_addr); + } + + /* We use MD view base address "0" to observe registers */ + priv->ext_reg_base =3D priv->bar23_addr - ATR_PCIE_REG_TRSL_ADDR; + + return 0; +} + +static int mtk_mhccif_irq_cb(int irq_id, void *data) +{ + struct mtk_md_dev *mdev =3D data; + struct mtk_pci_priv *priv; + + priv =3D mdev->hw_priv; + queue_work(system_highpri_wq, &priv->mhccif_work); + + return 0; +} + +static int mtk_mhccif_init(struct mtk_md_dev *mdev) +{ + struct mtk_pci_priv *priv =3D mdev->hw_priv; + int ret; + + INIT_LIST_HEAD(&priv->mhccif_cb_list); + spin_lock_init(&priv->mhccif_lock); + INIT_WORK(&priv->mhccif_work, mtk_mhccif_isr_work); + + /* The write below goes through the BAR 2/3 ATR window, so this + * function must run after mtk_pci_atr_init(). + * Mask every channel; consumers unmask the channels they need. + * Do NOT ack here: EP2RC status bits are latched one-shot + * notifications (e.g. BOOT_FLOW_SYNC raised before the driver + * loads) that the device never re-sends. Masking blocks + * delivery but preserves the bit for delivery on unmask; an + * ack would erase it and the boot flow would hang silently. + */ + mtk_pci_write32(mdev, MTK_MHCCIF_RC_BASE_ADDR + + MHCCIF_EP2RC_SW_INT_EAP_MASK_SET, U32_MAX); + + ret =3D mtk_pci_get_irq_id(mdev, MTK_IRQ_SRC_MHCCIF); + if (ret < 0) { + dev_err(mdev->dev, "Failed to get mhccif_irq_id. ret=3D%d\n", ret); + return ret; + } + priv->mhccif_irq_id =3D ret; + + ret =3D mtk_pci_register_irq(mdev, priv->mhccif_irq_id, mtk_mhccif_irq_cb= , mdev); + if (ret) { + dev_err(mdev->dev, "Failed to register mhccif_irq callback\n"); + return ret; + } + + return 0; +} + +static void mtk_mhccif_exit(struct mtk_md_dev *mdev) +{ + struct mtk_pci_priv *priv =3D mdev->hw_priv; + struct mtk_mhccif_cb *cb, *tmp; + + mtk_pci_unregister_irq(mdev, priv->mhccif_irq_id); + cancel_work_sync(&priv->mhccif_work); + /* the work unmasks the source on its way out */ + mtk_pci_mask_irq(mdev, priv->mhccif_irq_id); + + list_for_each_entry_safe(cb, tmp, &priv->mhccif_cb_list, entry) { + list_del(&cb->entry); + kfree(cb); + } +} + +static irqreturn_t mtk_pci_irq_handler(struct mtk_md_dev *mdev, u32 irq_st= ate) +{ + struct mtk_pci_priv *priv =3D mdev->hw_priv; + int irq_id; + + /* Check whether each set bit has a callback, if has, call it */ + do { + int (*cb)(int irq_id, void *data); + + irq_id =3D fls(irq_state) - 1; + irq_state &=3D ~BIT(irq_id); + cb =3D READ_ONCE(priv->irq_cb_list[irq_id]); + if (likely(cb)) { + smp_rmb(); /* Ensure data is read after callback */ + cb(irq_id, priv->irq_cb_data[irq_id]); + } else { + dev_err_ratelimited(mdev->dev, + "Unhandled irq_id=3D%d, no callback for it.\n", + irq_id); + mtk_pci_clear_irq(mdev, irq_id); + } + } while (irq_state); + + return IRQ_HANDLED; +} + +static irqreturn_t mtk_pci_irq_msix(int irq, void *data) +{ + struct mtk_pci_irq_desc *irq_desc =3D data; + struct mtk_md_dev *mdev =3D irq_desc->mdev; + struct mtk_pci_priv *priv; + u32 irq_state, irq_enable; + + priv =3D mdev->hw_priv; + irq_state =3D mtk_pci_mac_read32(priv, REG_MSIX_ISTATUS_HOST_GRP0_0); + irq_enable =3D mtk_pci_mac_read32(priv, REG_IMASK_HOST_MSIX_GRP0_0); + irq_state &=3D irq_enable; + + if (unlikely(irq_state =3D=3D U32_MAX && irq_enable =3D=3D U32_MAX)) + return IRQ_NONE; /* device gone */ + + irq_state &=3D irq_desc->msix_bits; /* scope to this vector */ + if (unlikely(!irq_state)) + return IRQ_NONE; + + /* Mask the bit; the consumer unmasks it when it is done */ + mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_CLR_GRP0_0, irq_state); + + return mtk_pci_irq_handler(mdev, irq_state); +} + +static int mtk_pci_request_irq_msix(struct mtk_md_dev *mdev) +{ + struct mtk_pci_priv *priv =3D mdev->hw_priv; + struct mtk_pci_irq_desc *irq_desc; + struct pci_dev *pdev; + int ret, i; + + pdev =3D to_pci_dev(mdev->dev); + irq_desc =3D priv->irq_desc; + + priv->irq_cnt =3D MTK_IRQ_CNT_MAX; + + for (i =3D 0; i < MTK_IRQ_CNT_MAX; i++) { + irq_desc[i].mdev =3D mdev; + irq_desc[i].msix_bits =3D BIT(i); + snprintf(irq_desc[i].name, MTK_IRQ_NAME_LEN, "msix%d-%s", i, mdev->dev_s= tr); + ret =3D pci_request_irq(pdev, i, mtk_pci_irq_msix, NULL, + &irq_desc[i], "%s", irq_desc[i].name); + if (ret) { + dev_err(mdev->dev, "Failed to request %s: ret=3D%d\n", + irq_desc[i].name, ret); + for (i--; i >=3D 0; i--) + pci_free_irq(pdev, i, &irq_desc[i]); + priv->irq_cnt =3D 0; + return ret; + } + } + + return 0; +} + +static int mtk_pci_request_irq(struct mtk_md_dev *mdev) +{ + struct pci_dev *pdev =3D to_pci_dev(mdev->dev); + int ret; + + ret =3D pci_alloc_irq_vectors(pdev, MTK_IRQ_CNT_MAX, + MTK_IRQ_CNT_MAX, PCI_IRQ_MSIX); + if (ret < 0) { + dev_err(mdev->dev, + "Unable to alloc %d MSI-X vectors: ret=3D%d\n", + MTK_IRQ_CNT_MAX, ret); + return ret; + } + + ret =3D mtk_pci_request_irq_msix(mdev); + if (ret) + pci_free_irq_vectors(pdev); + + return ret; +} + +static void mtk_pci_free_irq(struct mtk_md_dev *mdev) +{ + struct pci_dev *pdev =3D to_pci_dev(mdev->dev); + struct mtk_pci_priv *priv =3D mdev->hw_priv; + int i; + + for (i =3D 0; i < priv->irq_cnt; i++) + pci_free_irq(pdev, i, &priv->irq_desc[i]); + + pci_free_irq_vectors(pdev); +} + +static int mtk_pci_probe(struct pci_dev *pdev, const struct pci_device_id = *id) +{ + struct device *dev =3D &pdev->dev; + struct mtk_pci_priv *priv; + struct mtk_md_dev *mdev; + int ret; + + mdev =3D devm_kzalloc(dev, sizeof(*mdev), GFP_KERNEL); + if (!mdev) { + ret =3D -ENOMEM; + goto log_err; + } + mdev->dev =3D dev; + + priv =3D devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL); + if (!priv) { + ret =3D -ENOMEM; + goto log_err; + } + + pci_set_drvdata(pdev, mdev); + priv->mdev =3D mdev; + mdev->hw_ver =3D pdev->device; + mdev->hw_priv =3D priv; + mdev->dev =3D dev; + snprintf(mdev->dev_str, MTK_DEV_STR_LEN, "%02x%02x%d", + pdev->bus->number, PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn)); + if (pdev->state_saved) + pci_restore_state(pdev); + + ret =3D pci_enable_device(pdev); + if (ret) { + dev_err(mdev->dev, "Failed to enable pci device.\n"); + goto log_err; + } + + ret =3D dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64)); + if (ret) { + dev_err(mdev->dev, "Failed to set DMA Mask and Coherent. (ret=3D%d)\n", = ret); + goto disable_dev; + } + + ret =3D mtk_pci_bar_init(mdev); + if (ret) + goto disable_dev; + + ret =3D mtk_pci_atr_init(mdev); + if (ret) + goto disable_dev; + + spin_lock_init(&priv->irq_cb_lock); + + ret =3D mtk_mhccif_init(mdev); + if (ret) + goto disable_dev; + + /* Mask every source before the handlers go in, so a source + * left enabled by firmware or a previous bind cannot fire + * into a half-initialised driver. + */ + mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_CLR_GRP0_0, U32_MAX); + + ret =3D mtk_pci_request_irq(mdev); + if (ret) + goto free_mhccif; + + pci_set_master(pdev); + mtk_pci_unmask_irq(mdev, priv->mhccif_irq_id); + + if (!mtk_pci_link_check(mdev)) { + ret =3D -ENOLINK; + goto clear_master; + } + + ret =3D pci_save_state(pdev); + if (ret) { + dev_err(mdev->dev, "Failed to save PCI state: %d\n", ret); + goto clear_master; + } + + priv->saved_state =3D pci_store_saved_state(pdev); + if (!priv->saved_state) { + ret =3D -ENOMEM; + goto clear_master; + } + + return 0; + +clear_master: + pci_clear_master(pdev); + mtk_pci_free_irq(mdev); +free_mhccif: + mtk_mhccif_exit(mdev); +disable_dev: + pci_disable_device(pdev); +log_err: + dev_err(dev, "Failed to probe device, ret=3D%d\n", ret); + + return ret; +} + +static void mtk_pci_remove(struct pci_dev *pdev) +{ + struct mtk_md_dev *mdev =3D pci_get_drvdata(pdev); + struct mtk_pci_priv *priv =3D mdev->hw_priv; + struct device *dev =3D &pdev->dev; + int ret; + + /* Silence every source before tearing anything down. */ + mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_CLR_GRP0_0, U32_MAX); + + /* Unregisters the callback (masks and synchronises the vector), + * then cancels the work. With the callback gone the work cannot + * be requeued. + */ + mtk_mhccif_exit(mdev); + mtk_pci_free_irq(mdev); + pci_clear_master(pdev); + pci_disable_device(pdev); + + /* Reset last: the endpoint comes back at power-on defaults, + * so no MMIO, config write or MSI-X teardown may follow it. + */ + ret =3D mtk_pci_pldr(mdev); + if (ret && mtk_pci_link_check(mdev)) { + dev_warn(dev, "PLDR failed (%d), trying MHCCIF reset\n", ret); + if (mtk_pci_send_ext_evt(mdev, DEV_EVT_H2D_DEVICE_RESET)) + dev_err(dev, "MHCCIF reset failed\n"); + } + + pci_load_and_free_saved_state(pdev, &priv->saved_state); +} + +static pci_ers_result_t mtk_pci_error_detected(struct pci_dev *pdev, + pci_channel_state_t state) +{ + struct mtk_md_dev *mdev =3D pci_get_drvdata(pdev); + + dev_err(mdev->dev, "AER detected: pci_channel_state_t=3D%u\n", state); + + /* AER recovery not supported, disconnect the device */ + return PCI_ERS_RESULT_DISCONNECT; +} + +static const struct pci_error_handlers mtk_pci_err_handler =3D { + .error_detected =3D mtk_pci_error_detected, +}; + +static struct pci_driver mtk_pci_drv =3D { + .name =3D "mtk_pci_drv", + .id_table =3D t9xx_pci_table, + .probe =3D mtk_pci_probe, + .remove =3D mtk_pci_remove, + .err_handler =3D &mtk_pci_err_handler +}; + +module_pci_driver(mtk_pci_drv); + +MODULE_DESCRIPTION("MediaTek T9xx PCIe WWAN driver"); +MODULE_LICENSE("GPL"); diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.h b/drivers/net/wwan/t9xx/p= cie/mtk_pci.h new file mode 100644 index 000000000000..a155846dbba3 --- /dev/null +++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.h @@ -0,0 +1,143 @@ +/* SPDX-License-Identifier: GPL-2.0-only + * + * Copyright (c) 2022, MediaTek Inc. + */ + +#ifndef __MTK_PCI_H__ +#define __MTK_PCI_H__ + +#include + +#include "../mtk_dev.h" + +enum mtk_irq_src { + MTK_IRQ_SRC_MIN, + MTK_IRQ_SRC_MHCCIF, + MTK_IRQ_SRC_CLDMA0, + MTK_IRQ_SRC_CLDMA1, + MTK_IRQ_SRC_MAX +}; + +enum mtk_atr_src_port { + ATR_SRC_PCI_WIN0 =3D 0, + ATR_SRC_PCI_WIN1, + ATR_SRC_AXIS_0, + ATR_SRC_AXIS_1, + ATR_SRC_AXIS_2, + ATR_SRC_AXIS_3, +}; + +enum mtk_atr_dst_port { + ATR_DST_PCI_TRX =3D 0, + ATR_DST_AXIM_0 =3D 4, +}; + +#define MTK_PCI_CLASS 0x0D4000 +#define MTK_PCI_VENDOR_ID 0x14C3 +#define CEI_PCI_VENDOR_ID 0x03F0 + +#define MTK_BAR_0_1_IDX 0 +#define MTK_BAR_2_3_IDX 2 + +#define MTK_IRQ_CNT_MAX 32 +#define MTK_IRQ_NAME_LEN 32 + +#define ATR_PORT_OFFSET 0x100 +#define ATR_TABLE_OFFSET 0x20 +#define ATR_TABLE_NUM_PER_ATR 8 +#define ATR_PCIE_REG_TRSL_ADDR 0x10000000 +#define ATR_PCIE_REG_SIZE 0x00400000 +#define ATR_PCIE_REG_PORT ATR_SRC_PCI_WIN0 +#define ATR_PCIE_REG_TABLE_NUM 1 +#define ATR_PCIE_REG_TRSL_PORT ATR_DST_AXIM_0 +#define ATR_PCIE_DEV_DMA_SRC_ADDR 0x00000000 +#define ATR_PCIE_DEV_DMA_TRANSPARENT 1 +#define ATR_PCIE_DEV_DMA_SIZE 0 +#define ATR_PCIE_DEV_DMA_TABLE_NUM 0 +#define ATR_PCIE_DEV_DMA_TRSL_ADDR 0x00000000 + +struct mtk_pci_irq_desc { + struct mtk_md_dev *mdev; + u32 msix_bits; + char name[MTK_IRQ_NAME_LEN]; +}; + +struct mtk_pci_priv { + struct mtk_md_dev *mdev; + void __iomem *bar23_addr; + void __iomem *mac_reg_base; + void __iomem *ext_reg_base; + int irq_cnt; + /* irq_cb_lock: protects irq_cb_list[] and irq_cb_data[] */ + spinlock_t irq_cb_lock; + void *irq_cb_data[MTK_IRQ_CNT_MAX]; + + int (*irq_cb_list[MTK_IRQ_CNT_MAX])(int irq_id, void *data); + struct mtk_pci_irq_desc irq_desc[MTK_IRQ_CNT_MAX]; + struct list_head mhccif_cb_list; + /* mhccif_lock: lock to protect mhccif_cb_list */ + spinlock_t mhccif_lock; + struct work_struct mhccif_work; + int mhccif_irq_id; + struct pci_saved_state *saved_state; +}; + +struct mtk_atr_cfg { + u64 src_addr; + u64 trsl_addr; + u64 size; + u32 port; /* Port number */ + u32 table; /* Table number (8 tables for each port) */ + u32 trsl_id; + u32 trsl_param; + u32 transparent; +}; + +/* BAR 0/1 MMIO access */ +static inline u32 mtk_pci_mac_read32(struct mtk_pci_priv *priv, u64 addr) +{ + return ioread32(priv->mac_reg_base + addr); +} + +static inline void mtk_pci_mac_write32(struct mtk_pci_priv *priv, u64 addr= , u32 val) +{ + iowrite32(val, priv->mac_reg_base + addr); +} + +/* BAR 2/3 MMIO access */ +static inline u32 mtk_pci_read32(struct mtk_md_dev *mdev, u64 addr) +{ + return ioread32(((struct mtk_pci_priv *)mdev->hw_priv)->ext_reg_base + ad= dr); +} + +static inline void mtk_pci_write32(struct mtk_md_dev *mdev, u64 addr, u32 = val) +{ + iowrite32(val, ((struct mtk_pci_priv *)mdev->hw_priv)->ext_reg_base + add= r); +} + +/* Device operations */ +u32 mtk_pci_get_dev_state(struct mtk_md_dev *mdev); +void mtk_pci_ack_dev_state(struct mtk_md_dev *mdev, u32 state); +/* IRQ Related operations */ +int mtk_pci_get_irq_id(struct mtk_md_dev *mdev, enum mtk_irq_src irq_src); +int mtk_pci_get_virq_id(struct mtk_md_dev *mdev, int irq_id); +int mtk_pci_register_irq(struct mtk_md_dev *mdev, int irq_id, + int (*irq_cb)(int irq_id, void *data), void *data); +int mtk_pci_unregister_irq(struct mtk_md_dev *mdev, int irq_id); +int mtk_pci_mask_irq(struct mtk_md_dev *mdev, int irq_id); +int mtk_pci_unmask_irq(struct mtk_md_dev *mdev, int irq_id); +int mtk_pci_clear_irq(struct mtk_md_dev *mdev, int irq_id); +/* External event related */ +int mtk_pci_register_ext_evt(struct mtk_md_dev *mdev, u32 chs, + int (*evt_cb)(u32 status, void *data), void *data); +void mtk_pci_unregister_ext_evt(struct mtk_md_dev *mdev, u32 chs); +void mtk_pci_mask_ext_evt(struct mtk_md_dev *mdev, u32 chs); +void mtk_pci_unmask_ext_evt(struct mtk_md_dev *mdev, u32 chs); +void mtk_pci_clear_ext_evt(struct mtk_md_dev *mdev, u32 chs); +int mtk_pci_send_ext_evt(struct mtk_md_dev *mdev, u32 ch); +int mtk_pci_pldr(struct mtk_md_dev *mdev); +bool mtk_pci_link_check(struct mtk_md_dev *mdev); +int mtk_pci_setup_atr(struct mtk_md_dev *mdev, struct mtk_atr_cfg *cfg); +void mtk_pci_atr_disable(struct mtk_pci_priv *priv); + +#endif /* __MTK_PCI_H__ */ diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h b/drivers/net/wwan/t9= xx/pcie/mtk_pci_reg.h new file mode 100644 index 000000000000..a97ad6630dcf --- /dev/null +++ b/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0-only + * + * Copyright (c) 2022, MediaTek Inc. + */ + +#ifndef __MTK_PCI_REG_H__ +#define __MTK_PCI_REG_H__ + +#define REG_ATR_PCIE_WIN0_T0_SRC_ADDR_LSB 0x0600 +#define REG_ATR_PCIE_WIN0_T0_SRC_ADDR_MSB 0x0604 +#define REG_ATR_PCIE_WIN0_T0_TRSL_ADDR_LSB 0x0608 +#define REG_ATR_PCIE_WIN0_T0_TRSL_ADDR_MSB 0x060C +#define REG_ATR_PCIE_WIN0_T0_TRSL_PARAM 0x0610 +#define REG_PCIE_DEBUG_DUMMY_7 0x0D1C +#define REG_MSIX_ISTATUS_HOST_GRP0_0 0x0F00 +#define REG_IMASK_HOST_MSIX_SET_GRP0_0 0x3000 +#define REG_IMASK_HOST_MSIX_CLR_GRP0_0 0x3080 +#define REG_IMASK_HOST_MSIX_GRP0_0 0x3100 + +/* mhccif registers */ +#define MHCCIF_RC2EP_SW_BSY 0x4 +#define MHCCIF_RC2EP_SW_TCHNUM 0xC +#define MHCCIF_RC2EP_EVT_DEVICE_RESET BIT(13) + +#define MHCCIF_EP2RC_SW_INT_STS 0x10 +#define MHCCIF_EP2RC_SW_INT_ACK 0x14 +#define MHCCIF_EP2RC_SW_INT_EAP_MASK 0x20 +#define MHCCIF_EP2RC_SW_INT_EAP_MASK_SET 0x30 +#define MHCCIF_EP2RC_SW_INT_EAP_MASK_CLR 0x40 +#define MHCCIF_EP2RC_EVT_BOOT_FLOW_SYNC BIT(5) +#define MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_SAP BIT(15) +#define MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_MD BIT(16) + +#endif /* __MTK_PCI_REG_H__ */ --=20 2.34.1 From nobody Wed Sep 30 11:25:46 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A915B44F557; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790754372; cv=none; b=ecLb6lpgml8VpsEZDwnurqviHKG2aTzSHVezFJfTjTf0hEQto/bBfq8cV9+E1WzIi3KI66BbsMVNESe9sQO8ywR4YSPEaWRJro2BdEBM4aOur7ADkGWz47IBlI3y5Wrwwrj965GC5d4xWaooADVnYJ1LULJUiYyssD4ETdH/x7A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790754372; c=relaxed/simple; bh=nvGBz05PQL7TBFxG78iW6YO30fpji/1etEFgt0zEPxw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=pnCF1xBhYv88zd7zZWj/yfy5/l812REHLNrpEUDJBAxWT0Yqgyfdt3rpdnAerW65+MZtIMHRKwCtTW3sWAm0BTSxkXYfec3NltGvuYRZOH7UNa7DHAkF55URO4C/0CfGh8pXb5qTmUiLMp4kG8MDimdosHXIb2RZukxiPTtN3+A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=NhgabHwN; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="NhgabHwN" Received: by smtp.kernel.org (Postfix) with ESMTPS id 5E22FC4AF19; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1790754372; bh=nvGBz05PQL7TBFxG78iW6YO30fpji/1etEFgt0zEPxw=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=NhgabHwNC0oUzZoTpyLAeH35BUF1WGf6IF5J1dJEKQ3DaVz2WgwnTweYbM0JuHpWe JIxGOXDiNjccugp2fpfaF9XP90rVJq0+BM8UoJISR2/KtDqR23OROiBnOseTVMYsfV uJo+fCsVcQBPacLXXbahvWR5oCtgtxsUSZmmiT/EkdhOl1401BefuHoewg1jNFOIio Hpi2hgqB2+Efi/6BxWwSubzskuaSiXi8bXNQoQVyQ7Odm3Ci0AdFttwUtq17Ef+Mt0 0dX2H9ADymhc67d6qkAoNDUdHBo6L6o2IKuvsUv39Gj4UPNOk7eJtUI32myI89AkWM AGyjgnNS4tYvA== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41A22CA5FC4; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) From: Jack Wu via B4 Relay Date: Wed, 30 Sep 2026 15:46:09 +0800 Subject: [PATCH v9 2/6] net: wwan: t9xx: Add control plane transaction layer Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260930-t9xx_driver_v1-v9-2-3248b9256956@compal.com> References: <20260930-t9xx_driver_v1-v9-0-3248b9256956@compal.com> In-Reply-To: <20260930-t9xx_driver_v1-v9-0-3248b9256956@compal.com> To: Loic Poulain , Sergey Ryazanov , Johannes Berg , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jack Wu , Wen-Zhi Huang , Shi-Wei Yeh , Minano Tseng , Matthias Brugger , AngeloGioacchino Del Regno , Simon Horman , Jonathan Corbet , Shuah Khan , Robert Yu , Jeff Chang Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, linux-doc@vger.kernel.org X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1790754369; l=4860; i=jackbb_wu@compal.com; s=20260526; h=from:subject:message-id; bh=uRaR7LXWts9xSG1pICFj0duFvYSLuyCLHNcuT1Xp8Iw=; b=FQGFJTsopdan3Dy37SlJ1AuYAdM49N5ISKdHxFy85ZfIVQsSr0OZBdYOuFOrLSrrVbbwynDV2 WUG/rK8Zt5DA6qW1pXYYxzuWZ0+S5u6METIfeMV2wWv+3Dr2ZMCsHMN X-Developer-Key: i=jackbb_wu@compal.com; a=ed25519; pk=VH1prTWixNl8OEUPPSfII3p46MzJpQN8J3+ecE1tZXg= X-Endpoint-Received: by B4 Relay for jackbb_wu@compal.com/20260526 with auth_id=793 X-Original-From: Jack Wu Reply-To: jackbb_wu@compal.com From: Jack Wu Introduce the control plane and transaction layer framework for the T9XX WWAN driver. This patch adds the core data structures (mtk_ctrl_blk, mtk_ctrl_trans) and initialization entry points that subsequent patches build upon. The control plane block (ctrl_blk) serves as the central context for managing the transaction layer, port layer, and FSM interactions. The actual DMA engine and TX/RX service implementations are added in subsequent patches. Signed-off-by: Jack Wu --- drivers/net/wwan/t9xx/Makefile | 1 + drivers/net/wwan/t9xx/mtk_ctrl_plane.c | 44 +++++++++++++++++++++++++= ++++ drivers/net/wwan/t9xx/mtk_ctrl_plane.h | 22 +++++++++++++++ drivers/net/wwan/t9xx/mtk_dev.h | 3 ++ drivers/net/wwan/t9xx/pcie/mtk_pci.c | 1 + drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h | 21 ++++++++++++++ 6 files changed, 92 insertions(+) diff --git a/drivers/net/wwan/t9xx/Makefile b/drivers/net/wwan/t9xx/Makefile index bd6063c85c2b..0a07507f1ddf 100644 --- a/drivers/net/wwan/t9xx/Makefile +++ b/drivers/net/wwan/t9xx/Makefile @@ -6,4 +6,5 @@ ccflags-y +=3D -I$(src) obj-$(CONFIG_MTK_T9XX) +=3D mtk_t9xx.o =20 mtk_t9xx-y :=3D \ + mtk_ctrl_plane.o \ pcie/mtk_pci.o diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.c b/drivers/net/wwan/t9xx= /mtk_ctrl_plane.c new file mode 100644 index 000000000000..fa2ab8c3e757 --- /dev/null +++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.c @@ -0,0 +1,44 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022, MediaTek Inc. + * Copyright (c) 2022-2023, Intel Corporation. + */ + +#include + +#include "mtk_ctrl_plane.h" + +/** + * mtk_ctrl_init() - Initialize the control plane block. + * @mdev: Pointer to the MTK modem device. + * + * Allocates and initializes the control plane block + * associated with @mdev. + * + * Return: 0 on success, -ENOMEM on allocation failure. + */ +int mtk_ctrl_init(struct mtk_md_dev *mdev) +{ + struct mtk_ctrl_blk *ctrl_blk; + + ctrl_blk =3D devm_kzalloc(mdev->dev, sizeof(*ctrl_blk), GFP_KERNEL); + if (!ctrl_blk) + return -ENOMEM; + + ctrl_blk->mdev =3D mdev; + mdev->ctrl_blk =3D ctrl_blk; + + return 0; +} + +/** + * mtk_ctrl_exit() - Clean up the control plane block. + * @mdev: Pointer to the MTK modem device. + * + * Clears the control plane block pointer. The allocation + * itself is managed by devres and freed on driver detach. + */ +void mtk_ctrl_exit(struct mtk_md_dev *mdev) +{ + mdev->ctrl_blk =3D NULL; +} diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h b/drivers/net/wwan/t9xx= /mtk_ctrl_plane.h new file mode 100644 index 000000000000..c141876ef95d --- /dev/null +++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0-only + * + * Copyright (c) 2022, MediaTek Inc. + */ + +#ifndef __MTK_CTRL_PLANE_H__ +#define __MTK_CTRL_PLANE_H__ + +#include +#include + +#include "mtk_dev.h" + +struct mtk_ctrl_blk { + struct mtk_md_dev *mdev; + struct mtk_ctrl_trans *trans; +}; + +int mtk_ctrl_init(struct mtk_md_dev *mdev); +void mtk_ctrl_exit(struct mtk_md_dev *mdev); + +#endif /* __MTK_CTRL_PLANE_H__ */ diff --git a/drivers/net/wwan/t9xx/mtk_dev.h b/drivers/net/wwan/t9xx/mtk_de= v.h index 329e5e6695df..84fcd980110b 100644 --- a/drivers/net/wwan/t9xx/mtk_dev.h +++ b/drivers/net/wwan/t9xx/mtk_dev.h @@ -31,12 +31,15 @@ enum mtk_dev_evt_d2h { DEV_EVT_D2H_ASYNC_HS_NOTIFY_MD =3D BIT(6), }; =20 +struct mtk_ctrl_blk; + /* mtk_md_dev defines the structure of MTK modem device */ struct mtk_md_dev { struct device *dev; void *hw_priv; u32 hw_ver; char dev_str[MTK_DEV_STR_LEN]; + struct mtk_ctrl_blk *ctrl_blk; }; =20 #endif /* __MTK_DEV_H__ */ diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.c b/drivers/net/wwan/t9xx/p= cie/mtk_pci.c index 34ee823119fc..c2c2b824ff31 100644 --- a/drivers/net/wwan/t9xx/pcie/mtk_pci.c +++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.c @@ -12,6 +12,7 @@ #include =20 #include "mtk_dev.h" +#include "mtk_trans_ctrl.h" #include "mtk_pci.h" #include "mtk_pci_reg.h" =20 diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h b/drivers/net/wwan= /t9xx/pcie/mtk_trans_ctrl.h new file mode 100644 index 000000000000..d6de4c43b529 --- /dev/null +++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h @@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0-only + * + * Copyright (c) 2022, MediaTek Inc. + */ + +#ifndef __MTK_TRANS_CTRL_H__ +#define __MTK_TRANS_CTRL_H__ + +#include +#include +#include +#include + +#include "mtk_dev.h" + +struct mtk_ctrl_trans { + struct mtk_ctrl_blk *ctrl_blk; + struct mtk_md_dev *mdev; +}; + +#endif --=20 2.34.1 From nobody Wed Sep 30 11:25:46 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A8314446823; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790754372; cv=none; b=Ilc95KWE/S4WIYXLtXTlE7IWmbAM1xd7AeA91wuQvdBZvWp3xXdojmSE/PQy44tuGGNBuXggdnooPi2akBQXCNC/egvl0N/OtU5EHZ8WXjsr0//EUZHp5clFtXlDJnVSd/rVhHz1gWl0ZG2PEYRGK5czv9nY4PyFm4CjFdJsJwI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790754372; c=relaxed/simple; bh=C6nhAA6vsZSkGYZQzrucxhxD/CJRdS6NB83PQa7zOwU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=jGWNZ6FqVQ9TvPB3escIrvLncvx44QTjUL64zwqIhrFaEGKda8tLm48G0z+tXsVnyK2yLWvc140ZbC2i8tjG0E9I3uELyyXKIZomhM4OL/DC4F9ZpgbV7bgnlBRh8NnRwbIc+/gvyshLYagxBtfVtwQLBgVDA34Z+b3nw5F/xzQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=sWxyS5YA; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="sWxyS5YA" Received: by smtp.kernel.org (Postfix) with ESMTPS id 6FFA8C4AF16; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1790754372; bh=C6nhAA6vsZSkGYZQzrucxhxD/CJRdS6NB83PQa7zOwU=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=sWxyS5YAbMac9yRVTdEp0FS9R2fowK01evg1mKljBQgccbfW4yWpv6c/IJz10/MUf 5lgWsm+GEZxsfM4CK11ROx9zKYO3ufFhptofI38xfXC0eCMIQ2Fa2d4s4RZ1Gw6S2h njZfr9Z3j2jT/mW0Age+1wiLrJwDq4QPHnktZ3Hyn//KdmRIylANpVsEhuAFT+doJu Te1BUTL+//hgHn7Gh/yAK7QfN8Iuu8yAB1CU5sywbDEM/f/2ZX7R1uj55s/rY00O7e fDZy2kqaUuVteKVlOXay7hRAakHXaZzUkcBtnq3PS5N19PO0mbNMapceGKLVk5TppD eyX4ZMZ8fIiJg== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B620CA5FC6; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) From: Jack Wu via B4 Relay Date: Wed, 30 Sep 2026 15:46:10 +0800 Subject: [PATCH v9 3/6] net: wwan: t9xx: Add control DMA interface Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260930-t9xx_driver_v1-v9-3-3248b9256956@compal.com> References: <20260930-t9xx_driver_v1-v9-0-3248b9256956@compal.com> In-Reply-To: <20260930-t9xx_driver_v1-v9-0-3248b9256956@compal.com> To: Loic Poulain , Sergey Ryazanov , Johannes Berg , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jack Wu , Wen-Zhi Huang , Shi-Wei Yeh , Minano Tseng , Matthias Brugger , AngeloGioacchino Del Regno , Simon Horman , Jonathan Corbet , Shuah Khan , Robert Yu , Jeff Chang Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, linux-doc@vger.kernel.org X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1790754369; l=98710; i=jackbb_wu@compal.com; s=20260526; h=from:subject:message-id; bh=qi9HuyX+aA8l7o4VUrkOBPiaVve8i3Cbf3KCe2HnIFw=; b=2DgDKKLEBwDxfebDqT4ECkNEry7x4nWaPkSe0jQcpGlhKhj8uZUZ/ptT6y00n+jIlSKjuFTQ+ YzLc/Myk5HyBOrsyuYAo1WPUF0bpjgc7T4lB4wlvp1RIlkkDsza+7y7 X-Developer-Key: i=jackbb_wu@compal.com; a=ed25519; pk=VH1prTWixNl8OEUPPSfII3p46MzJpQN8J3+ecE1tZXg= X-Endpoint-Received: by B4 Relay for jackbb_wu@compal.com/20260526 with auth_id=793 X-Original-From: Jack Wu Reply-To: jackbb_wu@compal.com From: Jack Wu Cross Layer Direct Memory Access(CLDMA) is the hardware interface used by the control plane and designated to translate data between the host and the device. It supports 8 hardware queues for the device AP and modem respectively. CLDMA driver uses General Purpose Descriptor (GPD) to describe transaction information that can be recognized by CLDMA hardware. Once CLDMA hardware transaction is started, it would fetch and parse GPD to transfer data correctly. To facilitate the CLDMA transaction, a GPD ring for each queue is used. Once the transaction is started, CLDMA hardware will traverse the GPD ring to transfer data between the host and the device until no GPD is available. CLDMA TX flow: Once a TX service receives the TX data from the port layer, it uses APIs exported by the CLDMA driver to configure GPD with the DMA address of TX data. After that, the service triggers CLDMA to fetch the first available GPD to transfer data. CLDMA RX flow: When there is RX data from the MD, CLDMA hardware asserts an interrupt to notify the host to fetch data and dispatch it to FSM (for handshake messages) or the port layer. After CLDMA opening is finished, All RX GPDs are fulfilled and ready to receive data from the device. With this commit, probe registers the transport plane: the queue info table for the AP and modem control channels is set up and the transaction context is attached to the control plane. Nothing is brought up yet - the FSM listener that calls mtk_pcie_hif_init(), and with it the CLDMA instances and the TRB service thread, arrives with the later "Add FSM thread" patch; the userspace-visible port interfaces arrive with "Add control port". Signed-off-by: Jack Wu --- drivers/net/wwan/t9xx/Makefile | 5 +- drivers/net/wwan/t9xx/mtk_ctrl_plane.h | 50 +- drivers/net/wwan/t9xx/pcie/mtk_cldma.c | 1520 +++++++++++++++++++++++= ++++ drivers/net/wwan/t9xx/pcie/mtk_cldma.h | 165 +++ drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.c | 561 ++++++++++ drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h | 135 +++ drivers/net/wwan/t9xx/pcie/mtk_pci.c | 41 + drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h | 1 + drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c | 615 +++++++++++ drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h | 68 ++ 10 files changed, 3159 insertions(+), 2 deletions(-) diff --git a/drivers/net/wwan/t9xx/Makefile b/drivers/net/wwan/t9xx/Makefile index 0a07507f1ddf..74bef150e3a1 100644 --- a/drivers/net/wwan/t9xx/Makefile +++ b/drivers/net/wwan/t9xx/Makefile @@ -7,4 +7,7 @@ obj-$(CONFIG_MTK_T9XX) +=3D mtk_t9xx.o =20 mtk_t9xx-y :=3D \ mtk_ctrl_plane.o \ - pcie/mtk_pci.o + pcie/mtk_pci.o \ + pcie/mtk_trans_ctrl.o \ + pcie/mtk_cldma.o \ + pcie/mtk_cldma_drv.o diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h b/drivers/net/wwan/t9xx= /mtk_ctrl_plane.h index c141876ef95d..0dcc0ec2c45d 100644 --- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h +++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h @@ -11,9 +11,57 @@ =20 #include "mtk_dev.h" =20 +#define Q_MTU_3_5K (0xE00) +#define Q_FRAG_3_5K (0xE00) + +enum mtk_ccci_ch { + /* to sAP */ + CCCI_SAP_CONTROL_RX =3D 0x1000, + CCCI_SAP_CONTROL_TX =3D 0x1001, + /* to MD */ + CCCI_CONTROL_RX =3D 0x2000, + CCCI_CONTROL_TX =3D 0x2001, +}; + +enum mtk_trb_cmd_type { + TRB_CMD_MIN, + TRB_CMD_ENABLE, + TRB_CMD_TX, + TRB_CMD_DISABLE, + TRB_CMD_STOP, + TRB_CMD_RECOVER, + TRB_CMD_MAX, +}; + +enum mtk_hif_dev_ctrl_cmd { + HIF_CTRL_CMD_CHECK_TX_FULL, +}; + +struct trb_open_priv { + u8 log_rg_offset; + u32 tx_mtu; + u32 rx_mtu; + u32 tx_frag_size; + u32 rx_frag_size; + int (*rx_done)(struct sk_buff *skb, void *priv, bool force_recv); +}; + +struct trb { + u32 channel_id; + enum mtk_trb_cmd_type cmd; + int status; + struct kref kref; + void *priv; + int (*trb_complete)(struct sk_buff *skb); +}; + +union ctrl_hif_cmd_data { + u32 rx_ch; +}; + struct mtk_ctrl_blk { struct mtk_md_dev *mdev; - struct mtk_ctrl_trans *trans; + void *ctrl_hw_priv; }; =20 int mtk_ctrl_init(struct mtk_md_dev *mdev); diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma.c b/drivers/net/wwan/t9xx= /pcie/mtk_cldma.c new file mode 100644 index 000000000000..0f281bfb38cc --- /dev/null +++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma.c @@ -0,0 +1,1520 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022, MediaTek Inc. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "mtk_pci.h" +#include "mtk_cldma.h" +#include "mtk_cldma_drv.h" +#include "mtk_dev.h" + +#define DMA_POOL_NAME_LEN (64) +#define WAIT_HWO_ROUND (10) +#define WAIT_HWO_TIME (5) +#define NO_BUDGET (0) + +static const int mtk_cldma_hw_id_tbl[NR_CLDMA] =3D { + [CLDMA0] =3D CLDMA0_HW_ID, + [CLDMA1] =3D CLDMA1_HW_ID, +}; + +static void mtk_cldma_clr_bd_dsc(struct cldma_drv_info *drv_info, + struct bd_dsc *bd_dsc_pool, int nr_bds) +{ + struct bd_dsc *bd_dsc; + int i; + + for (i =3D 0; i < nr_bds; i++) { + bd_dsc =3D bd_dsc_pool + i; + dma_unmap_single(drv_info->mdev->dev, bd_dsc->data_dma_addr, + bd_dsc->data_len, DMA_TO_DEVICE); + bd_dsc->data_dma_addr =3D 0; + bd_dsc->data_len =3D 0; + if (bd_dsc->bd->tx_bd.bd_flags & CLDMA_BD_FLAG_EOL) { + bd_dsc->bd->tx_bd.bd_flags &=3D ~CLDMA_BD_FLAG_EOL; + break; + } + } +} + +static void mtk_cldma_tx_done_work(struct work_struct *work) +{ + struct txq *txq =3D container_of(work, struct txq, tx_done_work); + struct cldma_drv_info *drv_info; + struct mtk_ctrl_trans *trans; + struct mtk_md_dev *mdev; + struct sk_buff *skb; + struct trb_srv *srv; + struct tx_req *req; + unsigned int state; + bool was_starved; + struct trb *trb; + int i, hif_id; + u32 txqno; + + drv_info =3D txq->drv_info; + hif_id =3D drv_info->hif_id; + txqno =3D txq->txqno; + mdev =3D drv_info->mdev; + trans =3D drv_info->cd->trans; + +again: + for (i =3D 0; i < txq->nr_gpds; i++) { + spin_lock(&txq->ring_lock); + + req =3D txq->req_pool + txq->free_idx; + + /* Ownership is decided by the slot alone: an empty slot has no + * skb, and one the hardware still owns has HWO set. The + * producer publishes both under ring_lock, so a slot can never + * be observed half-filled and there is no need to bound the + * walk by wr_idx. + */ + if (!req->skb || (req->gpd->tx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO)) { + spin_unlock(&txq->ring_lock); + break; + } + + dma_rmb(); /* read descriptor fields after HWO check */ + + if (txq->nr_bds) + mtk_cldma_clr_bd_dsc(drv_info, req->bd_dsc_pool, txq->nr_bds); + else + dma_unmap_single(mdev->dev, req->data_dma_addr, + req->data_len, DMA_TO_DEVICE); + + skb =3D req->skb; + req->data_dma_addr =3D 0; + req->data_len =3D 0; + req->skb =3D NULL; + + txq->free_idx =3D (txq->free_idx + 1) % txq->nr_gpds; + was_starved =3D atomic_fetch_inc(&txq->req_budget) =3D=3D NO_BUDGET; + + spin_unlock(&txq->ring_lock); + + trb =3D (struct trb *)skb->cb; + trb->status =3D 0; + trb->trb_complete(skb); + + /* The service may already have been torn down; completing the + * TRB is still correct, there is just nobody left to wake. + */ + srv =3D trans->trb_srv[trans->srv_cfg[hif_id][txqno]]; + if (was_starved && srv) + wake_up(&srv->trb_waitq); + } + + state =3D mtk_cldma_check_intr_status(drv_info, DIR_TX, txqno, QUEUE_XFER= _DONE); + if (state) { + if (unlikely(state =3D=3D LINK_ERROR_VAL)) + goto out; + + mtk_cldma_clr_intr_status(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE); + + cond_resched(); + + goto again; + } + +out: + mtk_cldma_unmask_intr(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE); +} + +/* Clamp every length the device reports before it reaches skb_put(). + * + * In BD mode the bound is the per-BD data_allow_len, not frag_size or mtu: + * on the last BD of a packet data_allow_len is smaller than frag_size. T= he + * running total is then clamped against req->mtu, so a device that inflat= es + * data_allow_len cannot grow the packet past the buffer. + * + * In non-BD mode the bound is req->mtu directly. data_allow_len lives in= a + * buffer the device can write, whereas req->mtu is host-only state and is= by + * construction the value data_allow_len was programmed with, so a well-fo= rmed + * descriptor gives an identical result. + * + * Returns -EPROTO when the device over-reports so the caller drops the pa= cket + * instead of delivering a malformed skb. + */ +static int mtk_cldma_rx_skb_adjust(struct mtk_md_dev *mdev, struct rxq *rx= q, + struct rx_req *req) +{ + u32 recv_len, allow_len, total_len =3D 0; + struct bd_dsc *bd_dsc; + int ret =3D 0; + int i; + + for (i =3D 0; i < rxq->nr_bds; i++) { + bd_dsc =3D req->bd_dsc_pool + i; + if (bd_dsc->data_dma_addr) { + dma_unmap_single(mdev->dev, bd_dsc->data_dma_addr, + req->frag_size, DMA_FROM_DEVICE); + bd_dsc->data_dma_addr =3D 0; + } + recv_len =3D le16_to_cpu(bd_dsc->bd->rx_bd.data_recv_len); + allow_len =3D le16_to_cpu(bd_dsc->bd->rx_bd.data_allow_len); + if (recv_len > allow_len) { + ret =3D -EPROTO; + recv_len =3D allow_len; + } + total_len +=3D recv_len; + if (total_len > req->mtu) { + ret =3D -EPROTO; + recv_len -=3D min(total_len - req->mtu, recv_len); + total_len =3D req->mtu; + } + bd_dsc->skb->len =3D 0; + skb_reset_tail_pointer(bd_dsc->skb); + skb_put(bd_dsc->skb, recv_len); + if (req->skb !=3D bd_dsc->skb) { + req->skb->len +=3D bd_dsc->skb->len; + req->skb->data_len +=3D bd_dsc->skb->len; + } + bd_dsc->bd->rx_bd.data_recv_len =3D 0; + bd_dsc->skb =3D NULL; + } + if (!rxq->nr_bds) { + if (req->data_dma_addr) { + dma_unmap_single(mdev->dev, req->data_dma_addr, + req->mtu, DMA_FROM_DEVICE); + req->data_dma_addr =3D 0; + } + recv_len =3D le16_to_cpu(req->gpd->rx_gpd.data_recv_len); + if (recv_len > req->mtu) { + ret =3D -EPROTO; + recv_len =3D req->mtu; + } + req->skb->len =3D 0; + skb_reset_tail_pointer(req->skb); + skb_put(req->skb, recv_len); + } + + req->gpd->rx_gpd.data_recv_len =3D 0; + + return ret; +} + +static int mtk_cldma_reload_rx_skb(struct mtk_md_dev *mdev, struct rxq *rx= q, + struct rx_req *req) +{ + struct sk_buff *tail =3D NULL; + struct bd_dsc *bd_dsc; + int nr_bds; + int i, ret; + + nr_bds =3D rxq->nr_bds; + + for (i =3D 0; i < nr_bds; i++) { + bd_dsc =3D req->bd_dsc_pool + i; + bd_dsc->skb =3D __dev_alloc_skb(req->frag_size, GFP_KERNEL); + if (!bd_dsc->skb) { + dev_warn_ratelimited(mdev->dev, "Failed to alloc SKB\n"); + ret =3D -ENOMEM; + goto err_free_skb; + } + bd_dsc->skb->next =3D NULL; + bd_dsc->data_dma_addr =3D dma_map_single(mdev->dev, bd_dsc->skb->data, + req->frag_size, DMA_FROM_DEVICE); + ret =3D dma_mapping_error(mdev->dev, bd_dsc->data_dma_addr); + if (unlikely(ret)) { + dev_warn_ratelimited(mdev->dev, "Failed to map SKB data\n"); + ret =3D -EFAULT; + goto err_free_skb; + } + bd_dsc->bd->rx_bd.data_buff_ptr_h =3D + cpu_to_le32((u64)(bd_dsc->data_dma_addr) >> 32); + bd_dsc->bd->rx_bd.data_buff_ptr_l =3D + cpu_to_le32(bd_dsc->data_dma_addr); + if (tail) { + tail->next =3D bd_dsc->skb; + tail =3D bd_dsc->skb; + continue; + } + if (!req->skb) { + req->skb =3D bd_dsc->skb; + } else { + skb_shinfo(req->skb)->frag_list =3D bd_dsc->skb; + tail =3D bd_dsc->skb; + } + } + if (!nr_bds) { + req->skb =3D __dev_alloc_skb(req->mtu, GFP_KERNEL); + if (!req->skb) { + ret =3D -ENOMEM; + goto err_free_skb; + } + + req->data_dma_addr =3D dma_map_single(mdev->dev, req->skb->data, + req->mtu, DMA_FROM_DEVICE); + ret =3D dma_mapping_error(mdev->dev, req->data_dma_addr); + if (unlikely(ret)) { + dev_warn_ratelimited(mdev->dev, "Failed to map SKB data\n"); + ret =3D -EFAULT; + goto err_free_skb; + } + req->gpd->rx_gpd.data_buff_ptr_h =3D cpu_to_le32((u64)req->data_dma_addr= >> 32); + req->gpd->rx_gpd.data_buff_ptr_l =3D cpu_to_le32(req->data_dma_addr); + } + return 0; + +err_free_skb: + if (nr_bds) { + if (req->skb) + skb_shinfo(req->skb)->frag_list =3D NULL; + for (i =3D 0; i < nr_bds; i++) { + bd_dsc =3D req->bd_dsc_pool + i; + if (!bd_dsc->skb) + break; + if (!dma_mapping_error(mdev->dev, bd_dsc->data_dma_addr)) + dma_unmap_single(mdev->dev, bd_dsc->data_dma_addr, + req->frag_size, DMA_FROM_DEVICE); + bd_dsc->data_dma_addr =3D 0; + bd_dsc->skb->next =3D NULL; + dev_kfree_skb_any(bd_dsc->skb); + bd_dsc->skb =3D NULL; + } + } else { + req->data_dma_addr =3D 0; + if (req->skb) + dev_kfree_skb_any(req->skb); + } + req->skb =3D NULL; + + return ret; +} + +static int mtk_cldma_check_rx_req(struct cldma_drv_info *drv_info, struct = rxq *rxq) +{ + struct rx_req *req =3D rxq->req_pool + rxq->free_idx; + u64 curr_addr; + int i; + + curr_addr =3D mtk_cldma_get_rx_curr_addr(drv_info, rxq->rxqno); + if (unlikely(!curr_addr)) + return -ENXIO; + + if (req->gpd_dma_addr =3D=3D curr_addr) + return -EAGAIN; + for (i =3D 0; i < WAIT_HWO_ROUND; i++) { + udelay(WAIT_HWO_TIME); + if (!(READ_ONCE(req->gpd->rx_gpd.gpd_flags) & CLDMA_GPD_FLAG_HWO)) + break; + } + if (i =3D=3D WAIT_HWO_ROUND) { + dev_err((drv_info->mdev)->dev, "Failed to check HWO=3D0\n"); + return -EAGAIN; + } + + return 0; +} + +static bool mtk_cldma_rx_check_again(struct rxq *rxq) +{ + struct cldma_drv_info *drv_info; + bool need_check_again =3D false; + u32 state; + int rxqno; + + drv_info =3D rxq->drv_info; + rxqno =3D rxq->rxqno; + + do { + state =3D mtk_cldma_check_intr_status(drv_info, DIR_RX, + rxqno, QUEUE_XFER_DONE); + if (state) { + if (unlikely(state =3D=3D LINK_ERROR_VAL)) + break; + + mtk_cldma_clr_intr_status(drv_info, DIR_RX, + rxqno, QUEUE_XFER_DONE); + cond_resched(); + return true; + } + } while (need_check_again); + + return false; +} + +/* Point the device at the GPD it should fill next and start the queue. + * free_idx is owned by mtk_cldma_rx_done_work(), so this may only run + * from that worker, at a point where every consumed GPD is re-armed. + */ +static void mtk_cldma_rxq_restart(struct cldma_drv_info *drv_info, struct = rxq *rxq) +{ + mtk_cldma_setup_start_addr(drv_info, DIR_RX, rxq->rxqno, + rxq->req_pool[rxq->free_idx].gpd_dma_addr); + mtk_cldma_start_queue(drv_info, DIR_RX, rxq->rxqno); +} + +static void mtk_cldma_rx_done_work(struct work_struct *work) +{ + struct rx_req *req =3D NULL, *pre_req =3D NULL; + struct rxq *rxq =3D container_of(work, struct rxq, rx_done_work); + struct cldma_drv_info *drv_info; + struct mtk_md_dev *mdev; + struct sk_buff *rx_skb; + int i, ret, idx, len_err; + + drv_info =3D rxq->drv_info; + mdev =3D drv_info->mdev; + +again: + for (i =3D 0; i < rxq->nr_gpds; i++) { + req =3D rxq->req_pool + rxq->free_idx; + if (!req->skb) { + dev_err(mdev->dev, + "Failed to get valid req cldma%d rxq%d req%d\n", + drv_info->hw_id, rxq->rxqno, rxq->free_idx); + goto out; + } + + if (req->gpd->rx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO) + break; + + dma_rmb(); /* read descriptor fields after HWO check */ + + len_err =3D mtk_cldma_rx_skb_adjust(mdev, rxq, req); + rx_skb =3D req->skb; + req->skb =3D NULL; + + ret =3D mtk_cldma_reload_rx_skb(mdev, rxq, req); + if (ret) { + /* Alloc failed =E2=80=94 recycle old buffer, drop packet. + * BD mode cannot recycle directly (BDP flag mismatch), + * so accept the stall and let reset recovery handle it. + */ + if (rxq->nr_bds) { + dev_kfree_skb_any(rx_skb); + goto out; + } + + skb_trim(rx_skb, 0); + req->skb =3D rx_skb; + req->data_dma_addr =3D dma_map_single(mdev->dev, + rx_skb->data, + req->mtu, + DMA_FROM_DEVICE); + if (dma_mapping_error(mdev->dev, req->data_dma_addr)) { + req->data_dma_addr =3D 0; + /* Keep rx_skb in req->skb for clean stall. + * HWO is not set =E2=80=94 HW won't touch this slot. + * Queue stalls until modem reset recovery. + */ + goto out; + } else { + req->gpd->rx_gpd.data_buff_ptr_h =3D + cpu_to_le32((u64)req->data_dma_addr >> 32); + req->gpd->rx_gpd.data_buff_ptr_l =3D + cpu_to_le32(req->data_dma_addr); + } + } else if (len_err) { + dev_err_ratelimited(mdev->dev, + "Drop oversized packet on cldma%d rxq%d\n", + drv_info->hw_id, rxq->rxqno); + dev_kfree_skb_any(rx_skb); + } else { + do { + ret =3D rxq->rx_done(rx_skb, rxq->arg, + atomic_read(&rxq->need_exit) ? true : false); + if (ret =3D=3D -EAGAIN) + usleep_range(1000, 2000); + } while (ret =3D=3D -EAGAIN); + } + + wmb(); /* ensure addr set done before HWO setup done */ + + idx =3D rxq->free_idx =3D=3D 0 ? rxq->nr_gpds - 1 : rxq->free_idx - 1; + pre_req =3D rxq->req_pool + idx; + pre_req->gpd->rx_gpd.gpd_flags |=3D CLDMA_GPD_FLAG_HWO; + rxq->free_idx =3D (rxq->free_idx + 1) % rxq->nr_gpds; + } + + ret =3D mtk_cldma_check_rx_req(drv_info, rxq); + if (!ret) + goto again; + + /* -ENXIO means the queue has no start address programmed, which is + * exactly what a restart installs, so consume need_restart first. + * Only the resume below is skipped in that case. + */ + if (!atomic_read(&rxq->need_exit)) { + if (atomic_xchg(&rxq->need_restart, 0)) + mtk_cldma_rxq_restart(drv_info, rxq); + else if (ret !=3D -ENXIO) + mtk_cldma_resume_queue(drv_info, DIR_RX, rxq->rxqno); + } + + if (ret =3D=3D -ENXIO) + goto out; + + if (mtk_cldma_rx_check_again(rxq)) + goto again; + +out: + mtk_cldma_unmask_intr(drv_info, DIR_RX, rxq->rxqno, QUEUE_XFER_DONE); + mtk_cldma_clear_ip_busy(drv_info); +} + +static int mtk_cldma_alloc_tx_bd(struct cldma_drv_info *drv_info, struct t= xq *txq, + struct tx_req *req) +{ + struct bd_dsc *bd_dsc, *last_bd_dsc =3D NULL; + int i; + + req->bd_dsc_pool =3D kcalloc(txq->nr_bds, sizeof(*bd_dsc), + GFP_KERNEL); + if (!req->bd_dsc_pool) + return -ENOMEM; + + for (i =3D 0; i < txq->nr_bds; i++) { + bd_dsc =3D req->bd_dsc_pool + i; + bd_dsc->bd =3D dma_pool_zalloc(drv_info->bd_dma_pool, GFP_KERNEL, + &bd_dsc->bd_dma_addr); + if (!bd_dsc->bd) + return -ENOMEM; + if (!last_bd_dsc) { + req->gpd->tx_gpd.data_buff_ptr_h =3D + cpu_to_le32((u64)(bd_dsc->bd_dma_addr) >> 32); + req->gpd->tx_gpd.data_buff_ptr_l =3D + cpu_to_le32(bd_dsc->bd_dma_addr); + } else { + last_bd_dsc->bd->tx_bd.next_bd_ptr_h =3D + cpu_to_le32((u64)(bd_dsc->bd_dma_addr) >> 32); + last_bd_dsc->bd->tx_bd.next_bd_ptr_l =3D + cpu_to_le32(bd_dsc->bd_dma_addr); + } + last_bd_dsc =3D bd_dsc; + } + return 0; +} + +static struct txq *mtk_cldma_txq_alloc(struct cldma_drv_info *drv_info, st= ruct queue_info *que) +{ + struct mtk_md_dev *mdev; + struct bd_dsc *bd_dsc; + struct tx_req *next; + struct tx_req *req; + u16 tx_frag_size; + struct txq *txq; + int i, j, ret; + + mdev =3D drv_info->mdev; + + txq =3D kzalloc_obj(*txq); + if (!txq) + return NULL; + + txq->que =3D que; + txq->drv_info =3D drv_info; + txq->txqno =3D txq->que->txqno; + txq->nr_gpds =3D txq->que->tx_nr_gpds; + atomic_set(&txq->req_budget, txq->que->tx_nr_gpds); + spin_lock_init(&txq->ring_lock); + tx_frag_size =3D txq->que->tx_frag_size; + if (txq->que->tx_mtu > tx_frag_size && tx_frag_size) + txq->nr_bds =3D (txq->que->tx_mtu + tx_frag_size - 1) / tx_frag_size; + + txq->req_pool =3D kcalloc(txq->nr_gpds, sizeof(*req), GFP_KERNEL); + if (!txq->req_pool) + goto err_free_txq; + + for (i =3D 0; i < txq->nr_gpds; i++) { + req =3D txq->req_pool + i; + req->mtu =3D txq->que->tx_mtu; + req->frag_size =3D tx_frag_size; + req->gpd =3D dma_pool_zalloc(drv_info->gpd_dma_pool, GFP_KERNEL, &req->g= pd_dma_addr); + if (!req->gpd) + goto err_free_req; + if (txq->nr_bds) { + ret =3D mtk_cldma_alloc_tx_bd(drv_info, txq, req); + if (ret) + goto err_free_req; + req->gpd->tx_gpd.gpd_flags |=3D CLDMA_GPD_FLAG_BDP; + } + } + + for (i =3D 0; i < txq->nr_gpds; i++) { + req =3D txq->req_pool + i; + next =3D txq->req_pool + ((i + 1) % txq->nr_gpds); + req->gpd->tx_gpd.gpd_flags |=3D CLDMA_GPD_FLAG_IOC; + req->gpd->tx_gpd.next_gpd_ptr_h =3D cpu_to_le32((u64)(next->gpd_dma_addr= ) >> 32); + req->gpd->tx_gpd.next_gpd_ptr_l =3D cpu_to_le32(next->gpd_dma_addr); + } + + INIT_WORK(&txq->tx_done_work, mtk_cldma_tx_done_work); + + /* Publish the queue before unmasking its interrupts: an interrupt + * that fires in between must find a valid txq, or the sources it + * masks are never unmasked again. The release pairs with the + * acquire load in the ISR. + */ + smp_store_release(&drv_info->txq[txq->txqno], txq); + ret =3D mtk_cldma_stop_queue(drv_info, DIR_TX, txq->txqno); + if (ret) { + dev_warn(mdev->dev, "Failed to stop TX queue %d before alloc\n", txq->tx= qno); + WRITE_ONCE(drv_info->txq[txq->txqno], NULL); + goto err_free_req; + } + txq->tx_started =3D false; + mtk_cldma_setup_start_addr(drv_info, DIR_TX, txq->txqno, + txq->req_pool[0].gpd_dma_addr); + mtk_cldma_unmask_intr(drv_info, DIR_TX, txq->txqno, QUEUE_ERROR); + mtk_cldma_unmask_intr(drv_info, DIR_TX, txq->txqno, QUEUE_XFER_DONE); + + return txq; + +err_free_req: + for (i =3D 0; i < txq->nr_gpds; i++) { + req =3D txq->req_pool + i; + if (!req->gpd) + break; + if (req->bd_dsc_pool) { + for (j =3D 0; j < txq->nr_bds; j++) { + bd_dsc =3D req->bd_dsc_pool + j; + if (!bd_dsc->bd) + break; + dma_pool_free(drv_info->bd_dma_pool, bd_dsc->bd, + bd_dsc->bd_dma_addr); + } + kfree(req->bd_dsc_pool); + } + dma_pool_free(drv_info->gpd_dma_pool, req->gpd, req->gpd_dma_addr); + } + kfree(txq->req_pool); +err_free_txq: + kfree(txq); + return NULL; +} + +/* mtk_cldma_drv_init() rewrites IP-global configuration (including the in= terrupt + * mask) and mtk_cldma_drv_reset() wipes every queue of the instance, so a= ll + * queues still published in drv_info have to be re-armed, not only the one + * that triggered the reset. A queue already unpublished (NULL slot) is b= eing + * torn down and is deliberately left stopped. + */ +static void mtk_cldma_rearm_queues(struct cldma_drv_info *drv_info) +{ + struct txq *txq; + struct rxq *rxq; + int i; + + mtk_cldma_drv_init(drv_info); + + for (i =3D 0; i < HW_QUEUE_NUM; i++) { + txq =3D drv_info->txq[i]; + if (txq) { + spin_lock(&txq->ring_lock); + mtk_cldma_setup_start_addr(drv_info, DIR_TX, i, + txq->req_pool[txq->free_idx].gpd_dma_addr); + mtk_cldma_unmask_intr(drv_info, DIR_TX, i, QUEUE_ERROR); + mtk_cldma_unmask_intr(drv_info, DIR_TX, i, QUEUE_XFER_DONE); + if (READ_ONCE(txq->tx_started)) + mtk_cldma_start_queue(drv_info, DIR_TX, i); + spin_unlock(&txq->ring_lock); + } + + rxq =3D drv_info->rxq[i]; + if (rxq) { + mtk_cldma_unmask_intr(drv_info, DIR_RX, i, QUEUE_ERROR); + mtk_cldma_unmask_intr(drv_info, DIR_RX, i, QUEUE_XFER_DONE); + /* free_idx belongs to rx_done_work, which may be halfway + * through a packet: let it program the start address + * once it reaches a consistent point. + */ + atomic_set(&rxq->need_restart, 1); + queue_work(drv_info->wq, &rxq->rx_done_work); + } + } +} + +static void mtk_cldma_txq_free(struct cldma_drv_info *drv_info, u32 txqno) +{ + struct mtk_md_dev *mdev; + struct bd_dsc *bd_dsc; + struct tx_req *req; + struct txq *txq; + struct trb *trb; + int irq_id; + int i, j, ret; + + mdev =3D drv_info->mdev; + + txq =3D drv_info->txq[txqno]; + /* Unpublish before teardown; pairs with the acquire loads of this + * slot. No release is needed, there is no prior store to expose. + */ + WRITE_ONCE(drv_info->txq[txqno], NULL); + /* stop HW tx transaction; -ENODEV means the link is dead, so the + * device cannot be walking the ring and the free may proceed. + */ + ret =3D mtk_cldma_stop_queue(drv_info, DIR_TX, txqno); + if (ret =3D=3D -ETIMEDOUT) { + dev_err(mdev->dev, "TX queue %d stop timed out, resetting CLDMA%d\n", + txqno, drv_info->hw_id); + mtk_cldma_drv_reset(drv_info); + ret =3D mtk_cldma_stop_queue(drv_info, DIR_TX, txqno); + /* the reset took the whole instance down with this queue */ + mtk_cldma_rearm_queues(drv_info); + } + txq->tx_started =3D false; + + irq_id =3D mtk_pci_get_virq_id(mdev, drv_info->pci_ext_irq_id); + synchronize_irq(irq_id); + /* flush on-going work */ + flush_work(&txq->tx_done_work); + mtk_cldma_mask_intr(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE); + mtk_cldma_mask_intr(drv_info, DIR_TX, txqno, QUEUE_ERROR); + + if (ret =3D=3D -ETIMEDOUT) { + /* Still a live DMA target: leak the ring rather than hand it + * back to the allocator. + */ + dev_err(mdev->dev, "TX queue %d cannot be stopped, leaking its ring\n", + txqno); + return; + } + + /* Free tx req resource. No ring_lock is taken here: txq was already + * unpublished from drv_info->txq[] above, so no new producer can enter, + * and synchronize_irq() plus flush_work() have retired the consumer. + */ + for (i =3D 0; i < txq->nr_gpds; i++) { + req =3D txq->req_pool + txq->free_idx; + if (req->skb && req->data_len) { + if (!txq->nr_bds) + dma_unmap_single(mdev->dev, req->data_dma_addr, + req->data_len, DMA_TO_DEVICE); + for (j =3D 0; j < txq->nr_bds; j++) { + bd_dsc =3D req->bd_dsc_pool + j; + if (!bd_dsc->data_dma_addr) + continue; + dma_unmap_single(mdev->dev, bd_dsc->data_dma_addr, + bd_dsc->data_len, DMA_TO_DEVICE); + } + trb =3D (struct trb *)req->skb->cb; + trb->status =3D -EPIPE; + trb->trb_complete(req->skb); + } + for (j =3D 0; j < txq->nr_bds; j++) { + bd_dsc =3D req->bd_dsc_pool + j; + dma_pool_free(drv_info->bd_dma_pool, bd_dsc->bd, + bd_dsc->bd_dma_addr); + } + kfree(req->bd_dsc_pool); + dma_pool_free(drv_info->gpd_dma_pool, req->gpd, req->gpd_dma_addr); + txq->free_idx =3D (txq->free_idx + 1) % txq->nr_gpds; + } + + kfree(txq->req_pool); + kfree(txq); +} + +static int mtk_cldma_alloc_rx_bd(struct cldma_drv_info *drv_info, struct r= x_req *req, + int nr_bds) +{ + struct bd_dsc *bd_dsc, *last_bd_dsc =3D NULL; + struct sk_buff *tail =3D NULL; + struct mtk_md_dev *mdev; + u32 left_size; + int ret; + int i; + + mdev =3D drv_info->mdev; + left_size =3D req->mtu; + + req->bd_dsc_pool =3D kcalloc(nr_bds, sizeof(*bd_dsc), + GFP_KERNEL); + if (!req->bd_dsc_pool) + return -ENOMEM; + for (i =3D 0; i < nr_bds; i++) { + bd_dsc =3D req->bd_dsc_pool + i; + bd_dsc->bd =3D dma_pool_zalloc(drv_info->bd_dma_pool, GFP_KERNEL, + &bd_dsc->bd_dma_addr); + if (!bd_dsc->bd) + return -ENOMEM; + + bd_dsc->skb =3D __dev_alloc_skb(req->frag_size, GFP_KERNEL); + if (!bd_dsc->skb) + return -ENOMEM; + bd_dsc->skb->next =3D NULL; + bd_dsc->data_dma_addr =3D + dma_map_single(mdev->dev, bd_dsc->skb->data, + req->frag_size, DMA_FROM_DEVICE); + ret =3D dma_mapping_error(mdev->dev, bd_dsc->data_dma_addr); + if (unlikely(ret)) + return -ENOMEM; + + bd_dsc->bd->rx_bd.data_buff_ptr_h =3D + cpu_to_le32((u64)(bd_dsc->data_dma_addr) >> 32); + bd_dsc->bd->rx_bd.data_buff_ptr_l =3D + cpu_to_le32(bd_dsc->data_dma_addr); + bd_dsc->bd->rx_bd.data_allow_len =3D + cpu_to_le16(min(req->frag_size, left_size)); + left_size -=3D min(req->frag_size, left_size); + if (!last_bd_dsc) { + req->gpd->rx_gpd.data_buff_ptr_h =3D + cpu_to_le32((u64)(bd_dsc->bd_dma_addr) >> 32); + req->gpd->rx_gpd.data_buff_ptr_l =3D + cpu_to_le32(bd_dsc->bd_dma_addr); + } else { + last_bd_dsc->bd->rx_bd.next_bd_ptr_h =3D + cpu_to_le32((u64)(bd_dsc->bd_dma_addr) >> 32); + last_bd_dsc->bd->rx_bd.next_bd_ptr_l =3D + cpu_to_le32(bd_dsc->bd_dma_addr); + } + last_bd_dsc =3D bd_dsc; + if (tail) { + tail->next =3D bd_dsc->skb; + tail =3D bd_dsc->skb; + continue; + } + if (!req->skb) { + req->skb =3D bd_dsc->skb; + } else { + skb_shinfo(req->skb)->frag_list =3D bd_dsc->skb; + tail =3D bd_dsc->skb; + } + } + last_bd_dsc->bd->rx_bd.bd_flags |=3D CLDMA_BD_FLAG_EOL; + return 0; +} + +static void mtk_cldma_rxq_alloc_cancel(struct cldma_drv_info *drv_info, st= ruct rx_req *req, + int nr_bds) +{ + struct mtk_md_dev *mdev; + struct bd_dsc *bd_dsc; + int i; + + mdev =3D drv_info->mdev; + + if (nr_bds) { + if (req->skb) + skb_shinfo(req->skb)->frag_list =3D NULL; + if (req->bd_dsc_pool) { + for (i =3D 0; i < nr_bds; i++) { + bd_dsc =3D req->bd_dsc_pool + i; + if (!bd_dsc->bd) + break; + if (bd_dsc->skb) { + if (!dma_mapping_error(mdev->dev, bd_dsc->data_dma_addr)) + dma_unmap_single(mdev->dev, bd_dsc->data_dma_addr, + req->frag_size, DMA_FROM_DEVICE); + bd_dsc->data_dma_addr =3D 0; + bd_dsc->skb->next =3D NULL; + dev_kfree_skb_any(bd_dsc->skb); + } + dma_pool_free(drv_info->bd_dma_pool, bd_dsc->bd, + bd_dsc->bd_dma_addr); + } + kfree(req->bd_dsc_pool); + } + } else { + if (req->skb) { + if (!dma_mapping_error(mdev->dev, req->data_dma_addr)) + dma_unmap_single(mdev->dev, req->data_dma_addr, + req->mtu, DMA_FROM_DEVICE); + req->data_dma_addr =3D 0; + dev_kfree_skb_any(req->skb); + } + } + dma_pool_free(drv_info->gpd_dma_pool, req->gpd, req->gpd_dma_addr); +} + +static struct rxq *mtk_cldma_rxq_alloc(struct cldma_drv_info *drv_info, st= ruct queue_info *que, + struct sk_buff *skb) +{ + struct trb_open_priv *trb_open_priv =3D (struct trb_open_priv *)skb->data; + struct trb *trb =3D (struct trb *)skb->cb; + struct mtk_md_dev *mdev; + struct rx_req *next; + struct rx_req *req; + u16 rx_frag_size; + struct rxq *rxq; + int ret; + int i; + + mdev =3D drv_info->mdev; + + rxq =3D kzalloc_obj(*rxq); + if (!rxq) + return NULL; + + rxq->que =3D que; + rxq->drv_info =3D drv_info; + rxq->rxqno =3D rxq->que->rxqno; + if (rxq->que->rx_nr_gpds < MIN_GPD_NUM) { + dev_err(mdev->dev, + "Failed to alloc cldma%d rxq%d due to gpd number < 2\n", + drv_info->hw_id, rxq->rxqno); + goto err_free_rxq; + } + rxq->nr_gpds =3D rxq->que->rx_nr_gpds; + rxq->arg =3D trb->priv; + rxq->rx_done =3D trb_open_priv->rx_done; + atomic_set(&rxq->need_exit, 0); + atomic_set(&rxq->need_restart, 0); + rx_frag_size =3D rxq->que->rx_frag_size; + if (rxq->que->rx_mtu > rx_frag_size && rx_frag_size) + rxq->nr_bds =3D (rxq->que->rx_mtu + rx_frag_size - 1) / rx_frag_size; + + rxq->req_pool =3D kcalloc(rxq->nr_gpds, sizeof(*req), GFP_KERNEL); + if (!rxq->req_pool) + goto err_free_rxq; + + /* setup rx request */ + for (i =3D 0; i < rxq->nr_gpds; i++) { + req =3D rxq->req_pool + i; + req->mtu =3D rxq->que->rx_mtu; + req->frag_size =3D rx_frag_size; + req->gpd =3D dma_pool_zalloc(drv_info->gpd_dma_pool, GFP_KERNEL, &req->g= pd_dma_addr); + if (!req->gpd) + goto err_free_req; + if (rxq->nr_bds) { + ret =3D mtk_cldma_alloc_rx_bd(drv_info, req, rxq->nr_bds); + if (ret) + goto err_free_req; + req->gpd->rx_gpd.gpd_flags |=3D CLDMA_GPD_FLAG_BDP; + } else { + req->skb =3D __dev_alloc_skb(req->mtu, GFP_KERNEL); + if (!req->skb) + goto err_free_req; + req->data_dma_addr =3D dma_map_single(mdev->dev, req->skb->data, + req->mtu, DMA_FROM_DEVICE); + ret =3D dma_mapping_error(mdev->dev, req->data_dma_addr); + if (unlikely(ret)) + goto err_free_req; + } + } + + for (i =3D 0; i < rxq->nr_gpds; i++) { + req =3D rxq->req_pool + i; + next =3D rxq->req_pool + ((i + 1) % rxq->nr_gpds); + req->gpd->rx_gpd.gpd_flags |=3D CLDMA_GPD_FLAG_IOC; + req->gpd->rx_gpd.data_allow_len =3D cpu_to_le16(req->mtu); + req->gpd->rx_gpd.next_gpd_ptr_h =3D cpu_to_le32((u64)(next->gpd_dma_addr= ) >> 32); + req->gpd->rx_gpd.next_gpd_ptr_l =3D cpu_to_le32(next->gpd_dma_addr); + if (!rxq->nr_bds) { + req->gpd->rx_gpd.data_buff_ptr_h =3D + cpu_to_le32((u64)(req->data_dma_addr) >> 32); + req->gpd->rx_gpd.data_buff_ptr_l =3D cpu_to_le32(req->data_dma_addr); + } + if (i !=3D rxq->nr_gpds - 1) + req->gpd->rx_gpd.gpd_flags |=3D CLDMA_GPD_FLAG_HWO; + } + + INIT_WORK(&rxq->rx_done_work, mtk_cldma_rx_done_work); + + /* Pairs with the acquire load in the ISR, as for txq above. */ + smp_store_release(&drv_info->rxq[rxq->rxqno], rxq); + ret =3D mtk_cldma_stop_queue(drv_info, DIR_RX, rxq->rxqno); + if (ret) { + dev_warn(mdev->dev, "Failed to stop RX queue %d before alloc\n", rxq->rx= qno); + WRITE_ONCE(drv_info->rxq[rxq->rxqno], NULL); + goto err_free_req; + } + mtk_cldma_setup_start_addr(drv_info, DIR_RX, + rxq->rxqno, rxq->req_pool[0].gpd_dma_addr); + mtk_cldma_start_queue(drv_info, DIR_RX, rxq->rxqno); + mtk_cldma_unmask_intr(drv_info, DIR_RX, rxq->rxqno, QUEUE_ERROR); + mtk_cldma_unmask_intr(drv_info, DIR_RX, rxq->rxqno, QUEUE_XFER_DONE); + + return rxq; + +err_free_req: + for (i =3D 0; i < rxq->nr_gpds; i++) { + req =3D rxq->req_pool + i; + if (!req->gpd) + break; + mtk_cldma_rxq_alloc_cancel(drv_info, req, rxq->nr_bds); + } + + kfree(rxq->req_pool); +err_free_rxq: + kfree(rxq); + return NULL; +} + +static void mtk_cldma_rxq_free(struct cldma_drv_info *drv_info, u32 rxqno) +{ + struct mtk_md_dev *mdev; + struct bd_dsc *bd_dsc; + struct rx_req *req; + struct rxq *rxq; + int irq_id; + int i, j, ret; + + mdev =3D drv_info->mdev; + + rxq =3D drv_info->rxq[rxqno]; + /* Unpublish before teardown; pairs with the acquire loads of this + * slot. No release is needed, there is no prior store to expose. + */ + WRITE_ONCE(drv_info->rxq[rxqno], NULL); + + /* stop HW rx transaction; -ENODEV means the link is dead, so the + * device cannot be walking the ring and the free may proceed. + */ + atomic_set(&rxq->need_exit, 1); + ret =3D mtk_cldma_stop_queue(drv_info, DIR_RX, rxqno); + if (ret =3D=3D -ETIMEDOUT) { + dev_err(mdev->dev, "RX queue %d stop timed out, resetting CLDMA%d\n", + rxqno, drv_info->hw_id); + mtk_cldma_drv_reset(drv_info); + ret =3D mtk_cldma_stop_queue(drv_info, DIR_RX, rxqno); + /* the reset took the whole instance down with this queue */ + mtk_cldma_rearm_queues(drv_info); + } + + irq_id =3D mtk_pci_get_virq_id(mdev, drv_info->pci_ext_irq_id); + synchronize_irq(irq_id); + /* flush on-going work */ + flush_work(&rxq->rx_done_work); + /* mask L2 RX interrupt again to avoid race condition causing use-after-f= ree issue */ + mtk_cldma_mask_intr(drv_info, DIR_RX, rxqno, QUEUE_XFER_DONE); + mtk_cldma_mask_intr(drv_info, DIR_RX, rxqno, QUEUE_ERROR); + + if (ret =3D=3D -ETIMEDOUT) { + /* Still a live DMA target: leak the ring rather than hand it + * back to the allocator. + */ + dev_err(mdev->dev, "RX queue %d cannot be stopped, leaking its ring\n", + rxqno); + return; + } + + /* free rx req resource */ + for (i =3D 0; i < rxq->nr_gpds; i++) { + req =3D rxq->req_pool + rxq->free_idx; + if (!(req->gpd->rx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO) && + le16_to_cpu(req->gpd->rx_gpd.data_recv_len)) { + if (!mtk_cldma_rx_skb_adjust(mdev, rxq, req)) { + rxq->rx_done(req->skb, rxq->arg, true); + req->skb =3D NULL; + } + } + if (req->skb) { + if (!rxq->nr_bds) { + if (req->data_dma_addr) + dma_unmap_single(mdev->dev, req->data_dma_addr, + req->mtu, DMA_FROM_DEVICE); + dev_kfree_skb_any(req->skb); + } else if (req->bd_dsc_pool[0].skb) { + /* The head skb aliases bd_dsc_pool[0].skb and the + * frag_list chain aliases the other BD skbs: break + * the aliases and let the BD walk below free every + * skb exactly once, as rxq_alloc_cancel() does. + */ + skb_shinfo(req->skb)->frag_list =3D NULL; + } else { + /* The BD skbs are already detached, the head owns + * the whole chain through its frag_list. + */ + dev_kfree_skb_any(req->skb); + } + req->skb =3D NULL; + } + for (j =3D 0; j < rxq->nr_bds; j++) { + bd_dsc =3D req->bd_dsc_pool + j; + if (bd_dsc->skb) { + if (bd_dsc->data_dma_addr) + dma_unmap_single(mdev->dev, bd_dsc->data_dma_addr, + req->frag_size, DMA_FROM_DEVICE); + bd_dsc->skb->next =3D NULL; + dev_kfree_skb_any(bd_dsc->skb); + } + dma_pool_free(drv_info->bd_dma_pool, + bd_dsc->bd, bd_dsc->bd_dma_addr); + } + kfree(req->bd_dsc_pool); + dma_pool_free(drv_info->gpd_dma_pool, req->gpd, req->gpd_dma_addr); + rxq->free_idx =3D (rxq->free_idx + 1) % rxq->nr_gpds; + } + + kfree(rxq->req_pool); + kfree(rxq); +} + +/* The device reported a zero TX start address: it lost its queue state. */ +static int mtk_cldma_hw_recovery(struct cldma_drv_info *drv_info, u32 qno) +{ + u64 val; + + dev_err(drv_info->mdev->dev, + "CLDMA%d lost its queue state, re-initializing\n", + drv_info->hw_id); + + mtk_cldma_rearm_queues(drv_info); + + val =3D mtk_cldma_get_tx_start_addr(drv_info, qno); + if (!val || val =3D=3D U64_MAX) + return -EIO; + + return 0; +} + +static int mtk_cldma_start_xfer(struct cldma_drv_info *drv_info, u32 qno) +{ + struct txq *txq; + u64 val; + int ret; + + txq =3D drv_info->txq[qno]; + + val =3D mtk_cldma_get_tx_start_addr(drv_info, qno); + if (unlikely(val =3D=3D U64_MAX)) + return -EIO; + + if (unlikely(!val)) { + ret =3D mtk_cldma_hw_recovery(drv_info, qno); + if (ret) + return ret; + } + + /* Hold ring_lock across programming and kicking the queue so the + * start address given to the device is the one free_idx still + * names; tx_done_work advances free_idx under the same lock. + */ + spin_lock(&txq->ring_lock); + if (unlikely(!READ_ONCE(txq->tx_started))) { + mtk_cldma_setup_start_addr(drv_info, DIR_TX, qno, + txq->req_pool[txq->free_idx].gpd_dma_addr); + mtk_cldma_start_queue(drv_info, DIR_TX, qno); + WRITE_ONCE(txq->tx_started, true); + } else { + mtk_cldma_resume_queue(drv_info, DIR_TX, qno); + } + spin_unlock(&txq->ring_lock); + + return 0; +} + +int mtk_cldma_init(struct mtk_ctrl_trans *trans) +{ + struct cldma_dev *cd; + + cd =3D kzalloc_obj(*cd); + if (!cd) + return -ENOMEM; + + cd->trans =3D trans; + trans->dev =3D cd; + + return 0; +} + +void mtk_cldma_exit(struct mtk_ctrl_trans *trans) +{ + if (!trans->dev) + return; + + kfree(trans->dev); + trans->dev =3D NULL; +} + +static int mtk_cldma_open(struct cldma_dev *cd, struct sk_buff *skb) +{ + struct trb_open_priv *trb_open_priv =3D (struct trb_open_priv *)skb->data; + struct trb *trb =3D (struct trb *)skb->cb; + struct cldma_drv_info *drv_info; + struct queue_info *que; + struct txq *txq; + struct rxq *rxq; + int ret =3D 0; + + que =3D radix_tree_lookup(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF= ); + if (unlikely(!que)) { + trb->status =3D -EINVAL; + trb->trb_complete(skb); + return -EINVAL; + } + drv_info =3D cd->cldma_drv_info[que->hif_id]; + if (!drv_info) { + ret =3D -EIO; + goto out; + } + + if (que->tx_mtu =3D=3D 0 || que->rx_mtu =3D=3D 0) { + dev_err((cd->trans->mdev)->dev, + "Failed to enable cldma%d txq%d rxq%d due to wrong mtu\n", + drv_info->hw_id, que->txqno, que->rxqno); + ret =3D -EINVAL; + goto out; + } + + trb_open_priv->tx_mtu =3D que->tx_mtu; + trb_open_priv->rx_mtu =3D que->rx_mtu; + trb_open_priv->tx_frag_size =3D que->tx_frag_size; + trb_open_priv->rx_frag_size =3D que->rx_frag_size; + + if (drv_info->txq[que->txqno] || drv_info->rxq[que->rxqno]) { + ret =3D -EBUSY; + goto out; + } + + txq =3D mtk_cldma_txq_alloc(drv_info, que); + if (!txq) { + ret =3D -ENOMEM; + goto out; + } + + rxq =3D mtk_cldma_rxq_alloc(drv_info, que, skb); + if (!rxq) { + ret =3D -ENOMEM; + mtk_cldma_txq_free(drv_info, txq->txqno); + goto out; + } + +out: + if (ret) + cd->trans->usr_cnt[que->hif_id][que->txqno]--; + + trb->status =3D ret; + trb->trb_complete(skb); + + return ret; +} + +/* Complete every submitted-but-unkicked request with an error so no TRB + * is stranded in the ring when the doorbell cannot be rung. + */ +static void mtk_cldma_txq_flush(struct cldma_drv_info *drv_info, + struct txq *txq, int err) +{ + struct mtk_ctrl_trans *trans =3D drv_info->cd->trans; + int hif_id =3D drv_info->hif_id; + u32 txqno =3D txq->txqno; + struct sk_buff *skb; + struct trb_srv *srv; + struct tx_req *req; + bool was_starved; + struct trb *trb; + int i; + + for (i =3D 0; i < txq->nr_gpds; i++) { + spin_lock(&txq->ring_lock); + + req =3D txq->req_pool + txq->free_idx; + if (!req->skb) { + spin_unlock(&txq->ring_lock); + break; + } + + req->gpd->tx_gpd.gpd_flags &=3D ~CLDMA_GPD_FLAG_HWO; + + if (txq->nr_bds) + mtk_cldma_clr_bd_dsc(drv_info, req->bd_dsc_pool, txq->nr_bds); + else + dma_unmap_single(drv_info->mdev->dev, req->data_dma_addr, + req->data_len, DMA_TO_DEVICE); + + skb =3D req->skb; + req->data_dma_addr =3D 0; + req->data_len =3D 0; + req->skb =3D NULL; + + txq->free_idx =3D (txq->free_idx + 1) % txq->nr_gpds; + was_starved =3D atomic_fetch_inc(&txq->req_budget) =3D=3D NO_BUDGET; + + spin_unlock(&txq->ring_lock); + + trb =3D (struct trb *)skb->cb; + trb->status =3D err; + trb->trb_complete(skb); + + /* The service may already have been torn down; completing the + * TRB is still correct, there is just nobody left to wake. + */ + srv =3D trans->trb_srv[trans->srv_cfg[hif_id][txqno]]; + if (was_starved && srv) + wake_up(&srv->trb_waitq); + } +} + +static int mtk_cldma_tx(struct cldma_dev *cd, struct sk_buff *skb) +{ + struct trb *trb =3D (struct trb *)skb->cb; + struct cldma_drv_info *drv_info; + struct mtk_md_dev *mdev; + struct queue_info *que; + struct txq *txq; + int ret; + + que =3D radix_tree_lookup(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF= ); + if (unlikely(!que)) + return -EPIPE; + drv_info =3D cd->cldma_drv_info[que->hif_id]; + if (unlikely(!drv_info)) + return -EPIPE; + txq =3D drv_info->txq[que->txqno]; + if (unlikely(!txq)) + return -EPIPE; + + mdev =3D drv_info->mdev; + + ret =3D mtk_cldma_start_xfer(drv_info, que->txqno); + if (unlikely(ret)) { + dev_err(mdev->dev, "Failed to trigger cldma tx\n"); + mtk_cldma_txq_flush(drv_info, txq, ret); + } + + return ret; +} + +static int mtk_cldma_close(struct cldma_dev *cd, struct sk_buff *skb) +{ + struct trb *trb =3D (struct trb *)skb->cb; + struct cldma_drv_info *drv_info; + struct queue_info *que; + + que =3D radix_tree_lookup(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF= ); + if (unlikely(!que)) { + trb->status =3D -EPIPE; + trb->trb_complete(skb); + return -EPIPE; + } + drv_info =3D cd->cldma_drv_info[que->hif_id]; + if (unlikely(!drv_info)) { + trb->status =3D -EPIPE; + trb->trb_complete(skb); + return -EPIPE; + } + + if (drv_info->txq[que->txqno]) + mtk_cldma_txq_free(drv_info, que->txqno); + if (drv_info->rxq[que->rxqno]) + mtk_cldma_rxq_free(drv_info, que->rxqno); + + trb->status =3D 0; + trb->trb_complete(skb); + + return 0; +} + +static int mtk_cldma_txbuf_set(struct cldma_drv_info *drv_info, struct sk_= buff *skb, + struct tx_req *req, int nr_bds) +{ + struct sk_buff *curr_skb, *next_skb; + struct mtk_md_dev *mdev; + struct bd_dsc *bd_dsc; + int ret; + int i; + + mdev =3D drv_info->mdev; + + if (nr_bds) { + bd_dsc =3D req->bd_dsc_pool; + curr_skb =3D skb; + for (i =3D 0; i < nr_bds && curr_skb; i++) { + bd_dsc =3D req->bd_dsc_pool + i; + if (req->bd_dsc_pool =3D=3D bd_dsc) { + bd_dsc->data_len =3D skb->len - skb->data_len; + next_skb =3D skb_shinfo(skb)->frag_list; + } else { + bd_dsc->data_len =3D curr_skb->len; + next_skb =3D curr_skb->next; + } + bd_dsc->data_dma_addr =3D dma_map_single(mdev->dev, curr_skb->data, + bd_dsc->data_len, DMA_TO_DEVICE); + ret =3D dma_mapping_error(mdev->dev, bd_dsc->data_dma_addr); + if (unlikely(ret)) + goto err_unmap_buffer; + + bd_dsc->bd->tx_bd.data_buff_ptr_h =3D + cpu_to_le32((u64)(bd_dsc->data_dma_addr) >> 32); + bd_dsc->bd->tx_bd.data_buff_ptr_l =3D cpu_to_le32(bd_dsc->data_dma_addr= ); + bd_dsc->bd->tx_bd.data_buffer_len =3D cpu_to_le16(bd_dsc->data_len); + curr_skb =3D next_skb; + } + bd_dsc->bd->tx_bd.bd_flags =3D CLDMA_BD_FLAG_EOL; + } else { + /* Non-BD mode maps only the linear area; a nonlinear SKB + * here would be silently truncated to skb_headlen(). + */ + if (WARN_ON_ONCE(skb_is_nonlinear(skb))) + return -EINVAL; + + req->data_dma_addr =3D dma_map_single(mdev->dev, skb->data, + skb_headlen(skb), DMA_TO_DEVICE); + ret =3D dma_mapping_error(mdev->dev, req->data_dma_addr); + if (unlikely(ret)) { + req->data_dma_addr =3D 0; + goto err_exit; + } + + req->gpd->tx_gpd.data_buff_ptr_h =3D cpu_to_le32((u64)(req->data_dma_add= r) >> 32); + req->gpd->tx_gpd.data_buff_ptr_l =3D cpu_to_le32(req->data_dma_addr); + } + + return 0; + +err_unmap_buffer: + for (i =3D 0; i < nr_bds; i++) { + bd_dsc =3D req->bd_dsc_pool + i; + if (dma_mapping_error(mdev->dev, bd_dsc->data_dma_addr)) { + bd_dsc->data_dma_addr =3D 0; + break; + } + dma_unmap_single(mdev->dev, bd_dsc->data_dma_addr, + bd_dsc->data_len, DMA_TO_DEVICE); + bd_dsc->data_dma_addr =3D 0; + } +err_exit: + dev_err_ratelimited(mdev->dev, "Failed to map dma! error:%d\n", ret); + return -ENOMEM; +} + +int mtk_cldma_submit_tx(void *dev, struct sk_buff *skb) +{ + struct trb *trb =3D (struct trb *)skb->cb; + struct cldma_drv_info *drv_info; + struct cldma_dev *cd =3D dev; + struct queue_info *que; + struct tx_req *req; + struct txq *txq; + int ret; + + /* the CLDMA device is unpublished before it is freed, so a submitter + * that raced the teardown lands here with a NULL dev + */ + if (unlikely(!cd)) + return -EINVAL; + + que =3D radix_tree_lookup(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF= ); + if (unlikely(!que)) + return -EINVAL; + drv_info =3D cd->cldma_drv_info[que->hif_id]; + if (unlikely(!drv_info)) + return -EINVAL; + + txq =3D drv_info->txq[que->txqno]; + if (unlikely(!txq)) + return -EINVAL; + + spin_lock(&txq->ring_lock); + + if (!atomic_read(&txq->req_budget)) { + spin_unlock(&txq->ring_lock); + return -EAGAIN; + } + + req =3D txq->req_pool + txq->wr_idx; + req->gpd->tx_gpd.debug_id =3D 0x01; + ret =3D mtk_cldma_txbuf_set(drv_info, skb, req, txq->nr_bds); + if (ret) { + spin_unlock(&txq->ring_lock); + return ret; + } + + req->gpd->tx_gpd.data_buff_len =3D cpu_to_le16(skb->len); + + req->data_len =3D skb->len; + req->skb =3D skb; + + dma_wmb(); /* ensure req and data msg set done before HWO setup */ + + req->gpd->tx_gpd.gpd_flags |=3D CLDMA_GPD_FLAG_HWO; + + txq->wr_idx =3D (txq->wr_idx + 1) % txq->nr_gpds; + atomic_dec(&txq->req_budget); + + spin_unlock(&txq->ring_lock); + + return 0; +} + +int mtk_cldma_get_tx_budget(void *dev, enum mtk_hif_id hif_id, u32 qno) +{ + struct cldma_drv_info *drv_info; + struct cldma_dev *cd =3D dev; + struct txq *txq; + + if (unlikely(hif_id >=3D NR_CLDMA || qno >=3D HW_QUE_NUM || !cd)) + return -EINVAL; + + drv_info =3D cd->cldma_drv_info[hif_id]; + if (!drv_info) + return -EINVAL; + txq =3D drv_info->txq[qno]; + if (!txq) + return -EINVAL; + return atomic_read(&txq->req_budget); +} + +static int (*trb_act_tbl[TRB_CMD_MAX])(struct cldma_dev *cd, struct sk_buf= f *skb) =3D { + [TRB_CMD_ENABLE] =3D mtk_cldma_open, + [TRB_CMD_TX] =3D mtk_cldma_tx, + [TRB_CMD_DISABLE] =3D mtk_cldma_close, +}; + +int mtk_cldma_trb_process(void *dev, struct sk_buff *skb) +{ + struct cldma_dev *cd; + struct trb *trb; + + if (!dev || !skb) + return -EINVAL; + + cd =3D (struct cldma_dev *)dev; + trb =3D (struct trb *)skb->cb; + + if (!(trb->cmd > TRB_CMD_MIN && trb->cmd < TRB_CMD_STOP)) + return -EINVAL; + + return trb_act_tbl[trb->cmd](cd, skb); +} + +int mtk_cldma_check_ch_cfg(void *dev, struct queue_info *que) +{ + struct cldma_drv_info *drv_info; + struct cldma_dev *cd =3D dev; + struct mtk_md_dev *mdev; + struct txq *txq; + struct rxq *rxq; + + mdev =3D cd->trans->mdev; + drv_info =3D cd->cldma_drv_info[que->hif_id]; + + if (!drv_info) { + dev_err(mdev->dev, "CLDMA%d has not been initialized\n", + mtk_cldma_hw_id_tbl[que->hif_id]); + return -EINVAL; + } + + txq =3D drv_info->txq[que->txqno]; + rxq =3D drv_info->rxq[que->rxqno]; + if (!txq || !rxq) { + dev_err(mdev->dev, + "CLDMA%d txq%d rxq%d has not been enabled\n", + mtk_cldma_hw_id_tbl[que->hif_id], que->txqno, que->rxqno); + return -EINVAL; + } + + if (que->tx_mtu !=3D txq->que->tx_mtu || que->rx_mtu !=3D rxq->que->rx_mt= u) { + dev_err(mdev->dev, + "Channel:%08x tx_mtu:%08x rx_mtu:%08x do not match ch cfg\n", + que->tx_chl, que->tx_mtu, que->rx_mtu); + return -EINVAL; + } + + return 0; +} diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma.h b/drivers/net/wwan/t9xx= /pcie/mtk_cldma.h new file mode 100644 index 000000000000..fa6d79f8b5df --- /dev/null +++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma.h @@ -0,0 +1,165 @@ +/* SPDX-License-Identifier: GPL-2.0-only + * + * Copyright (c) 2022, MediaTek Inc. + */ + +#ifndef __MTK_CLDMA_H__ +#define __MTK_CLDMA_H__ + +#include +#include +#include +#include +#include +#include + +#include "mtk_ctrl_plane.h" +#include "mtk_trans_ctrl.h" + +struct mtk_fsm_param; + +#define TXQ(N) (N) +#define RXQ(N) (N) + +#define CLDMA_GPD_FLAG_HWO BIT(0) +#define CLDMA_GPD_FLAG_BDP BIT(1) +#define CLDMA_GPD_FLAG_IOC BIT(7) +#define CLDMA_BD_FLAG_EOL BIT(0) + +union gpd { + struct { + u8 gpd_flags; + u8 non_used1; + __le16 data_allow_len; + __le32 next_gpd_ptr_h; + __le32 next_gpd_ptr_l; + __le32 data_buff_ptr_h; + __le32 data_buff_ptr_l; + __le16 data_recv_len; + u8 non_used2; + u8 debug_id; + } rx_gpd; + + struct { + u8 gpd_flags; + u8 non_used1; + u8 non_used2; + u8 debug_id; + __le32 next_gpd_ptr_h; + __le32 next_gpd_ptr_l; + __le32 data_buff_ptr_h; + __le32 data_buff_ptr_l; + __le16 data_buff_len; + __le16 non_used3; + } tx_gpd; +} __packed; + +union bd { + struct { + u8 bd_flags; + u8 non_used1; + __le16 data_allow_len; + __le32 next_bd_ptr_h; + __le32 next_bd_ptr_l; + __le32 data_buff_ptr_h; + __le32 data_buff_ptr_l; + __le16 data_recv_len; + __le16 non_used2; + } rx_bd; + + struct { + u8 bd_flags; + u8 non_used1; + __le16 non_used2; + __le32 next_bd_ptr_h; + __le32 next_bd_ptr_l; + __le32 data_buff_ptr_h; + __le32 data_buff_ptr_l; + __le16 data_buffer_len; + u8 extension_len; + u8 non_used3; + } tx_bd; +} __packed; + +struct bd_dsc { + union bd *bd; + struct sk_buff *skb; + dma_addr_t bd_dma_addr; + dma_addr_t data_dma_addr; + size_t data_len; +}; + +struct rx_req { + union gpd *gpd; + u32 mtu; + struct sk_buff *skb; + size_t data_len; + dma_addr_t gpd_dma_addr; + dma_addr_t data_dma_addr; + u32 frag_size; + struct bd_dsc *bd_dsc_pool; +}; + +struct rxq { + struct cldma_drv_info *drv_info; + u32 rxqno; + struct queue_info *que; + struct work_struct rx_done_work; + struct rx_req *req_pool; + u32 nr_gpds; + u32 free_idx; + void *arg; + int (*rx_done)(struct sk_buff *skb, void *priv, bool force_recv); + u32 nr_bds; + atomic_t need_exit; + /* set when the queue must be programmed and started again after an IP + * reset; consumed by mtk_cldma_rx_done_work(), the owner of free_idx + */ + atomic_t need_restart; +}; + +struct tx_req { + union gpd *gpd; + u32 mtu; + size_t data_len; + dma_addr_t data_dma_addr; + dma_addr_t gpd_dma_addr; + struct sk_buff *skb; + int (*trb_complete)(struct sk_buff *skb); + u32 frag_size; + struct bd_dsc *bd_dsc_pool; +}; + +struct txq { + struct cldma_drv_info *drv_info; + u32 txqno; + struct queue_info *que; + struct work_struct tx_done_work; + struct tx_req *req_pool; + u32 nr_gpds; + atomic_t req_budget; + /* ring_lock: serializes the producer (TRB service thread) against the + * consumer (tx_done_work) over req_budget, wr_idx, free_idx and the + * tx_req/GPD slots they index. + */ + spinlock_t ring_lock; + u32 wr_idx; + u32 free_idx; + bool tx_started; + u32 nr_bds; +}; + +struct cldma_dev { + struct cldma_drv_info *cldma_drv_info[NR_CLDMA]; + struct mtk_ctrl_trans *trans; +}; + +int mtk_cldma_init(struct mtk_ctrl_trans *trans); +void mtk_cldma_exit(struct mtk_ctrl_trans *trans); +int mtk_cldma_submit_tx(void *dev, struct sk_buff *skb); +int mtk_cldma_get_tx_budget(void *dev, enum mtk_hif_id hif_id, u32 qno); +int mtk_cldma_trb_process(void *dev, struct sk_buff *skb); +void mtk_cldma_fsm_state_listener(struct mtk_fsm_param *param, struct mtk_= ctrl_trans *trans); +int mtk_cldma_check_ch_cfg(void *dev, struct queue_info *que); + +#endif diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.c b/drivers/net/wwan/= t9xx/pcie/mtk_cldma_drv.c new file mode 100644 index 000000000000..8ab153da2083 --- /dev/null +++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.c @@ -0,0 +1,561 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2023, MediaTek Inc. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "mtk_cldma_drv.h" +#include "mtk_dev.h" +#include "mtk_pci.h" +#include "mtk_pci_reg.h" + +#define WAIT_QUEUE_STOP (70) + +#define CLDMA0_BASE_ADDR (0x1021C000) +#define CLDMA1_BASE_ADDR (0x1021E000) + +#define CLDMA_RX_SKB_POOL_MAX_SIZE (64) +#define CLDMA_RX_SKB_RELOAD_THRESHOLD (16) + +/* L2TISAR0 */ +#define TQ_ERR_INT_OFFSET (16) +#define TQ_ERR_INT_BITMASK (0x00FF0000) +#define TQ_ACTIVE_START_ERR_INT_OFFSET (24) +#define TQ_ACTIVE_START_ERR_INT_BITMASK (0xFF000000) + +/* L2RISAR0 */ +#define RQ_ERR_INT_OFFSET (16) +#define RQ_ERR_INT_BITMASK (0x00FF0000) +#define RQ_ACTIVE_START_ERR_INT_OFFSET (24) +#define RQ_ACTIVE_START_ERR_INT_BITMASK (0xFF000000) + +/* CLDMA IN(Tx) */ +#define REG_CLDMA_UL_START_ADDRL_0 (0x0004) +#define REG_CLDMA_UL_START_ADDRH_0 (0x0008) +#define REG_CLDMA_UL_CURRENT_ADDRL_0 (0x0044) +#define REG_CLDMA_UL_CURRENT_ADDRH_0 (0x0048) +#define REG_CLDMA_UL_STATUS (0x0084) +#define REG_CLDMA_UL_START_CMD (0x0088) +#define REG_CLDMA_UL_RESUME_CMD (0x008C) +#define REG_CLDMA_UL_STOP_CMD (0x0090) +#define REG_CLDMA_UL_ERROR (0x0094) +#define REG_CLDMA_UL_CFG (0x0098) +#define REG_CLDMA_UL_DUMMY_0 (0x009C) + +/* CLDMA OUT(Rx) */ +#define REG_CLDMA_SO_ERROR (0x0400 + 0x0100) +#define REG_CLDMA_SO_START_CMD (0x0400 + 0x01BC) +#define REG_CLDMA_SO_RESUME_CMD (0x0400 + 0x01C0) +#define REG_CLDMA_SO_STOP_CMD (0x0400 + 0x01C4) +#define REG_CLDMA_SO_DUMMY_0 (0x0400 + 0x0108) +#define REG_CLDMA_SO_CFG (0x0400 + 0x0004) +#define REG_CLDMA_SO_START_ADDRL_0 (0x0400 + 0x0078) +#define REG_CLDMA_SO_START_ADDRH_0 (0x0400 + 0x007C) +#define REG_CLDMA_SO_CUR_ADDRL_0 (0x0400 + 0x00B8) +#define REG_CLDMA_SO_CUR_ADDRH_0 (0x0400 + 0x00BC) +#define REG_CLDMA_SO_STATUS (0x0400 + 0x00F8) +#define REG_CLDMA_DEBUG_ID_EN (0x0400 + 0x00FC) +#define REG_CLDMA_SO_LAST_UPDATE_ADDRL_0 (0x0400 + 0x01C8) +#define REG_CLDMA_SO_LAST_UPDATE_ADDRH_0 (0x0400 + 0x01CC) + +/* CLDMA MISC */ +#define REG_CLDMA_L2TISAR0 (0x0800 + 0x0010) +#define REG_CLDMA_L2TISAR1 (0x0800 + 0x0014) +#define REG_CLDMA_L2TIMR0 (0x0800 + 0x0018) +#define REG_CLDMA_L2TIMR1 (0x0800 + 0x001C) +#define REG_CLDMA_L2TIMCR0 (0x0800 + 0x0020) +#define REG_CLDMA_L2TIMCR1 (0x0800 + 0x0024) +#define REG_CLDMA_L2TIMSR0 (0x0800 + 0x0028) +#define REG_CLDMA_L2TIMSR1 (0x0800 + 0x002C) +#define REG_CLDMA_L3TISAR0 (0x0800 + 0x0030) +#define REG_CLDMA_L3TISAR1 (0x0800 + 0x0034) +#define REG_CLDMA_L2RISAR0 (0x0800 + 0x0050) +#define REG_CLDMA_L2RISAR1 (0x0800 + 0x0054) +#define REG_CLDMA_L3RISAR0 (0x0800 + 0x0070) +#define REG_CLDMA_L3RISAR1 (0x0800 + 0x0074) +#define REG_CLDMA_IP_BUSY (0x0800 + 0x00B4) +#define REG_CLDMA_L3TISAR2 (0x0800 + 0x00C0) + +#define REG_CLDMA_L2RIMR0 (0x0800 + 0x00E8) +#define REG_CLDMA_L2RIMR1 (0x0800 + 0x00EC) +#define REG_CLDMA_L2RIMCR0 (0x0800 + 0x00F0) +#define REG_CLDMA_L2RIMCR1 (0x0800 + 0x00F4) +#define REG_CLDMA_L2RIMSR0 (0x0800 + 0x00F8) +#define REG_CLDMA_L2RIMSR1 (0x0800 + 0x00FC) + +#define REG_CLDMA_INT_EAP_USIP_MASK (0x0800 + 0x011C) + +#define REG_CLDMA_IP_BUSY_TO_PCIE_MASK (0x0800 + 0x0194) +#define REG_CLDMA_IP_BUSY_TO_PCIE_MASK_SET (0x0800 + 0x0198) +#define REG_CLDMA_IP_BUSY_TO_PCIE_MASK_CLR (0x0800 + 0x019C) + +#define REG_CLDMA_IP_BUSY_TO_AP_MASK (0x0800 + 0x0200) +#define REG_CLDMA_IP_BUSY_TO_AP_MASK_SET (0x0800 + 0x0204) +#define REG_CLDMA_IP_BUSY_TO_AP_MASK_CLR (0x0800 + 0x0208) +#define REG_CLDMA_IP_BUSY_TO_MD_MASK_SET (0x0800 + 0x0210) +#define REG_CLDMA_RX_WORK_TO_REG_MASK_SET (0x0800 + 0x021C) + +/* CLDMA RESET */ +#define REG_INFRA_RST0_SET (0x120) +#define REG_INFRA_RST0_CLR (0x124) +#define REG_CLDMA0_RST_SET_BIT (8) +#define REG_CLDMA0_RST_CLR_BIT (8) + +struct cldma_hw_regs mtk_cldma_regs_m9xx =3D { + .cldma0_base_addr =3D CLDMA0_BASE_ADDR, + .cldma1_base_addr =3D CLDMA1_BASE_ADDR, + .cldma_rx_skb_pool_max_size =3D CLDMA_RX_SKB_POOL_MAX_SIZE, + .cldma_rx_skb_reload_threshold =3D CLDMA_RX_SKB_RELOAD_THRESHOLD, + .tq_err_int_offset =3D TQ_ERR_INT_OFFSET, + .tq_err_int_bitmask =3D TQ_ERR_INT_BITMASK, + .tq_active_start_err_int_offset =3D TQ_ACTIVE_START_ERR_INT_OFFSET, + .tq_active_start_err_int_bitmask =3D TQ_ACTIVE_START_ERR_INT_BITMASK, + .rq_err_int_offset =3D RQ_ERR_INT_OFFSET, + .rq_err_int_bitmask =3D RQ_ERR_INT_BITMASK, + .rq_active_start_err_int_offset =3D RQ_ACTIVE_START_ERR_INT_OFFSET, + .rq_active_start_err_int_bitmask =3D RQ_ACTIVE_START_ERR_INT_BITMASK, + .reg_cldma_ul_start_addrl_0 =3D REG_CLDMA_UL_START_ADDRL_0, + .reg_cldma_ul_start_addrh_0 =3D REG_CLDMA_UL_START_ADDRH_0, + .reg_cldma_ul_current_addrl_0 =3D REG_CLDMA_UL_CURRENT_ADDRL_0, + .reg_cldma_ul_current_addrh_0 =3D REG_CLDMA_UL_CURRENT_ADDRH_0, + .reg_cldma_ul_status =3D REG_CLDMA_UL_STATUS, + .reg_cldma_ul_start_cmd =3D REG_CLDMA_UL_START_CMD, + .reg_cldma_ul_resume_cmd =3D REG_CLDMA_UL_RESUME_CMD, + .reg_cldma_ul_stop_cmd =3D REG_CLDMA_UL_STOP_CMD, + .reg_cldma_ul_error =3D REG_CLDMA_UL_ERROR, + .reg_cldma_ul_cfg =3D REG_CLDMA_UL_CFG, + .reg_cldma_ul_dummy_0 =3D REG_CLDMA_UL_DUMMY_0, + .reg_cldma_so_error =3D REG_CLDMA_SO_ERROR, + .reg_cldma_so_start_cmd =3D REG_CLDMA_SO_START_CMD, + .reg_cldma_so_resume_cmd =3D REG_CLDMA_SO_RESUME_CMD, + .reg_cldma_so_stop_cmd =3D REG_CLDMA_SO_STOP_CMD, + .reg_cldma_so_dummy_0 =3D REG_CLDMA_SO_DUMMY_0, + .reg_cldma_so_cfg =3D REG_CLDMA_SO_CFG, + .reg_cldma_so_start_addrl_0 =3D REG_CLDMA_SO_START_ADDRL_0, + .reg_cldma_so_start_addrh_0 =3D REG_CLDMA_SO_START_ADDRH_0, + .reg_cldma_so_current_addrl_0 =3D REG_CLDMA_SO_CUR_ADDRL_0, + .reg_cldma_so_current_addrh_0 =3D REG_CLDMA_SO_CUR_ADDRH_0, + .reg_cldma_so_status =3D REG_CLDMA_SO_STATUS, + .reg_cldma_debug_id_en =3D REG_CLDMA_DEBUG_ID_EN, + .reg_cldma_so_last_update_addrl_0 =3D REG_CLDMA_SO_LAST_UPDATE_ADDRL_0, + .reg_cldma_so_last_update_addrh_0 =3D REG_CLDMA_SO_LAST_UPDATE_ADDRH_0, + .reg_cldma_l2tisar0 =3D REG_CLDMA_L2TISAR0, + .reg_cldma_l2tisar1 =3D REG_CLDMA_L2TISAR1, + .reg_cldma_l2timr0 =3D REG_CLDMA_L2TIMR0, + .reg_cldma_l2timr1 =3D REG_CLDMA_L2TIMR1, + .reg_cldma_l2timcr0 =3D REG_CLDMA_L2TIMCR0, + .reg_cldma_l2timcr1 =3D REG_CLDMA_L2TIMCR1, + .reg_cldma_l2timsr0 =3D REG_CLDMA_L2TIMSR0, + .reg_cldma_l2timsr1 =3D REG_CLDMA_L2TIMSR1, + .reg_cldma_l3tisar0 =3D REG_CLDMA_L3TISAR0, + .reg_cldma_l3tisar1 =3D REG_CLDMA_L3TISAR1, + .reg_cldma_l3tisar2 =3D REG_CLDMA_L3TISAR2, + .reg_cldma_l2risar0 =3D REG_CLDMA_L2RISAR0, + .reg_cldma_l2risar1 =3D REG_CLDMA_L2RISAR1, + .reg_cldma_l2rimr0 =3D REG_CLDMA_L2RIMR0, + .reg_cldma_l2rimr1 =3D REG_CLDMA_L2RIMR1, + .reg_cldma_l2rimcr0 =3D REG_CLDMA_L2RIMCR0, + .reg_cldma_l2rimcr1 =3D REG_CLDMA_L2RIMCR1, + .reg_cldma_l2rimsr0 =3D REG_CLDMA_L2RIMSR0, + .reg_cldma_l2rimsr1 =3D REG_CLDMA_L2RIMSR1, + .reg_cldma_l3risar0 =3D REG_CLDMA_L3RISAR0, + .reg_cldma_l3risar1 =3D REG_CLDMA_L3RISAR1, + .reg_cldma_ip_busy =3D REG_CLDMA_IP_BUSY, + .reg_cldma_int_mask =3D REG_CLDMA_INT_EAP_USIP_MASK, + .reg_cldma_ip_busy_to_pcie_mask =3D REG_CLDMA_IP_BUSY_TO_PCIE_MASK, + .reg_cldma_ip_busy_to_pcie_mask_set =3D REG_CLDMA_IP_BUSY_TO_PCIE_MASK_SE= T, + .reg_cldma_ip_busy_to_pcie_mask_clr =3D REG_CLDMA_IP_BUSY_TO_PCIE_MASK_CL= R, + .reg_cldma_ip_busy_to_ap_mask =3D REG_CLDMA_IP_BUSY_TO_AP_MASK, + .reg_cldma_ip_busy_to_ap_mask_set =3D REG_CLDMA_IP_BUSY_TO_AP_MASK_SET, + .reg_cldma_ip_busy_to_ap_mask_clr =3D REG_CLDMA_IP_BUSY_TO_AP_MASK_CLR, + .reg_cldma_ip_busy_to_md_mask_set =3D REG_CLDMA_IP_BUSY_TO_MD_MASK_SET, + .reg_cldma_rx_work_to_reg_mask_set =3D REG_CLDMA_RX_WORK_TO_REG_MASK_SET, + .reg_infra_rst0_set =3D REG_INFRA_RST0_SET, + .reg_infra_rst0_clr =3D REG_INFRA_RST0_CLR, +}; + +void mtk_cldma_drv_reset(struct cldma_drv_info *drv_info) +{ + struct cldma_hw_regs *hw_regs; + struct mtk_md_dev *mdev; + u32 val; + + mdev =3D drv_info->mdev; + hw_regs =3D drv_info->hw_regs; + + val =3D mtk_pci_read32(mdev, REG_DEV_INFRA_BASE + hw_regs->reg_infra_rst0= _set); + val |=3D 1 << (REG_CLDMA0_RST_SET_BIT + drv_info->hw_id); + mtk_pci_write32(mdev, REG_DEV_INFRA_BASE + hw_regs->reg_infra_rst0_set, v= al); + udelay(1); + val =3D mtk_pci_read32(mdev, REG_DEV_INFRA_BASE + hw_regs->reg_infra_rst0= _clr); + val |=3D 1 << (REG_CLDMA0_RST_CLR_BIT + drv_info->hw_id); + mtk_pci_write32(mdev, REG_DEV_INFRA_BASE + hw_regs->reg_infra_rst0_clr, v= al); +} + +void mtk_cldma_drv_init(struct cldma_drv_info *drv_info) +{ + struct cldma_hw_regs *hw_regs; + struct mtk_md_dev *mdev; + int base; + u32 val; + + mdev =3D drv_info->mdev; + base =3D drv_info->base_addr; + hw_regs =3D drv_info->hw_regs; + + /* set CLDMA to 64 bit mode GPD */ + val =3D mtk_pci_read32(mdev, base + hw_regs->reg_cldma_ul_cfg); + val =3D (val & (~(0x7 << 5))) | ((0x4) << 5); + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ul_cfg, val); + + val =3D mtk_pci_read32(mdev, base + hw_regs->reg_cldma_so_cfg); + val =3D (val & (~(0x7 << 10))) | ((0x4) << 10) | (1 << 2); + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_so_cfg, val); + + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_rx_work_to_reg_mask_set, = ALLQ); + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ip_busy_to_pcie_mask_set, + ALLQ << 16); + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ip_busy_to_pcie_mask_clr, + ALLQ << 24); + + /* enable interrupt to PCIe */ + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_int_mask, 0); + + /* disable illegal memory check */ + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ul_dummy_0, 1); + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_so_dummy_0, 1); +} + +void mtk_cldma_setup_start_addr(struct cldma_drv_info *drv_info, enum mtk_= tx_rx dir, + u32 qno, dma_addr_t addr) +{ + struct cldma_hw_regs *hw_regs; + unsigned int addr_l; + unsigned int addr_h; + int base; + + hw_regs =3D drv_info->hw_regs; + base =3D drv_info->base_addr; + + if (dir =3D=3D DIR_TX) { + addr_l =3D base + hw_regs->reg_cldma_ul_start_addrl_0 + qno * HW_QUEUE_N= UM; + addr_h =3D base + hw_regs->reg_cldma_ul_start_addrh_0 + qno * HW_QUEUE_N= UM; + } else { + addr_l =3D base + hw_regs->reg_cldma_so_start_addrl_0 + qno * HW_QUEUE_N= UM; + addr_h =3D base + hw_regs->reg_cldma_so_start_addrh_0 + qno * HW_QUEUE_N= UM; + } + + mtk_pci_write32(drv_info->mdev, addr_l, (u32)addr); + mtk_pci_write32(drv_info->mdev, addr_h, (u32)((u64)addr >> 32)); +} + +void mtk_cldma_mask_intr(struct cldma_drv_info *drv_info, enum mtk_tx_rx d= ir, + u32 qno, enum mtk_intr_type type) +{ + struct cldma_hw_regs *hw_regs; + int base; + u32 addr; + u32 val; + + hw_regs =3D drv_info->hw_regs; + base =3D drv_info->base_addr; + + if (dir =3D=3D DIR_TX) + addr =3D base + hw_regs->reg_cldma_l2timsr0; + else + addr =3D base + hw_regs->reg_cldma_l2rimsr0; + + if (qno =3D=3D ALLQ) + val =3D qno << type; + else + val =3D BIT(qno) << type; + + mtk_pci_write32(drv_info->mdev, addr, val); +} + +void mtk_cldma_unmask_intr(struct cldma_drv_info *drv_info, enum mtk_tx_rx= dir, + u32 qno, enum mtk_intr_type type) +{ + struct cldma_hw_regs *hw_regs; + int base; + u32 addr; + u32 val; + + hw_regs =3D drv_info->hw_regs; + base =3D drv_info->base_addr; + + if (dir =3D=3D DIR_TX) + addr =3D base + hw_regs->reg_cldma_l2timcr0; + else + addr =3D base + hw_regs->reg_cldma_l2rimcr0; + + if (qno =3D=3D ALLQ) + val =3D qno << type; + else + val =3D BIT(qno) << type; + + mtk_pci_write32(drv_info->mdev, addr, val); +} + +void mtk_cldma_clr_intr_status(struct cldma_drv_info *drv_info, enum mtk_t= x_rx dir, + u32 qno, enum mtk_intr_type type) +{ + struct cldma_hw_regs *hw_regs; + struct mtk_md_dev *mdev; + int base; + u32 addr; + u32 val; + + hw_regs =3D drv_info->hw_regs; + base =3D drv_info->base_addr; + mdev =3D drv_info->mdev; + + if (type =3D=3D QUEUE_ERROR) { + if (dir =3D=3D DIR_TX) { + val =3D mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l3tisar0); + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l3tisar0, val); + val =3D mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l3tisar1); + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l3tisar1, val); + val =3D mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l3tisar2); + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l3tisar2, val); + } else { + val =3D mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l3risar0); + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l3risar0, val); + val =3D mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l3risar1); + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l3risar1, val); + } + } + + if (dir =3D=3D DIR_TX) + addr =3D base + hw_regs->reg_cldma_l2tisar0; + else + addr =3D base + hw_regs->reg_cldma_l2risar0; + + if (qno =3D=3D ALLQ) + val =3D qno << type; + else + val =3D BIT(qno) << type; + + mtk_pci_write32(mdev, addr, val); + val =3D mtk_pci_read32(mdev, addr); +} + +u32 mtk_cldma_check_intr_status(struct cldma_drv_info *drv_info, enum mtk_= tx_rx dir, + u32 qno, enum mtk_intr_type type) +{ + struct cldma_hw_regs *hw_regs; + u32 addr, val, sta; + int base; + + hw_regs =3D drv_info->hw_regs; + base =3D drv_info->base_addr; + + if (dir =3D=3D DIR_TX) + addr =3D base + hw_regs->reg_cldma_l2tisar0; + else + addr =3D base + hw_regs->reg_cldma_l2risar0; + + val =3D mtk_pci_read32(drv_info->mdev, addr); + if (val =3D=3D LINK_ERROR_VAL) + sta =3D val; + else if (qno =3D=3D ALLQ) + sta =3D (val >> type) & 0xFF; + else + sta =3D (val >> type) & BIT(qno); + + return sta; +} + +void mtk_cldma_start_queue(struct cldma_drv_info *drv_info, enum mtk_tx_rx= dir, u32 qno) +{ + struct cldma_hw_regs *hw_regs; + u32 val =3D BIT(qno); + int base; + u32 addr; + + hw_regs =3D drv_info->hw_regs; + base =3D drv_info->base_addr; + + if (dir =3D=3D DIR_TX) + addr =3D base + hw_regs->reg_cldma_ul_start_cmd; + else + addr =3D base + hw_regs->reg_cldma_so_start_cmd; + + mtk_pci_write32(drv_info->mdev, addr, val); +} + +void mtk_cldma_resume_queue(struct cldma_drv_info *drv_info, enum mtk_tx_r= x dir, u32 qno) +{ + struct cldma_hw_regs *hw_regs; + u32 val =3D BIT(qno); + int base; + u32 addr; + + hw_regs =3D drv_info->hw_regs; + base =3D drv_info->base_addr; + + if (dir =3D=3D DIR_TX) + addr =3D base + hw_regs->reg_cldma_ul_resume_cmd; + else + addr =3D base + hw_regs->reg_cldma_so_resume_cmd; + + mtk_pci_write32(drv_info->mdev, addr, val); +} + +static u32 mtk_cldma_queue_status(struct cldma_drv_info *drv_info, enum mt= k_tx_rx dir, u32 qno) +{ + struct cldma_hw_regs *hw_regs; + int base; + u32 addr; + u32 val; + + hw_regs =3D drv_info->hw_regs; + base =3D drv_info->base_addr; + + if (dir =3D=3D DIR_TX) + addr =3D base + hw_regs->reg_cldma_ul_status; + else + addr =3D base + hw_regs->reg_cldma_so_status; + + val =3D mtk_pci_read32(drv_info->mdev, addr); + + if (qno =3D=3D ALLQ || val =3D=3D LINK_ERROR_VAL) + return val; + + return val & BIT(qno); +} + +int mtk_cldma_stop_queue(struct cldma_drv_info *drv_info, enum mtk_tx_rx d= ir, u32 qno) +{ + u32 val =3D (qno =3D=3D ALLQ) ? qno : BIT(qno); + struct cldma_hw_regs *hw_regs; + unsigned int active; + int base; + u32 addr; + + hw_regs =3D drv_info->hw_regs; + base =3D drv_info->base_addr; + + if (dir =3D=3D DIR_TX) + addr =3D base + hw_regs->reg_cldma_ul_stop_cmd; + else + addr =3D base + hw_regs->reg_cldma_so_stop_cmd; + + mtk_pci_write32(drv_info->mdev, addr, val); + + if (read_poll_timeout(mtk_cldma_queue_status, active, + active =3D=3D LINK_ERROR_VAL || !active, + WAIT_QUEUE_STOP, WAIT_QUEUE_STOP * 10, false, + drv_info, dir, qno)) + return -ETIMEDOUT; + + /* An all-ones read means the link is dead, not that the queue + * stopped; let the caller tell the two apart. + */ + if (active =3D=3D LINK_ERROR_VAL) + return -ENODEV; + + return 0; +} + +void mtk_cldma_clear_ip_busy(struct cldma_drv_info *drv_info) +{ + mtk_pci_write32(drv_info->mdev, drv_info->base_addr + + drv_info->hw_regs->reg_cldma_ip_busy, 0x01); +} + +void mtk_cldma_get_intr_status(struct cldma_drv_info *drv_info, u32 *tx_st= a, u32 *rx_sta) +{ + struct cldma_hw_regs *hw_regs; + struct mtk_md_dev *mdev; + u32 tx_mask, rx_mask; + int base; + + mdev =3D drv_info->mdev; + base =3D drv_info->base_addr; + hw_regs =3D drv_info->hw_regs; + + *tx_sta =3D mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l2tisar0); + tx_mask =3D mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l2timr0); + *rx_sta =3D mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l2risar0); + rx_mask =3D mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l2rimr0); + + *tx_sta =3D (*tx_sta) & (~tx_mask); + *rx_sta =3D (*rx_sta) & (~rx_mask); + + if (*tx_sta) { + /* TX XFER_DONE and QUEUE_ERROR mask */ + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l2timsr0, *tx_sta); + /* TX XFER_DONE clear */ + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l2tisar0, + (*tx_sta) & (0xFF << QUEUE_XFER_DONE)); + } + + if (*rx_sta) { + /* RX XFER_DONE and QUEUE_ERROR mask */ + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l2rimsr0, *rx_sta); + /* RX XFER_DONE clear */ + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l2risar0, + (*rx_sta) & (0xFF << QUEUE_XFER_DONE)); + } +} + +u64 mtk_cldma_get_tx_start_addr(struct cldma_drv_info *drv_info, u32 qno) +{ + struct cldma_hw_regs *hw_regs =3D drv_info->hw_regs; + struct mtk_md_dev *mdev =3D drv_info->mdev; + int base =3D drv_info->base_addr; + u32 addr_h, addr_l; + + addr_l =3D mtk_pci_read32(mdev, + base + hw_regs->reg_cldma_ul_start_addrl_0 + qno * HW_QUEUE_NUM); + addr_h =3D mtk_pci_read32(mdev, + base + hw_regs->reg_cldma_ul_start_addrh_0 + qno * HW_QUEUE_NUM); + + return ((u64)addr_h << 32) | addr_l; +} + +u64 mtk_cldma_get_rx_curr_addr(struct cldma_drv_info *drv_info, u32 qno) +{ + struct cldma_hw_regs *hw_regs; + u32 curr_addr_h, curr_addr_l; + struct mtk_md_dev *mdev; + u64 curr_addr; + int base; + u64 addr; + + hw_regs =3D drv_info->hw_regs; + base =3D drv_info->base_addr; + mdev =3D drv_info->mdev; + + addr =3D base + hw_regs->reg_cldma_so_current_addrh_0 + + (u64)qno * HW_QUEUE_NUM; + curr_addr_h =3D mtk_pci_read32(mdev, addr); + addr =3D base + hw_regs->reg_cldma_so_current_addrl_0 + + (u64)qno * HW_QUEUE_NUM; + curr_addr_l =3D mtk_pci_read32(mdev, addr); + curr_addr =3D ((u64)curr_addr_h << 32) | curr_addr_l; + if (curr_addr_h =3D=3D LINK_ERROR_VAL && curr_addr_l =3D=3D LINK_ERROR_VA= L) + curr_addr =3D 0; + return curr_addr; +} diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h b/drivers/net/wwan/= t9xx/pcie/mtk_cldma_drv.h new file mode 100644 index 000000000000..d7abf4b48895 --- /dev/null +++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h @@ -0,0 +1,135 @@ +/* SPDX-License-Identifier: GPL-2.0-only + * + * Copyright (c) 2023, MediaTek Inc. + */ + +#ifndef __MTK_CLDMA_DRV_H__ +#define __MTK_CLDMA_DRV_H__ + +#define HW_QUEUE_NUM (8) +#define ALLQ (0xFF) +#define LINK_ERROR_VAL (0xFFFFFFFF) +#define CLDMA0_HW_ID (0) +#define CLDMA1_HW_ID (1) + +struct cldma_hw_regs { + u8 cldma_rx_skb_pool_max_size; + u8 cldma_rx_skb_reload_threshold; + u8 tq_err_int_offset; + u8 tq_active_start_err_int_offset; + u8 rq_err_int_offset; + u8 rq_active_start_err_int_offset; + u16 reg_cldma_so_cfg; + u16 reg_cldma_so_start_addrl_0; + u16 reg_cldma_so_start_addrh_0; + u16 reg_cldma_so_current_addrl_0; + u16 reg_cldma_so_current_addrh_0; + u16 reg_cldma_so_status; + u16 reg_cldma_debug_id_en; + u16 reg_cldma_so_last_update_addrl_0; + u16 reg_cldma_so_last_update_addrh_0; + u16 reg_cldma_l2rimr0; + u16 reg_cldma_l2rimr1; + u16 reg_cldma_l2rimcr0; + u16 reg_cldma_l2rimcr1; + u16 reg_cldma_l2rimsr0; + u16 reg_cldma_l2rimsr1; + u16 reg_cldma_int_mask; + u16 reg_cldma_ip_busy_to_pcie_mask; + u16 reg_cldma_ip_busy_to_pcie_mask_set; + u16 reg_cldma_ip_busy_to_pcie_mask_clr; + u16 reg_cldma_ip_busy_to_ap_mask; + u16 reg_cldma_ip_busy_to_ap_mask_set; + u16 reg_cldma_ip_busy_to_ap_mask_clr; + u16 reg_cldma_ip_busy_to_md_mask_set; + u16 reg_cldma_rx_work_to_reg_mask_set; + u16 reg_infra_rst0_set; + u16 reg_infra_rst0_clr; + u32 tq_err_int_bitmask; + u32 tq_active_start_err_int_bitmask; + u32 rq_err_int_bitmask; + u32 cldma0_base_addr; + u32 cldma1_base_addr; + u32 rq_active_start_err_int_bitmask; + u32 reg_cldma_ul_start_addrl_0; + u32 reg_cldma_ul_start_addrh_0; + u32 reg_cldma_ul_current_addrl_0; + u32 reg_cldma_ul_current_addrh_0; + u32 reg_cldma_ul_status; + u32 reg_cldma_ul_start_cmd; + u32 reg_cldma_ul_resume_cmd; + u32 reg_cldma_ul_stop_cmd; + u32 reg_cldma_ul_error; + u32 reg_cldma_ul_cfg; + u32 reg_cldma_ul_dummy_0; + u32 reg_cldma_so_error; + u32 reg_cldma_so_start_cmd; + u32 reg_cldma_so_resume_cmd; + u32 reg_cldma_so_stop_cmd; + u32 reg_cldma_so_dummy_0; + u32 reg_cldma_l2tisar0; + u32 reg_cldma_l2tisar1; + u32 reg_cldma_l2timr0; + u32 reg_cldma_l2timr1; + u32 reg_cldma_l2timcr0; + u32 reg_cldma_l2timcr1; + u32 reg_cldma_l2timsr0; + u32 reg_cldma_l2timsr1; + u32 reg_cldma_l2risar0; + u32 reg_cldma_l2risar1; + u32 reg_cldma_l3tisar0; + u32 reg_cldma_l3tisar1; + u32 reg_cldma_l3tisar2; + u32 reg_cldma_l3risar0; + u32 reg_cldma_l3risar1; + u32 reg_cldma_ip_busy; +}; + +enum mtk_intr_type { + QUEUE_XFER_DONE =3D 0, + QUEUE_ERROR =3D 16, +}; + +enum mtk_tx_rx { + DIR_TX, + DIR_RX, +}; + +struct cldma_drv_info { + int hif_id; + int hw_id; + int base_addr; + int pci_ext_irq_id; + struct mtk_md_dev *mdev; + struct cldma_dev *cd; + struct txq *txq[HW_QUEUE_NUM]; + struct rxq *rxq[HW_QUEUE_NUM]; + struct dma_pool *gpd_dma_pool; + struct dma_pool *bd_dma_pool; + struct workqueue_struct *wq; + struct cldma_hw_regs *hw_regs; +}; + +extern struct cldma_hw_regs mtk_cldma_regs_m9xx; + +void mtk_cldma_drv_init(struct cldma_drv_info *drv_info); +void mtk_cldma_drv_reset(struct cldma_drv_info *drv_info); +void mtk_cldma_setup_start_addr(struct cldma_drv_info *drv_info, enum mtk_= tx_rx dir, + u32 qno, dma_addr_t addr); +void mtk_cldma_mask_intr(struct cldma_drv_info *drv_info, enum mtk_tx_rx d= ir, + u32 qno, enum mtk_intr_type type); +void mtk_cldma_unmask_intr(struct cldma_drv_info *drv_info, enum mtk_tx_rx= dir, + u32 qno, enum mtk_intr_type type); +void mtk_cldma_clr_intr_status(struct cldma_drv_info *drv_info, enum mtk_t= x_rx dir, + u32 qno, enum mtk_intr_type type); +u32 mtk_cldma_check_intr_status(struct cldma_drv_info *drv_info, enum mtk_= tx_rx dir, + u32 qno, enum mtk_intr_type type); +void mtk_cldma_start_queue(struct cldma_drv_info *drv_info, enum mtk_tx_rx= dir, u32 qno); +void mtk_cldma_resume_queue(struct cldma_drv_info *drv_info, enum mtk_tx_r= x dir, u32 qno); +int mtk_cldma_stop_queue(struct cldma_drv_info *drv_info, enum mtk_tx_rx d= ir, u32 qno); +void mtk_cldma_clear_ip_busy(struct cldma_drv_info *drv_info); +void mtk_cldma_get_intr_status(struct cldma_drv_info *drv_info, u32 *tx_st= a, u32 *rx_sta); +u64 mtk_cldma_get_tx_start_addr(struct cldma_drv_info *drv_info, u32 qno); +u64 mtk_cldma_get_rx_curr_addr(struct cldma_drv_info *drv_info, u32 qno); + +#endif diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.c b/drivers/net/wwan/t9xx/p= cie/mtk_pci.c index c2c2b824ff31..6c31a5d80693 100644 --- a/drivers/net/wwan/t9xx/pcie/mtk_pci.c +++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.c @@ -872,6 +872,29 @@ static void mtk_pci_free_irq(struct mtk_md_dev *mdev) pci_free_irq_vectors(pdev); } =20 +static int mtk_pci_dev_init(struct mtk_md_dev *mdev) +{ + int ret; + + ret =3D mtk_trans_ctrl_init(mdev); + if (ret) { + dev_err(mdev->dev, "Failed to initialize control plane: %d\n", ret); + return ret; + } + + return 0; +} + +static void mtk_pci_dev_exit(struct mtk_md_dev *mdev) +{ + mtk_trans_ctrl_exit(mdev); +} + +static int mtk_pci_dev_start(struct mtk_md_dev *mdev) +{ + return 0; +} + static int mtk_pci_probe(struct pci_dev *pdev, const struct pci_device_id = *id) { struct device *dev =3D &pdev->dev; @@ -938,6 +961,12 @@ static int mtk_pci_probe(struct pci_dev *pdev, const s= truct pci_device_id *id) if (ret) goto free_mhccif; =20 + ret =3D mtk_pci_dev_init(mdev); + if (ret) { + dev_err(mdev->dev, "Failed to init dev.\n"); + goto free_irq; + } + pci_set_master(pdev); mtk_pci_unmask_irq(mdev, priv->mhccif_irq_id); =20 @@ -958,10 +987,20 @@ static int mtk_pci_probe(struct pci_dev *pdev, const = struct pci_device_id *id) goto clear_master; } =20 + ret =3D mtk_pci_dev_start(mdev); + if (ret) { + dev_err(mdev->dev, "Failed to start dev.\n"); + goto free_saved_state; + } + return 0; =20 +free_saved_state: + pci_load_and_free_saved_state(pdev, &priv->saved_state); clear_master: pci_clear_master(pdev); + mtk_pci_dev_exit(mdev); +free_irq: mtk_pci_free_irq(mdev); free_mhccif: mtk_mhccif_exit(mdev); @@ -980,6 +1019,8 @@ static void mtk_pci_remove(struct pci_dev *pdev) struct device *dev =3D &pdev->dev; int ret; =20 + mtk_pci_dev_exit(mdev); + /* Silence every source before tearing anything down. */ mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_CLR_GRP0_0, U32_MAX); =20 diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h b/drivers/net/wwan/t9= xx/pcie/mtk_pci_reg.h index a97ad6630dcf..f65a8065d945 100644 --- a/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h +++ b/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h @@ -16,6 +16,7 @@ #define REG_IMASK_HOST_MSIX_SET_GRP0_0 0x3000 #define REG_IMASK_HOST_MSIX_CLR_GRP0_0 0x3080 #define REG_IMASK_HOST_MSIX_GRP0_0 0x3100 +#define REG_DEV_INFRA_BASE 0x10001000 =20 /* mhccif registers */ #define MHCCIF_RC2EP_SW_BSY 0x4 diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c b/drivers/net/wwan= /t9xx/pcie/mtk_trans_ctrl.c new file mode 100644 index 000000000000..531940b06e2a --- /dev/null +++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c @@ -0,0 +1,615 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022, MediaTek Inc. + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "mtk_cldma.h" +#include "mtk_ctrl_plane.h" +#include "mtk_dev.h" +#include "mtk_pci.h" +#include "mtk_trans_ctrl.h" + +#define QUEUE_CHL_MASK 0xFFFF +#define TRB_SRV_NUM (1) + +static const int mtk_srv_cfg[NR_CLDMA][HW_QUE_NUM] =3D { + {0}, + {0}, +}; + +/* the number of RX GPDs should be at least two */ +static const struct queue_info mtk_queue_info[] =3D { + {CCCI_CONTROL_TX, CCCI_CONTROL_RX, CLDMA1, TXQ(0), RXQ(0), + Q_MTU_3_5K, Q_MTU_3_5K, TX_GPD_NUM, RX_GPD_NUM, Q_FRAG_3_5K, Q_FRAG_3_5K= , 0}, + {CCCI_SAP_CONTROL_TX, CCCI_SAP_CONTROL_RX, CLDMA0, TXQ(0), RXQ(0), + Q_MTU_3_5K, Q_MTU_3_5K, TX_GPD_NUM, RX_GPD_NUM, Q_FRAG_3_5K, Q_FRAG_3_5K= , 0}, +}; + +static bool mtk_queue_list_is_full(struct mtk_ctrl_trans *trans, struct qu= eue_info *que) +{ + return skb_queue_len_lockless(&trans->trans_list[que->hif_id].skb_list[qu= e->txqno]) >=3D + SKB_LIST_MAX_LEN; +} + +static bool mtk_ctrl_chs_is_busy_or_empty(struct trb_srv *srv) +{ + struct srv_que *srv_que; + int i; + + for (i =3D 0; i < NR_CLDMA; i++) { + list_for_each_entry(srv_que, &srv->srv_q_list[i], list) { + struct sk_buff *skb; + struct trb *trb; + + skb =3D skb_peek(&srv->trans->trans_list[i].skb_list[srv_que->qno]); + if (!skb) + continue; + + /* ENABLE and DISABLE are software-only and are queued at + * the head, so gating them on TX budget would make a queue + * that cannot drain impossible to close. + */ + trb =3D (struct trb *)skb->cb; + if (trb->cmd !=3D TRB_CMD_TX || + mtk_cldma_get_tx_budget(srv->trans->dev, i, srv_que->qno)) + return false; + } + } + + return true; +} + +static void mtk_ctrl_ch_flush(struct sk_buff_head *skb_list) +{ + struct sk_buff *skb; + struct trb *trb; + + while (!skb_queue_empty(skb_list)) { + skb =3D skb_dequeue(skb_list); + trb =3D (struct trb *)skb->cb; + trb->status =3D -EIO; + trb->trb_complete(skb); + } +} + +static void mtk_ctrl_chs_flush(struct trb_srv *srv) +{ + struct srv_que *srv_que; + int i; + + for (i =3D 0; i < NR_CLDMA; i++) + list_for_each_entry(srv_que, &srv->srv_q_list[i], list) + mtk_ctrl_ch_flush(&srv->trans->trans_list[i].skb_list[srv_que->qno]); +} + +static int mtk_ch_status_check(struct mtk_ctrl_trans *trans, struct sk_buf= f *skb) +{ + struct trb *trb =3D (struct trb *)skb->cb; + struct trb_open_priv *trb_open_priv; + struct queue_info *que; + int ret =3D 0; + + que =3D radix_tree_lookup(&trans->queue_tbl, trb->channel_id & QUEUE_CHL_= MASK); + + switch (trb->cmd) { + case TRB_CMD_ENABLE: + trb_open_priv =3D (struct trb_open_priv *)skb->data; + trb_open_priv->log_rg_offset =3D que->log_rg_offset; + trans->usr_cnt[que->hif_id][que->txqno]++; + if (trans->usr_cnt[que->hif_id][que->txqno] =3D=3D 1) + break; + trb_open_priv->tx_mtu =3D que->tx_mtu; + trb_open_priv->rx_mtu =3D que->rx_mtu; + trb_open_priv->tx_frag_size =3D que->tx_frag_size; + trb_open_priv->rx_frag_size =3D que->rx_frag_size; + if (mtk_cldma_check_ch_cfg(trans->dev, que)) { + trb->status =3D -EINVAL; + ret =3D -EINVAL; + } else { + trb->status =3D -EBUSY; + ret =3D -EBUSY; + } + trb->trb_complete(skb); + break; + case TRB_CMD_DISABLE: + if (trans->usr_cnt[que->hif_id][que->txqno] > 0) { + trans->usr_cnt[que->hif_id][que->txqno]--; + if (!trans->usr_cnt[que->hif_id][que->txqno]) + break; + } + trb->status =3D -EBUSY; + trb->trb_complete(skb); + ret =3D -EBUSY; + break; + default: + dev_err((trans->mdev)->dev, "Invalid trb command(%d)\n", trb->cmd); + ret =3D -EINVAL; + break; + } + return ret; +} + +/* Single consumer per srv_que =E2=80=94 only this kthread dequeues from s= kb_list. + * The list lock is held around every list read (peek, is_last, peek_next, + * unlink) so a producer inserting a DISABLE at the head cannot race the + * traversal, but it is dropped across submit and dispatch: those paths + * may allocate with GFP_KERNEL and thus sleep. Dropping the lock there + * is safe because no other consumer can steal the peeked skb. + */ +static void mtk_ctrl_trb_handler(struct trb_srv *srv, struct trans_list *t= rans_list, u32 qno) +{ + struct sk_buff_head *skb_list =3D &trans_list->skb_list[qno]; + struct mtk_ctrl_trans *trans =3D srv->trans; + struct sk_buff *skb, *skb_next; + struct trb *trb, *trb_next; + unsigned long flags; + bool kick =3D false; + int loop =3D 0; + int err; + + do { + spin_lock_irqsave(&skb_list->lock, flags); + skb =3D skb_peek(skb_list); + if (!skb) { + spin_unlock_irqrestore(&skb_list->lock, flags); + break; + } + trb =3D (struct trb *)skb->cb; + + switch (trb->cmd) { + case TRB_CMD_ENABLE: + case TRB_CMD_DISABLE: + __skb_unlink(skb, skb_list); + spin_unlock_irqrestore(&skb_list->lock, flags); + err =3D mtk_ch_status_check(trans, skb); + if (!err) { + kick =3D true; + if (trb->cmd =3D=3D TRB_CMD_DISABLE) + mtk_ctrl_ch_flush(skb_list); + } + break; + case TRB_CMD_TX: + spin_unlock_irqrestore(&skb_list->lock, flags); + err =3D mtk_cldma_submit_tx(trans->dev, skb); + if (err) { + if (trans_list->tx_burst_cnt[qno]) { + kick =3D true; + break; + } + if (err =3D=3D -EAGAIN) + return; + + skb_unlink(skb, skb_list); + trb->status =3D err; + trb->trb_complete(skb); + break; + } + + trans_list->tx_burst_cnt[qno]++; + spin_lock_irqsave(&skb_list->lock, flags); + if (trans_list->tx_burst_cnt[qno] >=3D TX_BURST_MAX_CNT || + skb_queue_is_last(skb_list, skb)) { + kick =3D true; + } else { + skb_next =3D skb_peek_next(skb, skb_list); + trb_next =3D (struct trb *)skb_next->cb; + if (trb_next->cmd !=3D TRB_CMD_TX) + kick =3D true; + } + + __skb_unlink(skb, skb_list); + spin_unlock_irqrestore(&skb_list->lock, flags); + break; + default: + __skb_unlink(skb, skb_list); + spin_unlock_irqrestore(&skb_list->lock, flags); + trb->status =3D -EINVAL; + trb->trb_complete(skb); + break; + } + + if (kick) { + err =3D mtk_cldma_trb_process(trans->dev, skb); + if (err) + dev_err_ratelimited((trans->mdev)->dev, + "Failed to process trb on queue %u: %d\n", + qno, err); + trans_list->tx_burst_cnt[qno] =3D 0; + kick =3D false; + } + + loop++; + } while (loop < TRB_NUM_PER_ROUND); +} + +static void mtk_ctrl_trb_process(struct trb_srv *srv) +{ + struct mtk_ctrl_trans *trans =3D srv->trans; + struct srv_que *srv_que; + int i; + + for (i =3D 0; i < NR_CLDMA; i++) + list_for_each_entry(srv_que, &srv->srv_q_list[i], list) + mtk_ctrl_trb_handler(srv, &trans->trans_list[i], srv_que->qno); +} + +static int mtk_ctrl_trb_thread(void *args) +{ + struct trb_srv *srv =3D args; + + for (;;) { + wait_event_interruptible(srv->trb_waitq, + !mtk_ctrl_chs_is_busy_or_empty(srv) || + kthread_should_stop() || kthread_should_park()); + if (kthread_should_stop()) + break; + + if (kthread_should_park()) + kthread_parkme(); + + do { + mtk_ctrl_trb_process(srv); + cond_resched(); + } while (!mtk_ctrl_chs_is_busy_or_empty(srv) && !kthread_should_stop() && + !kthread_should_park()); + } + mtk_ctrl_chs_flush(srv); + return 0; +} + +static int mtk_ctrl_trb_srv_init(struct mtk_ctrl_trans *trans) +{ + struct srv_que *srv_que; + struct trb_srv *srv; + int i, j; + int ret; + + for (i =3D 0; i < trans->trb_srv_num; i++) { + srv =3D kzalloc_obj(*srv); + if (!srv) { + ret =3D -ENOMEM; + goto err_free_srv; + } + + srv->trans =3D trans; + srv->srv_id =3D i; + trans->trb_srv[i] =3D srv; + + init_waitqueue_head(&srv->trb_waitq); + for (j =3D 0; j < NR_CLDMA; j++) + INIT_LIST_HEAD(&srv->srv_q_list[j]); + } + + for (i =3D 0; i < NR_CLDMA; i++) + for (j =3D 0; j < HW_QUE_NUM; j++) { + if (trans->srv_cfg[i][j] < 0 || + trans->srv_cfg[i][j] >=3D trans->trb_srv_num) + trans->srv_cfg[i][j] =3D 0; + srv_que =3D kzalloc_obj(*srv_que); + if (!srv_que) { + ret =3D -ENOMEM; + goto err_free_srv_que; + } + srv_que->hif_id =3D i; + srv_que->qno =3D j; + list_add_tail(&srv_que->list, + &trans->trb_srv[trans->srv_cfg[i][j]]->srv_q_list[i]); + } + + for (i =3D 0; i < trans->trb_srv_num; i++) { + trans->trb_srv[i]->trb_thread =3D kthread_run(mtk_ctrl_trb_thread, trans= ->trb_srv[i], + "mtk_trb_srv%d_%s", i, + trans->mdev->dev_str); + if (IS_ERR(trans->trb_srv[i]->trb_thread)) { + ret =3D PTR_ERR(trans->trb_srv[i]->trb_thread); + trans->trb_srv[i]->trb_thread =3D NULL; + goto err_stop_kthread; + } + } + + return 0; +err_stop_kthread: + while (--i >=3D 0) + kthread_stop(trans->trb_srv[i]->trb_thread); +err_free_srv_que: + for (i =3D 0; i < trans->trb_srv_num; i++) { + for (j =3D 0; j < NR_CLDMA; j++) { + struct srv_que *next_srv_que; + + list_for_each_entry_safe(srv_que, next_srv_que, + &trans->trb_srv[i]->srv_q_list[j], list) { + list_del(&srv_que->list); + kfree(srv_que); + } + } + } +err_free_srv: + for (i =3D 0; i < trans->trb_srv_num; i++) { + if (!trans->trb_srv[i]) + break; + kfree(trans->trb_srv[i]); + trans->trb_srv[i] =3D NULL; + } + + return ret; +} + +static void mtk_ctrl_trb_srv_exit(struct mtk_ctrl_trans *trans) +{ + struct srv_que *srv_que, *next_srv_que; + struct trb_srv *srv; + int i, j; + + for (i =3D 0; i < trans->trb_srv_num; i++) { + srv =3D trans->trb_srv[i]; + if (!srv) + continue; + kthread_stop(srv->trb_thread); + for (j =3D 0; j < NR_CLDMA; j++) { + list_for_each_entry_safe(srv_que, next_srv_que, + &trans->trb_srv[i]->srv_q_list[j], list) { + list_del(&srv_que->list); + kfree(srv_que); + } + } + kfree(srv); + trans->trb_srv[i] =3D NULL; + } +} + +static void mtk_ctrl_remove_radix_tree(struct mtk_ctrl_trans *trans) +{ + struct radix_tree_iter iter; + struct queue_info *queue; + void __rcu **slot; + + radix_tree_for_each_slot(slot, &trans->queue_tbl, &iter, 0) { + queue =3D radix_tree_deref_slot(slot); + if (!queue) + continue; + radix_tree_delete(&trans->queue_tbl, iter.index); + kfree(queue); + } +} + +int mtk_pcie_hif_init(struct mtk_md_dev *mdev) +{ + struct mtk_ctrl_blk *ctrl_blk =3D mdev->ctrl_blk; + struct queue_info *queue, *queue_info; + struct mtk_ctrl_trans *trans; + int i, j; + int ret; + + trans =3D ctrl_blk->ctrl_hw_priv; + trans->ctrl_blk =3D ctrl_blk; + queue_info =3D trans->queue_info; + + INIT_RADIX_TREE(&trans->queue_tbl, GFP_KERNEL); + for (i =3D 0; i < trans->queue_info_num; i++) { + queue =3D kmemdup(queue_info + i, sizeof(*queue), GFP_KERNEL); + if (!queue) { + ret =3D -ENOMEM; + goto err_free_radix_tree; + } + if (queue->txqno >=3D HW_QUE_NUM || queue->rxqno >=3D HW_QUE_NUM || + queue->hif_id >=3D NR_CLDMA) { + dev_err(mdev->dev, "Failed to get correct queue info %x\n", + queue->rx_chl); + kfree(queue); + ret =3D -EINVAL; + goto err_free_radix_tree; + } + ret =3D radix_tree_insert(&trans->queue_tbl, queue->rx_chl & QUEUE_CHL_M= ASK, queue); + if (ret) { + dev_err(mdev->dev, "Insert %x fail, ret: %d", queue->rx_chl, ret); + kfree(queue); + goto err_free_radix_tree; + } + } + + for (i =3D 0; i < NR_CLDMA; i++) { + for (j =3D 0; j < HW_QUE_NUM; j++) { + skb_queue_head_init(&trans->trans_list[i].skb_list[j]); + trans->trans_list[i].tx_burst_cnt[j] =3D 0; + /* usr_cnt tracks the queues rebuilt by mtk_cldma_init() + * below, so it must be reset with them. Otherwise a + * count left over from a torn-down cycle makes the + * channel permanently unopenable. + */ + trans->usr_cnt[i][j] =3D 0; + } + } + ret =3D mtk_cldma_init(trans); + if (ret) + goto err_free_radix_tree; + + ret =3D mtk_ctrl_trb_srv_init(trans); + if (ret) + goto err_cldma_exit; + + atomic_set(&trans->available, 1); + + return 0; + +err_cldma_exit: + mtk_cldma_exit(trans); +err_free_radix_tree: + mtk_ctrl_remove_radix_tree(trans); + + return ret; +} + +int mtk_pcie_hif_exit(struct mtk_md_dev *mdev) +{ + struct mtk_ctrl_blk *ctrl_blk =3D mdev->ctrl_blk; + struct mtk_ctrl_trans *trans; + + trans =3D ctrl_blk->ctrl_hw_priv; + + mutex_lock(&trans->submit_lock); + atomic_set(&trans->available, 0); + mutex_unlock(&trans->submit_lock); + + /* Join the service threads before freeing what they dereference: they + * read trans->dev without a lock, so clearing it cannot stop a + * consumer that has already loaded the pointer. + */ + mtk_ctrl_trb_srv_exit(trans); + mtk_cldma_exit(trans); + + /* Late submitters may still hold the lock and walk the tree. */ + mutex_lock(&trans->submit_lock); + mtk_ctrl_remove_radix_tree(trans); + mutex_unlock(&trans->submit_lock); + + return 0; +} + +int mtk_pcie_hif_submit_skb(struct mtk_md_dev *mdev, struct sk_buff *skb, = bool force_send) +{ + struct mtk_ctrl_blk *ctrl_blk =3D mdev->ctrl_blk; + struct mtk_ctrl_trans *trans; + struct queue_info *que; + struct trb *trb; + int ret; + + trans =3D ctrl_blk->ctrl_hw_priv; + trb =3D (struct trb *)skb->cb; + + if (trb->cmd =3D=3D TRB_CMD_STOP || trb->cmd =3D=3D TRB_CMD_RECOVER) { + trb->trb_complete(skb); + return 0; + } + + mutex_lock(&trans->submit_lock); + + if (!atomic_read(&trans->available)) { + ret =3D -EIO; + goto unlock; + } + + que =3D radix_tree_lookup(&trans->queue_tbl, trb->channel_id & QUEUE_CHL_= MASK); + if (!que) { + dev_warn(mdev->dev, "lookup que fail, ch_id: %x\n", + trb->channel_id); + ret =3D -EINVAL; + goto unlock; + } + + if (mtk_queue_list_is_full(trans, que) && !force_send) { + ret =3D -EAGAIN; + goto unlock; + } + + if (trb->cmd =3D=3D TRB_CMD_DISABLE) { + struct sk_buff *entry =3D NULL; + struct sk_buff_head *list; + struct sk_buff *iter; + unsigned long flags; + + /* A disable may overtake queued data, so teardown does not + * wait for a TX backlog, but it must never overtake a pending + * ENABLE for the same queue: the two do not commute, and a + * DISABLE consumed before its ENABLE closes nothing while the + * ENABLE then arms rings nobody owns. + */ + list =3D &trans->trans_list[que->hif_id].skb_list[que->txqno]; + spin_lock_irqsave(&list->lock, flags); + skb_queue_walk(list, iter) { + if (((struct trb *)iter->cb)->cmd !=3D TRB_CMD_ENABLE) { + entry =3D iter; + break; + } + } + if (entry) + __skb_queue_before(list, entry, skb); + else + __skb_queue_tail(list, skb); + spin_unlock_irqrestore(&list->lock, flags); + } else { + skb_queue_tail(&trans->trans_list[que->hif_id].skb_list[que->txqno], skb= ); + } + + wake_up(&trans->trb_srv[trans->srv_cfg[que->hif_id][que->txqno]]->trb_wai= tq); + ret =3D 0; + +unlock: + mutex_unlock(&trans->submit_lock); + return ret; +} + +int mtk_pcie_hif_cmd_func(struct mtk_md_dev *mdev, int cmd, void *data) +{ + struct mtk_ctrl_blk *ctrl_blk =3D mdev->ctrl_blk; + struct mtk_ctrl_trans *trans; + struct queue_info *que; + int ret; + + switch (cmd) { + case HIF_CTRL_CMD_CHECK_TX_FULL: + trans =3D ctrl_blk->ctrl_hw_priv; + mutex_lock(&trans->submit_lock); + if (!atomic_read(&trans->available)) { + ret =3D -EIO; + break; + } + que =3D radix_tree_lookup(&trans->queue_tbl, + ((union ctrl_hif_cmd_data *)data)->rx_ch & QUEUE_CHL_MASK); + if (!que) { + dev_warn(mdev->dev, "Failed to find que to check tx full\n"); + ret =3D -EINVAL; + break; + } + ret =3D mtk_queue_list_is_full(trans, que); + break; + default: + return -EINVAL; + } + mutex_unlock(&trans->submit_lock); + + return ret; +} + +int mtk_trans_ctrl_init(struct mtk_md_dev *mdev) +{ + struct mtk_ctrl_trans *trans; + struct mtk_ctrl_blk *ctrl_blk; + int err; + + trans =3D devm_kzalloc(mdev->dev, sizeof(*trans), GFP_KERNEL); + if (!trans) + return -ENOMEM; + trans->mdev =3D mdev; + mutex_init(&trans->submit_lock); + atomic_set(&trans->available, 0); + + memcpy(trans->srv_cfg, mtk_srv_cfg, sizeof(mtk_srv_cfg)); + trans->queue_info =3D (struct queue_info *)mtk_queue_info; + trans->queue_info_num =3D ARRAY_SIZE(mtk_queue_info); + trans->trb_srv_num =3D TRB_SRV_NUM; + + err =3D mtk_ctrl_init(mdev); + if (err) + return err; + + ctrl_blk =3D mdev->ctrl_blk; + ctrl_blk->ctrl_hw_priv =3D trans; + + return 0; +} + +int mtk_trans_ctrl_exit(struct mtk_md_dev *mdev) +{ + mtk_ctrl_exit(mdev); + + return 0; +} diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h b/drivers/net/wwan= /t9xx/pcie/mtk_trans_ctrl.h index d6de4c43b529..f9144c60edf3 100644 --- a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h +++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h @@ -8,14 +8,82 @@ =20 #include #include +#include #include #include =20 #include "mtk_dev.h" =20 +#define TRB_SRV_MAX_NUM (1) +#define HW_QUE_NUM (8) +#define TX_GPD_NUM (16) +#define RX_GPD_NUM (TX_GPD_NUM) +#define MIN_GPD_NUM (2) +#define SKB_LIST_MAX_LEN (16) +#define TRB_NUM_PER_ROUND (TX_GPD_NUM) +#define TX_BURST_MAX_CNT (TX_GPD_NUM / 4 + 1) + +enum mtk_hif_id { + CLDMA0, + CLDMA1, + NR_CLDMA +}; + +struct queue_info { + u32 tx_chl; + u32 rx_chl; + enum mtk_hif_id hif_id; + u32 txqno; + u32 rxqno; + u32 tx_mtu; + u32 rx_mtu; + u32 tx_nr_gpds; + u32 rx_nr_gpds; + u32 tx_frag_size; + u32 rx_frag_size; + u8 log_rg_offset; +}; + +struct trans_list { + struct sk_buff_head skb_list[HW_QUE_NUM]; + u8 tx_burst_cnt[HW_QUE_NUM]; +}; + struct mtk_ctrl_trans { struct mtk_ctrl_blk *ctrl_blk; + struct trb_srv *trb_srv[TRB_SRV_MAX_NUM]; + struct trans_list trans_list[NR_CLDMA]; + void *dev; + struct radix_tree_root queue_tbl; struct mtk_md_dev *mdev; + int usr_cnt[NR_CLDMA][HW_QUE_NUM]; + struct mutex submit_lock; /* protects available flag and submit/exit path= s */ + atomic_t available; + int srv_cfg[NR_CLDMA][HW_QUE_NUM]; + struct queue_info *queue_info; + int queue_info_num; + int trb_srv_num; +}; + +struct srv_que { + u32 hif_id; + u32 qno; + struct list_head list; }; =20 +struct trb_srv { + u32 srv_id; + struct list_head srv_q_list[NR_CLDMA]; + struct mtk_ctrl_trans *trans; + wait_queue_head_t trb_waitq; + struct task_struct *trb_thread; +}; + +int mtk_pcie_hif_init(struct mtk_md_dev *mdev); +int mtk_pcie_hif_exit(struct mtk_md_dev *mdev); +int mtk_pcie_hif_submit_skb(struct mtk_md_dev *mdev, struct sk_buff *skb, = bool force_send); +int mtk_pcie_hif_cmd_func(struct mtk_md_dev *mdev, int cmd, void *data); +int mtk_trans_ctrl_init(struct mtk_md_dev *mdev); +int mtk_trans_ctrl_exit(struct mtk_md_dev *mdev); + #endif --=20 2.34.1 From nobody Wed Sep 30 11:25:46 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DCB7545562A; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790754373; cv=none; b=smA3t0c7Eyq8VUJFHgqT+txvVSN59ivrXs51LvNYzWqS7gD7gB9ue/OalhYVCp9E2EkDIG6Rxxluaqp3S3mG6nn5+k+EhgbMvVcBmLSYK0DhtWHGpoXFU03Dh+3OZEOgPLXLcUcuNhQ7bxiOqjCfLtTI+ad0jV/uV6XHq3cfnHM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790754373; c=relaxed/simple; bh=IE22r1C/iN6BMq2BoHJ3u4IncyesF4EhaqyICiOI+Xs=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=YLcBO6ryccY0/n7kvKKLHhtHlq3kAaUoiih3QM4dKq78fhu/UnHdt6fB8PTYnjMVV4BUyKf+ElqQoJ6QgIbjFiQxZKXyJqCmt5UpiHZVsThRHh3mZIBiwAhFsVMc3FyOV6QgWSpYJtWgihS3jNoUxmhbDYhg9YrOPPPiRjkSMYA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=igkMprGU; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="igkMprGU" Received: by smtp.kernel.org (Postfix) with ESMTPS id 8B1ABC4AF49; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1790754372; bh=IE22r1C/iN6BMq2BoHJ3u4IncyesF4EhaqyICiOI+Xs=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=igkMprGU5Q/km2oPl0uBh7rYbRSNKa6F76QW+C+wibXvaZHl8lDmYIRv6C7WkUbfT TgiCgtSzr+j8hxA7d8fDaOE2HJAZpgeCaVYYvFjSliWMPSV2lVBZpvfuu7hemCx/YJ FLtZOU6cAfQ4UL1ws3NyojO/Lerlpo2z0kQ1UKk2q0oNHLoGaqbl1i11ErzGRXZzAy jvoNCvg8y44lRb3Jq5RfQffdWid8fK5eSUA9TMRXIpt4HSKZzO/7L4qRrs+JuMrmx4 EEtSdQ2ONIaiM/Fo7SjRjSgQfvNokbuNcgTuZxteF/aV/s6t1nXSUWpMQLlAyiGflP BlH+jnu0KHGhQ== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7413ACA5FBE; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) From: Jack Wu via B4 Relay Date: Wed, 30 Sep 2026 15:46:11 +0800 Subject: [PATCH v9 4/6] net: wwan: t9xx: Add control port Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260930-t9xx_driver_v1-v9-4-3248b9256956@compal.com> References: <20260930-t9xx_driver_v1-v9-0-3248b9256956@compal.com> In-Reply-To: <20260930-t9xx_driver_v1-v9-0-3248b9256956@compal.com> To: Loic Poulain , Sergey Ryazanov , Johannes Berg , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jack Wu , Wen-Zhi Huang , Shi-Wei Yeh , Minano Tseng , Matthias Brugger , AngeloGioacchino Del Regno , Simon Horman , Jonathan Corbet , Shuah Khan , Robert Yu , Jeff Chang Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, linux-doc@vger.kernel.org X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1790754369; l=37936; i=jackbb_wu@compal.com; s=20260526; h=from:subject:message-id; bh=HnOTZ0DVGXPbKNaMuGqGIblYQ2jH9F3f/l+fOiWrzSs=; b=yN/4bfsCWm3SS/ts/vfEWe3NJVZ8MAGig9bhzIHlAliTX94ce24NEz6D4YUqp4979yTNGzsnf f3HUSdmxVkjAJjdzbwQuKr+XBrrYc3cdzdGqjXwTwULN0BxxTl848cZ X-Developer-Key: i=jackbb_wu@compal.com; a=ed25519; pk=VH1prTWixNl8OEUPPSfII3p46MzJpQN8J3+ecE1tZXg= X-Endpoint-Received: by B4 Relay for jackbb_wu@compal.com/20260526 with auth_id=793 X-Original-From: Jack Wu Reply-To: jackbb_wu@compal.com From: Jack Wu The control port consists of port I/O and port manager. Port I/O provides a common operation as defined by "struct port_ops", and the operation is managed by the "port manager". It provides interfaces to internal users, the implemented internal interfaces are open, close, write and recv_register. The port manager defines and implements port management interfaces and structures. It is responsible for port creation, destroying, and managing port states. It sends data from port I/O to CLDMA via TRB ( Transaction Request Block ), and dispatches received data from CLDMA to port I/O. Signed-off-by: Jack Wu --- drivers/net/wwan/t9xx/Makefile | 2 + drivers/net/wwan/t9xx/mtk_ctrl_plane.c | 18 +- drivers/net/wwan/t9xx/mtk_ctrl_plane.h | 10 +- drivers/net/wwan/t9xx/mtk_port.c | 693 ++++++++++++++++++++++++= ++++ drivers/net/wwan/t9xx/mtk_port.h | 131 ++++++ drivers/net/wwan/t9xx/mtk_port_io.c | 258 +++++++++++ drivers/net/wwan/t9xx/mtk_port_io.h | 35 ++ drivers/net/wwan/t9xx/pcie/mtk_pci.c | 25 +- drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c | 22 +- drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h | 1 + 10 files changed, 1187 insertions(+), 8 deletions(-) diff --git a/drivers/net/wwan/t9xx/Makefile b/drivers/net/wwan/t9xx/Makefile index 74bef150e3a1..d44266009362 100644 --- a/drivers/net/wwan/t9xx/Makefile +++ b/drivers/net/wwan/t9xx/Makefile @@ -7,6 +7,8 @@ obj-$(CONFIG_MTK_T9XX) +=3D mtk_t9xx.o =20 mtk_t9xx-y :=3D \ mtk_ctrl_plane.o \ + mtk_port.o \ + mtk_port_io.o \ pcie/mtk_pci.o \ pcie/mtk_trans_ctrl.o \ pcie/mtk_cldma.o \ diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.c b/drivers/net/wwan/t9xx= /mtk_ctrl_plane.c index fa2ab8c3e757..10f0d3267b3b 100644 --- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.c +++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.c @@ -7,19 +7,22 @@ #include =20 #include "mtk_ctrl_plane.h" +#include "mtk_port.h" =20 /** * mtk_ctrl_init() - Initialize the control plane block. * @mdev: Pointer to the MTK modem device. + * @port_layer_cfg: Port layer configuration. * * Allocates and initializes the control plane block * associated with @mdev. * - * Return: 0 on success, -ENOMEM on allocation failure. + * Return: 0 on success, negative error code on failure. */ -int mtk_ctrl_init(struct mtk_md_dev *mdev) +int mtk_ctrl_init(struct mtk_md_dev *mdev, struct mtk_port_layer_cfg *port= _layer_cfg) { struct mtk_ctrl_blk *ctrl_blk; + int err; =20 ctrl_blk =3D devm_kzalloc(mdev->dev, sizeof(*ctrl_blk), GFP_KERNEL); if (!ctrl_blk) @@ -28,7 +31,15 @@ int mtk_ctrl_init(struct mtk_md_dev *mdev) ctrl_blk->mdev =3D mdev; mdev->ctrl_blk =3D ctrl_blk; =20 + err =3D mtk_port_mngr_init(ctrl_blk, port_layer_cfg->port_cfg, + port_layer_cfg->port_cnt); + if (err) + goto err_free_mem; + return 0; + +err_free_mem: + return err; } =20 /** @@ -40,5 +51,8 @@ int mtk_ctrl_init(struct mtk_md_dev *mdev) */ void mtk_ctrl_exit(struct mtk_md_dev *mdev) { + struct mtk_ctrl_blk *ctrl_blk =3D mdev->ctrl_blk; + + mtk_port_mngr_exit(ctrl_blk); mdev->ctrl_blk =3D NULL; } diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h b/drivers/net/wwan/t9xx= /mtk_ctrl_plane.h index 0dcc0ec2c45d..933ba6193b70 100644 --- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h +++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h @@ -11,8 +11,8 @@ =20 #include "mtk_dev.h" =20 -#define Q_MTU_3_5K (0xE00) -#define Q_FRAG_3_5K (0xE00) +#define Q_MTU_3_5K (0xE00) +#define Q_FRAG_3_5K (0xE00) =20 enum mtk_ccci_ch { /* to sAP */ @@ -59,12 +59,16 @@ union ctrl_hif_cmd_data { u32 rx_ch; }; =20 +struct mtk_port_layer_cfg; +struct mtk_port_mngr; + struct mtk_ctrl_blk { struct mtk_md_dev *mdev; + struct mtk_port_mngr *port_mngr; void *ctrl_hw_priv; }; =20 -int mtk_ctrl_init(struct mtk_md_dev *mdev); +int mtk_ctrl_init(struct mtk_md_dev *mdev, struct mtk_port_layer_cfg *port= _layer_cfg); void mtk_ctrl_exit(struct mtk_md_dev *mdev); =20 #endif /* __MTK_CTRL_PLANE_H__ */ diff --git a/drivers/net/wwan/t9xx/mtk_port.c b/drivers/net/wwan/t9xx/mtk_p= ort.c new file mode 100644 index 000000000000..a8ff06ab4de4 --- /dev/null +++ b/drivers/net/wwan/t9xx/mtk_port.c @@ -0,0 +1,693 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022, MediaTek Inc. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "mtk_port.h" +#include "mtk_port_io.h" +#include "mtk_trans_ctrl.h" + +#define MTK_DFLT_TRB_TIMEOUT (5 * HZ) +#define MTK_DFLT_TRB_STATUS (0x1) +#define MTK_TRB_HEADER_ADDED (0xADDED) +#define MTK_CHECK_RX_SEQ_MASK (0x7fff) + +#define MTK_PORT_ENUM_VER (0) +#define MTK_PORT_ENUM_HEAD_PATTERN (0x5a5a5a5a) +#define MTK_PORT_ENUM_TAIL_PATTERN (0xa5a5a5a5) + +#define MTK_PORT_SEARCH_FROM_RADIX_TREE(p, s) ({\ + struct mtk_port *_p; \ + _p =3D rcu_dereference_raw(*(s)); \ + if (!_p) \ + continue; \ + p =3D _p; \ +}) + +#define MTK_PORT_INTERNAL_NODE_CHECK(p, s, i) ({\ + if (radix_tree_is_internal_node(p)) { \ + s =3D radix_tree_iter_retry(&(i));\ + continue; \ + } \ +}) + +struct mtk_port_info { + __le16 channel; + __le16 reserved; +} __packed; + +struct mtk_port_enum_msg { + __le32 head_pattern; + __le16 port_cnt; + __le16 version; + __le32 tail_pattern; + u8 data[]; +} __packed; + +/* mutex lock for the port refcount */ +DEFINE_MUTEX(port_mngr_grp_mtx); + +/* This function working always under mutex lock port_mngr_grp_mtx */ +void mtk_port_release(struct kref *port_kref) +{ + struct mtk_port *port; + + port =3D container_of(port_kref, struct mtk_port, kref); + ports_ops[port->info.type]->exit(port); + kfree_rcu(port, rcu); +} + +static int mtk_port_tbl_add(struct mtk_port_mngr *port_mngr, struct mtk_po= rt *port) +{ + int ret; + + mutex_lock(&port_mngr->port_tbl_mtx); + ret =3D radix_tree_insert(&port_mngr->port_tbl[MTK_PORT_TBL_TYPE(port->in= fo.rx_ch)], + port->info.rx_ch & 0xFFF, port); + if (!ret) + port_mngr->port_cnt++; + mutex_unlock(&port_mngr->port_tbl_mtx); + + if (ret) + dev_err(port_mngr->ctrl_blk->mdev->dev, + "port(%s) add to port_tbl failed, return %d\n", + port->info.name, ret); + + return ret; +} + +static void mtk_port_tbl_del(struct mtk_port_mngr *port_mngr, struct mtk_p= ort *port) +{ + mutex_lock(&port_mngr->port_tbl_mtx); + radix_tree_delete(&port_mngr->port_tbl[MTK_PORT_TBL_TYPE(port->info.rx_ch= )], + port->info.rx_ch & 0xFFF); + port_mngr->port_cnt--; + mutex_unlock(&port_mngr->port_tbl_mtx); +} + +static struct mtk_port *mtk_port_alloc_and_add(struct mtk_port_mngr *port_= mngr, + struct mtk_port_cfg *dflt_info) +{ + struct mtk_port *port; + int ret; + + port =3D kzalloc_obj(*port, GFP_KERNEL); + if (!port) { + ret =3D -ENOMEM; + goto err_alloc_port; + } + memcpy(&port->info, dflt_info, sizeof(*dflt_info)); + + ret =3D mtk_port_tbl_add(port_mngr, port); + if (ret < 0) { + dev_err(port_mngr->ctrl_blk->mdev->dev, + "Failed to add port(%s) to port tbl\n", dflt_info->name); + goto err_free_port; + } + + port->port_mngr =3D port_mngr; + ret =3D ports_ops[port->info.type]->init(port); + if (ret < 0) { + mtk_port_tbl_del(port_mngr, port); + goto err_free_port; + } + + return port; + +err_free_port: + kfree(port); +err_alloc_port: + return ERR_PTR(ret); +} + +static void mtk_port_free(struct mtk_port_mngr *port_mngr, struct mtk_port= *port) +{ + mutex_lock(&port_mngr_grp_mtx); + mtk_port_tbl_del(port_mngr, port); + kref_put(&port->kref, mtk_port_release); + mutex_unlock(&port_mngr_grp_mtx); +} + +static struct mtk_port *mtk_port_search_by_id(struct mtk_port_mngr *port_m= ngr, int rx_ch) +{ + int tbl_type =3D MTK_PORT_TBL_TYPE(rx_ch); + + if (tbl_type < PORT_TBL_SAP || tbl_type >=3D PORT_TBL_MAX) + return NULL; + + return radix_tree_lookup(&port_mngr->port_tbl[tbl_type], MTK_CH_ID(rx_ch)= ); +} + +struct mtk_port *mtk_port_search_by_name(struct mtk_port_mngr *port_mngr, = char *name) +{ + int tbl_type =3D PORT_TBL_SAP; + struct radix_tree_iter iter; + struct mtk_port *port; + void __rcu **slot; + + do { + radix_tree_for_each_slot(slot, &port_mngr->port_tbl[tbl_type], &iter, 0)= { + MTK_PORT_SEARCH_FROM_RADIX_TREE(port, slot); + MTK_PORT_INTERNAL_NODE_CHECK(port, slot, iter); + if (!strncmp(port->info.name, name, MTK_DFLT_PORT_NAME_LEN)) + return port; + } + tbl_type++; + } while (tbl_type < PORT_TBL_MAX); + + return NULL; +} + +static int mtk_port_tbl_create(struct mtk_port_mngr *port_mngr, struct mtk= _port_cfg *cfg, + const int port_cnt) +{ + struct mtk_port_cfg *dflt_port; + struct mtk_port *port; + int i; + + INIT_RADIX_TREE(&port_mngr->port_tbl[PORT_TBL_SAP], GFP_KERNEL); + INIT_RADIX_TREE(&port_mngr->port_tbl[PORT_TBL_MD], GFP_KERNEL); + + /* copy ports from static port cfg table */ + for (i =3D 0; i < port_cnt; i++) { + dflt_port =3D cfg + i; + if (!mtk_port_search_by_id(port_mngr, dflt_port->rx_ch)) { + port =3D mtk_port_alloc_and_add(port_mngr, dflt_port); + if (IS_ERR(port)) + return PTR_ERR(port); + } + } + + return 0; +} + +static void mtk_port_tbl_destroy(struct mtk_port_mngr *port_mngr) +{ + struct radix_tree_iter iter; + struct mtk_port *port; + void __rcu **slot; + int tbl_type; + + tbl_type =3D PORT_TBL_SAP; + do { + radix_tree_for_each_slot(slot, &port_mngr->port_tbl[tbl_type], &iter, 0)= { + MTK_PORT_SEARCH_FROM_RADIX_TREE(port, slot); + MTK_PORT_INTERNAL_NODE_CHECK(port, slot, iter); + ports_ops[port->info.type]->disable(port); + } + + while (radix_tree_gang_lookup(&port_mngr->port_tbl[tbl_type], + (void **)&port, 0, 1)) + mtk_port_free(port_mngr, port); + } while (++tbl_type < PORT_TBL_MAX); +} + +void mtk_port_trb_init(struct mtk_port *port, struct trb *trb, enum mtk_tr= b_cmd_type cmd, + int (*trb_complete)(struct sk_buff *skb)) +{ + kref_init(&trb->kref); + trb->channel_id =3D port->info.rx_ch; + trb->status =3D MTK_DFLT_TRB_STATUS; + trb->priv =3D port; + trb->cmd =3D cmd; + trb->trb_complete =3D trb_complete; +} + +void mtk_port_trb_free(struct kref *trb_kref) +{ + struct trb *trb =3D container_of(trb_kref, struct trb, kref); + struct sk_buff *skb, *frag_skb, *next_skb; + + skb =3D container_of((char *)trb, struct sk_buff, cb[0]); + /* Free frag_list for scatter gather TX */ + if (trb->cmd =3D=3D TRB_CMD_TX && skb_has_frag_list(skb)) { + frag_skb =3D skb_shinfo(skb)->frag_list; + while (frag_skb) { + next_skb =3D frag_skb->next; + frag_skb->next =3D NULL; + dev_kfree_skb_any(frag_skb); + frag_skb =3D next_skb; + } + skb_shinfo(skb)->frag_list =3D NULL; + skb->data_len =3D 0; + } + dev_kfree_skb_any(skb); +} + +static int mtk_port_open_trb_complete(struct sk_buff *skb) +{ + struct trb_open_priv *trb_open_priv =3D (struct trb_open_priv *)skb->data; + struct trb *trb =3D (struct trb *)skb->cb; + struct mtk_port *port =3D trb->priv; + + if (!trb->status) { + port->tx_mtu =3D trb_open_priv->tx_mtu; + port->rx_mtu =3D trb_open_priv->rx_mtu; + port->tx_frag_size =3D trb_open_priv->tx_frag_size; + port->rx_frag_size =3D trb_open_priv->rx_frag_size; + port->tx_mtu -=3D MTK_CCCI_H_ELEN; + port->rx_mtu -=3D MTK_CCCI_H_ELEN; + } + + wake_up_all(&port->trb_wq); + + kref_put(&trb->kref, mtk_port_trb_free); + return 0; +} + +static int mtk_port_close_trb_complete(struct sk_buff *skb) +{ + struct trb *trb =3D (struct trb *)skb->cb; + struct mtk_port *port =3D trb->priv; + + wake_up_all(&port->trb_wq); + wake_up_all(&port->rx_wq); + kref_put(&trb->kref, mtk_port_trb_free); + + return 0; +} + +static int mtk_port_tx_complete(struct sk_buff *skb) +{ + struct trb *trb =3D (struct trb *)skb->cb; + struct mtk_port *port =3D trb->priv; + + if (trb->status < 0) + dev_warn(port->port_mngr->ctrl_blk->mdev->dev, + "Failed to send data: status:%d, port:%s\n", + trb->status, port->info.name); + + wake_up_all(&port->trb_wq); + kref_put(&trb->kref, mtk_port_trb_free); + + return 0; +} + +int mtk_port_status_check(struct mtk_port *port) +{ + if (!test_bit(PORT_S_ENABLE, &port->status)) + return -ENODEV; + + if (!test_bit(PORT_S_OPEN, &port->status) || test_bit(PORT_S_FLUSH, &port= ->status) || + !test_bit(PORT_S_WR, &port->status)) + return -EBADF; + + return 0; +} + +int mtk_port_send_data(struct mtk_port *port, void *data, bool blocking, b= ool force_send) +{ + struct mtk_port_mngr *port_mngr; + struct sk_buff *skb =3D data; + struct trb *trb; + int ret, len; + + port_mngr =3D port->port_mngr; + + trb =3D (struct trb *)skb->cb; + mtk_port_trb_init(port, trb, TRB_CMD_TX, mtk_port_tx_complete); + len =3D skb->len; + kref_get(&trb->kref); /* kref count 1->2 */ + + /* add ccci header */ + mtk_port_add_header(skb); + ret =3D mtk_port_status_check(port); + if (!ret) + ret =3D mtk_pcie_hif_submit_skb(port_mngr->ctrl_blk->mdev, skb, + force_send); + + if (ret < 0) { + kref_put(&trb->kref, mtk_port_trb_free); /* kref count 2->1 */ + kref_put(&trb->kref, mtk_port_trb_free); /* kref count 1->0 */ + port->tx_seq--; + goto out; + } + + if (!blocking) { + kref_put(&trb->kref, mtk_port_trb_free); + ret =3D len; + goto out; + } +start_wait: + + /* wait trb done, and no timeout in tx blocking mode */ + ret =3D wait_event_interruptible_timeout(port->trb_wq, + trb->status <=3D 0 || + test_bit(PORT_S_FLUSH, &port->status) || + !test_bit(PORT_S_WR, &port->status), + MTK_DFLT_TRB_TIMEOUT); + if (!ret) { + goto start_wait; + } else if (ret =3D=3D -ERESTARTSYS) { + ret =3D -EINTR; + } else if (ret > 0) { + if (test_bit(PORT_S_FLUSH, &port->status)) + ret =3D len; + else + ret =3D (!trb->status) ? len : trb->status; + } + kref_put(&trb->kref, mtk_port_trb_free); + +out: + return ret; +} + +static int mtk_port_check_rx_seq(struct mtk_port *port, struct mtk_ccci_he= ader *ccci_h) +{ + u16 seq_num, assert_bit, channel; + struct mtk_md_dev *mdev; + + seq_num =3D FIELD_GET(MTK_HDR_FLD_SEQ, le32_to_cpu(ccci_h->status)); + assert_bit =3D FIELD_GET(MTK_HDR_FLD_AST, le32_to_cpu(ccci_h->status)); + if (assert_bit && port->rx_seq && + ((seq_num - port->rx_seq) & MTK_CHECK_RX_SEQ_MASK) !=3D 1) { + mdev =3D port->port_mngr->ctrl_blk->mdev; + channel =3D FIELD_GET(MTK_HDR_FLD_CHN, le32_to_cpu(ccci_h->status)); + dev_warn(mdev->dev, + " seq num out-of-order %d->%d, len(%u)\n", + channel, seq_num, port->rx_seq, + le32_to_cpu(ccci_h->packet_len)); + + port->rx_seq =3D seq_num; + return -EPROTO; + } + + return 0; +} + +static int mtk_port_rx_dispatch_frag_skb(struct mtk_port *port, struct sk_= buff *skb) +{ + struct sk_buff *frag_skb, *frag_next; + int ret; + + frag_skb =3D skb_shinfo(skb)->frag_list; + skb->len -=3D skb->data_len; + skb->data_len =3D 0; + skb_shinfo(skb)->frag_list =3D NULL; + + ret =3D ports_ops[port->info.type]->recv(port, skb); + if (ret < 0) { + skb_shinfo(skb)->frag_list =3D frag_skb; + return ret; + } + + while (frag_skb) { + frag_next =3D frag_skb->next; + if (!frag_skb->len) { + frag_skb->next =3D NULL; + dev_kfree_skb_any(frag_skb); + frag_skb =3D frag_next; + continue; + } + frag_skb->next =3D NULL; + ret =3D ports_ops[port->info.type]->recv(port, frag_skb); + if (ret < 0) { + frag_skb->next =3D frag_next; + while (frag_skb) { + frag_next =3D frag_skb->next; + frag_skb->next =3D NULL; + dev_kfree_skb_any(frag_skb); + frag_skb =3D frag_next; + } + return -EIO; + } + frag_skb =3D frag_next; + } + + return 0; +} + +static int mtk_port_rx_dispatch(struct sk_buff *skb, void *priv, bool forc= e_recv) +{ + struct mtk_port_mngr *port_mngr; + struct mtk_ccci_header *ccci_h; + struct mtk_port *port =3D priv; + int ret =3D -EPROTO; + u16 channel; + + if (!skb || !priv) { + pr_err("Invalid input value in rx dispatch\n"); + return -EINVAL; + } + + port_mngr =3D port->port_mngr; + + ccci_h =3D mtk_port_strip_header(skb); + if (unlikely(!ccci_h)) { + dev_warn(port_mngr->ctrl_blk->mdev->dev, + "Unsupported: skb length(%d) is less than ccci header\n", + skb->len); + goto drop_data; + } + + channel =3D FIELD_GET(MTK_HDR_FLD_CHN, le32_to_cpu(ccci_h->status)); + port =3D mtk_port_search_by_id(port_mngr, channel); + if (unlikely(!port)) { + dev_warn(port_mngr->ctrl_blk->mdev->dev, + "Failed to find port by channel:%d\n", channel); + goto drop_data; + } + + ret =3D mtk_port_check_rx_seq(port, ccci_h); + if (unlikely(ret)) + goto drop_data; + + port->rx_seq =3D FIELD_GET(MTK_HDR_FLD_SEQ, le32_to_cpu(ccci_h->status)); + skb_pull(skb, sizeof(*ccci_h)); + + /* Support scatter gather transmission */ + if (port->rx_mtu > port->rx_frag_size) { + ret =3D mtk_port_rx_dispatch_frag_skb(port, skb); + /* -EIO means partial data dispatch complete, does not goto drop flow */ + if (ret < 0 && ret !=3D -EIO) + goto drop_frag_skb; + } else { + ret =3D ports_ops[port->info.type]->recv(port, skb); + if (ret < 0) + goto drop_data; + } + + return ret; + +drop_frag_skb: + { + struct sk_buff *frag_skb, *tmp; + + frag_skb =3D skb_shinfo(skb)->frag_list; + while (frag_skb) { + tmp =3D frag_skb->next; + frag_skb->next =3D NULL; + dev_kfree_skb_any(frag_skb); + frag_skb =3D tmp; + } + skb_shinfo(skb)->frag_list =3D NULL; + } +drop_data: + dev_kfree_skb_any(skb); + return ret; +} + +int mtk_port_add_header(struct sk_buff *skb) +{ + struct mtk_ccci_header *ccci_h; + struct mtk_port *port; + struct trb *trb; + + trb =3D (struct trb *)skb->cb; + if (trb->status =3D=3D MTK_TRB_HEADER_ADDED) + return 0; + + port =3D trb->priv; + if (!port) + return -EINVAL; + + ccci_h =3D skb_push(skb, sizeof(*ccci_h)); + + ccci_h->packet_header =3D cpu_to_le32(0); + ccci_h->packet_len =3D cpu_to_le32(skb->len); + ccci_h->ex_msg =3D cpu_to_le32(0); + ccci_h->status =3D cpu_to_le32(FIELD_PREP(MTK_HDR_FLD_CHN, port->info.tx_= ch) | + FIELD_PREP(MTK_HDR_FLD_SEQ, port->tx_seq++) | + FIELD_PREP(MTK_HDR_FLD_AST, 1)); + + trb->status =3D MTK_TRB_HEADER_ADDED; + + return 0; +} + +struct mtk_ccci_header *mtk_port_strip_header(struct sk_buff *skb) +{ + struct mtk_ccci_header *ccci_h; + + if (skb->len < sizeof(*ccci_h)) { + pr_err("Invalid input value\n"); + return NULL; + } + + ccci_h =3D (struct mtk_ccci_header *)skb->data; + + return ccci_h; +} + +int mtk_port_status_update(struct mtk_md_dev *mdev, void *data, u32 data_l= en) +{ + struct mtk_port_enum_msg *msg =3D data; + struct mtk_port_info *port_info; + struct mtk_port_mngr *port_mngr; + struct mtk_ctrl_blk *ctrl_blk; + struct mtk_port *port; + int port_id; + u16 ch_id; + + if (unlikely(!mdev || !msg)) + return -EINVAL; + + ctrl_blk =3D mdev->ctrl_blk; + port_mngr =3D ctrl_blk->port_mngr; + if (le16_to_cpu(msg->version) !=3D MTK_PORT_ENUM_VER || + le32_to_cpu(msg->head_pattern) !=3D MTK_PORT_ENUM_HEAD_PATTERN || + le32_to_cpu(msg->tail_pattern) !=3D MTK_PORT_ENUM_TAIL_PATTERN) + return -EPROTO; + + if (data_len < sizeof(*msg) + + le16_to_cpu(msg->port_cnt) * sizeof(*port_info)) + return -EPROTO; + + for (port_id =3D 0; port_id < le16_to_cpu(msg->port_cnt); port_id++) { + port_info =3D (struct mtk_port_info *)(msg->data + + (sizeof(*port_info) * port_id)); + ch_id =3D FIELD_GET(MTK_INFO_FLD_CHID, le16_to_cpu(port_info->channel)); + port =3D mtk_port_search_by_id(port_mngr, ch_id); + if (!port) + continue; + port->enable =3D FIELD_GET(MTK_INFO_FLD_EN, le16_to_cpu(port_info->chann= el)); + } + + return 0; +} + +int mtk_port_ch_enable(struct mtk_port *port) +{ + struct mtk_port_mngr *port_mngr =3D port->port_mngr; + struct trb_open_priv *trb_open_priv; + struct sk_buff *skb; + struct trb *trb; + int ret; + + skb =3D __dev_alloc_skb(Q_MTU_3_5K, GFP_KERNEL); + if (!skb) + return -ENOMEM; + + trb_open_priv =3D (struct trb_open_priv *)skb->data; + trb_open_priv->rx_done =3D mtk_port_rx_dispatch; + + skb_put(skb, sizeof(struct trb_open_priv)); + trb =3D (struct trb *)skb->cb; + mtk_port_trb_init(port, trb, TRB_CMD_ENABLE, mtk_port_open_trb_complete); + kref_get(&trb->kref); + + ret =3D mtk_pcie_hif_submit_skb(port_mngr->ctrl_blk->mdev, skb, true); + if (ret) { + dev_err(port_mngr->ctrl_blk->mdev->dev, + "Failed to submit trb for port(%s), ret=3D%d\n", + port->info.name, ret); + kref_put(&trb->kref, mtk_port_trb_free); + kref_put(&trb->kref, mtk_port_trb_free); + return ret; + } + + ret =3D wait_event_timeout(port->trb_wq, trb->status <=3D 0, + MTK_DFLT_TRB_TIMEOUT); + if (!ret) + ret =3D -ETIMEDOUT; + else + ret =3D trb->status; + + kref_put(&trb->kref, mtk_port_trb_free); + + return ret; +} + +int mtk_port_ch_disable(struct mtk_port *port) +{ + struct mtk_port_mngr *port_mngr =3D port->port_mngr; + struct sk_buff *skb; + struct trb *trb; + int ret; + + skb =3D __dev_alloc_skb(Q_MTU_3_5K, GFP_KERNEL); + if (!skb) + return -ENOMEM; + + trb =3D (struct trb *)skb->cb; + mtk_port_trb_init(port, trb, TRB_CMD_DISABLE, mtk_port_close_trb_complete= ); + kref_get(&trb->kref); + + ret =3D mtk_pcie_hif_submit_skb(port_mngr->ctrl_blk->mdev, skb, true); + if (ret) { + dev_warn(port_mngr->ctrl_blk->mdev->dev, + "Failed to submit trb for port(%s), ret=3D%d\n", + port->info.name, ret); + kref_put(&trb->kref, mtk_port_trb_free); + kref_put(&trb->kref, mtk_port_trb_free); + return ret; + } + + ret =3D wait_event_timeout(port->trb_wq, trb->status <=3D 0, + MTK_DFLT_TRB_TIMEOUT); + if (!ret) + ret =3D -ETIMEDOUT; + else + ret =3D trb->status; + + kref_put(&trb->kref, mtk_port_trb_free); + + return ret; +} + +int mtk_port_mngr_init(struct mtk_ctrl_blk *ctrl_blk, struct mtk_port_cfg = *port_cfg, int port_cnt) +{ + struct mtk_port_mngr *port_mngr; + int ret =3D -ENOMEM; + + port_mngr =3D devm_kzalloc(ctrl_blk->mdev->dev, sizeof(*port_mngr), GFP_K= ERNEL); + if (unlikely(!port_mngr)) { + dev_err((ctrl_blk->mdev)->dev, "Failed to alloc memory for port_mngr\n"); + goto err_out; + } + + port_mngr->ctrl_blk =3D ctrl_blk; + mutex_init(&port_mngr->port_tbl_mtx); + + ret =3D mtk_port_tbl_create(port_mngr, port_cfg, port_cnt); + if (unlikely(ret)) { + dev_err((ctrl_blk->mdev)->dev, "Failed to create port_tbl\n"); + goto err_free_port_mngr; + } + + ctrl_blk->port_mngr =3D port_mngr; + + return ret; + +err_free_port_mngr: + mtk_port_tbl_destroy(port_mngr); +err_out: + return ret; +} + +void mtk_port_mngr_exit(struct mtk_ctrl_blk *ctrl_blk) +{ + struct mtk_port_mngr *port_mngr =3D ctrl_blk->port_mngr; + + mtk_port_tbl_destroy(port_mngr); + + ctrl_blk->port_mngr =3D NULL; +} diff --git a/drivers/net/wwan/t9xx/mtk_port.h b/drivers/net/wwan/t9xx/mtk_p= ort.h new file mode 100644 index 000000000000..9b8f63ce3f3c --- /dev/null +++ b/drivers/net/wwan/t9xx/mtk_port.h @@ -0,0 +1,131 @@ +/* SPDX-License-Identifier: GPL-2.0-only + * + * Copyright (c) 2022, MediaTek Inc. + */ + +#ifndef __MTK_PORT_H__ +#define __MTK_PORT_H__ + +#include +#include +#include +#include +#include + +#include "mtk_ctrl_plane.h" +#include "mtk_dev.h" + +#define MTK_PEER_ID_MASK (0xF000) +#define MTK_PEER_ID_SHIFT (12) +#define MTK_PEER_ID(ch) (((ch) & MTK_PEER_ID_MASK) >> MTK_PEER_ID_SHIFT) +#define MTK_CH_ID_MASK (0x0FFF) +#define MTK_CH_ID(ch) ((ch) & MTK_CH_ID_MASK) +#define MTK_DFLT_PORT_NAME_LEN (20) + +/* Mapping MTK_PEER_ID and mtk_port_tbl index */ +#define MTK_PORT_TBL_TYPE(ch) (MTK_PEER_ID(ch) - 1) + +/* ccci header length + reserved space that is used in exception flow */ +#define MTK_CCCI_H_ELEN (128) + +#define MTK_HDR_FLD_AST ((u32)BIT(31)) +#define MTK_HDR_FLD_SEQ GENMASK(30, 16) +#define MTK_HDR_FLD_CHN GENMASK(15, 0) + +#define MTK_INFO_FLD_EN ((u16)BIT(15)) +#define MTK_INFO_FLD_CHID GENMASK(14, 0) + +enum mtk_port_status { + PORT_S_ENABLE, + PORT_S_OPEN, + PORT_S_WR, + PORT_S_FLUSH, +}; + +enum mtk_port_flag { + PORT_F_DFLT =3D 0, + PORT_F_BLOCKING =3D BIT(1), +}; + +enum mtk_port_tbl { + PORT_TBL_SAP, + PORT_TBL_MD, + PORT_TBL_MAX +}; + +enum mtk_port_type { + PORT_TYPE_INTERNAL, + PORT_TYPE_MAX +}; + +struct mtk_internal_port { + void *arg; + int (*recv_cb)(void *arg, struct sk_buff *skb); +}; + +struct mtk_port_cfg { + enum mtk_ccci_ch tx_ch; + enum mtk_ccci_ch rx_ch; + enum mtk_port_type type; + char name[MTK_DFLT_PORT_NAME_LEN]; + unsigned char flags; +}; + +struct mtk_port { + struct mtk_port_cfg info; + struct kref kref; + struct rcu_head rcu; + bool enable; + unsigned long status; + unsigned short tx_seq; + unsigned short rx_seq; + unsigned int tx_mtu; + unsigned int rx_mtu; + u32 tx_frag_size; + u32 rx_frag_size; + struct sk_buff_head rx_skb_list; + unsigned int rx_data_len; + unsigned int rx_buf_size; + wait_queue_head_t trb_wq; + wait_queue_head_t rx_wq; + struct mtk_port_mngr *port_mngr; + struct mtk_internal_port i_priv; +}; + +struct mtk_port_mngr { + struct mtk_ctrl_blk *ctrl_blk; + struct radix_tree_root port_tbl[PORT_TBL_MAX]; + /* protects port_tbl[] and port_cnt against concurrent mutators */ + struct mutex port_tbl_mtx; + unsigned int port_cnt; +}; + +struct mtk_ccci_header { + __le32 packet_header; + __le32 packet_len; + __le32 status; + __le32 ex_msg; +}; + +struct mtk_port_layer_cfg { + struct mtk_port_cfg *port_cfg; + int port_cnt; +}; + +extern const struct port_ops *ports_ops[PORT_TYPE_MAX]; + +void mtk_port_release(struct kref *port_kref); +void mtk_port_trb_free(struct kref *trb_kref); +struct mtk_port *mtk_port_search_by_name(struct mtk_port_mngr *port_mngr, = char *name); +int mtk_port_add_header(struct sk_buff *skb); +struct mtk_ccci_header *mtk_port_strip_header(struct sk_buff *skb); +int mtk_port_status_check(struct mtk_port *port); +int mtk_port_send_data(struct mtk_port *port, void *data, bool blocking, b= ool force_send); +int mtk_port_status_update(struct mtk_md_dev *mdev, void *data, u32 data_l= en); +int mtk_port_ch_enable(struct mtk_port *port); +int mtk_port_ch_disable(struct mtk_port *port); +int mtk_port_mngr_init(struct mtk_ctrl_blk *ctrl_blk, struct mtk_port_cfg = *port_cfg, int port_cnt); +void mtk_port_mngr_exit(struct mtk_ctrl_blk *ctrl_blk); +void mtk_port_trb_init(struct mtk_port *port, struct trb *trb, enum mtk_tr= b_cmd_type cmd, + int (*trb_complete)(struct sk_buff *skb)); +#endif /* __MTK_PORT_H__ */ diff --git a/drivers/net/wwan/t9xx/mtk_port_io.c b/drivers/net/wwan/t9xx/mt= k_port_io.c new file mode 100644 index 000000000000..6cff0704b7fc --- /dev/null +++ b/drivers/net/wwan/t9xx/mtk_port_io.c @@ -0,0 +1,258 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022, MediaTek Inc. + */ +#include + +#include "mtk_port_io.h" + +static int mtk_port_get_locked(struct mtk_port *port) +{ + int ret =3D 0; + + mutex_lock(&port_mngr_grp_mtx); + if (!port) { + mutex_unlock(&port_mngr_grp_mtx); + pr_err("Port does not exist\n"); + return -ENODEV; + } + kref_get(&port->kref); + mutex_unlock(&port_mngr_grp_mtx); + + return ret; +} + +static void mtk_port_put_locked(struct mtk_port *port) +{ + mutex_lock(&port_mngr_grp_mtx); + kref_put(&port->kref, mtk_port_release); + mutex_unlock(&port_mngr_grp_mtx); +} + +static void mtk_port_struct_init(struct mtk_port *port) +{ + port->tx_seq =3D 0; + port->rx_seq =3D -1; + clear_bit(PORT_S_ENABLE, &port->status); + kref_init(&port->kref); + skb_queue_head_init(&port->rx_skb_list); + port->rx_buf_size =3D MTK_RX_BUF_SIZE; + init_waitqueue_head(&port->trb_wq); + init_waitqueue_head(&port->rx_wq); +} + +static int mtk_port_internal_init(struct mtk_port *port) +{ + mtk_port_struct_init(port); + port->enable =3D false; + + return 0; +} + +static void mtk_port_internal_exit(struct mtk_port *port) +{ + if (test_bit(PORT_S_ENABLE, &port->status)) + ports_ops[port->info.type]->disable(port); +} + +static void mtk_port_internal_enable(struct mtk_port *port) +{ + int ret; + + if (test_bit(PORT_S_ENABLE, &port->status)) + return; + + ret =3D mtk_port_ch_enable(port); + if (ret && ret !=3D -EBUSY) { + /* On -ETIMEDOUT the ENABLE trb may still be queued; queue a + * DISABLE behind it so the channel does not end up armed with + * no software owner. + */ + mtk_port_ch_disable(port); + return; + } + + set_bit(PORT_S_WR, &port->status); + set_bit(PORT_S_ENABLE, &port->status); +} + +static void mtk_port_internal_disable(struct mtk_port *port) +{ + int ret; + + if (!test_and_clear_bit(PORT_S_ENABLE, &port->status)) + return; + + /* PORT_S_WR is part of the blocking TX wait condition, and the waiter + * loops back on timeout, so without this wake it only notices at the + * next trb timeout rather than now. + */ + clear_bit(PORT_S_WR, &port->status); + wake_up_all(&port->trb_wq); + + /* Failure must not abort teardown or keep PORT_S_ENABLE set: + * usr_cnt and the queues are rebuilt together at the next + * FSM_STATE_ON, and a port left with the bit set could never + * be re-enabled (mtk_port_internal_enable() early-returns). + */ + ret =3D mtk_port_ch_disable(port); + if (ret) + dev_warn(port->port_mngr->ctrl_blk->mdev->dev, + "Failed to disable channel for port(%s): %d\n", + port->info.name, ret); +} + +static int mtk_port_internal_recv(struct mtk_port *port, struct sk_buff *s= kb) +{ + struct mtk_internal_port *priv; + int ret =3D -ENXIO; + + if (!test_bit(PORT_S_OPEN, &port->status)) + goto drop_data; + + priv =3D &port->i_priv; + if (!priv->recv_cb || !priv->arg) + goto drop_data; + + ret =3D priv->recv_cb(priv->arg, skb); + return ret; + +drop_data: + return ret; +} + +static int mtk_port_common_open(struct mtk_port *port) +{ + int ret =3D 0; + + if (!test_bit(PORT_S_ENABLE, &port->status)) + return -ENODEV; + + if (test_bit(PORT_S_OPEN, &port->status)) + return -EBUSY; + + skb_queue_purge(&port->rx_skb_list); + set_bit(PORT_S_OPEN, &port->status); + clear_bit(PORT_S_FLUSH, &port->status); + + return ret; +} + +static void mtk_port_common_close(struct mtk_port *port) +{ + clear_bit(PORT_S_OPEN, &port->status); + + skb_queue_purge(&port->rx_skb_list); + port->rx_data_len =3D 0; + + set_bit(PORT_S_FLUSH, &port->status); + wake_up_all(&port->trb_wq); + wake_up_all(&port->rx_wq); +} + +void *mtk_port_internal_open(struct mtk_md_dev *mdev, char *name, int flag) +{ + struct mtk_port_mngr *port_mngr; + struct mtk_ctrl_blk *ctrl_blk; + struct mtk_port *port; + int ret; + + ctrl_blk =3D mdev->ctrl_blk; + port_mngr =3D ctrl_blk->port_mngr; + + port =3D mtk_port_search_by_name(port_mngr, name); + if (port && port->info.type !=3D PORT_TYPE_INTERNAL) { + port =3D NULL; + goto out; + } + + ret =3D mtk_port_get_locked(port); + if (ret) + goto out; + + ret =3D mtk_port_common_open(port); + if (ret) { + mtk_port_put_locked(port); + port =3D NULL; + goto out; + } + + if (flag & O_NONBLOCK) + port->info.flags &=3D ~PORT_F_BLOCKING; + else + port->info.flags |=3D PORT_F_BLOCKING; +out: + return port; +} + +int mtk_port_internal_close(void *i_port) +{ + struct mtk_port *port =3D i_port; + int ret =3D 0; + + if (!port) { + ret =3D -EINVAL; + goto end; + } + + if (!test_bit(PORT_S_OPEN, &port->status)) { + pr_err("Port(%s) has been closed\n", port->info.name); + ret =3D -EBADF; + goto end; + } + + mtk_port_common_close(port); + mtk_port_put_locked(port); +end: + return ret; +} + +int mtk_port_internal_write(void *i_port, struct sk_buff *skb) +{ + struct mtk_port *port =3D i_port; + bool blocking; + + if (!port || !skb) { + if (skb) + dev_kfree_skb_any(skb); + pr_err_ratelimited("Internal write: invalid input\n"); + return -EINVAL; + } + + blocking =3D !!(port->info.flags & PORT_F_BLOCKING); + + return mtk_port_send_data(port, skb, blocking, blocking); +} + +void mtk_port_internal_recv_register(void *i_port, + int (*cb)(void *priv, struct sk_buff *skb), + void *arg) +{ + struct mtk_port *port =3D i_port; + struct mtk_internal_port *priv; + + priv =3D &port->i_priv; + priv->arg =3D arg; + priv->recv_cb =3D cb; +} + +int mtk_port_io_init(void) +{ + return 0; +} + +void mtk_port_io_exit(void) +{ +} + +static const struct port_ops port_internal_ops =3D { + .init =3D mtk_port_internal_init, + .exit =3D mtk_port_internal_exit, + .enable =3D mtk_port_internal_enable, + .disable =3D mtk_port_internal_disable, + .recv =3D mtk_port_internal_recv, +}; + +const struct port_ops *ports_ops[PORT_TYPE_MAX] =3D { + &port_internal_ops, +}; diff --git a/drivers/net/wwan/t9xx/mtk_port_io.h b/drivers/net/wwan/t9xx/mt= k_port_io.h new file mode 100644 index 000000000000..5a4e36c075a5 --- /dev/null +++ b/drivers/net/wwan/t9xx/mtk_port_io.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: GPL-2.0-only + * + * Copyright (c) 2022, MediaTek Inc. + */ + +#ifndef __MTK_PORT_IO_H__ +#define __MTK_PORT_IO_H__ + +#include + +#include "mtk_port.h" + +#define MTK_RX_BUF_SIZE (1024 * 1024) + +extern struct mutex port_mngr_grp_mtx; + +struct port_ops { + int (*init)(struct mtk_port *port); + void (*exit)(struct mtk_port *port); + void (*enable)(struct mtk_port *port); + void (*disable)(struct mtk_port *port); + int (*recv)(struct mtk_port *port, struct sk_buff *skb); +}; + +void *mtk_port_internal_open(struct mtk_md_dev *mdev, char *name, int flag= ); +int mtk_port_internal_close(void *i_port); +int mtk_port_internal_write(void *i_port, struct sk_buff *skb); +void mtk_port_internal_recv_register(void *i_port, + int (*cb)(void *priv, struct sk_buff *skb), + void *arg); + +int mtk_port_io_init(void); +void mtk_port_io_exit(void); + +#endif /* __MTK_PORT_IO_H__ */ diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.c b/drivers/net/wwan/t9xx/p= cie/mtk_pci.c index 6c31a5d80693..84b7fb3c82fd 100644 --- a/drivers/net/wwan/t9xx/pcie/mtk_pci.c +++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.c @@ -15,6 +15,8 @@ #include "mtk_trans_ctrl.h" #include "mtk_pci.h" #include "mtk_pci_reg.h" +#include "mtk_port.h" +#include "mtk_port_io.h" =20 #define MTK_MHCCIF_RC_BASE_ADDR 0x1000A000 =20 @@ -1069,7 +1071,28 @@ static struct pci_driver mtk_pci_drv =3D { .err_handler =3D &mtk_pci_err_handler }; =20 -module_pci_driver(mtk_pci_drv); +static int __init mtk_drv_init(void) +{ + int ret; + + ret =3D mtk_port_io_init(); + if (ret) + return ret; + + ret =3D pci_register_driver(&mtk_pci_drv); + if (ret) + mtk_port_io_exit(); + + return ret; +} +module_init(mtk_drv_init); + +static void __exit mtk_drv_exit(void) +{ + pci_unregister_driver(&mtk_pci_drv); + mtk_port_io_exit(); +} +module_exit(mtk_drv_exit); =20 MODULE_DESCRIPTION("MediaTek T9xx PCIe WWAN driver"); MODULE_LICENSE("GPL"); diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c b/drivers/net/wwan= /t9xx/pcie/mtk_trans_ctrl.c index 531940b06e2a..d485bd7e3bf9 100644 --- a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c +++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c @@ -16,6 +16,7 @@ #include "mtk_ctrl_plane.h" #include "mtk_dev.h" #include "mtk_pci.h" +#include "mtk_port.h" #include "mtk_trans_ctrl.h" =20 #define QUEUE_CHL_MASK 0xFFFF @@ -34,6 +35,18 @@ static const struct queue_info mtk_queue_info[] =3D { Q_MTU_3_5K, Q_MTU_3_5K, TX_GPD_NUM, RX_GPD_NUM, Q_FRAG_3_5K, Q_FRAG_3_5K= , 0}, }; =20 +static const struct mtk_port_cfg mtk_port_cfg_tbl[] =3D { + {CCCI_CONTROL_TX, CCCI_CONTROL_RX, PORT_TYPE_INTERNAL, "MDCTRL", + PORT_F_DFLT}, + {CCCI_SAP_CONTROL_TX, CCCI_SAP_CONTROL_RX, PORT_TYPE_INTERNAL, "SAPCTRL", + PORT_F_DFLT}, +}; + +static struct mtk_port_layer_cfg mtk_port_layer_cfg =3D { + .port_cfg =3D (struct mtk_port_cfg *)mtk_port_cfg_tbl, + .port_cnt =3D ARRAY_SIZE(mtk_port_cfg_tbl), +}; + static bool mtk_queue_list_is_full(struct mtk_ctrl_trans *trans, struct qu= eue_info *que) { return skb_queue_len_lockless(&trans->trans_list[que->hif_id].skb_list[qu= e->txqno]) >=3D @@ -164,6 +177,7 @@ static void mtk_ctrl_trb_handler(struct trb_srv *srv, s= truct trans_list *trans_l break; } trb =3D (struct trb *)skb->cb; + kref_get(&trb->kref); =20 switch (trb->cmd) { case TRB_CMD_ENABLE: @@ -185,8 +199,10 @@ static void mtk_ctrl_trb_handler(struct trb_srv *srv, = struct trans_list *trans_l kick =3D true; break; } - if (err =3D=3D -EAGAIN) + if (err =3D=3D -EAGAIN) { + kref_put(&trb->kref, mtk_port_trb_free); return; + } =20 skb_unlink(skb, skb_list); trb->status =3D err; @@ -227,6 +243,8 @@ static void mtk_ctrl_trb_handler(struct trb_srv *srv, s= truct trans_list *trans_l kick =3D false; } =20 + kref_put(&trb->kref, mtk_port_trb_free); + loop++; } while (loop < TRB_NUM_PER_ROUND); } @@ -597,7 +615,7 @@ int mtk_trans_ctrl_init(struct mtk_md_dev *mdev) trans->queue_info_num =3D ARRAY_SIZE(mtk_queue_info); trans->trb_srv_num =3D TRB_SRV_NUM; =20 - err =3D mtk_ctrl_init(mdev); + err =3D mtk_ctrl_init(mdev, &mtk_port_layer_cfg); if (err) return err; =20 diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h b/drivers/net/wwan= /t9xx/pcie/mtk_trans_ctrl.h index f9144c60edf3..1228f40f2d39 100644 --- a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h +++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h @@ -13,6 +13,7 @@ #include =20 #include "mtk_dev.h" +#include "mtk_port.h" =20 #define TRB_SRV_MAX_NUM (1) #define HW_QUE_NUM (8) --=20 2.34.1 From nobody Wed Sep 30 11:25:46 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E44CB4562B0; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790754373; cv=none; b=qHEEH4b50pIlPiLXK8oyPJEoPwA7mI5OB3GjAf7VXtZBdWDkNmG2yIh8m7ZYts2NIGStmUn8QMdERgbd/PtKfEmqvTlNvbj8DjHxgQn7RXn33f/PeSRJCnPonA+oOuuysKHQgTT4NoJGTKZyhGqEjTtOJbexh91JdIvtfCWc9+s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790754373; c=relaxed/simple; bh=14KnZbQtbYXGLOA8f42F5B9d2557Oz9FW8DYLxMKFHg=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=tudBGg5Wxw5QgTA63youiWhBKAN70ajMZI9LZswPZe7ZT2M8rE4kfTc0hUR4ap2MsxqVthIyPBRNFrojdp4iS1B+dIQEHnz9v0RNk0E9cSupG0m1XdlnF7fdE28JqRcSYF/RyEeiCEQBRrD5DVnfbtWC2BtFf5J1f3jLnNGbjX4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=cwKPzcZP; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="cwKPzcZP" Received: by smtp.kernel.org (Postfix) with ESMTPS id A6182C4AF1D; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1790754372; bh=14KnZbQtbYXGLOA8f42F5B9d2557Oz9FW8DYLxMKFHg=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=cwKPzcZPvi8k3w72qewrngZ94okd/c5J/Tl31iJZT7uJBVILQ2LgzO7IRYKEUjZ+B SW45St4MwTiYd5KTua5LqRhZ0D6UOWvFVAzYPi6PdKe03VzTRahPtJhzfvjVjZAFt3 FAzs9lGU2GasobmA1po7wrn5mSKeB8WiWMfvRqimmGgGoKTEiezAGgX+48jM1Cm+v4 DCi+M8Z3xf8UdDgaRwettzwvNATCUSNnGzqQP+H1Fiqt0VE9hPnYXiDnQxD2VuUYbD qJxJEmIAw+pHUxdC8MTY2JEGzlqJpSN3aECfK4VEjJC26jmFTJpVL5rEgeM7QfFt8c MBcI5N6HTYfFQ== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91D78CA5FC4; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) From: Jack Wu via B4 Relay Date: Wed, 30 Sep 2026 15:46:12 +0800 Subject: [PATCH v9 5/6] net: wwan: t9xx: Add FSM thread Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260930-t9xx_driver_v1-v9-5-3248b9256956@compal.com> References: <20260930-t9xx_driver_v1-v9-0-3248b9256956@compal.com> In-Reply-To: <20260930-t9xx_driver_v1-v9-0-3248b9256956@compal.com> To: Loic Poulain , Sergey Ryazanov , Johannes Berg , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jack Wu , Wen-Zhi Huang , Shi-Wei Yeh , Minano Tseng , Matthias Brugger , AngeloGioacchino Del Regno , Simon Horman , Jonathan Corbet , Shuah Khan , Robert Yu , Jeff Chang Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, linux-doc@vger.kernel.org X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1790754370; l=72658; i=jackbb_wu@compal.com; s=20260526; h=from:subject:message-id; bh=IYb8zkeG+iqxVIXsnT9brzEcF0h5RXv3NAiBpsuK32c=; b=MI/zhH4SKZJjOl/11zu74PPcNhZ5fFeur/up/Lb097gbmtN41Zi9S8pD+YzHnGn2awYOQ3VQB 5bi/pzIHbQCBXlAv747jclm0kgMcapxnn6rMYv2hkCfMqI5MMFbA20c X-Developer-Key: i=jackbb_wu@compal.com; a=ed25519; pk=VH1prTWixNl8OEUPPSfII3p46MzJpQN8J3+ecE1tZXg= X-Endpoint-Received: by B4 Relay for jackbb_wu@compal.com/20260526 with auth_id=793 X-Original-From: Jack Wu Reply-To: jackbb_wu@compal.com From: Jack Wu The FSM (Finite-state Machine) thread is responsible for synchronizing the actions of different modules. The asynchronous events from the device or the OS will trigger a state transition. The FSM thread will append it to the event queue when an event arrives. It handles the events sequentially. After processing the event, the FSM thread notifies other modules before and after the state transition. Five FSM states are defined. They can transition from one state to another, and self-transition in some states. Also wire the HS1/HS2/HS3 runtime feature handshake into the FSM: the host opens a control port on device readiness, sends HS1 with the supported feature set, parses the HS2 reply, and confirms with HS3. MD and SAP each run this sequence independently; the FSM advances to READY only after both complete. Every state transition also emits a KOBJ_CHANGE uevent carrying ":event_id=3D1, info=3Dstate=3D, fsm_flag=3D0x". It is informational only: no driver behaviour depends on it and userspace is not required to consume it. CLDMA hardware bring-up and teardown are driven by FSM state transitions: mtk_cldma_dev_init() on FSM_STATE_BOOTUP allocates DMA pools, creates a workqueue and registers the MSI-X IRQ, and mtk_cldma_dev_exit() tears them down on FSM_STATE_OFF. Registering that IRQ is what first makes the completion path built by the earlier patches run: mtk_cldma_isr() dispatches tx_done_work and rx_done_work, and queues err_work on a QUEUE_ERROR interrupt. err_work owns the recovery policy - it stops the offending queue, completes every outstanding TX request with -EPIPE rather than retrying it, and hands a stopped RX queue back to its worker to be restarted. A queue that will not stop within the timeout is still a live DMA target, so its ring, and the DMA pools its descriptors live in, are deliberately leaked instead of being returned to the allocator. Teardown is ordered to match: mtk_pcie_hif_exit() joins the trb service threads before mtk_cldma_exit() frees what they dereference. Signed-off-by: Jack Wu --- drivers/net/wwan/t9xx/Makefile | 1 + drivers/net/wwan/t9xx/mtk_ctrl_plane.c | 51 ++ drivers/net/wwan/t9xx/mtk_ctrl_plane.h | 1 + drivers/net/wwan/t9xx/mtk_dev.h | 1 + drivers/net/wwan/t9xx/mtk_fsm.c | 1174 +++++++++++++++++++++++= ++++ drivers/net/wwan/t9xx/mtk_fsm.h | 158 ++++ drivers/net/wwan/t9xx/mtk_port.c | 66 ++ drivers/net/wwan/t9xx/mtk_port.h | 2 + drivers/net/wwan/t9xx/mtk_utility.h | 29 + drivers/net/wwan/t9xx/pcie/mtk_cldma.c | 393 ++++++++- drivers/net/wwan/t9xx/pcie/mtk_cldma.h | 4 + drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h | 7 + drivers/net/wwan/t9xx/pcie/mtk_pci.c | 30 +- drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c | 30 +- drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h | 3 + 15 files changed, 1932 insertions(+), 18 deletions(-) diff --git a/drivers/net/wwan/t9xx/Makefile b/drivers/net/wwan/t9xx/Makefile index d44266009362..fcfddf2e5147 100644 --- a/drivers/net/wwan/t9xx/Makefile +++ b/drivers/net/wwan/t9xx/Makefile @@ -9,6 +9,7 @@ mtk_t9xx-y :=3D \ mtk_ctrl_plane.o \ mtk_port.o \ mtk_port_io.o \ + mtk_fsm.o \ pcie/mtk_pci.o \ pcie/mtk_trans_ctrl.o \ pcie/mtk_cldma.o \ diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.c b/drivers/net/wwan/t9xx= /mtk_ctrl_plane.c index 10f0d3267b3b..4ea51b4d6a7c 100644 --- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.c +++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.c @@ -5,9 +5,50 @@ */ =20 #include +#include +#include +#include +#include +#include +#include =20 #include "mtk_ctrl_plane.h" #include "mtk_port.h" +#include "mtk_trans_ctrl.h" + +static void mtk_ctrl_trans_fsm_state_handler(struct mtk_fsm_param *param, + struct mtk_ctrl_blk *ctrl_blk) +{ + struct mtk_md_dev *mdev =3D ctrl_blk->mdev; + int ret; + + switch (param->to) { + case FSM_STATE_OFF: + mtk_pcie_hif_exit(mdev); + mtk_pcie_hif_fsm_indication(mdev, param); + break; + case FSM_STATE_ON: + ret =3D mtk_pcie_hif_init(mdev); + if (ret) { + dev_err(mdev->dev, "Failed to init HIF: %d\n", ret); + mtk_fsm_hif_err_record(mdev, MTK_FSM_HIF_USER_CTRL, ret); + break; + } + fallthrough; + default: + mtk_pcie_hif_fsm_indication(mdev, param); + break; + } +} + +static void mtk_ctrl_fsm_state_listener(struct mtk_fsm_param *param, void = *data) +{ + struct mtk_ctrl_blk *ctrl_blk =3D data; + + mtk_port_mngr_fsm_state_handler(param, ctrl_blk->port_mngr); + mtk_ctrl_trans_fsm_state_handler(param, ctrl_blk); + mtk_port_mngr_fsm_state_handler_late(param, ctrl_blk->port_mngr); +} =20 /** * mtk_ctrl_init() - Initialize the control plane block. @@ -36,8 +77,17 @@ int mtk_ctrl_init(struct mtk_md_dev *mdev, struct mtk_po= rt_layer_cfg *port_layer if (err) goto err_free_mem; =20 + err =3D mtk_fsm_notifier_register(mdev, MTK_USER_CTRL, mtk_ctrl_fsm_state= _listener, + ctrl_blk, FSM_PRIO_1, false); + if (err) { + dev_err(mdev->dev, "Fail to register fsm notification(ret =3D %d)\n", er= r); + goto err_port_exit; + } + return 0; =20 +err_port_exit: + mtk_port_mngr_exit(ctrl_blk); err_free_mem: return err; } @@ -53,6 +103,7 @@ void mtk_ctrl_exit(struct mtk_md_dev *mdev) { struct mtk_ctrl_blk *ctrl_blk =3D mdev->ctrl_blk; =20 + mtk_fsm_notifier_unregister(mdev, MTK_USER_CTRL); mtk_port_mngr_exit(ctrl_blk); mdev->ctrl_blk =3D NULL; } diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h b/drivers/net/wwan/t9xx= /mtk_ctrl_plane.h index 933ba6193b70..1f13e25502ad 100644 --- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h +++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h @@ -10,6 +10,7 @@ #include =20 #include "mtk_dev.h" +#include "mtk_fsm.h" =20 #define Q_MTU_3_5K (0xE00) #define Q_FRAG_3_5K (0xE00) diff --git a/drivers/net/wwan/t9xx/mtk_dev.h b/drivers/net/wwan/t9xx/mtk_de= v.h index 84fcd980110b..658fd5b8431e 100644 --- a/drivers/net/wwan/t9xx/mtk_dev.h +++ b/drivers/net/wwan/t9xx/mtk_dev.h @@ -40,6 +40,7 @@ struct mtk_md_dev { u32 hw_ver; char dev_str[MTK_DEV_STR_LEN]; struct mtk_ctrl_blk *ctrl_blk; + struct mtk_md_fsm *fsm; }; =20 #endif /* __MTK_DEV_H__ */ diff --git a/drivers/net/wwan/t9xx/mtk_fsm.c b/drivers/net/wwan/t9xx/mtk_fs= m.c new file mode 100644 index 000000000000..94617933e771 --- /dev/null +++ b/drivers/net/wwan/t9xx/mtk_fsm.c @@ -0,0 +1,1174 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022, MediaTek Inc. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "mtk_fsm.h" +#include "mtk_pci.h" +#include "mtk_port.h" +#include "mtk_port_io.h" +#include "mtk_utility.h" + +#define EVT_TF_GATECLOSED (1) +#define MTK_FSM_INFO_LEN (64) + +#define FSM_HS_START_MASK (FSM_F_SAP_HS_START | FSM_F_MD_HS_START) +#define FSM_HS2_DONE_MASK (FSM_F_SAP_HS2_DONE | FSM_F_MD_HS2_DONE) + +#define RTFT_DATA_SIZE (3 * 1024) +#define EVT_HANDLER_TIMEOUT (HZ * 30) +#define BLOCKING_EVT_TIMEOUT (2 * EVT_HANDLER_TIMEOUT) + +#define REGION_BITMASK 0xF +#define DEVICE_CFG_SHIFT 24 +#define DEVICE_CFG_REGION_MASK 0x3 + +enum device_stage { + DEV_STAGE_IDLE =3D 4, + DEV_STAGE_MAX +}; + +enum device_cfg { + DEV_CFG_MD_ONLY =3D 1, +}; + +enum runtime_feature_support_type { + RTFT_TYPE_NOT_EXIST =3D 0, + RTFT_TYPE_NOT_SUPPORT =3D 1, + RTFT_TYPE_MUST_SUPPORT =3D 2, + RTFT_TYPE_OPTIONAL_SUPPORT =3D 3, + RTFT_TYPE_SUPPORT_BACKWARD_COMPAT =3D 4, +}; + +enum query_runtime_feature_id { + QUERY_RTFT_ID_MD_PORT_ENUM =3D 0, + QUERY_RTFT_ID_SAP_PORT_ENUM =3D 1, + QUERY_RTFT_ID_MD_PORT_CFG =3D 2, +}; + +enum ctrl_msg_id { + CTRL_MSG_HS1 =3D 0, + CTRL_MSG_HS2 =3D 1, + CTRL_MSG_HS3 =3D 2, +}; + +struct ctrl_msg_header { + __le32 id; + __le32 ex_msg; + __le32 data_len; + u8 reserved[]; +} __packed; + +struct runtime_feature_entry { + u8 feature_id; + struct runtime_feature_info support_info; + u8 reserved[2]; + __le32 data_len; + u8 data[]; +} __packed; + +struct feature_query { + __le32 head_pattern; + struct runtime_feature_info ft_set[FEATURE_CNT]; + __le32 tail_pattern; +}; + +static int mtk_fsm_send_hs1_msg(struct fsm_hs_info *hs_info) +{ + struct ctrl_msg_header *ctrl_msg_h; + struct feature_query *ft_query; + struct sk_buff *skb; + int ret, msg_size; + + msg_size =3D sizeof(*ctrl_msg_h) + sizeof(*ft_query); + skb =3D __dev_alloc_skb(msg_size, GFP_KERNEL); + if (!skb) + return -ENOMEM; + + skb_put(skb, msg_size); + ctrl_msg_h =3D (struct ctrl_msg_header *)skb->data; + ctrl_msg_h->id =3D cpu_to_le32(CTRL_MSG_HS1); + ctrl_msg_h->ex_msg =3D 0; + ctrl_msg_h->data_len =3D cpu_to_le32(sizeof(*ft_query)); + + ft_query =3D (struct feature_query *)(skb->data + sizeof(*ctrl_msg_h)); + ft_query->head_pattern =3D cpu_to_le32(FEATURE_QUERY_PATTERN); + memcpy(ft_query->ft_set, hs_info->query_ft_set, sizeof(hs_info->query_ft_= set)); + ft_query->tail_pattern =3D cpu_to_le32(FEATURE_QUERY_PATTERN); + + /* send handshake1 message to device */ + ret =3D mtk_port_internal_write(hs_info->ctrl_port, skb); + if (ret <=3D 0) + return ret; + + return 0; +} + +static int mtk_fsm_feature_set_match(enum runtime_feature_support_type *cu= r_ft_spt, + struct runtime_feature_info rtft_info_st, + struct runtime_feature_info rtft_info_cfg) +{ + int ret =3D 0; + + switch (FIELD_GET(FEATURE_TYPE, rtft_info_st.feature)) { + case RTFT_TYPE_NOT_EXIST: + fallthrough; + case RTFT_TYPE_NOT_SUPPORT: + /* The device refusing a feature the host declared mandatory + * must fail the handshake, symmetrically with the + * MUST_SUPPORT case below. + */ + if (FIELD_GET(FEATURE_TYPE, rtft_info_cfg.feature) =3D=3D RTFT_TYPE_MUST= _SUPPORT) + ret =3D -EPROTO; + else + *cur_ft_spt =3D RTFT_TYPE_NOT_EXIST; + break; + case RTFT_TYPE_MUST_SUPPORT: + if (FIELD_GET(FEATURE_TYPE, rtft_info_cfg.feature) =3D=3D RTFT_TYPE_NOT_= EXIST || + FIELD_GET(FEATURE_TYPE, rtft_info_cfg.feature) =3D=3D RTFT_TYPE_NOT_= SUPPORT) + ret =3D -EPROTO; + else + *cur_ft_spt =3D RTFT_TYPE_MUST_SUPPORT; + break; + case RTFT_TYPE_OPTIONAL_SUPPORT: + if (FIELD_GET(FEATURE_TYPE, rtft_info_cfg.feature) =3D=3D RTFT_TYPE_NOT_= EXIST || + FIELD_GET(FEATURE_TYPE, rtft_info_cfg.feature) =3D=3D RTFT_TYPE_NOT_= SUPPORT) { + *cur_ft_spt =3D RTFT_TYPE_NOT_SUPPORT; + } else { + if (FIELD_GET(FEATURE_VER, rtft_info_st.feature) =3D=3D + FIELD_GET(FEATURE_VER, rtft_info_cfg.feature)) + *cur_ft_spt =3D RTFT_TYPE_MUST_SUPPORT; + else + *cur_ft_spt =3D RTFT_TYPE_NOT_SUPPORT; + } + break; + case RTFT_TYPE_SUPPORT_BACKWARD_COMPAT: + if (FIELD_GET(FEATURE_VER, rtft_info_st.feature) >=3D + FIELD_GET(FEATURE_VER, rtft_info_cfg.feature)) + *cur_ft_spt =3D RTFT_TYPE_MUST_SUPPORT; + else + *cur_ft_spt =3D RTFT_TYPE_NOT_EXIST; + break; + default: + ret =3D -EPROTO; + } + + return ret; +} + +static int (*query_rtft_action[FEATURE_CNT])(struct mtk_md_dev *mdev, + void *rt_data, u32 data_len) =3D { + [QUERY_RTFT_ID_MD_PORT_ENUM] =3D mtk_port_status_update, + [QUERY_RTFT_ID_SAP_PORT_ENUM] =3D mtk_port_status_update, +}; + +/* The runtime data skb is owned by the caller (the FSM kthread, which got + * it attached to the STARTUP event); its pointer and length are read once + * from the skb itself, never from a shared field the rx path could rewrit= e. + */ +static int mtk_fsm_parse_hs2_msg(struct mtk_md_fsm *fsm, struct fsm_hs_inf= o *hs_info, + struct sk_buff *skb) +{ + enum runtime_feature_support_type cur_ft_spt; + struct runtime_feature_entry *rtft_entry; + unsigned int ft_id, offset, data_len; + unsigned int rt_data_len; + char *rt_data; + int ret =3D 0; + + if (!skb) + return -EINVAL; + + rt_data =3D skb->data; + rt_data_len =3D skb->len; + + offset =3D sizeof(struct feature_query); + for (ft_id =3D 0; ft_id < FEATURE_CNT; ft_id++) { + if (offset + sizeof(*rtft_entry) > rt_data_len) + break; + + rtft_entry =3D (struct runtime_feature_entry *)(rt_data + offset); + ret =3D mtk_fsm_feature_set_match(&cur_ft_spt, + rtft_entry->support_info, + hs_info->query_ft_set[ft_id]); + if (ret < 0) + break; + + data_len =3D le32_to_cpu(rtft_entry->data_len); + if (data_len > rt_data_len - offset - sizeof(*rtft_entry)) + break; + + if (cur_ft_spt =3D=3D RTFT_TYPE_MUST_SUPPORT && query_rtft_action[ft_id]= ) { + if (!data_len) { + dev_err(fsm->mdev->dev, + "RTFT feature %u: zero data_len\n", ft_id); + ret =3D -EPROTO; + break; + } + ret =3D query_rtft_action[ft_id](fsm->mdev, + rtft_entry->data, + data_len); + } + if (ret < 0) + break; + + offset +=3D sizeof(*rtft_entry) + data_len; + } + + if (ft_id !=3D FEATURE_CNT) { + dev_err((fsm->mdev)->dev, "Unable to handle mistake hs2 msg, ft_id=3D%d\= n", ft_id); + ret =3D -EPROTO; + } + + return ret; +} + +static int mtk_fsm_append_rtft_entries(struct mtk_md_dev *mdev, void *feat= ure_data, + unsigned int *len, struct fsm_hs_info *hs_info, + struct sk_buff *skb) +{ + struct runtime_feature_entry *rtft_entry; + int ft_id, ret =3D 0, rtdata_len =3D 0; + struct feature_query *ft_query; + + if (!skb || skb->len < sizeof(*ft_query)) { + ret =3D -EPROTO; + goto hs_err; + } + + ft_query =3D (struct feature_query *)skb->data; + if (le32_to_cpu(ft_query->head_pattern) !=3D FEATURE_QUERY_PATTERN || + le32_to_cpu(ft_query->tail_pattern) !=3D FEATURE_QUERY_PATTERN) { + ret =3D -EPROTO; + goto hs_err; + } + + /* parse runtime feature query and fill runtime feature entry */ + rtft_entry =3D feature_data; + for (ft_id =3D 0; ft_id < FEATURE_CNT && rtdata_len < RTFT_DATA_SIZE; ft_= id++) { + rtft_entry->feature_id =3D ft_id; + rtft_entry->data_len =3D 0; + + switch (FIELD_GET(FEATURE_TYPE, ft_query->ft_set[ft_id].feature)) { + case RTFT_TYPE_NOT_EXIST: + fallthrough; + case RTFT_TYPE_NOT_SUPPORT: + fallthrough; + case RTFT_TYPE_MUST_SUPPORT: + rtft_entry->support_info =3D ft_query->ft_set[ft_id]; + break; + case RTFT_TYPE_OPTIONAL_SUPPORT: + fallthrough; + case RTFT_TYPE_SUPPORT_BACKWARD_COMPAT: + rtft_entry->support_info.feature =3D FEATURE_TYPE_NOT; + rtft_entry->support_info.feature |=3D FEATURE_VER_0; + break; + } + + rtdata_len +=3D sizeof(*rtft_entry) + le32_to_cpu(rtft_entry->data_len); + rtft_entry =3D (struct runtime_feature_entry *)(feature_data + rtdata_le= n); + } + *len =3D rtdata_len; + return 0; + +hs_err: + *len =3D 0; + return ret; +} + +static int mtk_fsm_send_hs3_msg(struct fsm_hs_info *hs_info, struct sk_buf= f *rt_skb) +{ + struct mtk_md_fsm *fsm =3D container_of(hs_info, struct mtk_md_fsm, hs_in= fo[hs_info->id]); + unsigned int data_len, msg_size =3D 0; + struct ctrl_msg_header *ctrl_msg_h; + struct sk_buff *skb; + int ret; + + skb =3D __dev_alloc_skb(RTFT_DATA_SIZE, GFP_KERNEL); + if (!skb) + return -ENOMEM; + memset(skb->data, 0, RTFT_DATA_SIZE); + + msg_size +=3D sizeof(*ctrl_msg_h); + ctrl_msg_h =3D (struct ctrl_msg_header *)skb->data; + ctrl_msg_h->id =3D cpu_to_le32(CTRL_MSG_HS3); + ctrl_msg_h->ex_msg =3D 0; + ret =3D mtk_fsm_append_rtft_entries(fsm->mdev, + skb->data + sizeof(*ctrl_msg_h), + &data_len, hs_info, rt_skb); + if (ret) { + dev_kfree_skb(skb); + return ret; + } + + ctrl_msg_h->data_len =3D cpu_to_le32(data_len); + msg_size +=3D data_len; + skb_put(skb, msg_size); + ret =3D mtk_port_internal_write(hs_info->ctrl_port, skb); + if (ret <=3D 0) + return ret; + + return 0; +} + +/* Both handlers own the skb on every path and always return 0: the caller + * (mtk_port_rx_dispatch) frees the skb itself on a negative return, so + * returning an error after dev_kfree_skb() would free it twice. + * + * On success the skb is attached to the submitted event (event->data) and + * from then on belongs to the FSM kthread; hs_info->rt_data is kept only + * as the duplicate-HS2 guard and is cleared by the kthread once the event + * has been consumed. That store and the accesses below are the only + * concurrent users of the field, so they are marked: a stale non-NULL read + * at worst drops one retransmitted HS2, which the device repeats. + */ +static int mtk_fsm_sap_ctrl_msg_handler(void *__fsm, struct sk_buff *skb) +{ + struct ctrl_msg_header *ctrl_msg_h; + struct mtk_md_fsm *fsm =3D __fsm; + struct fsm_hs_info *hs_info; + int ret; + + if (skb->len < sizeof(*ctrl_msg_h)) { + dev_kfree_skb(skb); + return 0; + } + + ctrl_msg_h =3D (struct ctrl_msg_header *)skb->data; + skb_pull(skb, sizeof(*ctrl_msg_h)); + + hs_info =3D &fsm->hs_info[HS_ID_SAP]; + if (le32_to_cpu(ctrl_msg_h->id) !=3D CTRL_MSG_HS2) { + dev_err(fsm->mdev->dev, "Invalid SAP ctrl msg id\n"); + dev_kfree_skb(skb); + return 0; + } + + if (READ_ONCE(hs_info->rt_data)) { + dev_warn(fsm->mdev->dev, "Duplicate SAP HS2, dropping\n"); + dev_kfree_skb(skb); + return 0; + } + + WRITE_ONCE(hs_info->rt_data, skb); + ret =3D mtk_fsm_evt_submit(fsm->mdev, FSM_EVT_STARTUP, + hs_info->fsm_flag_hs2, skb, skb->len, 0); + if (ret =3D=3D FSM_EVT_RET_FAIL) { + WRITE_ONCE(hs_info->rt_data, NULL); + dev_kfree_skb(skb); + } + + return 0; +} + +static int mtk_fsm_md_ctrl_msg_handler(void *__fsm, struct sk_buff *skb) +{ + struct ctrl_msg_header *ctrl_msg_h; + struct mtk_md_fsm *fsm =3D __fsm; + struct fsm_hs_info *hs_info; + int ret; + + if (skb->len < sizeof(*ctrl_msg_h)) { + dev_kfree_skb(skb); + return 0; + } + + ctrl_msg_h =3D (struct ctrl_msg_header *)skb->data; + hs_info =3D &fsm->hs_info[HS_ID_MD]; + if (le32_to_cpu(ctrl_msg_h->id) !=3D CTRL_MSG_HS2) { + dev_err(fsm->mdev->dev, "Invalid ctrl msg id\n"); + dev_kfree_skb(skb); + return 0; + } + + if (READ_ONCE(hs_info->rt_data)) { + /* The guarded skb belongs to a still-queued event; only the + * new skb may be freed here. + */ + dev_warn(fsm->mdev->dev, "Duplicate MD HS2, dropping\n"); + dev_kfree_skb(skb); + return 0; + } + + skb_pull(skb, sizeof(*ctrl_msg_h)); + WRITE_ONCE(hs_info->rt_data, skb); + ret =3D mtk_fsm_evt_submit(fsm->mdev, FSM_EVT_STARTUP, + hs_info->fsm_flag_hs2, skb, skb->len, 0); + if (ret =3D=3D FSM_EVT_RET_FAIL) { + WRITE_ONCE(hs_info->rt_data, NULL); + dev_kfree_skb(skb); + } + + return 0; +} + +static int (*ctrl_msg_handler[HS_ID_MAX])(void *__fsm, struct sk_buff *skb= ) =3D { + [HS_ID_MD] =3D mtk_fsm_md_ctrl_msg_handler, + [HS_ID_SAP] =3D mtk_fsm_sap_ctrl_msg_handler, +}; + +static int mtk_fsm_idle_evt_handler(struct mtk_md_dev *mdev, + u32 dev_state, struct mtk_md_fsm *fsm) +{ + u32 dev_cfg =3D dev_state >> DEVICE_CFG_SHIFT & DEVICE_CFG_REGION_MASK; + int hs_id; + + if (dev_cfg =3D=3D DEV_CFG_MD_ONLY) + fsm->hs_done_flag =3D FSM_F_MD_HS_START | FSM_F_MD_HS2_DONE; + else + fsm->hs_done_flag =3D FSM_HS_START_MASK | FSM_HS2_DONE_MASK; + + /* On failure keep the handshake channels masked and report it, so + * the device's next boot-flow notification can retrigger us. + */ + if (mtk_fsm_evt_submit(mdev, FSM_EVT_STARTUP, FSM_F_DFLT, + NULL, 0, 0) =3D=3D FSM_EVT_RET_FAIL) { + dev_err(mdev->dev, "Failed to submit STARTUP evt, waiting for retry\n"); + return -ENOMEM; + } + + for (hs_id =3D 0; hs_id < HS_ID_MAX; hs_id++) + mtk_pci_unmask_ext_evt(mdev, fsm->hs_info[hs_id].mhccif_ch); + + return 0; +} + +static int mtk_fsm_early_bootup_handler(u32 status, void *__fsm) +{ + struct mtk_md_fsm *fsm =3D __fsm; + struct mtk_md_dev *mdev; + u32 dev_state, dev_stage; + + mdev =3D fsm->mdev; + mtk_pci_mask_ext_evt(mdev, status); + mtk_pci_clear_ext_evt(mdev, status); + + dev_state =3D mtk_pci_get_dev_state(mdev); + dev_stage =3D dev_state & REGION_BITMASK; + if (dev_stage >=3D DEV_STAGE_MAX) { + dev_err(mdev->dev, "Invalid dev state 0x%x\n", dev_state); + goto exit; + } + + if (dev_state =3D=3D fsm->last_dev_state) + goto exit; + + /* Only latch the state once it has been acted on; a failed submit + * leaves last_dev_state unchanged so the repeated notification is + * not filtered out. + */ + if (dev_stage =3D=3D DEV_STAGE_IDLE && + mtk_fsm_idle_evt_handler(mdev, dev_state, fsm)) + goto exit; + + fsm->last_dev_state =3D dev_state; + +exit: + /* Re-arm the channel masked on entry: the device notifies once + * per boot stage, so leaving it masked drops every later stage. + * A device still booting when the driver binds would never + * deliver its DEV_STAGE_IDLE notification, and a STARTUP submit + * that failed above would never see a second one to retry on. + */ + mtk_pci_unmask_ext_evt(mdev, status); + return 0; +} + +static int mtk_fsm_ctrl_ch_start(struct mtk_md_fsm *fsm, struct fsm_hs_inf= o *hs_info, int flag) +{ + if (!hs_info->ctrl_port) { + hs_info->ctrl_port =3D mtk_port_internal_open(fsm->mdev, hs_info->port_n= ame, flag); + if (!hs_info->ctrl_port) { + dev_err(fsm->mdev->dev, "Failed to open ctrl port(%s)\n", + hs_info->port_name); + return -ENODEV; + } + + mtk_port_internal_recv_register(hs_info->ctrl_port, + ctrl_msg_handler[hs_info->id], fsm); + } + + return 0; +} + +static void mtk_fsm_ctrl_ch_stop(struct mtk_md_fsm *fsm) +{ + struct fsm_hs_info *hs_info; + int hs_id; + + for (hs_id =3D 0; hs_id < HS_ID_MAX; hs_id++) { + hs_info =3D &fsm->hs_info[hs_id]; + if (hs_info->ctrl_port) { + mtk_port_internal_close(hs_info->ctrl_port); + hs_info->ctrl_port =3D NULL; + } + } +} + +static void mtk_fsm_switch_state(struct mtk_md_fsm *fsm, + enum mtk_fsm_state to_state, struct mtk_fsm_evt *event) +{ + char fsm_info[MTK_FSM_INFO_LEN]; + struct mtk_fsm_notifier *nt; + struct mtk_fsm_param param; + + param.from =3D fsm->state; + param.to =3D to_state; + param.evt_id =3D event ? event->id : FSM_EVT_MAX; + param.fsm_flag =3D event ? event->fsm_flag : FSM_F_DFLT; + + mutex_lock(&fsm->notifier_lock); + list_for_each_entry(nt, &fsm->pre_notifiers, entry) + nt->cb(¶m, nt->data); + + fsm->state =3D to_state; + fsm->fsm_flag |=3D event ? event->fsm_flag : FSM_F_DFLT; + + snprintf(fsm_info, MTK_FSM_INFO_LEN, + "state=3D%d, fsm_flag=3D0x%x", to_state, fsm->fsm_flag); + mtk_uevent_notify(fsm->mdev->dev, MTK_UEVENT_FSM, fsm_info); + + list_for_each_entry(nt, &fsm->post_notifiers, entry) + nt->cb(¶m, nt->data); + mutex_unlock(&fsm->notifier_lock); +} + +static int mtk_fsm_startup_act(struct mtk_md_fsm *fsm, struct mtk_fsm_evt = *event) +{ + enum mtk_fsm_state to_state =3D FSM_STATE_BOOTUP; + struct mtk_md_dev *mdev =3D fsm->mdev; + struct fsm_hs_info *hs_info; + struct sk_buff *skb =3D NULL; + int ret =3D 0; + + if (event->fsm_flag & FSM_HS2_DONE_MASK) { + /* The HS2 event owns its runtime-data skb. */ + skb =3D event->data; + hs_info =3D &fsm->hs_info[(event->fsm_flag & FSM_F_MD_HS2_DONE) ? + HS_ID_MD : HS_ID_SAP]; + } else { + /* HS1 events carry the hs_info; the bare STARTUP carries NULL. */ + hs_info =3D event->data; + } + + if (fsm->state !=3D FSM_STATE_ON && fsm->state !=3D FSM_STATE_BOOTUP) { + ret =3D -EPROTO; + goto free_skb; + } + + if (!(event->fsm_flag & (FSM_HS_START_MASK | FSM_HS2_DONE_MASK))) { + /* Bare STARTUP from the boot-flow notification: only the + * state switch. + */ + if (fsm->state !=3D FSM_STATE_BOOTUP) + mtk_fsm_switch_state(fsm, to_state, event); + return 0; + } + + if (event->fsm_flag & FSM_HS_START_MASK) { + /* An HS1 that raced ahead of the bare STARTUP must both + * switch ON to BOOTUP and run the handshake below; returning + * after the switch would consume the only trigger. + */ + mtk_fsm_switch_state(fsm, to_state, event); + + ret =3D mtk_fsm_ctrl_ch_start(fsm, hs_info, O_NONBLOCK); + if (!ret) + ret =3D mtk_fsm_send_hs1_msg(hs_info); + if (ret) + goto hs_err; + } else if (event->fsm_flag & FSM_HS2_DONE_MASK) { + ret =3D mtk_fsm_parse_hs2_msg(fsm, hs_info, skb); + if (!ret) + ret =3D mtk_fsm_send_hs3_msg(hs_info, skb); + dev_kfree_skb(skb); + skb =3D NULL; + /* re-open the duplicate guard */ + WRITE_ONCE(hs_info->rt_data, NULL); + if (ret) + goto hs_err; + mtk_fsm_switch_state(fsm, to_state, event); + } + + if (((fsm->fsm_flag | event->fsm_flag) & fsm->hs_done_flag) =3D=3D fsm->h= s_done_flag) { + int user; + + for (user =3D 0; user < MTK_FSM_HIF_USER_MAX; user++) { + if (fsm->hif_err[user]) { + dev_err(mdev->dev, + "Refusing READY: HIF user %d init failed with %d\n", + user, fsm->hif_err[user]); + return fsm->hif_err[user]; + } + } + + to_state =3D FSM_STATE_READY; + mtk_fsm_switch_state(fsm, to_state, NULL); + } + + return 0; + +free_skb: + if (skb) { + dev_kfree_skb(skb); + WRITE_ONCE(hs_info->rt_data, NULL); + } +hs_err: + for (int hs_id =3D 0; hs_id < HS_ID_MAX; hs_id++) + mtk_pci_unmask_ext_evt(mdev, fsm->hs_info[hs_id].mhccif_ch); + dev_err(mdev->dev, "Failed to hs with device %d:0x%x, ret=3D%d", + fsm->state, fsm->fsm_flag, ret); + return ret; +} + +static void mtk_fsm_evt_release(struct kref *kref) +{ + struct mtk_fsm_evt *event =3D container_of(kref, struct mtk_fsm_evt, kref= ); + + kfree(event); +} + +static void mtk_fsm_evt_put(struct mtk_fsm_evt *event) +{ + kref_put(&event->kref, mtk_fsm_evt_release); +} + +static void mtk_fsm_evt_finish(struct mtk_md_fsm *fsm, + struct mtk_fsm_evt *event, int retval) +{ + if (event->mode & EVT_MODE_BLOCKING) { + event->status =3D retval; + wake_up(&fsm->evt_waitq); + } + mtk_fsm_evt_put(event); +} + +static void mtk_fsm_evt_cleanup(struct mtk_md_fsm *fsm, struct list_head *= evtq) +{ + struct mtk_fsm_evt *event, *tmp; + + list_for_each_entry_safe(event, tmp, evtq, entry) { + list_del(&event->entry); + mtk_fsm_evt_finish(fsm, event, FSM_EVT_RET_FAIL); + } +} + +static int mtk_fsm_enter_off_state(struct mtk_md_fsm *fsm, struct mtk_fsm_= evt *event) +{ + struct mtk_md_dev *mdev =3D fsm->mdev; + int hs_id; + + if (fsm->state =3D=3D FSM_STATE_OFF || fsm->state =3D=3D FSM_STATE_INVALI= D) + return -EPROTO; + + mtk_pci_mask_ext_evt(mdev, DEV_EVT_D2H_BOOT_FLOW_SYNC); + for (hs_id =3D 0; hs_id < HS_ID_MAX; hs_id++) + mtk_pci_mask_ext_evt(mdev, fsm->hs_info[hs_id].mhccif_ch); + + mtk_fsm_ctrl_ch_stop(fsm); + mtk_fsm_switch_state(fsm, FSM_STATE_OFF, event); + + return 0; +} + +static int mtk_fsm_dev_rm_act(struct mtk_md_fsm *fsm, struct mtk_fsm_evt *= event) +{ + unsigned long flags; + + spin_lock_irqsave(&fsm->evtq_lock, flags); + set_bit(EVT_TF_GATECLOSED, &fsm->t_flag); + mtk_fsm_evt_cleanup(fsm, &fsm->evtq); + spin_unlock_irqrestore(&fsm->evtq_lock, flags); + + return mtk_fsm_enter_off_state(fsm, event); +} + +static int mtk_fsm_hs1_handler(u32 status, void *__hs_info) +{ + struct fsm_hs_info *hs_info =3D __hs_info; + struct mtk_md_dev *mdev; + struct mtk_md_fsm *fsm; + + fsm =3D container_of(hs_info, struct mtk_md_fsm, hs_info[hs_info->id]); + mdev =3D fsm->mdev; + /* Only consume the notification once the event is queued; on a + * failed submit the channel stays unmasked and uncleared so the + * device's retry still reaches us. + */ + if (mtk_fsm_evt_submit(mdev, FSM_EVT_STARTUP, hs_info->fsm_flag_hs1, + hs_info, sizeof(*hs_info), 0) =3D=3D FSM_EVT_RET_FAIL) { + dev_err(mdev->dev, "Failed to submit HS1 evt(hs%d), waiting for retry\n", + hs_info->id); + return -ENOMEM; + } + mtk_pci_mask_ext_evt(mdev, hs_info->mhccif_ch); + mtk_pci_clear_ext_evt(mdev, hs_info->mhccif_ch); + + return 0; +} + +static void mtk_fsm_hs_info_init_by_hsid(struct mtk_md_fsm *fsm, int hs_id) +{ + struct fsm_hs_info *hs_info; + + if (hs_id < 0 || hs_id >=3D HS_ID_MAX) { + dev_warn((fsm->mdev)->dev, "hs_id =3D %d, invalid.\n", hs_id); + return; + } + + hs_info =3D &fsm->hs_info[hs_id]; + hs_info->id =3D hs_id; + hs_info->ctrl_port =3D NULL; + hs_info->rt_data =3D NULL; + switch (hs_id) { + case HS_ID_MD: + snprintf(hs_info->port_name, PORT_NAME_LEN, "MDCTRL"); + hs_info->mhccif_ch =3D DEV_EVT_D2H_ASYNC_HS_NOTIFY_MD; + hs_info->fsm_flag_hs1 =3D FSM_F_MD_HS_START; + hs_info->fsm_flag_hs2 =3D FSM_F_MD_HS2_DONE; + hs_info->query_ft_set[QUERY_RTFT_ID_MD_PORT_ENUM].feature =3D + FIELD_PREP(FEATURE_TYPE, RTFT_TYPE_MUST_SUPPORT); + hs_info->query_ft_set[QUERY_RTFT_ID_MD_PORT_ENUM].feature |=3D + FIELD_PREP(FEATURE_VER, 0); + hs_info->query_ft_set[QUERY_RTFT_ID_MD_PORT_CFG].feature =3D + FIELD_PREP(FEATURE_TYPE, RTFT_TYPE_NOT_SUPPORT); + break; + case HS_ID_SAP: + snprintf(hs_info->port_name, PORT_NAME_LEN, "SAPCTRL"); + hs_info->mhccif_ch =3D DEV_EVT_D2H_ASYNC_HS_NOTIFY_SAP; + hs_info->fsm_flag_hs1 =3D FSM_F_SAP_HS_START; + hs_info->fsm_flag_hs2 =3D FSM_F_SAP_HS2_DONE; + hs_info->query_ft_set[QUERY_RTFT_ID_SAP_PORT_ENUM].feature =3D + FIELD_PREP(FEATURE_TYPE, RTFT_TYPE_MUST_SUPPORT); + hs_info->query_ft_set[QUERY_RTFT_ID_SAP_PORT_ENUM].feature |=3D + FIELD_PREP(FEATURE_VER, 0); + break; + } +} + +static int mtk_fsm_hs_info_init(struct mtk_md_fsm *fsm) +{ + struct mtk_md_dev *mdev =3D fsm->mdev; + struct fsm_hs_info *hs_info; + int hs_id, ret; + + for (hs_id =3D 0; hs_id < HS_ID_MAX; hs_id++) { + mtk_fsm_hs_info_init_by_hsid(fsm, hs_id); + hs_info =3D &fsm->hs_info[hs_id]; + ret =3D mtk_pci_register_ext_evt(mdev, hs_info->mhccif_ch, + mtk_fsm_hs1_handler, hs_info); + if (ret) + goto err_unregister; + } + + return 0; + +err_unregister: + /* All or nothing: the caller must not have to know how far we got. */ + while (--hs_id >=3D 0) + mtk_pci_unregister_ext_evt(mdev, fsm->hs_info[hs_id].mhccif_ch); + + return ret; +} + +static void mtk_fsm_hs_info_exit(struct mtk_md_fsm *fsm) +{ + struct mtk_md_dev *mdev =3D fsm->mdev; + struct fsm_hs_info *hs_info; + int hs_id; + + for (hs_id =3D 0; hs_id < HS_ID_MAX; hs_id++) { + hs_info =3D &fsm->hs_info[hs_id]; + mtk_pci_unregister_ext_evt(mdev, hs_info->mhccif_ch); + } +} + +static int mtk_fsm_dev_add_act(struct mtk_md_fsm *fsm, struct mtk_fsm_evt = *event) +{ + if (fsm->state !=3D FSM_STATE_OFF && fsm->state !=3D FSM_STATE_INVALID) + return -EPROTO; + + /* a fresh device lifecycle starts with a clean HIF error record */ + memset(fsm->hif_err, 0, sizeof(fsm->hif_err)); + mtk_fsm_switch_state(fsm, FSM_STATE_ON, event); + mtk_pci_unmask_ext_evt(fsm->mdev, DEV_EVT_D2H_BOOT_FLOW_SYNC); + + return 0; +} + +static int (*evts_act_tbl[FSM_EVT_MAX])(struct mtk_md_fsm *__fsm, struct m= tk_fsm_evt *event) =3D { + [FSM_EVT_STARTUP] =3D mtk_fsm_startup_act, + [FSM_EVT_DEV_RM] =3D mtk_fsm_dev_rm_act, + [FSM_EVT_DEV_ADD] =3D mtk_fsm_dev_add_act, +}; + +int mtk_fsm_start(struct mtk_md_dev *mdev) +{ + struct mtk_md_fsm *fsm =3D mdev->fsm; + + if (!fsm) + return -EINVAL; + + if (!fsm->fsm_handler) + return -EFAULT; + + wake_up_process(fsm->fsm_handler); + return 0; +} + +/* Record a transport-plane init failure so the STARTUP path refuses the + * BOOTUP-to-READY promotion: the notifier callbacks are void and cannot + * reject a transition, so without this the FSM would advertise readiness + * on top of a data path that was never created. Each producer owns one + * slot and stores its latest result there, so a retried init that + * succeeds clears the failure it recorded before; DEV_ADD clears all. + */ +void mtk_fsm_hif_err_record(struct mtk_md_dev *mdev, enum mtk_fsm_hif_user= user, int err) +{ + struct mtk_md_fsm *fsm =3D mdev->fsm; + + if (fsm) + fsm->hif_err[user] =3D err; +} + +static void mkt_fsm_notifier_cleanup(struct mtk_md_dev *mdev, struct list_= head *ntq) +{ + struct mtk_fsm_notifier *nt, *tmp; + + list_for_each_entry_safe(nt, tmp, ntq, entry) { + list_del(&nt->entry); + dev_warn(mdev->dev, "Having to free notifier(%d) by FSM!\n", nt->id); + kfree(nt); + } +} + +static void mtk_fsm_notifier_insert(struct mtk_fsm_notifier *notifier, str= uct list_head *head) +{ + struct mtk_fsm_notifier *nt; + + list_for_each_entry(nt, head, entry) { + if (notifier->prio > nt->prio) { + list_add(¬ifier->entry, nt->entry.prev); + return; + } + } + list_add_tail(¬ifier->entry, head); +} + +int mtk_fsm_notifier_register(struct mtk_md_dev *mdev, enum mtk_user_id id, + void (*cb)(struct mtk_fsm_param *, void *data), + void *data, enum mtk_fsm_prio prio, bool is_pre) +{ + struct mtk_md_fsm *fsm =3D mdev->fsm; + struct mtk_fsm_notifier *notifier; + + if (!fsm) + return -EINVAL; + + if (id >=3D MTK_USER_MAX || !cb || prio >=3D FSM_PRIO_MAX) + return -EINVAL; + + notifier =3D kzalloc_obj(*notifier); + if (!notifier) + return -ENOMEM; + + INIT_LIST_HEAD(¬ifier->entry); + notifier->id =3D id; + notifier->cb =3D cb; + notifier->data =3D data; + notifier->prio =3D prio; + + mutex_lock(&fsm->notifier_lock); + if (is_pre) + mtk_fsm_notifier_insert(notifier, &fsm->pre_notifiers); + else + mtk_fsm_notifier_insert(notifier, &fsm->post_notifiers); + mutex_unlock(&fsm->notifier_lock); + + return 0; +} + +int mtk_fsm_notifier_unregister(struct mtk_md_dev *mdev, enum mtk_user_id = id) +{ + struct mtk_md_fsm *fsm =3D mdev->fsm; + struct mtk_fsm_notifier *nt, *tmp; + + if (!fsm) + return -EINVAL; + + mutex_lock(&fsm->notifier_lock); + list_for_each_entry_safe(nt, tmp, &fsm->pre_notifiers, entry) { + if (nt->id =3D=3D id) { + list_del(&nt->entry); + kfree(nt); + break; + } + } + list_for_each_entry_safe(nt, tmp, &fsm->post_notifiers, entry) { + if (nt->id =3D=3D id) { + list_del(&nt->entry); + kfree(nt); + break; + } + } + mutex_unlock(&fsm->notifier_lock); + return 0; +} + +int mtk_fsm_evt_submit(struct mtk_md_dev *mdev, + enum mtk_fsm_evt_id id, enum mtk_fsm_flag flag, + void *data, unsigned int len, unsigned char mode) +{ + struct mtk_md_fsm *fsm =3D mdev->fsm; + struct mtk_fsm_evt *event; + unsigned long flags; + int ret =3D 0; + + if (!fsm || id >=3D FSM_EVT_MAX) { + dev_err(mdev->dev, "Invalid param!\n"); + return FSM_EVT_RET_FAIL; + } + + if (test_bit(EVT_TF_GATECLOSED, &fsm->t_flag)) { + dev_err(mdev->dev, "Failed to submit evt, fsm has been removed!\n"); + return FSM_EVT_RET_FAIL; + } + + event =3D kzalloc(sizeof(*event), + (in_hardirq() || in_softirq() || irqs_disabled()) ? + GFP_ATOMIC : GFP_KERNEL); + if (!event) + return FSM_EVT_RET_FAIL; + + kref_init(&event->kref); + event->mdev =3D mdev; + event->id =3D id; + event->fsm_flag =3D flag; + event->status =3D FSM_EVT_RET_ONGOING; + event->data =3D data; + event->len =3D len; + event->mode =3D mode; + + spin_lock_irqsave(&fsm->evtq_lock, flags); + if (test_bit(EVT_TF_GATECLOSED, &fsm->t_flag)) { + spin_unlock_irqrestore(&fsm->evtq_lock, flags); + mtk_fsm_evt_put(event); + dev_err(mdev->dev, "Failed to add event, fsm dev has been removed!\n"); + return FSM_EVT_RET_FAIL; + } + + kref_get(&event->kref); + if (mode & EVT_MODE_TOHEAD) + list_add(&event->entry, &fsm->evtq); + else + list_add_tail(&event->entry, &fsm->evtq); + if (fsm->fsm_handler) + wake_up_process(fsm->fsm_handler); + spin_unlock_irqrestore(&fsm->evtq_lock, flags); + + if (mode & EVT_MODE_BLOCKING) { + ret =3D wait_event_timeout(fsm->evt_waitq, + (event->status !=3D 0), BLOCKING_EVT_TIMEOUT); + if (!ret && event->status !=3D FSM_EVT_RET_DONE) { + dev_err(mdev->dev, "Handling fsm blocking event timeout!\n"); + ret =3D -ETIMEDOUT; + } else { + ret =3D event->status; + } + } + mtk_fsm_evt_put(event); + + return ret; +} + +static int mtk_fsm_evt_handler(void *__fsm) +{ + struct mtk_md_fsm *fsm =3D __fsm; + struct mtk_fsm_evt *event; + unsigned long flags; + int ret; + +wake_up: + set_current_state(TASK_INTERRUPTIBLE); + while (!kthread_should_stop() && !list_empty(&fsm->evtq)) { + set_current_state(TASK_RUNNING); + spin_lock_irqsave(&fsm->evtq_lock, flags); + event =3D list_first_entry(&fsm->evtq, struct mtk_fsm_evt, entry); + list_del(&event->entry); + spin_unlock_irqrestore(&fsm->evtq_lock, flags); + + if (event->id < FSM_EVT_MAX) { + ret =3D evts_act_tbl[event->id](fsm, event); + if (ret) { + dev_err((fsm->mdev)->dev, + "Failed to handle evt, fsm state =3D %d, ret =3D %d\n", + fsm->state, ret); + mtk_fsm_evt_finish(fsm, event, FSM_EVT_RET_FAIL); + } else { + mtk_fsm_evt_finish(fsm, event, FSM_EVT_RET_DONE); + } + } else { + mtk_fsm_evt_finish(fsm, event, FSM_EVT_RET_DONE); + } + } + + if (kthread_should_stop()) { + set_current_state(TASK_RUNNING); + return 0; + } + + schedule(); + goto wake_up; +} + +int mtk_fsm_init(struct mtk_md_dev *mdev) +{ + struct mtk_md_fsm *fsm; + int ret; + + fsm =3D devm_kzalloc(mdev->dev, sizeof(*fsm), GFP_KERNEL); + if (!fsm) + return -ENOMEM; + + fsm->fsm_handler =3D kthread_create(mtk_fsm_evt_handler, fsm, "fsm_evt_th= read%d_%s", + mdev->hw_ver, mdev->dev_str); + if (IS_ERR(fsm->fsm_handler)) + return PTR_ERR(fsm->fsm_handler); + + /* Keep our own reference so that kthread_stop() in the teardown + * paths stays safe even if the thread has already exited (e.g. + * it was killed by an oops in a listener it called): without it + * the task_struct may be freed and kthread_stop() would hit a + * use-after-free through its internal get_task_struct(). + */ + get_task_struct(fsm->fsm_handler); + + fsm->mdev =3D mdev; + fsm->state =3D FSM_STATE_INVALID; + fsm->fsm_flag =3D FSM_F_DFLT; + + INIT_LIST_HEAD(&fsm->evtq); + spin_lock_init(&fsm->evtq_lock); + init_waitqueue_head(&fsm->evt_waitq); + + INIT_LIST_HEAD(&fsm->pre_notifiers); + INIT_LIST_HEAD(&fsm->post_notifiers); + mutex_init(&fsm->notifier_lock); + + ret =3D mtk_pci_register_ext_evt(mdev, DEV_EVT_D2H_BOOT_FLOW_SYNC, + mtk_fsm_early_bootup_handler, fsm); + if (ret) + goto err_stop_thread; + + ret =3D mtk_fsm_hs_info_init(fsm); + if (ret) + goto err_unregister_evt; + + mdev->fsm =3D fsm; + return 0; + +err_unregister_evt: + mtk_pci_unregister_ext_evt(mdev, DEV_EVT_D2H_BOOT_FLOW_SYNC); +err_stop_thread: + /* mdev->fsm is never published on failure, so mtk_fsm_exit() cannot + * stop the parked kthread for us; it must be stopped here. + */ + kthread_stop(fsm->fsm_handler); + put_task_struct(fsm->fsm_handler); + return ret; +} + +/* Close the event gate and join the handler thread, then release the + * control ports while the port table they reference is still alive. + * Called from the removal path before the transport plane is dismantled; + * every step is idempotent, so mtk_fsm_exit() may repeat them safely. + */ +int mtk_fsm_stop(struct mtk_md_dev *mdev) +{ + struct mtk_md_fsm *fsm =3D mdev->fsm; + struct task_struct *handler; + unsigned long flags; + + if (!fsm) + return -EINVAL; + + spin_lock_irqsave(&fsm->evtq_lock, flags); + set_bit(EVT_TF_GATECLOSED, &fsm->t_flag); + handler =3D fsm->fsm_handler; + fsm->fsm_handler =3D NULL; + spin_unlock_irqrestore(&fsm->evtq_lock, flags); + + if (handler) { + kthread_stop(handler); + put_task_struct(handler); + } + + spin_lock_irqsave(&fsm->evtq_lock, flags); + mtk_fsm_evt_cleanup(fsm, &fsm->evtq); + spin_unlock_irqrestore(&fsm->evtq_lock, flags); + + mtk_fsm_ctrl_ch_stop(fsm); + + return 0; +} + +int mtk_fsm_exit(struct mtk_md_dev *mdev) +{ + struct mtk_md_fsm *fsm =3D mdev->fsm; + struct task_struct *handler; + unsigned long flags; + + if (!fsm) + return -EINVAL; + + spin_lock_irqsave(&fsm->evtq_lock, flags); + set_bit(EVT_TF_GATECLOSED, &fsm->t_flag); + handler =3D fsm->fsm_handler; + fsm->fsm_handler =3D NULL; + spin_unlock_irqrestore(&fsm->evtq_lock, flags); + + if (handler) { + kthread_stop(handler); + put_task_struct(handler); + } + + spin_lock_irqsave(&fsm->evtq_lock, flags); + if (WARN_ON(!list_empty(&fsm->evtq))) + mtk_fsm_evt_cleanup(fsm, &fsm->evtq); + spin_unlock_irqrestore(&fsm->evtq_lock, flags); + + /* Release the port references and recv callbacks even when the + * DEV_RM event never ran (its enter-off path is the only other + * caller); mtk_fsm_ctrl_ch_stop() is idempotent. + */ + mtk_fsm_ctrl_ch_stop(fsm); + + for (int i =3D 0; i < HS_ID_MAX; i++) { + if (fsm->hs_info[i].rt_data) { + dev_kfree_skb(fsm->hs_info[i].rt_data); + fsm->hs_info[i].rt_data =3D NULL; + } + } + + mutex_lock(&fsm->notifier_lock); + mkt_fsm_notifier_cleanup(mdev, &fsm->pre_notifiers); + mkt_fsm_notifier_cleanup(mdev, &fsm->post_notifiers); + mutex_unlock(&fsm->notifier_lock); + + mtk_pci_unregister_ext_evt(mdev, DEV_EVT_D2H_BOOT_FLOW_SYNC); + mtk_fsm_hs_info_exit(fsm); + mdev->fsm =3D NULL; + + return 0; +} diff --git a/drivers/net/wwan/t9xx/mtk_fsm.h b/drivers/net/wwan/t9xx/mtk_fs= m.h new file mode 100644 index 000000000000..1284a6188a8e --- /dev/null +++ b/drivers/net/wwan/t9xx/mtk_fsm.h @@ -0,0 +1,158 @@ +/* SPDX-License-Identifier: GPL-2.0-only + * + * Copyright (c) 2022, MediaTek Inc. + */ + +#ifndef __MTK_FSM_H__ +#define __MTK_FSM_H__ + +#include + +#include "mtk_dev.h" + +#define FEATURE_CNT (64) +#define FEATURE_QUERY_PATTERN (0x49434343) + +#define FEATURE_TYPE GENMASK(3, 0) +#define FEATURE_VER GENMASK(7, 4) + +#define FEATURE_TYPE_NOT FIELD_PREP(FEATURE_TYPE, RTFT_TYPE_NOT_SUPPORT) +#define FEATURE_VER_0 FIELD_PREP(FEATURE_VER, 0) + +#define EVT_MODE_BLOCKING (0x01) +#define EVT_MODE_TOHEAD (0x02) + +#define FSM_EVT_RET_FAIL (-1) +#define FSM_EVT_RET_ONGOING (0) +#define FSM_EVT_RET_DONE (1) + +enum mtk_fsm_flag { + FSM_F_DFLT =3D 0, + FSM_F_SAP_HS_START =3D BIT(0), + FSM_F_SAP_HS2_DONE =3D BIT(1), + FSM_F_MD_HS_START =3D BIT(2), + FSM_F_MD_HS2_DONE =3D BIT(3), +}; + +enum mtk_fsm_state { + FSM_STATE_INVALID =3D 0, + FSM_STATE_OFF, + FSM_STATE_ON, + FSM_STATE_BOOTUP, + FSM_STATE_READY, +}; + +enum mtk_fsm_evt_id { + FSM_EVT_STARTUP =3D 0, + FSM_EVT_DEV_RM, + FSM_EVT_DEV_ADD, + FSM_EVT_MAX +}; + +enum mtk_fsm_prio { + FSM_PRIO_1 =3D 1, + FSM_PRIO_MAX +}; + +/* One slot per transport-plane producer, so that a retried init clears + * only its own record. + */ +enum mtk_fsm_hif_user { + MTK_FSM_HIF_USER_CTRL =3D 0, + MTK_FSM_HIF_USER_CLDMA0, + MTK_FSM_HIF_USER_CLDMA1, + MTK_FSM_HIF_USER_MAX +}; + +struct mtk_fsm_param { + enum mtk_fsm_state from; + enum mtk_fsm_state to; + enum mtk_fsm_evt_id evt_id; + enum mtk_fsm_flag fsm_flag; +}; + +#define PORT_NAME_LEN 20 + +enum handshake_info_id { + HS_ID_MD =3D 0, + HS_ID_SAP, + HS_ID_MAX +}; + +struct runtime_feature_info { + u8 feature; +}; + +struct fsm_hs_info { + unsigned char id; + void *ctrl_port; + char port_name[PORT_NAME_LEN]; + unsigned int mhccif_ch; + unsigned int fsm_flag_hs1; + unsigned int fsm_flag_hs2; + /* the feature that the device should support */ + struct runtime_feature_info query_ft_set[FEATURE_CNT]; + /* Duplicate-HS2 guard only: the in-flight runtime-data skb itself + * is owned by the STARTUP event it was submitted with. + */ + void *rt_data; +}; + +struct mtk_md_fsm { + struct mtk_md_dev *mdev; + struct task_struct *fsm_handler; + struct fsm_hs_info hs_info[HS_ID_MAX]; + unsigned int hs_done_flag; + unsigned long t_flag; + u32 last_dev_state; + /* last transport-plane init result per producer; any non-zero entry + * refuses READY + */ + int hif_err[MTK_FSM_HIF_USER_MAX]; + enum mtk_fsm_state state; + unsigned int fsm_flag; + struct list_head evtq; + /* protect evtq */ + spinlock_t evtq_lock; + /* waitq for fsm blocking submit */ + wait_queue_head_t evt_waitq; + struct list_head pre_notifiers; + struct list_head post_notifiers; + /* protects pre_notifiers and post_notifiers lists */ + struct mutex notifier_lock; +}; + +struct mtk_fsm_evt { + struct list_head entry; + struct kref kref; + struct mtk_md_dev *mdev; + enum mtk_fsm_evt_id id; + unsigned int fsm_flag; + int status; + unsigned char mode; + unsigned int len; + void *data; +}; + +struct mtk_fsm_notifier { + struct list_head entry; + enum mtk_user_id id; + void (*cb)(struct mtk_fsm_param *param, void *data); + void *data; + enum mtk_fsm_prio prio; +}; + +int mtk_fsm_init(struct mtk_md_dev *mdev); +int mtk_fsm_exit(struct mtk_md_dev *mdev); +int mtk_fsm_start(struct mtk_md_dev *mdev); +int mtk_fsm_stop(struct mtk_md_dev *mdev); +void mtk_fsm_hif_err_record(struct mtk_md_dev *mdev, enum mtk_fsm_hif_user= user, int err); +int mtk_fsm_notifier_register(struct mtk_md_dev *mdev, enum mtk_user_id id, + void (*cb)(struct mtk_fsm_param *, void *data), + void *data, enum mtk_fsm_prio prio, bool is_pre); +int mtk_fsm_notifier_unregister(struct mtk_md_dev *mdev, enum mtk_user_id = id); +int mtk_fsm_evt_submit(struct mtk_md_dev *mdev, + enum mtk_fsm_evt_id id, enum mtk_fsm_flag flag, + void *data, unsigned int len, unsigned char mode); + +#endif /* __MTK_FSM_H__ */ diff --git a/drivers/net/wwan/t9xx/mtk_port.c b/drivers/net/wwan/t9xx/mtk_p= ort.c index a8ff06ab4de4..d7c50519e1fd 100644 --- a/drivers/net/wwan/t9xx/mtk_port.c +++ b/drivers/net/wwan/t9xx/mtk_port.c @@ -550,6 +550,9 @@ int mtk_port_status_update(struct mtk_md_dev *mdev, voi= d *data, u32 data_len) if (unlikely(!mdev || !msg)) return -EINVAL; =20 + if (data_len < sizeof(*msg)) + return -EPROTO; + ctrl_blk =3D mdev->ctrl_blk; port_mngr =3D ctrl_blk->port_mngr; if (le16_to_cpu(msg->version) !=3D MTK_PORT_ENUM_VER || @@ -653,6 +656,69 @@ int mtk_port_ch_disable(struct mtk_port *port) return ret; } =20 +static void mtk_port_disable(struct mtk_port_mngr *port_mngr) +{ + struct radix_tree_iter iter; + struct mtk_port *port; + void __rcu **slot; + int tbl_type; + + tbl_type =3D PORT_TBL_SAP; + do { + radix_tree_for_each_slot(slot, &port_mngr->port_tbl[tbl_type], + &iter, 0) { + MTK_PORT_SEARCH_FROM_RADIX_TREE(port, slot); + MTK_PORT_INTERNAL_NODE_CHECK(port, slot, iter); + ports_ops[port->info.type]->disable(port); + } + } while (++tbl_type < PORT_TBL_MAX); +} + +void mtk_port_mngr_fsm_state_handler(struct mtk_fsm_param *fsm_param, void= *arg) +{ + struct mtk_port_mngr *port_mngr; + + if (!fsm_param || !arg) + return; + + port_mngr =3D arg; + + switch (fsm_param->to) { + case FSM_STATE_OFF: + mtk_port_disable(port_mngr); + break; + default: + break; + } +} + +void mtk_port_mngr_fsm_state_handler_late(struct mtk_fsm_param *fsm_param,= void *arg) +{ + struct mtk_port_mngr *port_mngr; + struct mtk_port *port; + + if (!fsm_param || !arg) + return; + + port_mngr =3D arg; + + switch (fsm_param->to) { + case FSM_STATE_BOOTUP: + if (fsm_param->fsm_flag & FSM_F_MD_HS_START) { + port =3D mtk_port_search_by_id(port_mngr, CCCI_CONTROL_RX); + if (port) + ports_ops[port->info.type]->enable(port); + } else if (fsm_param->fsm_flag & FSM_F_SAP_HS_START) { + port =3D mtk_port_search_by_id(port_mngr, CCCI_SAP_CONTROL_RX); + if (port) + ports_ops[port->info.type]->enable(port); + } + break; + default: + break; + } +} + int mtk_port_mngr_init(struct mtk_ctrl_blk *ctrl_blk, struct mtk_port_cfg = *port_cfg, int port_cnt) { struct mtk_port_mngr *port_mngr; diff --git a/drivers/net/wwan/t9xx/mtk_port.h b/drivers/net/wwan/t9xx/mtk_p= ort.h index 9b8f63ce3f3c..2fc0e450ab6b 100644 --- a/drivers/net/wwan/t9xx/mtk_port.h +++ b/drivers/net/wwan/t9xx/mtk_port.h @@ -124,6 +124,8 @@ int mtk_port_send_data(struct mtk_port *port, void *dat= a, bool blocking, bool fo int mtk_port_status_update(struct mtk_md_dev *mdev, void *data, u32 data_l= en); int mtk_port_ch_enable(struct mtk_port *port); int mtk_port_ch_disable(struct mtk_port *port); +void mtk_port_mngr_fsm_state_handler(struct mtk_fsm_param *fsm_param, void= *arg); +void mtk_port_mngr_fsm_state_handler_late(struct mtk_fsm_param *fsm_param,= void *arg); int mtk_port_mngr_init(struct mtk_ctrl_blk *ctrl_blk, struct mtk_port_cfg = *port_cfg, int port_cnt); void mtk_port_mngr_exit(struct mtk_ctrl_blk *ctrl_blk); void mtk_port_trb_init(struct mtk_port *port, struct trb *trb, enum mtk_tr= b_cmd_type cmd, diff --git a/drivers/net/wwan/t9xx/mtk_utility.h b/drivers/net/wwan/t9xx/mt= k_utility.h new file mode 100644 index 000000000000..a7b5f3f4d54e --- /dev/null +++ b/drivers/net/wwan/t9xx/mtk_utility.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0-only + * + * Copyright (c) 2022, MediaTek Inc. + */ + +#ifndef __MTK_UTILITY_H__ +#define __MTK_UTILITY_H__ + +#include +#include "mtk_dev.h" + +#define MTK_UEVENT_INFO_LEN 128 + +/* MTK uevent */ +enum mtk_uevent_id { + MTK_UEVENT_FSM =3D 1, +}; + +static inline void mtk_uevent_notify(struct device *dev, enum mtk_uevent_i= d id, const char *info) +{ + char buf[MTK_UEVENT_INFO_LEN]; + char *ext[2] =3D {NULL, NULL}; + + snprintf(buf, MTK_UEVENT_INFO_LEN, "%s:event_id=3D%d, info=3D%s", + dev->kobj.name, id, info); + ext[0] =3D buf; + kobject_uevent_env(&dev->kobj, KOBJ_CHANGE, ext); +} +#endif /* __MTK_UTILITY_H__ */ diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma.c b/drivers/net/wwan/t9xx= /pcie/mtk_cldma.c index 0f281bfb38cc..080ae29d3a88 100644 --- a/drivers/net/wwan/t9xx/pcie/mtk_cldma.c +++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma.c @@ -32,11 +32,184 @@ #define WAIT_HWO_TIME (5) #define NO_BUDGET (0) =20 +static void mtk_cldma_err_work(struct work_struct *work); + +static int mtk_cldma_isr(int irq_id, void *param) +{ + struct cldma_drv_info *drv_info =3D param; + u32 tx_err, rx_err; + struct mtk_md_dev *mdev; + u32 tx_done, rx_done; + u32 tx_sta, rx_sta; + struct txq *txq; + struct rxq *rxq; + int i; + + mdev =3D drv_info->mdev; + mtk_cldma_get_intr_status(drv_info, &tx_sta, &rx_sta); + tx_done =3D (tx_sta >> QUEUE_XFER_DONE) & 0xFF; + rx_done =3D (rx_sta >> QUEUE_XFER_DONE) & 0xFF; + tx_err =3D (tx_sta >> QUEUE_ERROR) & 0xFF; + rx_err =3D (rx_sta >> QUEUE_ERROR) & 0xFF; + + if (tx_err || rx_err) { + dev_err_ratelimited(mdev->dev, "CLDMA%d queue error: TX 0x%x RX 0x%x\n", + drv_info->hif_id, tx_err, rx_err); + for (i =3D 0; i < HW_QUEUE_NUM; i++) { + if (tx_err & BIT(i)) { + mtk_cldma_clr_intr_status(drv_info, DIR_TX, + i, QUEUE_ERROR); + mtk_cldma_unmask_intr(drv_info, DIR_TX, + i, QUEUE_ERROR); + } + if (rx_err & BIT(i)) { + mtk_cldma_clr_intr_status(drv_info, DIR_RX, + i, QUEUE_ERROR); + mtk_cldma_unmask_intr(drv_info, DIR_RX, + i, QUEUE_ERROR); + } + } + atomic_or(tx_err, &drv_info->tx_err_qs); + atomic_or(rx_err, &drv_info->rx_err_qs); + queue_work(drv_info->wq, &drv_info->err_work); + } + + if (tx_done) { + for (i =3D 0; i < HW_QUEUE_NUM; i++) { + /* pairs with smp_store_release() in txq_alloc */ + txq =3D smp_load_acquire(&drv_info->txq[i]); + if (!(tx_done & BIT(i)) || !txq) + continue; + queue_work(drv_info->wq, &txq->tx_done_work); + } + } + if (rx_done) { + for (i =3D 0; i < HW_QUEUE_NUM; i++) { + /* pairs with smp_store_release() in rxq_alloc */ + rxq =3D smp_load_acquire(&drv_info->rxq[i]); + if (!(rx_done & BIT(i)) || !rxq) + continue; + queue_work(drv_info->wq, &rxq->rx_done_work); + } + } + + mtk_pci_clear_irq(mdev, drv_info->pci_ext_irq_id); + mtk_pci_unmask_irq(mdev, drv_info->pci_ext_irq_id); + + return IRQ_HANDLED; +} + static const int mtk_cldma_hw_id_tbl[NR_CLDMA] =3D { [CLDMA0] =3D CLDMA0_HW_ID, [CLDMA1] =3D CLDMA1_HW_ID, }; =20 +static int mtk_cldma_dev_init(struct cldma_dev *cd, int hif_id) +{ + char gpd_pool_name[DMA_POOL_NAME_LEN]; + char bd_pool_name[DMA_POOL_NAME_LEN]; + struct cldma_drv_info *drv_info; + struct cldma_hw_regs *hw_regs; + struct mtk_md_dev *mdev; + unsigned int flag; + int hw_id, ret; + + if (!cd || hif_id >=3D NR_CLDMA) + return -EINVAL; + + if (cd->cldma_drv_info[hif_id]) + return 0; + + hw_id =3D mtk_cldma_hw_id_tbl[hif_id]; + mdev =3D cd->trans->mdev; + drv_info =3D kzalloc_obj(*drv_info); + if (!drv_info) + return -ENOMEM; + + drv_info->cd =3D cd; + drv_info->mdev =3D mdev; + drv_info->hif_id =3D hif_id; + drv_info->hw_id =3D hw_id; + drv_info->hw_regs =3D &mtk_cldma_regs_m9xx; + + hw_regs =3D drv_info->hw_regs; + snprintf(gpd_pool_name, DMA_POOL_NAME_LEN, "cldma%d_gpd_pool_%s", + hw_id, mdev->dev_str); + snprintf(bd_pool_name, DMA_POOL_NAME_LEN, "cldma%d_bd_pool_%s", + hw_id, mdev->dev_str); + drv_info->gpd_dma_pool =3D dma_pool_create(gpd_pool_name, mdev->dev, + sizeof(union gpd), 16, 0); + if (!drv_info->gpd_dma_pool) { + dev_err(mdev->dev, "Failed to alloc gpd dma pool for cldma%d\n", hw_id); + ret =3D -ENOMEM; + goto err_free_drv_info; + } + drv_info->bd_dma_pool =3D dma_pool_create(bd_pool_name, mdev->dev, + sizeof(union bd), 16, 0); + if (!drv_info->bd_dma_pool) { + dev_err(mdev->dev, "Failed to alloc bd dma pool for cldma%d\n", hw_id); + ret =3D -ENOMEM; + goto err_destroy_gpd_pool; + } + + switch (hif_id) { + case CLDMA0: + drv_info->pci_ext_irq_id =3D mtk_pci_get_irq_id(mdev, MTK_IRQ_SRC_CLDMA0= ); + drv_info->base_addr =3D hw_regs->cldma0_base_addr; + break; + case CLDMA1: + drv_info->pci_ext_irq_id =3D mtk_pci_get_irq_id(mdev, MTK_IRQ_SRC_CLDMA1= ); + drv_info->base_addr =3D hw_regs->cldma1_base_addr; + break; + default: + ret =3D -EINVAL; + goto err_destroy_dma_pool; + } + + flag =3D WQ_UNBOUND | WQ_MEM_RECLAIM | WQ_HIGHPRI; + drv_info->wq =3D alloc_workqueue("cldma%d_workq_%s", flag, 0, hw_id, mdev= ->dev_str); + if (!drv_info->wq) { + dev_err(mdev->dev, "Failed to alloc work queue for cldma%d\n", hw_id); + ret =3D -ENOMEM; + goto err_destroy_dma_pool; + } + + INIT_WORK(&drv_info->err_work, mtk_cldma_err_work); + atomic_set(&drv_info->tx_err_qs, 0); + atomic_set(&drv_info->rx_err_qs, 0); + + mtk_cldma_drv_init(drv_info); + + /* mask/clear PCI CLDMA L1 interrupt */ + mtk_pci_mask_irq(mdev, drv_info->pci_ext_irq_id); + mtk_pci_clear_irq(mdev, drv_info->pci_ext_irq_id); + + /* register CLDMA interrupt handler */ + ret =3D mtk_pci_register_irq(mdev, drv_info->pci_ext_irq_id, mtk_cldma_is= r, drv_info); + if (ret) + goto err_destroy_wq; + + /* unmask PCI CLDMA L1 interrupt */ + mtk_pci_unmask_irq(mdev, drv_info->pci_ext_irq_id); + + /* Publish only once drv_info is fully built; pairs with the acquire + * loads in the transfer paths below. + */ + smp_store_release(&cd->cldma_drv_info[hif_id], drv_info); + return 0; + +err_destroy_wq: + destroy_workqueue(drv_info->wq); +err_destroy_dma_pool: + dma_pool_destroy(drv_info->bd_dma_pool); +err_destroy_gpd_pool: + dma_pool_destroy(drv_info->gpd_dma_pool); +err_free_drv_info: + kfree(drv_info); + + return ret; +} + static void mtk_cldma_clr_bd_dsc(struct cldma_drv_info *drv_info, struct bd_dsc *bd_dsc_pool, int nr_bds) { @@ -529,6 +702,7 @@ static struct txq *mtk_cldma_txq_alloc(struct cldma_drv= _info *drv_info, struct q txq->nr_gpds =3D txq->que->tx_nr_gpds; atomic_set(&txq->req_budget, txq->que->tx_nr_gpds); spin_lock_init(&txq->ring_lock); + txq->is_stopping =3D false; tx_frag_size =3D txq->que->tx_frag_size; if (txq->que->tx_mtu > tx_frag_size && tx_frag_size) txq->nr_bds =3D (txq->que->tx_mtu + tx_frag_size - 1) / tx_frag_size; @@ -679,8 +853,11 @@ static void mtk_cldma_txq_free(struct cldma_drv_info *= drv_info, u32 txqno) =20 irq_id =3D mtk_pci_get_virq_id(mdev, drv_info->pci_ext_irq_id); synchronize_irq(irq_id); - /* flush on-going work */ + /* flush on-going work; the error worker may have loaded this txq + * before it was unpublished above, so it has to be retired too + */ flush_work(&txq->tx_done_work); + flush_work(&drv_info->err_work); mtk_cldma_mask_intr(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE); mtk_cldma_mask_intr(drv_info, DIR_TX, txqno, QUEUE_ERROR); =20 @@ -690,6 +867,7 @@ static void mtk_cldma_txq_free(struct cldma_drv_info *d= rv_info, u32 txqno) */ dev_err(mdev->dev, "TX queue %d cannot be stopped, leaking its ring\n", txqno); + drv_info->ring_leaked =3D true; return; } =20 @@ -987,8 +1165,11 @@ static void mtk_cldma_rxq_free(struct cldma_drv_info = *drv_info, u32 rxqno) =20 irq_id =3D mtk_pci_get_virq_id(mdev, drv_info->pci_ext_irq_id); synchronize_irq(irq_id); - /* flush on-going work */ + /* flush on-going work; the error worker may still be about to stop + * this queue number, which a later allocation could already reuse + */ flush_work(&rxq->rx_done_work); + flush_work(&drv_info->err_work); /* mask L2 RX interrupt again to avoid race condition causing use-after-f= ree issue */ mtk_cldma_mask_intr(drv_info, DIR_RX, rxqno, QUEUE_XFER_DONE); mtk_cldma_mask_intr(drv_info, DIR_RX, rxqno, QUEUE_ERROR); @@ -999,6 +1180,7 @@ static void mtk_cldma_rxq_free(struct cldma_drv_info *= drv_info, u32 rxqno) */ dev_err(mdev->dev, "RX queue %d cannot be stopped, leaking its ring\n", rxqno); + drv_info->ring_leaked =3D true; return; } =20 @@ -1072,6 +1254,67 @@ static int mtk_cldma_hw_recovery(struct cldma_drv_in= fo *drv_info, u32 qno) return 0; } =20 +static int mtk_cldma_dev_exit(struct cldma_dev *cd, int hif_id) +{ + struct cldma_drv_info *drv_info; + struct mtk_md_dev *mdev; + int virq_id; + int i; + + if (!cd || hif_id >=3D NR_CLDMA) + return -EINVAL; + + if (!cd->cldma_drv_info[hif_id]) + return 0; + + /* free cldma descriptor */ + drv_info =3D cd->cldma_drv_info[hif_id]; + mdev =3D cd->trans->mdev; + virq_id =3D mtk_pci_get_virq_id(mdev, drv_info->pci_ext_irq_id); + /* mask first so no new interrupt can fire, then wait out any + * in-flight handler before tearing the registration down + */ + mtk_pci_mask_irq(mdev, drv_info->pci_ext_irq_id); + synchronize_irq(virq_id); + mtk_pci_unregister_irq(mdev, drv_info->pci_ext_irq_id); + for (i =3D 0; i < HW_QUEUE_NUM; i++) { + if (drv_info->txq[i]) + mtk_cldma_txq_free(drv_info, drv_info->txq[i]->txqno); + if (drv_info->rxq[i]) + mtk_cldma_rxq_free(drv_info, drv_info->rxq[i]->rxqno); + } + + flush_workqueue(drv_info->wq); + destroy_workqueue(drv_info->wq); + + /* quiesce the IP before releasing descriptor memory: disable its + * interrupt output and reset it, so it cannot touch the rings again + */ + mtk_pci_write32(mdev, drv_info->base_addr + drv_info->hw_regs->reg_cldma_= int_mask, + LINK_ERROR_VAL); + mtk_cldma_drv_reset(drv_info); + + if (drv_info->ring_leaked) { + /* A ring is leaked and its descriptors live in these pools: + * the device may still master DMA into them, so handing them + * back to the allocator would open a use-after-free window. + */ + dev_err(mdev->dev, "CLDMA%d rings leaked, leaking DMA pools too\n", + drv_info->hw_id); + } else { + dma_pool_destroy(drv_info->bd_dma_pool); + dma_pool_destroy(drv_info->gpd_dma_pool); + } + + /* Unpublish before teardown; pairs with the acquire loads of this + * slot. No release is needed, there is no prior store to expose. + */ + WRITE_ONCE(cd->cldma_drv_info[hif_id], NULL); + kfree(drv_info); + + return 0; +} + static int mtk_cldma_start_xfer(struct cldma_drv_info *drv_info, u32 qno) { struct txq *txq; @@ -1095,6 +1338,11 @@ static int mtk_cldma_start_xfer(struct cldma_drv_inf= o *drv_info, u32 qno) * names; tx_done_work advances free_idx under the same lock. */ spin_lock(&txq->ring_lock); + if (unlikely(txq->is_stopping)) { + spin_unlock(&txq->ring_lock); + return -EPIPE; + } + if (unlikely(!READ_ONCE(txq->tx_started))) { mtk_cldma_setup_start_addr(drv_info, DIR_TX, qno, txq->req_pool[txq->free_idx].gpd_dma_addr); @@ -1124,11 +1372,22 @@ int mtk_cldma_init(struct mtk_ctrl_trans *trans) =20 void mtk_cldma_exit(struct mtk_ctrl_trans *trans) { - if (!trans->dev) + struct cldma_dev *cd =3D trans->dev; + int i; + + if (!cd) return; =20 - kfree(trans->dev); + /* Latch-and-clear up front: the caller holds trans->submit_lock, so + * publishing the NULL here makes any later submit path bail out in + * mtk_cldma_get_tx_budget() instead of walking freed queues. + */ trans->dev =3D NULL; + + for (i =3D 0; i < NR_CLDMA; i++) + mtk_cldma_dev_exit(cd, i); + + kfree(cd); } =20 static int mtk_cldma_open(struct cldma_dev *cd, struct sk_buff *skb) @@ -1147,7 +1406,8 @@ static int mtk_cldma_open(struct cldma_dev *cd, struc= t sk_buff *skb) trb->trb_complete(skb); return -EINVAL; } - drv_info =3D cd->cldma_drv_info[que->hif_id]; + /* pairs with smp_store_release() in mtk_cldma_dev_init */ + drv_info =3D smp_load_acquire(&cd->cldma_drv_info[que->hif_id]); if (!drv_info) { ret =3D -EIO; goto out; @@ -1250,6 +1510,82 @@ static void mtk_cldma_txq_flush(struct cldma_drv_inf= o *drv_info, } } =20 +/* Handle QUEUE_ERROR interrupts out of atomic context: stop the errored + * queues so the device stops walking their rings, and complete pending TX + * requests with an error so the failure is reported upward instead of + * being silently logged. + */ +static void mtk_cldma_err_work(struct work_struct *work) +{ + struct cldma_drv_info *drv_info =3D container_of(work, struct cldma_drv_i= nfo, err_work); + u32 tx_err, rx_err; + struct txq *txq; + struct rxq *rxq; + int i, ret; + + tx_err =3D atomic_xchg(&drv_info->tx_err_qs, 0); + rx_err =3D atomic_xchg(&drv_info->rx_err_qs, 0); + + for (i =3D 0; i < HW_QUEUE_NUM; i++) { + if (tx_err & BIT(i)) { + /* pairs with smp_store_release() in txq_alloc */ + txq =3D smp_load_acquire(&drv_info->txq[i]); + if (!txq) + continue; + /* Keep the producer out for as long as this queue is + * being stopped, so start_xfer cannot ring the + * doorbell between the stop and the flush. + */ + spin_lock(&txq->ring_lock); + txq->is_stopping =3D true; + spin_unlock(&txq->ring_lock); + + ret =3D mtk_cldma_stop_queue(drv_info, DIR_TX, i); + if (ret) { + /* the device may still be walking the ring: + * unmapping its buffers here would leave it + * writing into unmapped memory. is_stopping is + * left set so nothing submits to it again. + */ + dev_err(drv_info->mdev->dev, + "TX queue %d stop failed (%d), keeping its requests\n", + i, ret); + continue; + } + + spin_lock(&txq->ring_lock); + WRITE_ONCE(txq->tx_started, false); + spin_unlock(&txq->ring_lock); + mtk_cldma_txq_flush(drv_info, txq, -EPIPE); + + spin_lock(&txq->ring_lock); + txq->is_stopping =3D false; + spin_unlock(&txq->ring_lock); + } + if (rx_err & BIT(i)) { + /* pairs with smp_store_release() in rxq_alloc */ + rxq =3D smp_load_acquire(&drv_info->rxq[i]); + if (!rxq) + continue; + ret =3D mtk_cldma_stop_queue(drv_info, DIR_RX, i); + if (ret) { + dev_err(drv_info->mdev->dev, + "RX queue %d stop failed (%d), leaving it stopped\n", + i, ret); + continue; + } + + /* Hand the queue back to the worker that owns free_idx + * instead of programming it here: a stopped queue + * raises no further interrupt, so nothing else would + * ever re-arm it. + */ + atomic_set(&rxq->need_restart, 1); + queue_work(drv_info->wq, &rxq->rx_done_work); + } + } +} + static int mtk_cldma_tx(struct cldma_dev *cd, struct sk_buff *skb) { struct trb *trb =3D (struct trb *)skb->cb; @@ -1262,7 +1598,8 @@ static int mtk_cldma_tx(struct cldma_dev *cd, struct = sk_buff *skb) que =3D radix_tree_lookup(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF= ); if (unlikely(!que)) return -EPIPE; - drv_info =3D cd->cldma_drv_info[que->hif_id]; + /* pairs with smp_store_release() in mtk_cldma_dev_init */ + drv_info =3D smp_load_acquire(&cd->cldma_drv_info[que->hif_id]); if (unlikely(!drv_info)) return -EPIPE; txq =3D drv_info->txq[que->txqno]; @@ -1292,7 +1629,8 @@ static int mtk_cldma_close(struct cldma_dev *cd, stru= ct sk_buff *skb) trb->trb_complete(skb); return -EPIPE; } - drv_info =3D cd->cldma_drv_info[que->hif_id]; + /* pairs with smp_store_release() in mtk_cldma_dev_init */ + drv_info =3D smp_load_acquire(&cd->cldma_drv_info[que->hif_id]); if (unlikely(!drv_info)) { trb->status =3D -EPIPE; trb->trb_complete(skb); @@ -1402,7 +1740,8 @@ int mtk_cldma_submit_tx(void *dev, struct sk_buff *sk= b) que =3D radix_tree_lookup(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF= ); if (unlikely(!que)) return -EINVAL; - drv_info =3D cd->cldma_drv_info[que->hif_id]; + /* pairs with smp_store_release() in mtk_cldma_dev_init */ + drv_info =3D smp_load_acquire(&cd->cldma_drv_info[que->hif_id]); if (unlikely(!drv_info)) return -EINVAL; =20 @@ -1451,7 +1790,8 @@ int mtk_cldma_get_tx_budget(void *dev, enum mtk_hif_i= d hif_id, u32 qno) if (unlikely(hif_id >=3D NR_CLDMA || qno >=3D HW_QUE_NUM || !cd)) return -EINVAL; =20 - drv_info =3D cd->cldma_drv_info[hif_id]; + /* pairs with smp_store_release() in mtk_cldma_dev_init */ + drv_info =3D smp_load_acquire(&cd->cldma_drv_info[hif_id]); if (!drv_info) return -EINVAL; txq =3D drv_info->txq[qno]; @@ -1483,6 +1823,38 @@ int mtk_cldma_trb_process(void *dev, struct sk_buff = *skb) return trb_act_tbl[trb->cmd](cd, skb); } =20 +void mtk_cldma_fsm_state_listener(struct mtk_fsm_param *param, struct mtk_= ctrl_trans *trans) +{ + struct cldma_dev *cd =3D trans->dev; + enum mtk_fsm_hif_user user; + enum mtk_hif_id hif_id; + int ret; + + switch (param->to) { + case FSM_STATE_BOOTUP: + if (param->fsm_flag & FSM_F_SAP_HS_START) { + hif_id =3D CLDMA0; + user =3D MTK_FSM_HIF_USER_CLDMA0; + } else if (param->fsm_flag & FSM_F_MD_HS_START) { + hif_id =3D CLDMA1; + user =3D MTK_FSM_HIF_USER_CLDMA1; + } else { + break; + } + + ret =3D mtk_cldma_dev_init(cd, hif_id); + if (ret) + dev_err(trans->mdev->dev, "Failed to init CLDMA%d: %d\n", hif_id, ret); + /* Record the success too: HS1 is retried, so a later good init + * has to clear what the failed attempt left behind. + */ + mtk_fsm_hif_err_record(trans->mdev, user, ret); + break; + default: + break; + } +} + int mtk_cldma_check_ch_cfg(void *dev, struct queue_info *que) { struct cldma_drv_info *drv_info; @@ -1492,7 +1864,8 @@ int mtk_cldma_check_ch_cfg(void *dev, struct queue_in= fo *que) struct rxq *rxq; =20 mdev =3D cd->trans->mdev; - drv_info =3D cd->cldma_drv_info[que->hif_id]; + /* pairs with smp_store_release() in mtk_cldma_dev_init */ + drv_info =3D smp_load_acquire(&cd->cldma_drv_info[que->hif_id]); =20 if (!drv_info) { dev_err(mdev->dev, "CLDMA%d has not been initialized\n", diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma.h b/drivers/net/wwan/t9xx= /pcie/mtk_cldma.h index fa6d79f8b5df..bc898f2f52ce 100644 --- a/drivers/net/wwan/t9xx/pcie/mtk_cldma.h +++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma.h @@ -146,6 +146,10 @@ struct txq { u32 wr_idx; u32 free_idx; bool tx_started; + /* Set by err_work while it stops and flushes this queue; read under + * ring_lock by the producer so no request is submitted in between. + */ + bool is_stopping; u32 nr_bds; }; =20 diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h b/drivers/net/wwan/= t9xx/pcie/mtk_cldma_drv.h index d7abf4b48895..1917bd70b51d 100644 --- a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h +++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h @@ -108,6 +108,13 @@ struct cldma_drv_info { struct dma_pool *bd_dma_pool; struct workqueue_struct *wq; struct cldma_hw_regs *hw_regs; + struct work_struct err_work; + atomic_t tx_err_qs; + atomic_t rx_err_qs; + /* a ring could not be quiesced and was leaked on free; skip + * destroying the DMA pools its descriptors still live in + */ + bool ring_leaked; }; =20 extern struct cldma_hw_regs mtk_cldma_regs_m9xx; diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.c b/drivers/net/wwan/t9xx/p= cie/mtk_pci.c index 84b7fb3c82fd..000a481398a4 100644 --- a/drivers/net/wwan/t9xx/pcie/mtk_pci.c +++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.c @@ -878,23 +878,47 @@ static int mtk_pci_dev_init(struct mtk_md_dev *mdev) { int ret; =20 - ret =3D mtk_trans_ctrl_init(mdev); + ret =3D mtk_fsm_init(mdev); if (ret) { - dev_err(mdev->dev, "Failed to initialize control plane: %d\n", ret); + dev_err(mdev->dev, "Failed to initialize FSM: %d\n", ret); return ret; } =20 + ret =3D mtk_trans_ctrl_init(mdev); + if (ret) + goto free_fsm; + return 0; +free_fsm: + mtk_fsm_exit(mdev); + return ret; } =20 static void mtk_pci_dev_exit(struct mtk_md_dev *mdev) { + int ret; + + ret =3D mtk_fsm_evt_submit(mdev, FSM_EVT_DEV_RM, 0, NULL, 0, + EVT_MODE_BLOCKING | EVT_MODE_TOHEAD); + if (ret < 0 || ret =3D=3D FSM_EVT_RET_FAIL) + dev_err(mdev->dev, "FSM DEV_RM failed: %d, forcing cleanup\n", ret); + /* Close the event gate and park the FSM thread before the transport + * plane it drives is torn down. + */ + mtk_fsm_stop(mdev); mtk_trans_ctrl_exit(mdev); + mtk_fsm_exit(mdev); } =20 static int mtk_pci_dev_start(struct mtk_md_dev *mdev) { - return 0; + int ret; + + ret =3D mtk_fsm_evt_submit(mdev, FSM_EVT_DEV_ADD, 0, NULL, 0, 0); + if (ret =3D=3D FSM_EVT_RET_FAIL) + return -ENOMEM; + + return mtk_fsm_start(mdev); } =20 static int mtk_pci_probe(struct pci_dev *pdev, const struct pci_device_id = *id) diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c b/drivers/net/wwan= /t9xx/pcie/mtk_trans_ctrl.c index d485bd7e3bf9..658189dace01 100644 --- a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c +++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c @@ -473,19 +473,23 @@ int mtk_pcie_hif_exit(struct mtk_md_dev *mdev) =20 trans =3D ctrl_blk->ctrl_hw_priv; =20 + /* Hold submit_lock across the whole teardown: late submitters either + * see available and finish before the teardown starts, or block here + * and then bail out on !available. Also makes this exit idempotent, + * so both the FSM listener and device removal may call it. + */ mutex_lock(&trans->submit_lock); + if (!atomic_read(&trans->available)) { + mutex_unlock(&trans->submit_lock); + return 0; + } atomic_set(&trans->available, 0); - mutex_unlock(&trans->submit_lock); - /* Join the service threads before freeing what they dereference: they * read trans->dev without a lock, so clearing it cannot stop a * consumer that has already loaded the pointer. */ mtk_ctrl_trb_srv_exit(trans); mtk_cldma_exit(trans); - - /* Late submitters may still hold the lock and walk the tree. */ - mutex_lock(&trans->submit_lock); mtk_ctrl_remove_radix_tree(trans); mutex_unlock(&trans->submit_lock); =20 @@ -565,6 +569,15 @@ int mtk_pcie_hif_submit_skb(struct mtk_md_dev *mdev, s= truct sk_buff *skb, bool f return ret; } =20 +void mtk_pcie_hif_fsm_indication(struct mtk_md_dev *mdev, struct mtk_fsm_p= aram *param) +{ + struct mtk_ctrl_blk *ctrl_blk =3D mdev->ctrl_blk; + struct mtk_ctrl_trans *trans; + + trans =3D ctrl_blk->ctrl_hw_priv; + mtk_cldma_fsm_state_listener(param, trans); +} + int mtk_pcie_hif_cmd_func(struct mtk_md_dev *mdev, int cmd, void *data) { struct mtk_ctrl_blk *ctrl_blk =3D mdev->ctrl_blk; @@ -627,6 +640,13 @@ int mtk_trans_ctrl_init(struct mtk_md_dev *mdev) =20 int mtk_trans_ctrl_exit(struct mtk_md_dev *mdev) { + /* FSM_STATE_OFF normally tears the HIF down. If that never ran, the + * trb kthreads and the CLDMA irq callback would outlive the devm + * allocations they point at, so do it here. mtk_pcie_hif_exit() is + * idempotent under trans->submit_lock, so calling it again is safe. + */ + mtk_pcie_hif_exit(mdev); + mtk_ctrl_exit(mdev); =20 return 0; diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h b/drivers/net/wwan= /t9xx/pcie/mtk_trans_ctrl.h index 1228f40f2d39..b281f49ea4f0 100644 --- a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h +++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h @@ -80,9 +80,12 @@ struct trb_srv { struct task_struct *trb_thread; }; =20 +struct mtk_fsm_param; + int mtk_pcie_hif_init(struct mtk_md_dev *mdev); int mtk_pcie_hif_exit(struct mtk_md_dev *mdev); int mtk_pcie_hif_submit_skb(struct mtk_md_dev *mdev, struct sk_buff *skb, = bool force_send); +void mtk_pcie_hif_fsm_indication(struct mtk_md_dev *mdev, struct mtk_fsm_p= aram *param); int mtk_pcie_hif_cmd_func(struct mtk_md_dev *mdev, int cmd, void *data); int mtk_trans_ctrl_init(struct mtk_md_dev *mdev); int mtk_trans_ctrl_exit(struct mtk_md_dev *mdev); --=20 2.34.1 From nobody Wed Sep 30 11:25:46 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 429A44477FF; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790754373; cv=none; b=pK7Q/skc26vxu259C6O39K3hMvGCL440Sj8VLEUHwRHVdt18tX4/79QKV+/zof4AFjzj26dP4l7Djiz2pC8SuwGuIJiihicpGbV8THdpgw0yJ3PTlkG7BjOWHRGI0vGcsBKhLEb9gkdLjpuUU+Uf9iI6d24xVQTNNvapeN4cl94= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790754373; c=relaxed/simple; bh=rxboJwXnw64S2MkT8qOujdEdVmE8rzy7uXbYsd1of2c=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Y/PrFmYTRznbZiaPSGpeyqMRFp5xJ4xK7sxIV6tXtMUYSPVBXxIwUKlzMjwz1NIkBeG9yl/3nhJh672ZBHbQq54qmzLYo8ZxLmDa8pfSnA57ZBqRmfIssqz3m/v+oKu/1qDemm5rRZbVJOCOPBp4tfF6uRl/6mTZ9OOOCejiKro= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IMNv+oXc; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IMNv+oXc" Received: by smtp.kernel.org (Postfix) with ESMTPS id BED6DC2BCF4; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1790754372; bh=rxboJwXnw64S2MkT8qOujdEdVmE8rzy7uXbYsd1of2c=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=IMNv+oXcI/hLg2F1Y9KNfhKcILWSJLdb9C3DV11tpS9SXPMM0IBDiHx2PCvskDXcm MEYgr6J2QCtZud0taxwtv4GCREno9TDQgWm4Ype/KgmnwjfpDRMhDqUhDlKh4REsQp ap7axpw6iA3lsBy1SX+dieARN3HP04Aqh8GsnpZWsrxkcrN85ildNLICltSftHuStj xp2Y5YyiIEqghWRTdz6twRhjU9fS/ug1O4AB4ce6D1Walfwmp2bx4EZLqkOSm2h2Pk fHTn0eKXcCfG5ZnyGXO4vwbld6qGnKqECNN7a8LdhhE4K34ArLBWENY+5PZELJpsYe W+5VEvH7a3SmA== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC914CA5FB1; Wed, 30 Sep 2026 07:46:12 +0000 (UTC) From: Jack Wu via B4 Relay Date: Wed, 30 Sep 2026 15:46:13 +0800 Subject: [PATCH v9 6/6] net: wwan: t9xx: Add AT & MBIM WWAN ports Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260930-t9xx_driver_v1-v9-6-3248b9256956@compal.com> References: <20260930-t9xx_driver_v1-v9-0-3248b9256956@compal.com> In-Reply-To: <20260930-t9xx_driver_v1-v9-0-3248b9256956@compal.com> To: Loic Poulain , Sergey Ryazanov , Johannes Berg , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jack Wu , Wen-Zhi Huang , Shi-Wei Yeh , Minano Tseng , Matthias Brugger , AngeloGioacchino Del Regno , Simon Horman , Jonathan Corbet , Shuah Khan , Robert Yu , Jeff Chang Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, linux-doc@vger.kernel.org X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1790754370; l=18607; i=jackbb_wu@compal.com; s=20260526; h=from:subject:message-id; bh=Wsm5XjIyTY0HSBgc2z4hZCJCP2htulSVWx0Yv5u1uBQ=; b=iHEbg1mllAfoxm1nWHtF2hU2NRdxcl+tl7R2/YR1Je8oY4Jh0jFTnCJFOSnaORaBb8b2eLyTB a9P+4lR97cMDRKN0VzT7U79x3CECkrgTCe48PmVVLOmRU42dLwhHuym X-Developer-Key: i=jackbb_wu@compal.com; a=ed25519; pk=VH1prTWixNl8OEUPPSfII3p46MzJpQN8J3+ecE1tZXg= X-Endpoint-Received: by B4 Relay for jackbb_wu@compal.com/20260526 with auth_id=793 X-Original-From: Jack Wu Reply-To: jackbb_wu@compal.com From: Jack Wu Add AT & MBIM ports to the port infrastructure. mtk_port_wwan_init() only prepares the port object; wwan_create_port() is called from mtk_port_wwan_enable(). enable() runs on FSM_STATE_READY through the new mtk_port_enable_by_type(PORT_TBL_MD) hook, which enables every MD-table port the device marked enabled, internal ports included. CCCI_UART2 and CCCI_MBIM get CLDMA1 TXQ(5)/RXQ(5) and TXQ(2)/RXQ(2) in mtk_queue_info[]. The implemented WWAN port operations are start, stop, tx and tx_blocking. The tx path copies with skb_copy_bits(), handling the non-linear skb the WWAN core supplies for writes larger than one fragment. TX back-pressure is reported through wwan_port_txon()/txoff() rather than a .tx_poll operation: a queue-full submit failure pauses the port and a per-port tx_complete hook resumes it from TRB completion, so poll() sleeps on the WWAN core's own waitqueue - whose lifetime is pinned by the open file - instead of a driver waitqueue that wwan_remove_port() can outlive. Signed-off-by: Jack Wu --- drivers/net/wwan/t9xx/mtk_ctrl_plane.h | 4 + drivers/net/wwan/t9xx/mtk_port.c | 34 +++ drivers/net/wwan/t9xx/mtk_port.h | 9 + drivers/net/wwan/t9xx/mtk_port_io.c | 355 ++++++++++++++++++++++++= ++++ drivers/net/wwan/t9xx/mtk_port_io.h | 1 + drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c | 8 + 6 files changed, 411 insertions(+) diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h b/drivers/net/wwan/t9xx= /mtk_ctrl_plane.h index 1f13e25502ad..2b031160eaa9 100644 --- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h +++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h @@ -22,6 +22,10 @@ enum mtk_ccci_ch { /* to MD */ CCCI_CONTROL_RX =3D 0x2000, CCCI_CONTROL_TX =3D 0x2001, + CCCI_UART2_RX =3D 0x200A, + CCCI_UART2_TX =3D 0x200C, + CCCI_MBIM_RX =3D 0x20D0, + CCCI_MBIM_TX =3D 0x20D1, }; =20 enum mtk_trb_cmd_type { diff --git a/drivers/net/wwan/t9xx/mtk_port.c b/drivers/net/wwan/t9xx/mtk_p= ort.c index d7c50519e1fd..90ac551887a8 100644 --- a/drivers/net/wwan/t9xx/mtk_port.c +++ b/drivers/net/wwan/t9xx/mtk_port.c @@ -285,6 +285,10 @@ static int mtk_port_tx_complete(struct sk_buff *skb) "Failed to send data: status:%d, port:%s\n", trb->status, port->info.name); =20 + /* Runs in the trb_srv kthread, so the hook may sleep on a mutex. */ + if (ports_ops[port->info.type]->tx_complete) + ports_ops[port->info.type]->tx_complete(port); + wake_up_all(&port->trb_wq); kref_put(&trb->kref, mtk_port_trb_free); =20 @@ -656,6 +660,29 @@ int mtk_port_ch_disable(struct mtk_port *port) return ret; } =20 +static int mtk_port_enable_by_type(struct mtk_port_mngr *port_mngr, int tb= l_type) +{ + struct mtk_port **ports; + int ret, idx; + + if (tbl_type < 0 || tbl_type >=3D PORT_TBL_MAX) + return -EINVAL; + + ports =3D kcalloc(port_mngr->port_cnt, sizeof(struct mtk_port *), GFP_KER= NEL); + if (!ports) + return -ENOMEM; + + ret =3D radix_tree_gang_lookup(&port_mngr->port_tbl[tbl_type], + (void **)ports, 0, port_mngr->port_cnt); + for (idx =3D 0; idx < ret; idx++) { + if (ports[idx]->enable) + ports_ops[ports[idx]->info.type]->enable(ports[idx]); + } + + kfree(ports); + return 0; +} + static void mtk_port_disable(struct mtk_port_mngr *port_mngr) { struct radix_tree_iter iter; @@ -677,6 +704,7 @@ static void mtk_port_disable(struct mtk_port_mngr *port= _mngr) void mtk_port_mngr_fsm_state_handler(struct mtk_fsm_param *fsm_param, void= *arg) { struct mtk_port_mngr *port_mngr; + int ret; =20 if (!fsm_param || !arg) return; @@ -687,6 +715,12 @@ void mtk_port_mngr_fsm_state_handler(struct mtk_fsm_pa= ram *fsm_param, void *arg) case FSM_STATE_OFF: mtk_port_disable(port_mngr); break; + case FSM_STATE_READY: + ret =3D mtk_port_enable_by_type(port_mngr, PORT_TBL_MD); + if (ret) + dev_err(port_mngr->ctrl_blk->mdev->dev, + "Failed to enable MD ports: %d\n", ret); + break; default: break; } diff --git a/drivers/net/wwan/t9xx/mtk_port.h b/drivers/net/wwan/t9xx/mtk_p= ort.h index 2fc0e450ab6b..acfd52792dc7 100644 --- a/drivers/net/wwan/t9xx/mtk_port.h +++ b/drivers/net/wwan/t9xx/mtk_port.h @@ -55,6 +55,7 @@ enum mtk_port_tbl { =20 enum mtk_port_type { PORT_TYPE_INTERNAL, + PORT_TYPE_WWAN, PORT_TYPE_MAX }; =20 @@ -63,6 +64,13 @@ struct mtk_internal_port { int (*recv_cb)(void *arg, struct sk_buff *skb); }; =20 +struct mtk_wwan_port { + /* w_lock protects wwan_port when recv data and disable port at the same = time */ + struct mutex w_lock; + int w_type; + void *w_port; +}; + struct mtk_port_cfg { enum mtk_ccci_ch tx_ch; enum mtk_ccci_ch rx_ch; @@ -90,6 +98,7 @@ struct mtk_port { wait_queue_head_t rx_wq; struct mtk_port_mngr *port_mngr; struct mtk_internal_port i_priv; + struct mtk_wwan_port w_priv; }; =20 struct mtk_port_mngr { diff --git a/drivers/net/wwan/t9xx/mtk_port_io.c b/drivers/net/wwan/t9xx/mt= k_port_io.c index 6cff0704b7fc..0d8f63e2dbda 100644 --- a/drivers/net/wwan/t9xx/mtk_port_io.c +++ b/drivers/net/wwan/t9xx/mtk_port_io.c @@ -3,8 +3,13 @@ * Copyright (c) 2022, MediaTek Inc. */ #include +#include +#include +#include +#include =20 #include "mtk_port_io.h" +#include "mtk_trans_ctrl.h" =20 static int mtk_port_get_locked(struct mtk_port *port) { @@ -41,6 +46,73 @@ static void mtk_port_struct_init(struct mtk_port *port) init_waitqueue_head(&port->rx_wq); } =20 +/* Splits the source skb into CCCI packets and submits them. The source m= ay + * be non-linear: the WWAN core hands the tx ops a head skb whose linear a= rea + * is one fragment (caps.frag_len bytes) with the rest chained on frag_lis= t, + * so every read goes through skb_copy_bits(), which walks that chain and = is + * bounded by src->len by construction. + * + * Every packet is built before any is submitted, because the core reports= a + * partial write as a whole-write failure and userspace then re-sends the + * message, duplicating the prefix already on the wire. So no packet is + * submitted unless all of them were built, and only the first submit may + * fail with -EAGAIN; the rest carry force_send, leaving only failures that + * mean the channel itself is going away. -EINTR from a blocking wait + * arrives after the skb was submitted, so the packet counts as accepted. + */ +static int mtk_port_common_write(struct mtk_port *port, struct sk_buff *sr= c, bool blocking) +{ + u32 packet_size, left_cnt =3D src->len, cur_pos; + struct sk_buff_head list; + bool force_send =3D false; + struct sk_buff *skb; + int ret; + + ret =3D mtk_port_status_check(port); + if (ret) + return ret; + + __skb_queue_head_init(&list); + + while (left_cnt) { + skb =3D __dev_alloc_skb(port->tx_mtu, GFP_KERNEL); + if (!skb) { + ret =3D -ENOMEM; + goto err_purge; + } + + skb_reserve(skb, sizeof(struct mtk_ccci_header)); + + packet_size =3D min_t(u32, left_cnt, + port->tx_mtu - sizeof(struct mtk_ccci_header)); + cur_pos =3D src->len - left_cnt; + ret =3D skb_copy_bits(src, cur_pos, skb_put(skb, packet_size), packet_si= ze); + if (ret) { + dev_err(port->port_mngr->ctrl_blk->mdev->dev, + "Failed to copy data for port(%s)\n", port->info.name); + dev_kfree_skb_any(skb); + goto err_purge; + } + + __skb_queue_tail(&list, skb); + left_cnt -=3D packet_size; + } + + while ((skb =3D __skb_dequeue(&list))) { + ret =3D mtk_port_send_data(port, skb, blocking, force_send); + if (ret < 0 && ret !=3D -EINTR) + goto err_purge; + + force_send =3D true; + } + + return 0; + +err_purge: + __skb_queue_purge(&list); + return ret; +} + static int mtk_port_internal_init(struct mtk_port *port) { mtk_port_struct_init(port); @@ -253,6 +325,289 @@ static const struct port_ops port_internal_ops =3D { .recv =3D mtk_port_internal_recv, }; =20 +static int mtk_port_wwan_open(struct wwan_port *w_port) +{ + struct mtk_port *port; + int ret; + + port =3D wwan_port_get_drvdata(w_port); + ret =3D mtk_port_get_locked(port); + if (ret) + return ret; + + ret =3D mtk_port_common_open(port); + if (ret) { + mtk_port_put_locked(port); + return ret; + } + + /* WWAN_PORT_TX_OFF persists on the wwan_port across close/open + * cycles; start every session writable. + */ + wwan_port_txon(w_port); + + return 0; +} + +static void mtk_port_wwan_close(struct wwan_port *w_port) +{ + struct mtk_port *port =3D wwan_port_get_drvdata(w_port); + + /* Clear PORT_S_OPEN under the same lock mtk_port_wwan_recv() holds, so + * a receive that saw the port open has finished its wwan_port_rx() + * before stop() returns and the core purges its rx queue. + */ + mutex_lock(&port->w_priv.w_lock); + mtk_port_common_close(port); + mutex_unlock(&port->w_priv.w_lock); + + mtk_port_put_locked(port); +} + +/* Pause TX after a queue-full submit failure. The queue may have drained + * between that failure and the txoff below - the tx_complete that emptied + * it ran before this txoff and nothing later would ever resume TX - so + * recheck under the same lock and undo the txoff if the queue is no longer + * full. An error from the recheck also resumes TX: reporting it as "not + * writable" would leave the poller asleep with nothing left to wake it, + * while the next write returns the real errno. + */ +static void mtk_port_wwan_tx_pause(struct mtk_port *port) +{ + union ctrl_hif_cmd_data hif_cmd; + struct mtk_ctrl_blk *ctrl_blk; + int ret; + + ctrl_blk =3D port->port_mngr->ctrl_blk; + + mutex_lock(&port->w_priv.w_lock); + if (!port->w_priv.w_port) + goto unlock; + + wwan_port_txoff(port->w_priv.w_port); + + hif_cmd.rx_ch =3D port->info.rx_ch; + ret =3D mtk_pcie_hif_cmd_func(ctrl_blk->mdev, HIF_CTRL_CMD_CHECK_TX_FULL, + &hif_cmd); + if (ret <=3D 0) + wwan_port_txon(port->w_priv.w_port); +unlock: + mutex_unlock(&port->w_priv.w_lock); +} + +/* Called from the trb_srv kthread when a TX trb for this port completes. + * w_lock serializes it against mtk_port_wwan_tx_pause(), closing the + * txoff-after-drain window described there. + */ +static void mtk_port_wwan_tx_complete(struct mtk_port *port) +{ + mutex_lock(&port->w_priv.w_lock); + if (port->w_priv.w_port) + wwan_port_txon(port->w_priv.w_port); + mutex_unlock(&port->w_priv.w_lock); +} + +static int mtk_port_wwan_tx(struct wwan_port *w_port, struct sk_buff *skb,= bool blocking) +{ + struct mtk_port *port =3D wwan_port_get_drvdata(w_port); + int ret; + + if (unlikely(!skb->len)) { + consume_skb(skb); + return 0; + } + + ret =3D mtk_port_common_write(port, skb, blocking); + if (ret < 0) { + if (ret =3D=3D -EAGAIN) + mtk_port_wwan_tx_pause(port); + return ret; + } + + consume_skb(skb); + return 0; +} + +static int mtk_port_wwan_write(struct wwan_port *w_port, struct sk_buff *s= kb) +{ + return mtk_port_wwan_tx(w_port, skb, false); +} + +static int mtk_port_wwan_write_blocking(struct wwan_port *w_port, struct s= k_buff *skb) +{ + return mtk_port_wwan_tx(w_port, skb, true); +} + +/* No .tx_poll: it would poll_wait() on port->trb_wq, which lives in a + * mtk_port that wwan_remove_port() lets go of while the file stays open, + * and wake_up_pollfree() is not available to modules. TX back-pressure + * reaches poll() through wwan_port_txon()/txoff() on the core's own + * waitqueue instead, whose lifetime is pinned by the open file. + */ +static const struct wwan_port_ops wwan_ops =3D { + .start =3D mtk_port_wwan_open, + .stop =3D mtk_port_wwan_close, + .tx =3D mtk_port_wwan_write, + .tx_blocking =3D mtk_port_wwan_write_blocking, +}; + +static int mtk_port_wwan_init(struct mtk_port *port) +{ + mtk_port_struct_init(port); + port->enable =3D false; + + mutex_init(&port->w_priv.w_lock); + + switch (port->info.rx_ch) { + case CCCI_MBIM_RX: + port->w_priv.w_type =3D WWAN_PORT_MBIM; + break; + case CCCI_UART2_RX: + port->w_priv.w_type =3D WWAN_PORT_AT; + break; + default: + port->w_priv.w_type =3D WWAN_PORT_UNKNOWN; + break; + } + + return 0; +} + +static void mtk_port_wwan_exit(struct mtk_port *port) +{ + if (test_bit(PORT_S_ENABLE, &port->status)) + ports_ops[port->info.type]->disable(port); +} + +static void mtk_port_wwan_enable(struct mtk_port *port) +{ + struct mtk_port_mngr *port_mngr; + struct wwan_port_caps caps; + struct wwan_port *wp; + int ret; + + port_mngr =3D port->port_mngr; + + if (test_bit(PORT_S_ENABLE, &port->status)) + return; + + ret =3D mtk_port_ch_enable(port); + if (ret && ret !=3D -EBUSY) { + /* On -ETIMEDOUT the enable's outcome is not yet known: the + * ENABLE trb may still be queued. The DISABLE is queued + * behind it, so the channel cannot stay armed unowned. + */ + mtk_port_ch_disable(port); + return; + } + + /* tx_mtu is only valid once the channel open trb has completed. A zero + * frag_len would make wwan_port_fops_write() loop forever. + */ + if (!port->tx_mtu) { + dev_err(port_mngr->ctrl_blk->mdev->dev, + "Invalid tx_mtu for port(%s)\n", port->info.name); + mtk_port_ch_disable(port); + return; + } + + /* The core allocates frag_len + headroom_len and skb_put()s frag_len, + * so frag_len is the payload budget: subtract the CCCI header to make + * one core fragment exactly one CCCI packet. + */ + caps.frag_len =3D port->tx_mtu - sizeof(struct mtk_ccci_header); + caps.headroom_len =3D sizeof(struct mtk_ccci_header); + + /* These bits must be set before wwan_create_port(): the device node + * becomes openable inside it and mtk_port_common_open() rejects a + * port without PORT_S_ENABLE. w_port cannot be published first - it is + * this call's return value - so an RX frame arriving in between is + * dropped with -ENXIO by design. + */ + set_bit(PORT_S_WR, &port->status); + set_bit(PORT_S_ENABLE, &port->status); + + wp =3D wwan_create_port(port_mngr->ctrl_blk->mdev->dev, + port->w_priv.w_type, + &wwan_ops, &caps, port); + if (IS_ERR(wp)) { + dev_warn(port_mngr->ctrl_blk->mdev->dev, + "Failed to create wwan port for (%s)\n", port->info.name); + clear_bit(PORT_S_ENABLE, &port->status); + clear_bit(PORT_S_WR, &port->status); + mtk_port_ch_disable(port); + return; + } + + mutex_lock(&port->w_priv.w_lock); + port->w_priv.w_port =3D wp; + mutex_unlock(&port->w_priv.w_lock); +} + +static void mtk_port_wwan_disable(struct mtk_port *port) +{ + struct wwan_port *w_port; + int ret; + + if (!test_and_clear_bit(PORT_S_ENABLE, &port->status)) + return; + + /* PORT_S_WR is part of the blocking TX wait condition, and the waiter + * loops back on timeout, so without this wake it only notices at the + * next trb timeout rather than now. + */ + clear_bit(PORT_S_WR, &port->status); + wake_up_all(&port->trb_wq); + + /* w_lock must be dropped before wwan_remove_port(): that takes the + * core's ops_lock and forces stop(), which is close() taking w_lock. + */ + mutex_lock(&port->w_priv.w_lock); + w_port =3D port->w_priv.w_port; + port->w_priv.w_port =3D NULL; + mutex_unlock(&port->w_priv.w_lock); + + ret =3D mtk_port_ch_disable(port); + if (ret) + dev_warn(port->port_mngr->ctrl_blk->mdev->dev, + "Failed to disable channel for port(%s): %d\n", + port->info.name, ret); + + wwan_remove_port(w_port); +} + +static int mtk_port_wwan_recv(struct mtk_port *port, struct sk_buff *skb) +{ + /* Drop frames when nobody has the device open: wwan_port_rx() queues + * without bound and only a reader drains the queue, so accepting + * unsolicited traffic here would grow the rxq indefinitely. Both + * conditions are read under w_lock, which mtk_port_wwan_close() also + * takes, so a frame accepted here cannot land on a queue the core is + * about to purge. + */ + mutex_lock(&port->w_priv.w_lock); + if (!test_bit(PORT_S_OPEN, &port->status) || !port->w_priv.w_port) { + mutex_unlock(&port->w_priv.w_lock); + dev_dbg_ratelimited(port->port_mngr->ctrl_blk->mdev->dev, + "Drop RX for unopened port(%s)\n", port->info.name); + return -ENXIO; + } + + wwan_port_rx(port->w_priv.w_port, skb); + mutex_unlock(&port->w_priv.w_lock); + return 0; +} + +static const struct port_ops port_wwan_ops =3D { + .init =3D mtk_port_wwan_init, + .exit =3D mtk_port_wwan_exit, + .enable =3D mtk_port_wwan_enable, + .disable =3D mtk_port_wwan_disable, + .recv =3D mtk_port_wwan_recv, + .tx_complete =3D mtk_port_wwan_tx_complete, +}; + const struct port_ops *ports_ops[PORT_TYPE_MAX] =3D { &port_internal_ops, + &port_wwan_ops, }; diff --git a/drivers/net/wwan/t9xx/mtk_port_io.h b/drivers/net/wwan/t9xx/mt= k_port_io.h index 5a4e36c075a5..88aeb28d536b 100644 --- a/drivers/net/wwan/t9xx/mtk_port_io.h +++ b/drivers/net/wwan/t9xx/mtk_port_io.h @@ -20,6 +20,7 @@ struct port_ops { void (*enable)(struct mtk_port *port); void (*disable)(struct mtk_port *port); int (*recv)(struct mtk_port *port, struct sk_buff *skb); + void (*tx_complete)(struct mtk_port *port); }; =20 void *mtk_port_internal_open(struct mtk_md_dev *mdev, char *name, int flag= ); diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c b/drivers/net/wwan= /t9xx/pcie/mtk_trans_ctrl.c index 658189dace01..a1f1f86838f7 100644 --- a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c +++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c @@ -29,6 +29,10 @@ static const int mtk_srv_cfg[NR_CLDMA][HW_QUE_NUM] =3D { =20 /* the number of RX GPDs should be at least two */ static const struct queue_info mtk_queue_info[] =3D { + {CCCI_UART2_TX, CCCI_UART2_RX, CLDMA1, TXQ(5), RXQ(5), + Q_MTU_3_5K, Q_MTU_3_5K, TX_GPD_NUM, RX_GPD_NUM, Q_FRAG_3_5K, Q_FRAG_3_5K= , 0}, + {CCCI_MBIM_TX, CCCI_MBIM_RX, CLDMA1, TXQ(2), RXQ(2), + Q_MTU_3_5K, Q_MTU_3_5K, TX_GPD_NUM, RX_GPD_NUM, Q_FRAG_3_5K, Q_FRAG_3_5K= , 0}, {CCCI_CONTROL_TX, CCCI_CONTROL_RX, CLDMA1, TXQ(0), RXQ(0), Q_MTU_3_5K, Q_MTU_3_5K, TX_GPD_NUM, RX_GPD_NUM, Q_FRAG_3_5K, Q_FRAG_3_5K= , 0}, {CCCI_SAP_CONTROL_TX, CCCI_SAP_CONTROL_RX, CLDMA0, TXQ(0), RXQ(0), @@ -36,6 +40,10 @@ static const struct queue_info mtk_queue_info[] =3D { }; =20 static const struct mtk_port_cfg mtk_port_cfg_tbl[] =3D { + {CCCI_UART2_TX, CCCI_UART2_RX, PORT_TYPE_WWAN, "AT", + PORT_F_DFLT}, + {CCCI_MBIM_TX, CCCI_MBIM_RX, PORT_TYPE_WWAN, "MBIM", + PORT_F_DFLT}, {CCCI_CONTROL_TX, CCCI_CONTROL_RX, PORT_TYPE_INTERNAL, "MDCTRL", PORT_F_DFLT}, {CCCI_SAP_CONTROL_TX, CCCI_SAP_CONTROL_RX, PORT_TYPE_INTERNAL, "SAPCTRL", --=20 2.34.1