From nobody Sun Jun 28 08:39:27 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69743C433F5 for ; Wed, 9 Feb 2022 22:42:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236225AbiBIWmt (ORCPT ); Wed, 9 Feb 2022 17:42:49 -0500 Received: from gmail-smtp-in.l.google.com ([23.128.96.19]:38026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236161AbiBIWmb (ORCPT ); Wed, 9 Feb 2022 17:42:31 -0500 Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE5B1C0302E5 for ; Wed, 9 Feb 2022 14:42:31 -0800 (PST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 219IO5Ei014382; Wed, 9 Feb 2022 14:42:22 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=+IeGoPa54Olck2SHTVItqzt1UTNReHPKUw2QQ7FDvSQ=; b=UgFUboO12HbiZ30Ce+GxhofvKG01v79cJib5LEq/q8PYAEY3TrPAj9p06JEoXNErindY b800C1xSuTySP0dRz/wvx1DSW2DF3bgAxH5kL1H6euzvX7e1PonjUHxmwgb/j2KvSCV4 CBKNWplVvLqVEaRjMwuL3mGp6Z7cT6/KD1WKeKAWVmovuQZHyPNU7wf86OT8JrJQf8N0 4K/uAxGm9+I/GKL/8cwbkoBmVvvjzIk6b31buo0+yEvkMFIXNj0Df8wn+FE6iBKWzXei PpSrePc4SP2I+PnnbciOrkjG3np/t86F/a0zd6IP/qrGt8OML6r+i549Tob+asuOBs5B 7A== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3e4am93bx1-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Wed, 09 Feb 2022 14:42:22 -0800 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 9 Feb 2022 14:42:18 -0800 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Wed, 9 Feb 2022 14:42:18 -0800 Received: from rchin-dellt430.marvell.com (rchin-dellt430.marvell.com [10.85.176.141]) by maili.marvell.com (Postfix) with ESMTP id 8BE0E3F7094; Wed, 9 Feb 2022 14:42:18 -0800 (PST) From: Radha Mohan Chintakuntla To: , , CC: , , "Radha Mohan Chintakuntla" Subject: [PATCH 1/4] soc: octeontx2-sdp: Add SDP PF driver support Date: Wed, 9 Feb 2022 14:42:07 -0800 Message-ID: <20220209224210.153907-2-radhac@marvell.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20220209224210.153907-1-radhac@marvell.com> References: <20220209224210.153907-1-radhac@marvell.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Proofpoint-GUID: 4c8fRZPMrLGFbJnKTjzXLGuM0dFUQOw5 X-Proofpoint-ORIG-GUID: 4c8fRZPMrLGFbJnKTjzXLGuM0dFUQOw5 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-09_12,2022-02-09_01,2021-12-02_01 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This patch adds base support to the SDP PF driver under the RVU PFs found in Marvell OcteonTx2 SoCs. The driver configures the SRIOV and allocates interrupts. Signed-off-by: Radha Mohan Chintakuntla --- MAINTAINERS | 7 + drivers/soc/Kconfig | 1 + drivers/soc/Makefile | 1 + drivers/soc/marvell/Kconfig | 18 + drivers/soc/marvell/Makefile | 2 + drivers/soc/marvell/octeontx2-sdp/Makefile | 9 + drivers/soc/marvell/octeontx2-sdp/sdp.c | 363 +++++++++++++++++++++ drivers/soc/marvell/octeontx2-sdp/sdp.h | 61 ++++ 8 files changed, 462 insertions(+) create mode 100644 drivers/soc/marvell/Kconfig create mode 100644 drivers/soc/marvell/Makefile create mode 100644 drivers/soc/marvell/octeontx2-sdp/Makefile create mode 100644 drivers/soc/marvell/octeontx2-sdp/sdp.c create mode 100644 drivers/soc/marvell/octeontx2-sdp/sdp.h diff --git a/MAINTAINERS b/MAINTAINERS index 3e461db9cd91..fb48e0317d34 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -11558,6 +11558,13 @@ S: Supported F: Documentation/networking/device_drivers/ethernet/marvell/octeontx2.rst F: drivers/net/ethernet/marvell/octeontx2/af/ =20 +MARVELL OCTEONTX2 SDP PHYISCAL FUNCTION DRIVER +M: Radha Mohan Chintakuntla +M: Veerasenareddy Burry +L: linux-arm-kernel@lists.infradead.org +S: Supported +F: drivers/soc/marvell/octeontx2-sdp + MARVELL PRESTERA ETHERNET SWITCH DRIVER M: Taras Chornyi S: Supported diff --git a/drivers/soc/Kconfig b/drivers/soc/Kconfig index a8562678c437..bfb1d993d257 100644 --- a/drivers/soc/Kconfig +++ b/drivers/soc/Kconfig @@ -12,6 +12,7 @@ source "drivers/soc/fsl/Kconfig" source "drivers/soc/imx/Kconfig" source "drivers/soc/ixp4xx/Kconfig" source "drivers/soc/litex/Kconfig" +source "drivers/soc/marvell/Kconfig" source "drivers/soc/mediatek/Kconfig" source "drivers/soc/qcom/Kconfig" source "drivers/soc/renesas/Kconfig" diff --git a/drivers/soc/Makefile b/drivers/soc/Makefile index adb30c2d4fea..1b6f1574b759 100644 --- a/drivers/soc/Makefile +++ b/drivers/soc/Makefile @@ -17,6 +17,7 @@ obj-y +=3D imx/ obj-y +=3D ixp4xx/ obj-$(CONFIG_SOC_XWAY) +=3D lantiq/ obj-$(CONFIG_LITEX_SOC_CONTROLLER) +=3D litex/ +obj-y +=3D marvell/ obj-y +=3D mediatek/ obj-y +=3D amlogic/ obj-y +=3D qcom/ diff --git a/drivers/soc/marvell/Kconfig b/drivers/soc/marvell/Kconfig new file mode 100644 index 000000000000..4e8f8a60b3df --- /dev/null +++ b/drivers/soc/marvell/Kconfig @@ -0,0 +1,18 @@ +# +# MARVELL SoC drivers +# + +menu "Marvell SoC drivers" + +config OCTEONTX2_SDP_PF + tristate "OcteonTX2 SDP PF driver" + depends on ARM64 && PCI && OCTEONTX2_AF && OCTEONTX2_MBOX + default y + help + This driver configures the SDP RVU PF when OcteonTx is in PCIe EndPoint + mode. It sets up the SDP rings and the VFs to be used in packet transf= er + using the NIX block. + + To compile this as a module choose M here. + +endmenu diff --git a/drivers/soc/marvell/Makefile b/drivers/soc/marvell/Makefile new file mode 100644 index 000000000000..9a9a4435d64d --- /dev/null +++ b/drivers/soc/marvell/Makefile @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0 +obj-y +=3D octeontx2-sdp/ diff --git a/drivers/soc/marvell/octeontx2-sdp/Makefile b/drivers/soc/marve= ll/octeontx2-sdp/Makefile new file mode 100644 index 000000000000..16a5d353ea44 --- /dev/null +++ b/drivers/soc/marvell/octeontx2-sdp/Makefile @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Makefile for Marvell's OcteonTX2 SDP PF driver +# + +obj-$(CONFIG_OCTEONTX2_SDP_PF) +=3D octeontx2_sdp.o + +octeontx2_sdp-y :=3D sdp.o +ccflags-y +=3D -I$(srctree)/drivers/net/ethernet/marvell/octeontx2/af diff --git a/drivers/soc/marvell/octeontx2-sdp/sdp.c b/drivers/soc/marvell/= octeontx2-sdp/sdp.c new file mode 100644 index 000000000000..dda85b5c0264 --- /dev/null +++ b/drivers/soc/marvell/octeontx2-sdp/sdp.c @@ -0,0 +1,363 @@ +// SPDX-License-Identifier: GPL-2.0 +/* OcteonTX2 SDP driver + * + * Copyright (C) 2020 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "rvu.h" +#include "rvu_reg.h" +#include "rvu_struct.h" +#include "sdp.h" + +#define DRV_NAME "octeontx2-sdp" +#define DRV_VERSION "1.1" + +#define PCI_DEVID_OCTEONTX2_SDP_PF 0xA0F6 +/* PCI BARs */ +#define PCI_AF_REG_BAR_NUM 0 +#define PCI_CFG_REG_BAR_NUM 2 +#define MBOX_BAR_NUM 4 + +#define SDP_PPAIR_THOLD 0x400 + +static void +sdp_write64(struct sdp_dev *rvu, u64 b, u64 s, u64 o, u64 v) +{ + writeq(v, rvu->bar2 + ((b << 20) | (s << 12) | o)); +} + +static u64 sdp_read64(struct sdp_dev *rvu, u64 b, u64 s, u64 o) +{ + return readq(rvu->bar2 + ((b << 20) | (s << 12) | o)); +} + +static int sdp_check_pf_usable(struct sdp_dev *sdp) +{ + u64 rev; + + rev =3D sdp_read64(sdp, BLKADDR_RVUM, 0, + RVU_PF_BLOCK_ADDRX_DISC(BLKADDR_RVUM)); + rev =3D (rev >> 12) & 0xFF; + /* Check if AF has setup revision for RVUM block, + * otherwise this driver probe should be deferred + * until AF driver comes up. + */ + if (!rev) { + dev_warn(&sdp->pdev->dev, + "AF is not initialized, deferring probe\n"); + return -EPROBE_DEFER; + } + return 0; +} + +static int sdp_alloc_irqs(struct pci_dev *pdev) +{ + struct sdp_dev *sdp; + int err; + + sdp =3D pci_get_drvdata(pdev); + + /* Get number of MSIX vector count and allocate vectors first */ + sdp->msix_count =3D pci_msix_vec_count(pdev); + + err =3D pci_alloc_irq_vectors(pdev, sdp->msix_count, sdp->msix_count, + PCI_IRQ_MSIX); + + if (err < 0) { + dev_err(&pdev->dev, "pci_alloc_irq_vectors() failed %d\n", err); + return err; + } + + sdp->irq_names =3D kmalloc_array(sdp->msix_count, NAME_SIZE, GFP_KERNEL); + if (!sdp->irq_names) { + err =3D -ENOMEM; + goto err_irq_names; + } + + sdp->irq_allocated =3D kcalloc(sdp->msix_count, sizeof(bool), GFP_KERNEL); + if (!sdp->irq_allocated) { + err =3D -ENOMEM; + goto err_irq_allocated; + } + + return 0; + +err_irq_allocated: + kfree(sdp->irq_names); + sdp->irq_names =3D NULL; +err_irq_names: + pci_free_irq_vectors(pdev); + sdp->msix_count =3D 0; + + return err; +} + +static void sdp_free_irqs(struct pci_dev *pdev) +{ + struct sdp_dev *sdp; + int irq; + + sdp =3D pci_get_drvdata(pdev); + for (irq =3D 0; irq < sdp->msix_count; irq++) { + if (sdp->irq_allocated[irq]) + free_irq(pci_irq_vector(sdp->pdev, irq), sdp); + } + + pci_free_irq_vectors(pdev); + + kfree(sdp->irq_names); + kfree(sdp->irq_allocated); +} + +static int __sriov_disable(struct pci_dev *pdev) +{ + struct sdp_dev *sdp; + + sdp =3D pci_get_drvdata(pdev); + if (pci_vfs_assigned(pdev)) { + dev_err(&pdev->dev, "Disabing VFs while VFs are assigned\n"); + dev_err(&pdev->dev, "VFs will not be freed\n"); + return -EPERM; + } + + pci_disable_sriov(pdev); + + kfree(sdp->vf_info); + sdp->vf_info =3D NULL; + + return 0; +} + +static int __sriov_enable(struct pci_dev *pdev, int num_vfs) +{ + struct rvu_vf *vf_ptr; + int curr_vfs, vf =3D 0; + struct sdp_dev *sdp; + int err; + + curr_vfs =3D pci_num_vf(pdev); + if (!curr_vfs && !num_vfs) + return -EINVAL; + + if (curr_vfs) { + dev_err( + &pdev->dev, + "Virtual Functions are already enabled on this device\n"); + return -EINVAL; + } + if (num_vfs > SDP_MAX_VFS) + num_vfs =3D SDP_MAX_VFS; + + sdp =3D pci_get_drvdata(pdev); + + if (sdp_get_available_rsrcs(sdp)) { + dev_err(&pdev->dev, "Failed to get resource limits.\n"); + return -EFAULT; + } + + sdp->vf_info =3D kcalloc(num_vfs, sizeof(struct rvu_vf), GFP_KERNEL); + if (sdp->vf_info =3D=3D NULL) + return -ENOMEM; + + sdp->num_vfs =3D num_vfs; + + err =3D pci_enable_sriov(pdev, num_vfs); + if (err) { + dev_err(&pdev->dev, "Failed to enable to SRIOV VFs: %d\n", err); + goto err_enable_sriov; + } + + return num_vfs; +} + +static int sdp_sriov_configure(struct pci_dev *pdev, int num_vfs) +{ + if (num_vfs =3D=3D 0) + return __sriov_disable(pdev); + else + return __sriov_enable(pdev, num_vfs); +} + + +static int sdp_probe(struct pci_dev *pdev, const struct pci_device_id *id) +{ + uint64_t inst, regval; + struct sdp_dev *sdp; + struct device *dev; + int err; + + dev =3D &pdev->dev; + sdp =3D devm_kzalloc(dev, sizeof(struct sdp_dev), GFP_KERNEL); + if (sdp =3D=3D NULL) + return -ENOMEM; + + sdp->pdev =3D pdev; + pci_set_drvdata(pdev, sdp); + + mutex_init(&sdp->lock); + + err =3D pci_enable_device(pdev); + if (err) { + dev_err(dev, "Failed to enable PCI device\n"); + goto enable_failed; + } + + err =3D pci_request_regions(pdev, DRV_NAME); + if (err) { + dev_err(dev, "PCI request regions failed 0x%x\n", err); + goto map_failed; + } + + if (pci_sriov_get_totalvfs(pdev) <=3D 0) { + err =3D -ENODEV; + goto set_mask_failed; + } + + err =3D pci_set_dma_mask(pdev, DMA_BIT_MASK(48)); + if (err) { + dev_err(dev, "Unable to set DMA mask\n"); + goto set_mask_failed; + } + + err =3D pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(48)); + if (err) { + dev_err(dev, "Unable to set DMA mask\n"); + goto set_mask_failed; + } + + pci_set_master(pdev); + + /* CSR Space mapping */ + sdp->bar2 =3D pcim_iomap(pdev, PCI_CFG_REG_BAR_NUM, + pci_resource_len(pdev, PCI_CFG_REG_BAR_NUM)); + if (!sdp->bar2) { + dev_err(&pdev->dev, "Unable to map BAR2\n"); + err =3D -ENODEV; + goto set_mask_failed; + } + + err =3D sdp_check_pf_usable(sdp); + if (err) + goto pf_unusable; + + /* Map SDP register area */ + /* right now only 2 SDP blocks are supported */ + inst =3D list_empty(&sdp_dev_lst_head) ? 0 : 1; + sdp->sdp_base =3D ioremap(SDP_BASE(inst), SDP_REG_SIZE); + if (!sdp->sdp_base) { + dev_err(&pdev->dev, "Unable to map SDP CSR space\n"); + err =3D -ENODEV; + goto pf_unusable; + } + /* Map PF-AF mailbox memory */ + sdp->af_mbx_base =3D ioremap_wc(pci_resource_start(pdev, MBOX_BAR_NUM), + pci_resource_len(pdev, MBOX_BAR_NUM)); + if (!sdp->af_mbx_base) { + dev_err(&pdev->dev, "Unable to map BAR4\n"); + err =3D -ENODEV; + goto pf_unusable; + } + + if (sdp_alloc_irqs(pdev)) { + dev_err(&pdev->dev, + "Unable to allocate MSIX Interrupt vectors\n"); + err =3D -ENODEV; + goto alloc_irqs_failed; + } + + regval =3D readq(sdp->sdp_base + SDPX_GBL_CONTROL); + regval |=3D (1 << 2); /* BPFLR_D disable clearing BP in FLR */ + writeq(regval, sdp->sdp_base + SDPX_GBL_CONTROL); + + sdp_sriov_configure(sdp->pdev, sdp->info.max_vfs); + + spin_lock(&sdp_lst_lock); + list_add(&sdp->list, &sdp_dev_lst_head); + spin_unlock(&sdp_lst_lock); + + return 0; + +alloc_irqs_failed: + iounmap(sdp->af_mbx_base); +pf_unusable: + pcim_iounmap(pdev, sdp->bar2); +set_mask_failed: + pci_release_regions(pdev); +map_failed: + pci_disable_device(pdev); +enable_failed: + pci_set_drvdata(pdev, NULL); + devm_kfree(dev, sdp); + return err; +} + +static void sdp_remove(struct pci_dev *pdev) +{ + struct sdp_dev *sdp =3D pci_get_drvdata(pdev); + + + spin_lock(&sdp_lst_lock); + list_del(&sdp->list); + spin_unlock(&sdp_lst_lock); + + if (sdp->num_vfs) + __sriov_disable(pdev); + + sdp_free_irqs(pdev); + + if (sdp->af_mbx_base) + iounmap(sdp->af_mbx_base); + if (sdp->bar2) + pcim_iounmap(pdev, sdp->bar2); + + pci_release_regions(pdev); + pci_disable_device(pdev); + devm_kfree(&pdev->dev, sdp); +} + +static const struct pci_device_id rvu_sdp_id_table[] =3D { + {PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, PCI_DEVID_OCTEONTX2_SDP_PF)}, + {0} /* end of table */ +}; + +static struct pci_driver sdp_driver =3D { + .name =3D DRV_NAME, + .id_table =3D rvu_sdp_id_table, + .probe =3D sdp_probe, + .remove =3D sdp_remove, + .sriov_configure =3D sdp_sriov_configure, +}; + +static int __init otx2_sdp_init_module(void) +{ + pr_info("%s\n", DRV_NAME); + + spin_lock_init(&sdp_lst_lock); + return pci_register_driver(&sdp_driver); +} + +static void __exit otx2_sdp_exit_module(void) +{ + pci_unregister_driver(&sdp_driver); +} + +module_init(otx2_sdp_init_module); +module_exit(otx2_sdp_exit_module); +MODULE_AUTHOR("Marvell International Ltd."); +MODULE_DESCRIPTION("Marvell OcteonTX2 SDP PF Driver"); +MODULE_LICENSE("GPL v2"); +MODULE_VERSION(DRV_VERSION); +MODULE_DEVICE_TABLE(pci, rvu_sdp_id_table); diff --git a/drivers/soc/marvell/octeontx2-sdp/sdp.h b/drivers/soc/marvell/= octeontx2-sdp/sdp.h new file mode 100644 index 000000000000..c8ef7a2ade45 --- /dev/null +++ b/drivers/soc/marvell/octeontx2-sdp/sdp.h @@ -0,0 +1,61 @@ +/* SPDX-License-Identifier: GPL-2.0 + * OcteonTX2 SDP driver + * + * Copyright (C) 2022 Marvell International Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ +#ifndef SDP_H_ +#define SDP_H_ + +#include +#include +#include +#include "mbox.h" + +#define SDP_BASE(a) (0x86E080000000ull | a << 36) +#define SDP_REG_SIZE 0x42000000 + +#define SDPX_GBL_CONTROL (0x40080200ull) + +struct sdp_dev { + struct list_head list; + struct mutex lock; + struct pci_dev *pdev; + void __iomem *sdp_base; + void __iomem *bar2; + void __iomem *af_mbx_base; +#define SDP_VF_ENABLED 0x1 + u32 flags; + u32 num_vfs; + u16 chan_base; + u16 num_chan; + bool *irq_allocated; + char *irq_names; + int msix_count; + int pf; + u8 valid_ep_pem_mask; + u8 mac_mask; + + struct sdp_node_info info; + struct rvu_vf *vf_info; + struct free_rsrcs_rsp limits; /* Maximum limits for all VFs */ +}; + +struct rvu_vf { + struct work_struct pfvf_flr_work; + struct device_attribute in_use_attr; + struct pci_dev *pdev; + struct kobject *limits_kobj; + /* pointer to PF struct this PF belongs to */ + struct sdp_dev *sdp; + int vf_id; + int intr_idx; /* vf_id%64 actually */ + bool in_use; + bool got_flr; +}; + + +#endif --=20 2.24.1 From nobody Sun Jun 28 08:39:27 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A03FC433F5 for ; Wed, 9 Feb 2022 23:16:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229863AbiBIXQB (ORCPT ); Wed, 9 Feb 2022 18:16:01 -0500 Received: from gmail-smtp-in.l.google.com ([23.128.96.19]:49638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234489AbiBIXPW (ORCPT ); Wed, 9 Feb 2022 18:15:22 -0500 X-Greylist: delayed 1946 seconds by postgrey-1.37 at lindbergh.monkeyblade.net; Wed, 09 Feb 2022 15:14:58 PST Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6BFCE0536EC for ; Wed, 9 Feb 2022 15:14:58 -0800 (PST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 219KEmBO013146; Wed, 9 Feb 2022 14:42:21 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=pjqjXnoWw716hxwQyfAnuGxeaNUWvsbakPWMCUb7fnQ=; b=eIp20DHVr2VO7MK8aDPPd6oC6jI4v96IipwfkPhb7eRnuSAjLfdjIGHspl5g0s+DiR+d x9tquuTQttBhA+iSPJrGRjg8GnTArgGkTPpobEdVb7lvFAGYYVCCPN/iBKyMV1mGXHwC bcln38Zl9dyUn7hO/avFvOpNGwsdCKTQspZ8mWUCV2npv3wtu3C20iqo0iYgSU7Afkkm OY1hnuIMEEmU9iqPoAyyqqh7py9CLR5fQjeevFAwoo2sXzBpT7IhGKobXyb8jCm11P0r d+YYAIsmhDa+EKGHCJkktFYKLgDTi9RNeTCe01GZhTzHvsJaeClsSqGyM8w+LkuCO5Sc RA== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3e4cpj2tkj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Wed, 09 Feb 2022 14:42:20 -0800 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 9 Feb 2022 14:42:19 -0800 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Wed, 9 Feb 2022 14:42:18 -0800 Received: from rchin-dellt430.marvell.com (rchin-dellt430.marvell.com [10.85.176.141]) by maili.marvell.com (Postfix) with ESMTP id C860E3F709C; Wed, 9 Feb 2022 14:42:18 -0800 (PST) From: Radha Mohan Chintakuntla To: , , CC: , , "Radha Mohan Chintakuntla" Subject: [PATCH 2/4] soc: octeontx2-sdp: Add mailbox support Date: Wed, 9 Feb 2022 14:42:08 -0800 Message-ID: <20220209224210.153907-3-radhac@marvell.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20220209224210.153907-1-radhac@marvell.com> References: <20220209224210.153907-1-radhac@marvell.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Proofpoint-GUID: udlJOXs9gcNXcWPP8lGOjdhgb6XtnCuN X-Proofpoint-ORIG-GUID: udlJOXs9gcNXcWPP8lGOjdhgb6XtnCuN X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-09_12,2022-02-09_01,2021-12-02_01 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Added mailbox support to communicate between VFs to PF and between PF to AF (Administrative Function). Signed-off-by: Radha Mohan Chintakuntla --- drivers/soc/marvell/octeontx2-sdp/sdp.c | 799 ++++++++++++++++++++++++ drivers/soc/marvell/octeontx2-sdp/sdp.h | 22 +- 2 files changed, 820 insertions(+), 1 deletion(-) diff --git a/drivers/soc/marvell/octeontx2-sdp/sdp.c b/drivers/soc/marvell/= octeontx2-sdp/sdp.c index dda85b5c0264..fef063b6ce1f 100644 --- a/drivers/soc/marvell/octeontx2-sdp/sdp.c +++ b/drivers/soc/marvell/octeontx2-sdp/sdp.c @@ -33,6 +33,9 @@ =20 #define SDP_PPAIR_THOLD 0x400 =20 +static spinlock_t sdp_lst_lock; +LIST_HEAD(sdp_dev_lst_head); + static void sdp_write64(struct sdp_dev *rvu, u64 b, u64 s, u64 o, u64 v) { @@ -44,6 +47,555 @@ static u64 sdp_read64(struct sdp_dev *rvu, u64 b, u64 s= , u64 o) return readq(rvu->bar2 + ((b << 20) | (s << 12) | o)); } =20 +static void enable_af_mbox_int(struct pci_dev *pdev) +{ + struct sdp_dev *sdp; + + sdp =3D pci_get_drvdata(pdev); + /* Clear interrupt if any */ + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_INT, 0x1ULL); + + /* Now Enable AF-PF interrupt */ + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_INT_ENA_W1S, 0x1ULL); +} + +static void disable_af_mbox_int(struct pci_dev *pdev) +{ + struct sdp_dev *sdp; + + sdp =3D pci_get_drvdata(pdev); + /* Clear interrupt if any */ + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_INT, 0x1ULL); + + /* Now Disable AF-PF interrupt */ + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_INT_ENA_W1C, 0x1ULL); +} + +static int +forward_to_mbox(struct sdp_dev *sdp, struct otx2_mbox *mbox, int devid, + struct mbox_msghdr *req, int size, const char *mstr) +{ + struct mbox_msghdr *msg; + int res =3D 0; + + msg =3D otx2_mbox_alloc_msg(mbox, devid, size); + if (msg =3D=3D NULL) + return -ENOMEM; + + memcpy((uint8_t *)msg + sizeof(struct mbox_msghdr), + (uint8_t *)req + sizeof(struct mbox_msghdr), size); + msg->id =3D req->id; + msg->pcifunc =3D req->pcifunc; + msg->sig =3D req->sig; + msg->ver =3D req->ver; + + otx2_mbox_msg_send(mbox, devid); + res =3D otx2_mbox_wait_for_rsp(mbox, devid); + if (res =3D=3D -EIO) { + dev_err(&sdp->pdev->dev, "RVU %s MBOX timeout.\n", mstr); + goto err; + } else if (res) { + dev_err(&sdp->pdev->dev, + "RVU %s MBOX error: %d.\n", mstr, res); + res =3D -EFAULT; + goto err; + } + + return 0; +err: + return res; +} + +static int +handle_af_req(struct sdp_dev *sdp, struct rvu_vf *vf, struct mbox_msghdr *= req, + int size) +{ + /* We expect a request here */ + if (req->sig !=3D OTX2_MBOX_REQ_SIG) { + dev_err(&sdp->pdev->dev, + "UP MBOX msg with wrong signature %x, ID 0x%x\n", + req->sig, req->id); + return -EINVAL; + } + + /* If handling notifs in PF is required,add a switch-case here. */ + return forward_to_mbox(sdp, &sdp->pfvf_mbox_up, vf->vf_id, req, size, + "VF"); +} + + +static void sdp_afpf_mbox_handler_up(struct work_struct *work) +{ + struct sdp_dev *sdp =3D container_of(work, struct sdp_dev, mbox_wrk_up); + struct otx2_mbox *mbox =3D &sdp->afpf_mbox_up; + struct otx2_mbox_dev *mdev =3D mbox->dev; + struct mbox_hdr *req_hdr; + struct mbox_msghdr *msg; + int offset, id, err; + struct rvu_vf *vf; + + /* sync with mbox memory region */ + smp_rmb(); + + /* Process received mbox messages */ + req_hdr =3D (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); + offset =3D ALIGN(sizeof(*req_hdr), MBOX_MSG_ALIGN); + for (id =3D 0; id < req_hdr->num_msgs; id++) { + msg =3D (struct mbox_msghdr *)(mdev->mbase + mbox->rx_start + + offset); + + if ((msg->pcifunc >> RVU_PFVF_PF_SHIFT) !=3D sdp->pf || + (msg->pcifunc & RVU_PFVF_FUNC_MASK) <=3D sdp->num_vfs) + err =3D -EINVAL; + else { + vf =3D &sdp->vf_info[msg->pcifunc & RVU_PFVF_FUNC_MASK]; + err =3D handle_af_req(sdp, vf, msg, + msg->next_msgoff - offset); + } + if (err) + otx2_reply_invalid_msg(mbox, 0, msg->pcifunc, msg->id); + offset =3D msg->next_msgoff; + } + + otx2_mbox_msg_send(mbox, 0); +} + +static void sdp_afpf_mbox_handler(struct work_struct *work) +{ + struct nix_lf_alloc_rsp *alloc_rsp; + struct otx2_mbox *af_mbx, *vf_mbx; + struct mbox_msghdr *msg, *fwd; + struct free_rsrcs_rsp *rsp; + int offset, i, vf_id, size; + struct mbox_hdr *rsp_hdr; + struct sdp_dev *sdp; + struct rvu_vf *vf; + + /* Read latest mbox data */ + smp_rmb(); + + sdp =3D container_of(work, struct sdp_dev, mbox_wrk); + af_mbx =3D &sdp->afpf_mbox; + vf_mbx =3D &sdp->pfvf_mbox; + rsp_hdr =3D (struct mbox_hdr *)(af_mbx->dev->mbase + af_mbx->rx_start); + if (rsp_hdr->num_msgs =3D=3D 0) + return; + offset =3D ALIGN(sizeof(struct mbox_hdr), MBOX_MSG_ALIGN); + + for (i =3D 0; i < rsp_hdr->num_msgs; i++) { + msg =3D (struct mbox_msghdr *)(af_mbx->dev->mbase + + af_mbx->rx_start + offset); + size =3D msg->next_msgoff - offset; + + if (msg->id >=3D MBOX_MSG_MAX) { + dev_err(&sdp->pdev->dev, + "MBOX msg with unknown ID 0x%x\n", msg->id); + goto end; + } + + if (msg->sig !=3D OTX2_MBOX_RSP_SIG) { + dev_err(&sdp->pdev->dev, + "MBOX msg with wrong signature %x, ID 0x%x\n", + msg->sig, msg->id); + goto end; + } + + vf_id =3D (msg->pcifunc & RVU_PFVF_FUNC_MASK); + + if (msg->id =3D=3D MBOX_MSG_NIX_LF_ALLOC) { + alloc_rsp =3D (struct nix_lf_alloc_rsp *)msg; + if (vf_id =3D=3D 1) + alloc_rsp->rx_chan_cnt =3D sdp->info.vf_rings[0]; + else + alloc_rsp->rx_chan_cnt =3D sdp->info.vf_rings[1]; + alloc_rsp->tx_chan_cnt =3D alloc_rsp->rx_chan_cnt; + } + + if (vf_id > 0) { + if (vf_id > sdp->num_vfs) { + dev_err(&sdp->pdev->dev, + "MBOX msg to unknown VF: %d >=3D %d\n", + vf_id, sdp->num_vfs); + goto end; + } + vf =3D &sdp->vf_info[vf_id - 1]; + /* Ignore stale responses and VFs in FLR. */ + if (!vf->in_use || vf->got_flr) + goto end; + fwd =3D otx2_mbox_alloc_msg(vf_mbx, vf_id - 1, size); + if (!fwd) { + dev_err(&sdp->pdev->dev, + "Forwarding to VF%d failed.\n", vf_id); + goto end; + } + memcpy((uint8_t *)fwd + sizeof(struct mbox_msghdr), + (uint8_t *)msg + sizeof(struct mbox_msghdr), + size); + fwd->id =3D msg->id; + fwd->pcifunc =3D msg->pcifunc; + fwd->sig =3D msg->sig; + fwd->ver =3D msg->ver; + fwd->rc =3D msg->rc; + } else { + if (msg->ver < OTX2_MBOX_VERSION) { + dev_err(&sdp->pdev->dev, + "MBOX msg with version %04x !=3D %04x\n", + msg->ver, OTX2_MBOX_VERSION); + goto end; + } + + switch (msg->id) { + case MBOX_MSG_READY: + sdp->pf =3D (msg->pcifunc >> RVU_PFVF_PF_SHIFT) & + RVU_PFVF_PF_MASK; + break; + case MBOX_MSG_FREE_RSRC_CNT: + rsp =3D (struct free_rsrcs_rsp *)msg; + memcpy(&sdp->limits, msg, sizeof(*rsp)); + break; + case MBOX_MSG_SET_SDP_CHAN_INFO: + /* Nothing to do */ + break; + case MBOX_MSG_GET_SDP_CHAN_INFO: + /* Nothing to do */ + break; + default: + dev_err(&sdp->pdev->dev, + "Unsupported msg %d received.\n", + msg->id); + break; + } + } +end: + offset =3D msg->next_msgoff; + af_mbx->dev->msgs_acked++; + } + otx2_mbox_reset(af_mbx, 0); +} + +static int +reply_free_rsrc_cnt(struct sdp_dev *sdp, struct rvu_vf *vf, + struct mbox_msghdr *req, int size) +{ + struct free_rsrcs_rsp *rsp; + + rsp =3D (struct free_rsrcs_rsp *)otx2_mbox_alloc_msg(&sdp->pfvf_mbox, + vf->vf_id, + sizeof(*rsp)); + if (rsp =3D=3D NULL) + return -ENOMEM; + + rsp->hdr.id =3D MBOX_MSG_FREE_RSRC_CNT; + rsp->hdr.pcifunc =3D req->pcifunc; + rsp->hdr.sig =3D OTX2_MBOX_RSP_SIG; + return 0; +} + +static int +handle_vf_req(struct sdp_dev *sdp, struct rvu_vf *vf, struct mbox_msghdr *= req, + int size) +{ + int err =3D 0, chan_idx, chan_diff, reg_off =3D 0, vf_id; + uint64_t en_bp; + u16 chan_base; + u8 chan_cnt; + + /* Check if valid, if not reply with a invalid msg */ + if (req->sig !=3D OTX2_MBOX_REQ_SIG) { + dev_err(&sdp->pdev->dev, + "VF MBOX msg with wrong signature %x, ID 0x%x\n", + req->sig, req->id); + return -EINVAL; + } + + switch (req->id) { + case MBOX_MSG_READY: + if (req->ver < OTX2_MBOX_VERSION) { + dev_err(&sdp->pdev->dev, + "VF MBOX msg with version %04x !=3D %04x\n", + req->ver, OTX2_MBOX_VERSION); + return -EINVAL; + } + vf->in_use =3D true; + err =3D forward_to_mbox(sdp, &sdp->afpf_mbox, 0, req, size, "AF"); + break; + case MBOX_MSG_FREE_RSRC_CNT: + if (req->ver < OTX2_MBOX_VERSION) { + dev_err(&sdp->pdev->dev, + "VF MBOX msg with version %04x !=3D %04x\n", + req->ver, OTX2_MBOX_VERSION); + return -EINVAL; + } + err =3D reply_free_rsrc_cnt(sdp, vf, req, size); + break; + case MBOX_MSG_ATTACH_RESOURCES: + if (req->ver < OTX2_MBOX_VERSION) { + dev_err(&sdp->pdev->dev, + "VF MBOX msg with version %04x !=3D %04x\n", + req->ver, OTX2_MBOX_VERSION); + return -EINVAL; + } + err =3D forward_to_mbox(sdp, &sdp->afpf_mbox, 0, req, size, "AF"); + break; + case MBOX_MSG_NIX_LF_ALLOC: + chan_base =3D sdp->chan_base + sdp->info.num_pf_rings; + for (vf_id =3D 0; vf_id < vf->vf_id; vf_id++) + chan_base +=3D sdp->info.vf_rings[vf_id]; + chan_cnt =3D sdp->info.vf_rings[vf->vf_id]; + for (chan_idx =3D 0; chan_idx < chan_cnt; chan_idx++) { + chan_diff =3D chan_base + chan_idx - sdp->chan_base; + reg_off =3D 0; + while (chan_diff > 63) { + reg_off +=3D 1; + chan_diff -=3D 64; + } + + en_bp =3D readq(sdp->sdp_base + + SDPX_OUT_BP_ENX_W1S(reg_off)); + en_bp |=3D (1ULL << chan_diff); + writeq(en_bp, sdp->sdp_base + + SDPX_OUT_BP_ENX_W1S(reg_off)); + } + fallthrough; + default: + err =3D forward_to_mbox(sdp, &sdp->afpf_mbox, 0, req, size, "AF"); + break; + } + + return err; +} + +static irqreturn_t sdp_af_pf_mbox_intr(int irq, void *arg) +{ + struct sdp_dev *sdp =3D (struct sdp_dev *)arg; + struct otx2_mbox_dev *mdev; + struct otx2_mbox *mbox; + struct mbox_hdr *hdr; + + /* Read latest mbox data */ + smp_rmb(); + + mbox =3D &sdp->afpf_mbox; + mdev =3D &mbox->dev[0]; + hdr =3D (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); + /* Handle PF =3D> AF channel response */ + if (hdr->num_msgs) + queue_work(sdp->afpf_mbox_wq, &sdp->mbox_wrk); + + mbox =3D &sdp->afpf_mbox_up; + mdev =3D &mbox->dev[0]; + hdr =3D (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); + /* Handle AF =3D> PF request */ + if (hdr->num_msgs) + queue_work(sdp->afpf_mbox_wq, &sdp->mbox_wrk_up); + + /* Clear and ack the interrupt */ + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_INT, 0x1ULL); + + return IRQ_HANDLED; +} + +static void sdp_pfvf_mbox_handler_up(struct work_struct *work) +{ + struct otx2_mbox *af_mbx, *vf_mbx; + struct mbox_msghdr *msg, *fwd; + struct mbox_hdr *rsp_hdr; + struct sdp_dev *sdp; + int offset, i, size; + struct rvu_vf *vf; + + /* Read latest mbox data */ + smp_rmb(); + + vf =3D container_of(work, struct rvu_vf, mbox_wrk_up); + sdp =3D vf->sdp; + af_mbx =3D &sdp->afpf_mbox; + vf_mbx =3D &sdp->pfvf_mbox; + rsp_hdr =3D (struct mbox_hdr *)(vf_mbx->dev[vf->vf_id].mbase + + vf_mbx->rx_start); + if (rsp_hdr->num_msgs =3D=3D 0) + return; + offset =3D ALIGN(sizeof(struct mbox_hdr), MBOX_MSG_ALIGN); + + for (i =3D 0; i < rsp_hdr->num_msgs; i++) { + msg =3D (struct mbox_msghdr *)(vf_mbx->dev->mbase + + vf_mbx->rx_start + offset); + size =3D msg->next_msgoff - offset; + + if (msg->sig !=3D OTX2_MBOX_RSP_SIG) { + dev_err(&sdp->pdev->dev, + "UP MBOX msg with wrong signature %x, ID 0x%x\n", + msg->sig, msg->id); + goto end; + } + + /* override message value with actual values */ + msg->pcifunc =3D (sdp->pf << RVU_PFVF_PF_SHIFT) | vf->vf_id; + + fwd =3D otx2_mbox_alloc_msg(af_mbx, 0, size); + if (!fwd) { + dev_err(&sdp->pdev->dev, + "UP Forwarding from VF%d to AF failed.\n", + vf->vf_id); + goto end; + } + memcpy((uint8_t *)fwd + sizeof(struct mbox_msghdr), + (uint8_t *)msg + sizeof(struct mbox_msghdr), + size); + fwd->id =3D msg->id; + fwd->pcifunc =3D msg->pcifunc; + fwd->sig =3D msg->sig; + fwd->ver =3D msg->ver; + fwd->rc =3D msg->rc; +end: + offset =3D msg->next_msgoff; + vf_mbx->dev->msgs_acked++; + } + otx2_mbox_reset(vf_mbx, vf->vf_id); +} + +static void sdp_pfvf_mbox_handler(struct work_struct *work) +{ + struct rvu_vf *vf =3D container_of(work, struct rvu_vf, mbox_wrk); + struct sdp_dev *sdp =3D vf->sdp; + struct otx2_mbox_dev *mdev; + struct mbox_hdr *req_hdr; + struct mbox_msghdr *msg; + struct otx2_mbox *mbox; + int offset, id, err; + + mbox =3D &sdp->pfvf_mbox; + mdev =3D &mbox->dev[vf->vf_id]; + + /* sync with mbox memory region */ + smp_rmb(); + + /* Process received mbox messages */ + req_hdr =3D (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); + offset =3D ALIGN(sizeof(*req_hdr), MBOX_MSG_ALIGN); + for (id =3D 0; id < req_hdr->num_msgs; id++) { + msg =3D (struct mbox_msghdr *)(mdev->mbase + mbox->rx_start + + offset); + + /* Set which VF sent this message based on mbox IRQ */ + msg->pcifunc =3D ((u16)sdp->pf << RVU_PFVF_PF_SHIFT) | + ((vf->vf_id + 1) & RVU_PFVF_FUNC_MASK); + err =3D handle_vf_req(sdp, vf, msg, msg->next_msgoff - offset); + if (err) + otx2_reply_invalid_msg(mbox, vf->vf_id, msg->pcifunc, + msg->id); + offset =3D msg->next_msgoff; + } + /* Send mbox responses to VF */ + if (mdev->num_msgs) + otx2_mbox_msg_send(mbox, vf->vf_id); +} + +static irqreturn_t sdp_pf_vf_mbox_intr(int irq, void *arg) +{ + struct sdp_dev *sdp =3D (struct sdp_dev *)arg; + struct otx2_mbox_dev *mdev; + struct otx2_mbox *mbox; + struct mbox_hdr *hdr; + struct rvu_vf *vf; + int i, vfi; + u64 intr; + + /* Check which VF has raised an interrupt and schedule corresponding + * workq to process the MBOX + */ + for (i =3D 0; i < 2; i++) { + /* Read the interrupt bits */ + intr =3D sdp_read64(sdp, BLKADDR_RVUM, 0, + RVU_PF_VFPF_MBOX_INTX(i)); + + for (vfi =3D i * 64; vfi < sdp->num_vfs; vfi++) { + vf =3D &sdp->vf_info[vfi]; + if ((intr & (1ULL << vf->intr_idx)) =3D=3D 0) + continue; + mbox =3D &sdp->pfvf_mbox; + mdev =3D &mbox->dev[vf->vf_id]; + hdr =3D (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); + /* Handle VF =3D> PF channel request */ + if (hdr->num_msgs) + queue_work(sdp->pfvf_mbox_wq, &vf->mbox_wrk); + + mbox =3D &sdp->pfvf_mbox_up; + mdev =3D &mbox->dev[vf->vf_id]; + hdr =3D (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); + /* Handle PF =3D> VF channel response */ + if (hdr->num_msgs) + queue_work(sdp->pfvf_mbox_wq, &vf->mbox_wrk_up); + /* Clear the interrupt */ + sdp_write64(sdp, BLKADDR_RVUM, 0, + RVU_PF_VFPF_MBOX_INTX(i), + BIT_ULL(vf->intr_idx)); + } + } + + return IRQ_HANDLED; +} + +static void enable_vf_mbox_int(struct pci_dev *pdev) +{ + struct sdp_dev *sdp; + int ena_bits, idx; + + sdp =3D pci_get_drvdata(pdev); + + /* Clear any pending interrupts */ + for (idx =3D 0; idx < 2; idx++) { + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_VFPF_MBOX_INTX(idx), + ~0x0ULL); + } + + /* Enable VF MBOX interrupts */ + if (sdp->num_vfs > 64) { + sdp_write64(sdp, BLKADDR_RVUM, 0, + RVU_PF_VFPF_MBOX_INT_ENA_W1SX(0), + GENMASK_ULL(63, 0)); + ena_bits =3D (sdp->num_vfs - 64) - 1; + sdp_write64(sdp, BLKADDR_RVUM, 0, + RVU_PF_VFPF_MBOX_INT_ENA_W1SX(1), + GENMASK_ULL(ena_bits, 0)); + } else { + ena_bits =3D sdp->num_vfs - 1; + sdp_write64(sdp, BLKADDR_RVUM, 0, + RVU_PF_VFPF_MBOX_INT_ENA_W1SX(0), + GENMASK_ULL(ena_bits, 0)); + } +} + +static void disable_vf_mbox_int(struct pci_dev *pdev) +{ + struct sdp_dev *sdp; + int ena_bits, idx; + + sdp =3D pci_get_drvdata(pdev); + + /* Clear any pending interrupts */ + for (idx =3D 0; idx < 2; idx++) { + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_VFPF_MBOX_INTX(idx), + ~0x0ULL); + } + + /* Disable the MBOX interrupts for VFs */ + if (sdp->num_vfs > 64) { + sdp_write64(sdp, BLKADDR_RVUM, 0, + RVU_PF_VFPF_MBOX_INT_ENA_W1CX(0), + GENMASK_ULL(63, 0)); + ena_bits =3D (sdp->num_vfs - 64) - 1; + sdp_write64(sdp, BLKADDR_RVUM, 0, + RVU_PF_VFPF_MBOX_INT_ENA_W1CX(1), + GENMASK_ULL(ena_bits, 0)); + } else { + ena_bits =3D sdp->num_vfs - 1; + sdp_write64(sdp, BLKADDR_RVUM, 0, + RVU_PF_VFPF_MBOX_INT_ENA_W1CX(0), + GENMASK_ULL(ena_bits, 0)); + } +} + static int sdp_check_pf_usable(struct sdp_dev *sdp) { u64 rev; @@ -122,6 +674,138 @@ static void sdp_free_irqs(struct pci_dev *pdev) kfree(sdp->irq_allocated); } =20 +static int sdp_register_mbox_irq(struct pci_dev *pdev) +{ + int err, vec =3D RVU_PF_INT_VEC_VFPF_MBOX0, i; + struct sdp_dev *sdp; + + sdp =3D pci_get_drvdata(pdev); + + /* Register PF-AF interrupt handler */ + sprintf(&sdp->irq_names[RVU_PF_INT_VEC_AFPF_MBOX * NAME_SIZE], + "PF%02d_AF_MBOX_IRQ", pdev->devfn); + err =3D request_irq(pci_irq_vector(pdev, RVU_PF_INT_VEC_AFPF_MBOX), + sdp_af_pf_mbox_intr, 0, + &sdp->irq_names[RVU_PF_INT_VEC_AFPF_MBOX * NAME_SIZE], + sdp); + if (err) { + dev_err(&pdev->dev, + "request_irq() failed for AF_PF MSIX vector\n"); + return err; + } + sdp->irq_allocated[RVU_PF_INT_VEC_AFPF_MBOX] =3D true; + + err =3D otx2_mbox_init(&sdp->afpf_mbox, sdp->af_mbx_base, pdev, sdp->bar2, + MBOX_DIR_PFAF, 1); + if (err) { + dev_err(&pdev->dev, "Failed to initialize PF/AF MBOX\n"); + goto error; + } + err =3D otx2_mbox_init(&sdp->afpf_mbox_up, sdp->af_mbx_base, pdev, + sdp->bar2, MBOX_DIR_PFAF_UP, 1); + if (err) { + dev_err(&pdev->dev, "Failed to initialize PF/AF UP MBOX\n"); + goto error; + } + + /* Register for PF-VF mailbox interrupts + * There are 2 vectors starting at index 0x4 + */ + for (vec =3D RVU_PF_INT_VEC_VFPF_MBOX0, i =3D 0; + vec + i <=3D RVU_PF_INT_VEC_VFPF_MBOX1; i++) { + sprintf(&sdp->irq_names[(vec + i) * NAME_SIZE], + "PF%02d_VF_MBOX_IRQ%d", pdev->devfn, i); + err =3D request_irq(pci_irq_vector(pdev, vec + i), + sdp_pf_vf_mbox_intr, 0, + &sdp->irq_names[(vec + i) * NAME_SIZE], sdp); + if (err) { + dev_err(&pdev->dev, + "request_irq() failed for PFVF Mbox intr %d\n", + vec + i); + goto error; + } + sdp->irq_allocated[vec + i] =3D true; + } + + sdp->afpf_mbox_wq =3D alloc_workqueue( + "sdp_pfaf_mailbox", WQ_UNBOUND | WQ_HIGHPRI | WQ_MEM_RECLAIM, 1); + if (!sdp->afpf_mbox_wq) + goto error; + + INIT_WORK(&sdp->mbox_wrk, sdp_afpf_mbox_handler); + INIT_WORK(&sdp->mbox_wrk_up, sdp_afpf_mbox_handler_up); + + return err; + +error: + if (sdp->afpf_mbox_up.dev !=3D NULL) + otx2_mbox_destroy(&sdp->afpf_mbox_up); + if (sdp->afpf_mbox.dev !=3D NULL) + otx2_mbox_destroy(&sdp->afpf_mbox); + + return err; +} + +static int sdp_get_pcifunc(struct sdp_dev *sdp) +{ + struct msg_req *ready_req; + int res =3D 0; + + ready_req =3D (struct msg_req *) + otx2_mbox_alloc_msg_rsp(&sdp->afpf_mbox, 0, sizeof(ready_req), + sizeof(struct ready_msg_rsp)); + if (ready_req =3D=3D NULL) { + dev_err(&sdp->pdev->dev, "RVU MBOX failed to get message.\n"); + return -EFAULT; + } + + ready_req->hdr.id =3D MBOX_MSG_READY; + ready_req->hdr.sig =3D OTX2_MBOX_REQ_SIG; + otx2_mbox_msg_send(&sdp->afpf_mbox, 0); + res =3D otx2_mbox_wait_for_rsp(&sdp->afpf_mbox, 0); + if (res =3D=3D -EIO) { + dev_err(&sdp->pdev->dev, "RVU AF MBOX timeout.\n"); + } else if (res) { + dev_err(&sdp->pdev->dev, "RVU MBOX error: %d.\n", res); + res =3D -EFAULT; + } + return res; +} + +static int sdp_get_available_rsrcs(struct sdp_dev *sdp) +{ + struct mbox_msghdr *rsrc_req; + int res =3D 0; + + rsrc_req =3D otx2_mbox_alloc_msg(&sdp->afpf_mbox, 0, sizeof(*rsrc_req)); + if (rsrc_req =3D=3D NULL) { + dev_err(&sdp->pdev->dev, "RVU MBOX failed to get message.\n"); + return -EFAULT; + } + rsrc_req->id =3D MBOX_MSG_FREE_RSRC_CNT; + rsrc_req->sig =3D OTX2_MBOX_REQ_SIG; + rsrc_req->pcifunc =3D RVU_PFFUNC(sdp->pf, 0); + otx2_mbox_msg_send(&sdp->afpf_mbox, 0); + res =3D otx2_mbox_wait_for_rsp(&sdp->afpf_mbox, 0); + if (res =3D=3D -EIO) { + dev_err(&sdp->pdev->dev, "RVU AF MBOX timeout.\n"); + } else if (res) { + dev_err(&sdp->pdev->dev, + "RVU MBOX error: %d.\n", res); + res =3D -EFAULT; + } + return res; +} + +static void sdp_afpf_mbox_term(struct pci_dev *pdev) +{ + struct sdp_dev *sdp =3D pci_get_drvdata(pdev); + + destroy_workqueue(sdp->afpf_mbox_wq); + otx2_mbox_destroy(&sdp->afpf_mbox); + otx2_mbox_destroy(&sdp->afpf_mbox_up); +} + static int __sriov_disable(struct pci_dev *pdev) { struct sdp_dev *sdp; @@ -133,6 +817,20 @@ static int __sriov_disable(struct pci_dev *pdev) return -EPERM; } =20 + disable_vf_mbox_int(pdev); + + if (sdp->pfvf_mbox_wq) { + destroy_workqueue(sdp->pfvf_mbox_wq); + sdp->pfvf_mbox_wq =3D NULL; + } + if (sdp->pfvf_mbx_base) { + iounmap(sdp->pfvf_mbx_base); + sdp->pfvf_mbx_base =3D NULL; + } + + otx2_mbox_destroy(&sdp->pfvf_mbox); + otx2_mbox_destroy(&sdp->pfvf_mbox_up); + pci_disable_sriov(pdev); =20 kfree(sdp->vf_info); @@ -146,6 +844,7 @@ static int __sriov_enable(struct pci_dev *pdev, int num= _vfs) struct rvu_vf *vf_ptr; int curr_vfs, vf =3D 0; struct sdp_dev *sdp; + u64 pf_vf_mbox_base; int err; =20 curr_vfs =3D pci_num_vf(pdev); @@ -174,6 +873,66 @@ static int __sriov_enable(struct pci_dev *pdev, int nu= m_vfs) =20 sdp->num_vfs =3D num_vfs; =20 + /* + * Map PF-VF mailbox memory. + */ + pf_vf_mbox_base =3D readq((void __iomem *)((u64)sdp->bar2 + + RVU_PF_VF_BAR4_ADDR)); + + if (!pf_vf_mbox_base) { + dev_err(&pdev->dev, "PF-VF Mailbox address not configured\n"); + err =3D -ENOMEM; + goto err_mbox_mem_map; + } + sdp->pfvf_mbx_base =3D ioremap_wc(pf_vf_mbox_base, MBOX_SIZE * num_vfs); + if (!sdp->pfvf_mbx_base) { + dev_err(&pdev->dev, + "Mapping of PF-VF mailbox address failed\n"); + err =3D -ENOMEM; + goto err_mbox_mem_map; + } + err =3D otx2_mbox_init(&sdp->pfvf_mbox, sdp->pfvf_mbx_base, pdev, + sdp->bar2, MBOX_DIR_PFVF, num_vfs); + if (err) { + dev_err(&pdev->dev, + "Failed to initialize PF/VF MBOX for %d VFs\n", + num_vfs); + goto err_mbox_init; + } + + err =3D otx2_mbox_init(&sdp->pfvf_mbox_up, sdp->pfvf_mbx_base, pdev, + sdp->bar2, MBOX_DIR_PFVF_UP, num_vfs); + if (err) { + dev_err(&pdev->dev, + "Failed to initialize PF/VF MBOX UP for %d VFs\n", + num_vfs); + goto err_mbox_up_init; + } + + /* Allocate a single workqueue for VF/PF mailbox because access to + * AF/PF mailbox has to be synchronized. + */ + sdp->pfvf_mbox_wq =3D + alloc_workqueue("sdp_pfvf_mailbox", + WQ_UNBOUND | WQ_HIGHPRI | WQ_MEM_RECLAIM, 1); + if (sdp->pfvf_mbox_wq =3D=3D NULL) { + dev_err(&pdev->dev, + "Workqueue allocation failed for PF-VF MBOX\n"); + err =3D -ENOMEM; + goto err_workqueue_alloc; + } + + for (vf =3D 0; vf < num_vfs; vf++) { + vf_ptr =3D &sdp->vf_info[vf]; + vf_ptr->vf_id =3D vf; + vf_ptr->sdp =3D (void *)sdp; + vf_ptr->intr_idx =3D vf % 64; + INIT_WORK(&vf_ptr->mbox_wrk, sdp_pfvf_mbox_handler); + INIT_WORK(&vf_ptr->mbox_wrk_up, sdp_pfvf_mbox_handler_up); + } + + enable_vf_mbox_int(pdev); + err =3D pci_enable_sriov(pdev, num_vfs); if (err) { dev_err(&pdev->dev, "Failed to enable to SRIOV VFs: %d\n", err); @@ -181,6 +940,23 @@ static int __sriov_enable(struct pci_dev *pdev, int nu= m_vfs) } =20 return num_vfs; + +err_enable_sriov: + disable_vf_mbox_int(pdev); +err_workqueue_alloc: + destroy_workqueue(sdp->pfvf_mbox_wq); + if (sdp->pfvf_mbox_up.dev !=3D NULL) + otx2_mbox_destroy(&sdp->pfvf_mbox_up); +err_mbox_up_init: + if (sdp->pfvf_mbox.dev !=3D NULL) + otx2_mbox_destroy(&sdp->pfvf_mbox); +err_mbox_init: + iounmap(sdp->pfvf_mbx_base); +err_mbox_mem_map: + kfree(sdp->vf_info); + + return err; + } =20 static int sdp_sriov_configure(struct pci_dev *pdev, int num_vfs) @@ -278,6 +1054,22 @@ static int sdp_probe(struct pci_dev *pdev, const stru= ct pci_device_id *id) goto alloc_irqs_failed; } =20 + if (sdp_register_mbox_irq(pdev) !=3D 0) { + dev_err(&pdev->dev, + "Unable to allocate MBOX Interrupt vectors\n"); + err =3D -ENODEV; + goto reg_mbox_irq_failed; + } + + enable_af_mbox_int(pdev); + + if (sdp_get_pcifunc(sdp)) { + dev_err(&pdev->dev, + "Failed to retrieve pcifunc from AF\n"); + err =3D -ENODEV; + goto get_pcifunc_failed; + } + regval =3D readq(sdp->sdp_base + SDPX_GBL_CONTROL); regval |=3D (1 << 2); /* BPFLR_D disable clearing BP in FLR */ writeq(regval, sdp->sdp_base + SDPX_GBL_CONTROL); @@ -290,6 +1082,11 @@ static int sdp_probe(struct pci_dev *pdev, const stru= ct pci_device_id *id) =20 return 0; =20 +get_pcifunc_failed: + disable_af_mbox_int(pdev); + sdp_afpf_mbox_term(pdev); +reg_mbox_irq_failed: + sdp_free_irqs(pdev); alloc_irqs_failed: iounmap(sdp->af_mbx_base); pf_unusable: @@ -316,6 +1113,8 @@ static void sdp_remove(struct pci_dev *pdev) if (sdp->num_vfs) __sriov_disable(pdev); =20 + disable_af_mbox_int(pdev); + sdp_afpf_mbox_term(pdev); sdp_free_irqs(pdev); =20 if (sdp->af_mbx_base) diff --git a/drivers/soc/marvell/octeontx2-sdp/sdp.h b/drivers/soc/marvell/= octeontx2-sdp/sdp.h index c8ef7a2ade45..6ee61aeb9750 100644 --- a/drivers/soc/marvell/octeontx2-sdp/sdp.h +++ b/drivers/soc/marvell/octeontx2-sdp/sdp.h @@ -15,9 +15,19 @@ #include #include "mbox.h" =20 +#define RVU_PFVF_PF_SHIFT 10 +#define RVU_PFVF_PF_MASK 0x3F +#define RVU_PFVF_FUNC_SHIFT 0 +#define RVU_PFVF_FUNC_MASK 0x3FF + +#define RVU_PFFUNC(pf, func) \ + ((((pf) & RVU_PFVF_PF_MASK) << RVU_PFVF_PF_SHIFT) | \ + (((func) & RVU_PFVF_FUNC_MASK) << RVU_PFVF_FUNC_SHIFT)) + #define SDP_BASE(a) (0x86E080000000ull | a << 36) #define SDP_REG_SIZE 0x42000000 =20 +#define SDPX_OUT_BP_ENX_W1S(a) (0x80280ull | a << 4) #define SDPX_GBL_CONTROL (0x40080200ull) =20 struct sdp_dev { @@ -27,6 +37,7 @@ struct sdp_dev { void __iomem *sdp_base; void __iomem *bar2; void __iomem *af_mbx_base; + void __iomem *pfvf_mbx_base; #define SDP_VF_ENABLED 0x1 u32 flags; u32 num_vfs; @@ -38,13 +49,22 @@ struct sdp_dev { int pf; u8 valid_ep_pem_mask; u8 mac_mask; - + struct otx2_mbox pfvf_mbox; /* MBOXes for VF =3D> PF channel */ + struct otx2_mbox pfvf_mbox_up; /* MBOXes for PF =3D> VF channel */ + struct otx2_mbox afpf_mbox; /* MBOX for PF =3D> AF channel */ + struct otx2_mbox afpf_mbox_up; /* MBOX for AF =3D> PF channel */ + struct work_struct mbox_wrk; + struct work_struct mbox_wrk_up; + struct workqueue_struct *afpf_mbox_wq; /* MBOX handler */ + struct workqueue_struct *pfvf_mbox_wq; /* VF MBOX handler */ struct sdp_node_info info; struct rvu_vf *vf_info; struct free_rsrcs_rsp limits; /* Maximum limits for all VFs */ }; =20 struct rvu_vf { + struct work_struct mbox_wrk; + struct work_struct mbox_wrk_up; struct work_struct pfvf_flr_work; struct device_attribute in_use_attr; struct pci_dev *pdev; --=20 2.24.1 From nobody Sun Jun 28 08:39:27 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A9CEC433EF for ; Wed, 9 Feb 2022 22:43:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236262AbiBIWm7 (ORCPT ); Wed, 9 Feb 2022 17:42:59 -0500 Received: from gmail-smtp-in.l.google.com ([23.128.96.19]:37986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236164AbiBIWmb (ORCPT ); Wed, 9 Feb 2022 17:42:31 -0500 Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4639CE02E45D for ; Wed, 9 Feb 2022 14:42:29 -0800 (PST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 219IO5Ej014382; Wed, 9 Feb 2022 14:42:24 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=u+eAZNiCKCIeT8xnn50IgpVJYz9Cz9Yacj4xxNzogLU=; b=IVGUx48H29DfpvQBLy+vBtxGkFGtnnpFtwjlgMhnFHFmm1nciby6KrujPHRyBuM6Ci19 dhUWnb8EalT2vQZH6PBtA3EsK6qi4k8ZfT4+EwNshfr74/902vXtqainWafXeuyScAwB yeMyIJvLGX7Sc0in8jEcm6BEd5rqrwGj1hSaqs9yfYabV4JdZcgUtvkK60Le563OtbeJ d43h+8VfQy3iL0c/zMy+0WzAmxf4PmoL1t6c2wT16pmZcLcrsPO5OEIhFJirrtoeeLZW 1y29P8U+2pc4MB8Y5dAZ+CKaT6eVyfdYkEHhN8Kj2qHm8GX1XF096Fi8XafNBHP2SEG7 7Q== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3e4am93bx1-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Wed, 09 Feb 2022 14:42:23 -0800 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 9 Feb 2022 14:42:19 -0800 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 9 Feb 2022 14:42:19 -0800 Received: from rchin-dellt430.marvell.com (rchin-dellt430.marvell.com [10.85.176.141]) by maili.marvell.com (Postfix) with ESMTP id 0F2093F70A4; Wed, 9 Feb 2022 14:42:19 -0800 (PST) From: Radha Mohan Chintakuntla To: , , CC: , , "Radha Mohan Chintakuntla" Subject: [PATCH 3/4] soc: octeontx2-sdp: Add FLR handling support Date: Wed, 9 Feb 2022 14:42:09 -0800 Message-ID: <20220209224210.153907-4-radhac@marvell.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20220209224210.153907-1-radhac@marvell.com> References: <20220209224210.153907-1-radhac@marvell.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Proofpoint-GUID: 6vw5brJFqggaoPcn1DwsGfxGvgfB3xH6 X-Proofpoint-ORIG-GUID: 6vw5brJFqggaoPcn1DwsGfxGvgfB3xH6 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-09_12,2022-02-09_01,2021-12-02_01 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This patch adds support to enable VF FLR interrupts and handling them. Signed-off-by: Radha Mohan Chintakuntla --- drivers/soc/marvell/octeontx2-sdp/sdp.c | 208 ++++++++++++++++++++++++ 1 file changed, 208 insertions(+) diff --git a/drivers/soc/marvell/octeontx2-sdp/sdp.c b/drivers/soc/marvell/= octeontx2-sdp/sdp.c index fef063b6ce1f..ce1e8c424bff 100644 --- a/drivers/soc/marvell/octeontx2-sdp/sdp.c +++ b/drivers/soc/marvell/octeontx2-sdp/sdp.c @@ -615,6 +615,202 @@ static int sdp_check_pf_usable(struct sdp_dev *sdp) return 0; } =20 +static void __handle_vf_flr(struct sdp_dev *sdp, struct rvu_vf *vf_ptr) +{ + if (vf_ptr->in_use) { + /* Using the same MBOX workqueue here, so that we can + * synchronize with other VF->PF messages being forwarded to + * AF + */ + vf_ptr->got_flr =3D true; + queue_work(sdp->pfvf_mbox_wq, &vf_ptr->pfvf_flr_work); + } else + sdp_write64(sdp, BLKADDR_RVUM, 0, + RVU_PF_VFTRPENDX(vf_ptr->vf_id / 64), + BIT_ULL(vf_ptr->intr_idx)); +} + +static irqreturn_t sdp_pf_vf_flr_intr(int irq, void *arg) +{ + struct sdp_dev *sdp =3D (struct sdp_dev *)arg; + struct rvu_vf *vf_ptr; + int vf, i; + u64 intr; + + /* Check which VF FLR has been raised and process accordingly */ + for (i =3D 0; i < 2; i++) { + /* Read the interrupt bits */ + intr =3D sdp_read64(sdp, BLKADDR_RVUM, 0, RVU_PF_VFFLR_INTX(i)); + + for (vf =3D i * 64; vf < sdp->num_vfs; vf++) { + vf_ptr =3D &sdp->vf_info[vf]; + if (intr & (1ULL << vf_ptr->intr_idx)) { + /* Clear the interrupts */ + sdp_write64(sdp, BLKADDR_RVUM, 0, + RVU_PF_VFFLR_INTX(i), + BIT_ULL(vf_ptr->intr_idx)); + __handle_vf_flr(sdp, vf_ptr); + } + } + } + + return IRQ_HANDLED; +} + +static int sdp_register_flr_irq(struct pci_dev *pdev) +{ + struct sdp_dev *sdp; + int err, vec, i; + + sdp =3D pci_get_drvdata(pdev); + + /* Register for VF FLR interrupts + * There are 2 vectors starting at index 0x0 + */ + for (vec =3D RVU_PF_INT_VEC_VFFLR0, i =3D 0; + vec + i <=3D RVU_PF_INT_VEC_VFFLR1; i++) { + sprintf(&sdp->irq_names[(vec + i) * NAME_SIZE], + "SDP_PF%02d_VF_FLR%d", pdev->bus->number, i); + err =3D request_irq(pci_irq_vector(pdev, vec + i), + sdp_pf_vf_flr_intr, 0, + &sdp->irq_names[(vec + i) * NAME_SIZE], sdp); + if (err) { + dev_err(&pdev->dev, + "request_irq() failed for PFVF FLR intr %d\n", + vec); + goto reg_fail; + } + sdp->irq_allocated[vec + i] =3D true; + } + + return 0; + +reg_fail: + + return err; +} + +static void enable_vf_flr_int(struct pci_dev *pdev) +{ + struct sdp_dev *sdp; + int ena_bits, idx; + + sdp =3D pci_get_drvdata(pdev); + + /* Clear any pending interrupts */ + for (idx =3D 0; idx < 2; idx++) { + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_VFTRPENDX(idx), + ~0x0ULL); + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_VFFLR_INTX(idx), + ~0x0ULL); + } + + /* Enable for FLR interrupts for VFs */ + if (sdp->num_vfs > 64) { + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_VFFLR_INT_ENA_W1SX(0), + GENMASK_ULL(63, 0)); + ena_bits =3D (sdp->num_vfs - 64) - 1; + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_VFFLR_INT_ENA_W1SX(1), + GENMASK_ULL(ena_bits, 0)); + } else { + ena_bits =3D sdp->num_vfs - 1; + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_VFFLR_INT_ENA_W1SX(0), + GENMASK_ULL(ena_bits, 0)); + } +} + +static int send_flr_msg(struct otx2_mbox *mbox, int dev_id, int pcifunc) +{ + struct msg_req *req; + + req =3D (struct msg_req *) + otx2_mbox_alloc_msg(mbox, dev_id, sizeof(*req)); + if (req =3D=3D NULL) + return -ENOMEM; + + req->hdr.pcifunc =3D pcifunc; + req->hdr.id =3D MBOX_MSG_VF_FLR; + req->hdr.sig =3D OTX2_MBOX_REQ_SIG; + + otx2_mbox_msg_send(mbox, 0); + + return 0; +} + +static void sdp_send_flr_msg(struct sdp_dev *sdp, struct rvu_vf *vf) +{ + int res, pcifunc; + + pcifunc =3D (vf->sdp->pf << RVU_PFVF_PF_SHIFT) | + ((vf->vf_id + 1) & RVU_PFVF_FUNC_MASK); + + if (send_flr_msg(&sdp->afpf_mbox, 0, pcifunc) !=3D 0) { + dev_err(&sdp->pdev->dev, "Sending FLR to AF failed\n"); + return; + } + + res =3D otx2_mbox_wait_for_rsp(&sdp->afpf_mbox, 0); + if (res =3D=3D -EIO) { + dev_err(&sdp->pdev->dev, "RVU AF MBOX timeout.\n"); + } else if (res) { + dev_err(&sdp->pdev->dev, + "RVU MBOX error: %d.\n", res); + } +} + +static void sdp_pfvf_flr_handler(struct work_struct *work) +{ + struct rvu_vf *vf =3D container_of(work, struct rvu_vf, pfvf_flr_work); + struct sdp_dev *sdp =3D vf->sdp; + struct otx2_mbox *mbox; + + mbox =3D &sdp->pfvf_mbox; + + sdp_send_flr_msg(sdp, vf); + + /* Disable interrupts from AF and wait for any pending + * responses to be handled for this VF and then reset the + * mailbox + */ + disable_af_mbox_int(sdp->pdev); + flush_workqueue(sdp->afpf_mbox_wq); + otx2_mbox_reset(mbox, vf->vf_id); + vf->in_use =3D false; + vf->got_flr =3D false; + enable_af_mbox_int(sdp->pdev); + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_VFTRPENDX(vf->vf_id / 64), + BIT_ULL(vf->intr_idx)); +} + +static void disable_vf_flr_int(struct pci_dev *pdev) +{ + struct sdp_dev *sdp; + int ena_bits, idx; + + sdp =3D pci_get_drvdata(pdev); + + /* Clear any pending interrupts */ + for (idx =3D 0; idx < 2; idx++) { + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_VFTRPENDX(idx), + ~0x0ULL); + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_VFFLR_INTX(idx), + ~0x0ULL); + } + + /* Disable the FLR interrupts for VFs */ + if (sdp->num_vfs > 64) { + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_VFFLR_INT_ENA_W1CX(0), + GENMASK_ULL(63, 0)); + ena_bits =3D (sdp->num_vfs - 64) - 1; + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_VFFLR_INT_ENA_W1CX(1), + GENMASK_ULL(ena_bits, 0)); + } else { + ena_bits =3D sdp->num_vfs - 1; + sdp_write64(sdp, BLKADDR_RVUM, 0, RVU_PF_VFFLR_INT_ENA_W1CX(0), + GENMASK_ULL(ena_bits, 0)); + } +} + static int sdp_alloc_irqs(struct pci_dev *pdev) { struct sdp_dev *sdp; @@ -817,6 +1013,7 @@ static int __sriov_disable(struct pci_dev *pdev) return -EPERM; } =20 + disable_vf_flr_int(pdev); disable_vf_mbox_int(pdev); =20 if (sdp->pfvf_mbox_wq) { @@ -929,9 +1126,11 @@ static int __sriov_enable(struct pci_dev *pdev, int n= um_vfs) vf_ptr->intr_idx =3D vf % 64; INIT_WORK(&vf_ptr->mbox_wrk, sdp_pfvf_mbox_handler); INIT_WORK(&vf_ptr->mbox_wrk_up, sdp_pfvf_mbox_handler_up); + INIT_WORK(&vf_ptr->pfvf_flr_work, sdp_pfvf_flr_handler); } =20 enable_vf_mbox_int(pdev); + enable_vf_flr_int(pdev); =20 err =3D pci_enable_sriov(pdev, num_vfs); if (err) { @@ -942,6 +1141,7 @@ static int __sriov_enable(struct pci_dev *pdev, int nu= m_vfs) return num_vfs; =20 err_enable_sriov: + disable_vf_flr_int(pdev); disable_vf_mbox_int(pdev); err_workqueue_alloc: destroy_workqueue(sdp->pfvf_mbox_wq); @@ -1061,6 +1261,13 @@ static int sdp_probe(struct pci_dev *pdev, const str= uct pci_device_id *id) goto reg_mbox_irq_failed; } =20 + if (sdp_register_flr_irq(pdev) !=3D 0) { + dev_err(&pdev->dev, + "Unable to allocate FLR Interrupt vectors\n"); + err =3D -ENODEV; + goto reg_flr_irq_failed; + } + enable_af_mbox_int(pdev); =20 if (sdp_get_pcifunc(sdp)) { @@ -1084,6 +1291,7 @@ static int sdp_probe(struct pci_dev *pdev, const stru= ct pci_device_id *id) =20 get_pcifunc_failed: disable_af_mbox_int(pdev); +reg_flr_irq_failed: sdp_afpf_mbox_term(pdev); reg_mbox_irq_failed: sdp_free_irqs(pdev); --=20 2.24.1 From nobody Sun Jun 28 08:39:27 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9972DC433F5 for ; Wed, 9 Feb 2022 22:42:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236196AbiBIWmk (ORCPT ); Wed, 9 Feb 2022 17:42:40 -0500 Received: from gmail-smtp-in.l.google.com ([23.128.96.19]:37988 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234420AbiBIWma (ORCPT ); Wed, 9 Feb 2022 17:42:30 -0500 Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6E45C0302D7 for ; Wed, 9 Feb 2022 14:42:30 -0800 (PST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 219IO5Ek014382; Wed, 9 Feb 2022 14:42:24 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=IRHFx3h5FBrQu2UY7xrQzm1gPaslNEwFcYm95KOm78w=; b=C18tnaYoWRwaK/jc+TNzIVx+iOaRNZaxWuRo9EkONzV83jKb0SYaEOvQsrVOTwQiLtp6 7H7cpS+WGjdRSkKIBKMuj1cZAjUscvgXuDLnATeidbEs0I1XfyXoquOxKPBnFt+0mTWC Ic+CCd011Kjs1eiky4bUl00hTi5s4m0T/hWeeCuOibswkxZNycLHBHVVQ/fTTD2Ly+vD FFhNANJdvS81etEPlKQmEMdK0W2tZe2PgolCVQrmayLbqgjV+dWD0HdHFrHqPSK4teI1 TnWjj2GTTuxU3t+DR9Ah0KZZFWXaPBJ0giohJ+KcfNbh6PBqWx/AElJsKo1Vz38E+L/t dA== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3e4am93bx1-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Wed, 09 Feb 2022 14:42:24 -0800 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 9 Feb 2022 14:42:19 -0800 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Wed, 9 Feb 2022 14:42:19 -0800 Received: from rchin-dellt430.marvell.com (rchin-dellt430.marvell.com [10.85.176.141]) by maili.marvell.com (Postfix) with ESMTP id 4A9A93F70A7; Wed, 9 Feb 2022 14:42:19 -0800 (PST) From: Radha Mohan Chintakuntla To: , , CC: , , "Radha Mohan Chintakuntla" Subject: [PATCH 4/4] soc: octeontx2-sdp: setup the SDP channel configuration Date: Wed, 9 Feb 2022 14:42:10 -0800 Message-ID: <20220209224210.153907-5-radhac@marvell.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20220209224210.153907-1-radhac@marvell.com> References: <20220209224210.153907-1-radhac@marvell.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Proofpoint-GUID: _UsGQAe4JH15u4wlqlluNdrLXTYCRlmP X-Proofpoint-ORIG-GUID: _UsGQAe4JH15u4wlqlluNdrLXTYCRlmP X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-09_12,2022-02-09_01,2021-12-02_01 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This patch sets up the SDP channels to NIX block for packet transmit and receive. Signed-off-by: Radha Mohan Chintakuntla --- drivers/soc/marvell/octeontx2-sdp/sdp.c | 186 ++++++++++++++++++++++++ 1 file changed, 186 insertions(+) diff --git a/drivers/soc/marvell/octeontx2-sdp/sdp.c b/drivers/soc/marvell/= octeontx2-sdp/sdp.c index ce1e8c424bff..a5f707bf17cf 100644 --- a/drivers/soc/marvell/octeontx2-sdp/sdp.c +++ b/drivers/soc/marvell/octeontx2-sdp/sdp.c @@ -1167,6 +1167,152 @@ static int sdp_sriov_configure(struct pci_dev *pdev= , int num_vfs) return __sriov_enable(pdev, num_vfs); } =20 +static int sdp_parse_rinfo(struct pci_dev *pdev, + struct sdp_node_info *info) +{ + u32 vf_ring_cnts, vf_rings; + struct device_node *dev; + struct device *sdev; + const void *ptr; + int len, vfid; + + sdev =3D &pdev->dev; + dev =3D of_find_node_by_name(NULL, "rvu-sdp"); + if (dev =3D=3D NULL) { + dev_err(sdev, "can't find FDT dev %s\n", "rvu-sdp"); + return -EINVAL; + } + + ptr =3D of_get_property(dev, "num-rvu-vfs", &len); + if (ptr =3D=3D NULL) { + dev_err(sdev, "SDP DTS: Failed to get num-rvu-vfs\n"); + return -EINVAL; + } + + if (len !=3D sizeof(u32)) { + dev_err(sdev, "SDP DTS: Wrong field length: num-rvu-vfs\n"); + return -EINVAL; + } + info->max_vfs =3D be32_to_cpup((u32 *)ptr); + + if (info->max_vfs > pci_sriov_get_totalvfs(pdev)) { + dev_err(sdev, "SDP DTS: Invalid field value: num-rvu-vfs\n"); + return -EINVAL; + } + + ptr =3D of_get_property(dev, "num-pf-rings", &len); + if (ptr =3D=3D NULL) { + dev_err(sdev, "SDP DTS: Failed to get num-pf-rings\n"); + return -EINVAL; + } + if (len !=3D sizeof(u32)) { + dev_err(sdev, "SDP DTS: Wrong field length: num-pf-rings\n"); + return -EINVAL; + } + info->num_pf_rings =3D be32_to_cpup((u32 *)ptr); + + ptr =3D of_get_property(dev, "pf-srn", &len); + if (ptr =3D=3D NULL) { + dev_err(sdev, "SDP DTS: Failed to get pf-srn\n"); + return -EINVAL; + } + if (len !=3D sizeof(u32)) { + dev_err(sdev, "SDP DTS: Wrong field length: pf-srn\n"); + return -EINVAL; + } + info->pf_srn =3D be32_to_cpup((u32 *)ptr); + + ptr =3D of_get_property(dev, "num-vf-rings", &len); + if (ptr =3D=3D NULL) { + dev_err(sdev, "SDP DTS: Failed to get num-vf-rings\n"); + return -EINVAL; + } + + vf_ring_cnts =3D len / sizeof(u32); + if (vf_ring_cnts > info->max_vfs) { + dev_err(sdev, "SDP DTS: Wrong field length: num-vf-rings\n"); + return -EINVAL; + } + + for (vfid =3D 0; vfid < info->max_vfs; vfid++) { + if (vfid < vf_ring_cnts) { + if (of_property_read_u32_index(dev, "num-vf-rings", + vfid, &vf_rings)) { + dev_err(sdev, "SDP DTS: Failed to get vf ring count\n"); + return -EINVAL; + } + info->vf_rings[vfid] =3D vf_rings; + } else { + /* + * Rest of the VFs use the same last ring count + * specified + */ + info->vf_rings[vfid] =3D info->vf_rings[vf_ring_cnts - 1]; + } + } + dev_info(sdev, "pf start ring number:%d num_pf_rings:%d max_vfs:%d vf_rin= g_cnts:%d\n", + info->pf_srn, info->num_pf_rings, info->max_vfs, vf_ring_cnts); + + return 0; +} + +static int get_chan_info(struct sdp_dev *sdp) +{ + struct sdp_get_chan_info_msg *rsp; + struct msg_req *req; + int res =3D 0; + + req =3D (struct msg_req *) otx2_mbox_alloc_msg(&sdp->afpf_mbox, 0, sizeof= (*req)); + if (req =3D=3D NULL) { + dev_err(&sdp->pdev->dev, "RVU Mbox failed to alloc\n"); + return -EFAULT; + } + req->hdr.id =3D MBOX_MSG_GET_SDP_CHAN_INFO; + req->hdr.sig =3D OTX2_MBOX_REQ_SIG; + req->hdr.pcifunc =3D RVU_PFFUNC(sdp->pf, 0); + + otx2_mbox_msg_send(&sdp->afpf_mbox, 0); + res =3D otx2_mbox_wait_for_rsp(&sdp->afpf_mbox, 0); + if (res =3D=3D -EIO) + dev_err(&sdp->pdev->dev, "RVU AF Mbox timeout\n"); + else if (res) { + dev_err(&sdp->pdev->dev, "RVU Mbox error: %d\n", res); + res =3D -EFAULT; + } + rsp =3D (struct sdp_get_chan_info_msg *)otx2_mbox_get_rsp(&sdp->afpf_mbox= , 0, + &req->hdr); + sdp->chan_base =3D rsp->chan_base; + sdp->num_chan =3D rsp->num_chan; + + return res; +} +static int send_chan_info(struct sdp_dev *sdp, struct sdp_node_info *info) +{ + struct sdp_chan_info_msg *cinfo; + int res =3D 0; + + cinfo =3D (struct sdp_chan_info_msg *) + otx2_mbox_alloc_msg(&sdp->afpf_mbox, 0, sizeof(*cinfo)); + if (cinfo =3D=3D NULL) { + dev_err(&sdp->pdev->dev, "RVU MBOX failed to get message.\n"); + return -EFAULT; + } + cinfo->hdr.id =3D MBOX_MSG_SET_SDP_CHAN_INFO; + cinfo->hdr.sig =3D OTX2_MBOX_REQ_SIG; + cinfo->hdr.pcifunc =3D RVU_PFFUNC(sdp->pf, 0); + + memcpy(&cinfo->info, info, sizeof(struct sdp_node_info)); + otx2_mbox_msg_send(&sdp->afpf_mbox, 0); + res =3D otx2_mbox_wait_for_rsp(&sdp->afpf_mbox, 0); + if (res =3D=3D -EIO) { + dev_err(&sdp->pdev->dev, "RVU AF MBOX timeout.\n"); + } else if (res) { + dev_err(&sdp->pdev->dev, "RVU MBOX error: %d.\n", res); + res =3D -EFAULT; + } + + return res; +} =20 static int sdp_probe(struct pci_dev *pdev, const struct pci_device_id *id) { @@ -1277,6 +1423,44 @@ static int sdp_probe(struct pci_dev *pdev, const str= uct pci_device_id *id) goto get_pcifunc_failed; } =20 + err =3D get_chan_info(sdp); + if (err) { + dev_err(&pdev->dev, "SDP get channel info failed\n"); + goto get_chan_info_failed; + } + + dev_info(&sdp->pdev->dev, "SDP chan base: 0x%x, num chan: 0x%x\n", + sdp->chan_base, sdp->num_chan); + + err =3D sdp_parse_rinfo(pdev, &sdp->info); + if (err) { + err =3D -EINVAL; + goto get_rinfo_failed; + } + + /* To differentiate a PF between SDP0 or SDP1 we make use of the + * revision ID field in the config space. The revision is filled + * by the firmware. The lower 4 bits field is used here. + * 0 means SDP0 + * 1 means SDP1 + */ + if (pdev->revision & 0x0F) + sdp->info.node_id =3D 1; + else + sdp->info.node_id =3D 0; + + /* + * For 98xx there are 2xSDPs so start the PF ring from 128 for SDP1 + * SDP0 has PCI revid 0 and SDP1 has PCI revid 1 + */ + sdp->info.pf_srn =3D (pdev->revision & 0x0F) ? 128 : sdp->info.pf_srn; + + err =3D send_chan_info(sdp, &sdp->info); + if (err) { + err =3D -EINVAL; + goto get_rinfo_failed; + } + regval =3D readq(sdp->sdp_base + SDPX_GBL_CONTROL); regval |=3D (1 << 2); /* BPFLR_D disable clearing BP in FLR */ writeq(regval, sdp->sdp_base + SDPX_GBL_CONTROL); @@ -1289,6 +1473,8 @@ static int sdp_probe(struct pci_dev *pdev, const stru= ct pci_device_id *id) =20 return 0; =20 +get_chan_info_failed: +get_rinfo_failed: get_pcifunc_failed: disable_af_mbox_int(pdev); reg_flr_irq_failed: --=20 2.24.1