From nobody Sun Feb 8 20:13:33 2026 Received: from fllvem-ot03.ext.ti.com (fllvem-ot03.ext.ti.com [198.47.19.245]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F06881D514E; Mon, 28 Apr 2025 07:21:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.47.19.245 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745824876; cv=none; b=eMiaLcMPhR3HWLIAMIfFR/6+G8jj9eOsUD7CaSPcmp45wKe+t9Tu1rwirRABCVn60Te6d3Vn4sm0nDnoL6j1ltVDvtsz29TV6suS0KaF2gpkY1WX1dwZhEyxsVG2ddp0wBpHEj+Kln+LDtBe31F1mxfxzut3gbvFD2TbN25DZRM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745824876; c=relaxed/simple; bh=cVTdVvpTF2b7AcMsFe+HcHyjAgx7BqnnsT6fycHw0/g=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=pvJOJq1RjNQgbhPWJMnIFAMb9oypAiDRZGrk4h+a6mOVrPqxVI6D5xC/H1p/kyimkhYFllGsyRZN3X3XVrCpgHgdJ41iuUbnGDD/qv2kMnCDtk11qnJdbSxE6UWgCxXhHEfR+EM5nPLIIml5Wchid+P6kFSTF2EP1qjzq0PmFXk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ti.com; spf=pass smtp.mailfrom=ti.com; dkim=pass (1024-bit key) header.d=ti.com header.i=@ti.com header.b=wLXUeqfK; arc=none smtp.client-ip=198.47.19.245 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ti.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ti.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ti.com header.i=@ti.com header.b="wLXUeqfK" Received: from fllv0034.itg.ti.com ([10.64.40.246]) by fllvem-ot03.ext.ti.com (8.15.2/8.15.2) with ESMTPS id 53S7KuMk2728379 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 28 Apr 2025 02:20:56 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1745824856; bh=hju8e8bkKiVvaXrgWmT1aJHhrIUpIH//AB39ScrzTHE=; h=From:To:Subject:Date:In-Reply-To:References; b=wLXUeqfKXwyXJ6Y/Bi35hKwKJj8bUsQ43jK3o0lVEIOPtkyJFKEqgA1Y6q/otZC5a YtCc8K4dbMxI0B0xStqz8y3kbINjDkLZ2NcdHnKRgBuLr7OBTzJ+2VGdFSbA50py7i lwkp8Us4BcEISeUleWseJc/MtRZhlai/mkLU290s= Received: from DFLE111.ent.ti.com (dfle111.ent.ti.com [10.64.6.32]) by fllv0034.itg.ti.com (8.15.2/8.15.2) with ESMTPS id 53S7KusE073324 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 28 Apr 2025 02:20:56 -0500 Received: from DFLE113.ent.ti.com (10.64.6.34) by DFLE111.ent.ti.com (10.64.6.32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.23; Mon, 28 Apr 2025 02:20:55 -0500 Received: from lelvsmtp6.itg.ti.com (10.180.75.249) by DFLE113.ent.ti.com (10.64.6.34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.23 via Frontend Transport; Mon, 28 Apr 2025 02:20:55 -0500 Received: from uda0498651.dhcp.ti.com (uda0498651.dhcp.ti.com [172.24.227.7]) by lelvsmtp6.itg.ti.com (8.15.2/8.15.2) with ESMTP id 53S7KdMa068873; Mon, 28 Apr 2025 02:20:51 -0500 From: Sai Sree Kartheek Adivi To: , , , , , , , , , , , , , , , Subject: [PATCH 1/8] dt-bindings: dma: ti: Add document for K3 BCDMA V2 Date: Mon, 28 Apr 2025 12:50:25 +0530 Message-ID: <20250428072032.946008-2-s-adivi@ti.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250428072032.946008-1-s-adivi@ti.com> References: <20250428072032.946008-1-s-adivi@ti.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-C2ProcessedOrg: 333ef613-75bf-4e12-a4b1-8e3623f5dcea Content-Type: text/plain; charset="utf-8" New binding document for Texas Instruments K3 Block Copy DMA (BCDMA) V2. BCDMA V2 is introduced as part of AM62L. Signed-off-by: Sai Sree Kartheek Adivi --- .../bindings/dma/ti/k3-bcdma-v2.yaml | 97 +++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 Documentation/devicetree/bindings/dma/ti/k3-bcdma-v2.ya= ml diff --git a/Documentation/devicetree/bindings/dma/ti/k3-bcdma-v2.yaml b/Do= cumentation/devicetree/bindings/dma/ti/k3-bcdma-v2.yaml new file mode 100644 index 0000000000000..af4aa3839fd66 --- /dev/null +++ b/Documentation/devicetree/bindings/dma/ti/k3-bcdma-v2.yaml @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +# Copyright (C) 2024-2025 Texas Instruments Incorporated +# Author: Sai Sree Kartheek Adivi +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/dma/ti/k3-bcdma-v2.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Texas Instruments K3 DMSS BCDMA V2 + +maintainers: + - Sai Sree Kartheek Adivi + +description: | + The BCDMA V2 is intended to perform similar functions as the TR + mode channels of K3 UDMA-P. + BCDMA V2 includes block copy channels and Split channels. + + Block copy channels mainly used for memory to memory transfers, but with + optional triggers a block copy channel can service peripherals by access= ing + directly to memory mapped registers or area. + + Split channels can be used to service PSI-L based peripherals. + The peripherals can be PSI-L native or legacy, non PSI-L native peripher= als + with PDMAs. PDMA is tasked to act as a bridge between the PSI-L fabric a= nd the + legacy peripheral. + +allOf: + - $ref: /schemas/dma/dma-controller.yaml# + +properties: + compatible: + const: ti,dmss-bcdma-v2 + + reg: + items: + - description: BCDMA Control /Status Registers region + - description: Block Copy Channel Realtime Registers region + - description: Channel Realtime Registers region + - description: Ring Realtime Registers region + + reg-names: + items: + - const: gcfg + - const: bchanrt + - const: chanrt + - const: ringrt + + "#dma-cells": + const: 4 + description: | + cell 1: Trigger type for the channel + 0 - disable / no trigger + 1 - internal channel event + 2 - external signal + 3 - timer manager event + + cell 2: parameter for the trigger: + if cell 1 is 0 (disable / no trigger): + Unused, ignored + if cell 1 is 1 (internal channel event): + channel number whose TR event should trigger the current channel. + if cell 1 is 2 or 3 (external signal or timer manager event): + index of global interfaces that come into the DMA. + + Please refer to the device documentation for global interface in= dexes. + + cell 3: Channel number for the peripheral + + Please refer to the device documentation for the channel map. + + cell 4: ASEL value for the channel + +required: + - compatible + - "#dma-cells" + - reg + - reg-names + +unevaluatedProperties: false + +examples: + - |+ + cbass_main { + #address-cells =3D <2>; + #size-cells =3D <2>; + + main_bcdma: dma-controller@485c4000 { + compatible =3D "ti,dmss-bcdma-v2"; + reg =3D <0x00 0x485c4000 0x00 0x4000>, + <0x00 0x48880000 0x00 0x10000>, + <0x00 0x48800000 0x00 0x80000>, + <0x00 0x47000000 0x00 0x200000>; + reg-names =3D "gcfg", "bchanrt", "chanrt", "ringrt"; + #dma-cells =3D <4>; + }; + }; --=20 2.34.1 From nobody Sun Feb 8 20:13:33 2026 Received: from lelvem-ot01.ext.ti.com (lelvem-ot01.ext.ti.com [198.47.23.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ED9D71DE892; Mon, 28 Apr 2025 07:21:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.47.23.234 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745824891; cv=none; b=gpMvvGLyY4k1cRKN3EFRjRKY0Yj+1BF7lIwcj80CCseBTuU4yIFOgRKt3gAwFAIEPQBIN31wkuWx3ZskBj3hFtdU7tbJDLQ5inKIYqMCFYyWumCBXT/NEVgSI5HzEnsKWKh9ClSGG+ftTlbdp+r1YvX7L7V7Uw7oNtBvCJuYTRg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745824891; c=relaxed/simple; bh=e41CFdgQUEAxPvzmXET0SVjM34QoLgMk/Ehiricj41o=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ufds7rpNT9zOrN0io5LjRr0kAYpfFVwjbtEphdiAVQ7h4sblX37q3HDCzGdmj0wbCjA8lZXSmj2OFDim5IHEMi+Xgz8JH9ZZjtK/oNQIz9HtV89ayIEgtpd2DDJVMWSk1t4dZYHkMx8Us4PqgEc5sqZO6Dkom4tshKUmD9LGW3s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ti.com; spf=pass smtp.mailfrom=ti.com; dkim=pass (1024-bit key) header.d=ti.com header.i=@ti.com header.b=J6Fk5y0X; arc=none smtp.client-ip=198.47.23.234 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ti.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ti.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ti.com header.i=@ti.com header.b="J6Fk5y0X" Received: from lelv0266.itg.ti.com ([10.180.67.225]) by lelvem-ot01.ext.ti.com (8.15.2/8.15.2) with ESMTPS id 53S7L3pH2713970 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 28 Apr 2025 02:21:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1745824863; bh=rWm00ecZW//noVOBD0ZolXeFNkMKumFFuTm6AnT3FXY=; h=From:To:Subject:Date:In-Reply-To:References; b=J6Fk5y0XLegsHh1dF41d7vojDkM91lABDPgLF91MmFeeM3z+OUxejL7sfBDVSXYp4 i7AS7nEzJZ++wY6GpWlaofBhm3KsZ1C4zzC0FmcdRCx6oiRwCJ6pKUkc3Iup/dL2xu jEb3LDnOxv58lnmw4MAX5ZG7JHlgBwK85H9KorcI= Received: from DFLE115.ent.ti.com (dfle115.ent.ti.com [10.64.6.36]) by lelv0266.itg.ti.com (8.15.2/8.15.2) with ESMTPS id 53S7L3dh121356 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 28 Apr 2025 02:21:03 -0500 Received: from DFLE107.ent.ti.com (10.64.6.28) by DFLE115.ent.ti.com (10.64.6.36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.23; Mon, 28 Apr 2025 02:21:02 -0500 Received: from lelvsmtp6.itg.ti.com (10.180.75.249) by DFLE107.ent.ti.com (10.64.6.28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.23 via Frontend Transport; Mon, 28 Apr 2025 02:21:02 -0500 Received: from uda0498651.dhcp.ti.com (uda0498651.dhcp.ti.com [172.24.227.7]) by lelvsmtp6.itg.ti.com (8.15.2/8.15.2) with ESMTP id 53S7KdMb068873; Mon, 28 Apr 2025 02:20:58 -0500 From: Sai Sree Kartheek Adivi To: , , , , , , , , , , , , , , , Subject: [PATCH 2/8] dt-bindings: dma: ti: Add document for K3 PKTDMA V2 Date: Mon, 28 Apr 2025 12:50:26 +0530 Message-ID: <20250428072032.946008-3-s-adivi@ti.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250428072032.946008-1-s-adivi@ti.com> References: <20250428072032.946008-1-s-adivi@ti.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-C2ProcessedOrg: 333ef613-75bf-4e12-a4b1-8e3623f5dcea Content-Type: text/plain; charset="utf-8" New binding document for Texas Instruments K3 Packet DMA (PKTDMA) V2. PKTDMA V2 is introduced as part of AM62L. Signed-off-by: Sai Sree Kartheek Adivi --- .../bindings/dma/ti/k3-pktdma-v2.yaml | 73 +++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 Documentation/devicetree/bindings/dma/ti/k3-pktdma-v2.y= aml diff --git a/Documentation/devicetree/bindings/dma/ti/k3-pktdma-v2.yaml b/D= ocumentation/devicetree/bindings/dma/ti/k3-pktdma-v2.yaml new file mode 100644 index 0000000000000..a6aae96af44df --- /dev/null +++ b/Documentation/devicetree/bindings/dma/ti/k3-pktdma-v2.yaml @@ -0,0 +1,73 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +# Copyright (C) 2024-2025 Texas Instruments Incorporated +# Author: Sai Sree Kartheek Adivi +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/dma/ti/k3-pktdma-v2.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Texas Instruments K3 DMSS PKTDMA V2 + +maintainers: + - Sai Sree Kartheek Adivi + +description: | + The PKTDMA V2 is intended to perform similar functions as the packet + mode channels of K3 UDMA-P. + PKTDMA V2 only includes Split channels to service PSI-L based peripheral= s. + + The peripherals can be PSI-L native or legacy, non PSI-L native peripher= als + with PDMAs. PDMA is tasked to act as a bridge between the PSI-L fabric a= nd the + legacy peripheral. + +allOf: + - $ref: /schemas/dma/dma-controller.yaml# + +properties: + compatible: + const: ti,dmss-pktdma-v2 + + "#dma-cells": + const: 2 + description: | + cell 1: Channel number for the peripheral + + Please refer to the device documentation for the channel map. + + cell 2: ASEL value for the channel + + reg: + items: + - description: Packet DMA Control /Status Registers region + - description: Channel Realtime Registers region + - description: Ring Realtime Registers region + + reg-names: + items: + - const: gcfg + - const: chanrt + - const: ringrt + +required: + - compatible + - "#dma-cells" + - reg + - reg-names + +unevaluatedProperties: false + +examples: + - |+ + cbass_main { + #address-cells =3D <2>; + #size-cells =3D <2>; + + main_pktdma: dma-controller@485c0000 { + compatible =3D "ti,dmss-pktdma-v2"; + reg =3D <0x00 0x485c0000 0x00 0x4000>, + <0x00 0x48900000 0x00 0x80000>, + <0x00 0x47200000 0x00 0x100000>; + reg-names =3D "gcfg", "chanrt", "ringrt"; + #dma-cells =3D <2>; + }; + }; --=20 2.34.1 From nobody Sun Feb 8 20:13:33 2026 Received: from lelvem-ot02.ext.ti.com (lelvem-ot02.ext.ti.com [198.47.23.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D071A1E22E9; Mon, 28 Apr 2025 07:21:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.47.23.235 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745824905; cv=none; b=q23od3shusqGZ65bNLyNBaPojVqjtHCMqH0hBrMIkAagET8I+YL+fNj7CWvN6/SBRyKrdKVVAcl2SVXPnXsRyQ1+Ff+kvaDX+tU/5GIQCrM/tTrhVmxpk6b4yBmzTLRIFmOZ33etM7EYA40wrH07+ZsPxUsZcMNibT26VdoE1pE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745824905; c=relaxed/simple; bh=OTRaCrvn5e8Ti7zZU+AtEhdsf5buFsE7ntHNlwLUETk=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=hdiknKcIePelXWkA/AFd5ZBZnZakBpWnArHLmR4kAZ7LzimiaDPGcEMLpodR6HlXPj1u+136IS9fOWj56h5U/pMge8vLu0x0zCvh7Z0Yx/gsko7x36PhoexEBNF/brvxjTbKWOVPVEsKoKwdFhMuUPLRqICMwkUZPRA8tgpi/dQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ti.com; spf=pass smtp.mailfrom=ti.com; dkim=pass (1024-bit key) header.d=ti.com header.i=@ti.com header.b=BiT7WLtO; arc=none smtp.client-ip=198.47.23.235 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ti.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ti.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ti.com header.i=@ti.com header.b="BiT7WLtO" Received: from fllv0035.itg.ti.com ([10.64.41.0]) by lelvem-ot02.ext.ti.com (8.15.2/8.15.2) with ESMTPS id 53S7L91e3475453 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 28 Apr 2025 02:21:09 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1745824869; bh=XEv0w4iLfO/eJDIEgWDO8fg3o79UUvjiniP1ddEX/Fw=; h=From:To:Subject:Date:In-Reply-To:References; b=BiT7WLtO6zVnWpgNQsrI0gTNfc6VF8t7i6BjxTzhPJH5v78SAGLDwaCSMn2aW2R92 HCv0ezteWG2GVAmfe6k/6M5zG/GFPTLA8x1ioCz7CGypCthRpFHR60Z97lnAPUH5u2 XZ3JCGfQmc4jGW54ikhdpTmBZvKz5Wh1BuRA1euI= Received: from DFLE111.ent.ti.com (dfle111.ent.ti.com [10.64.6.32]) by fllv0035.itg.ti.com (8.15.2/8.15.2) with ESMTPS id 53S7L9Gi016383 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 28 Apr 2025 02:21:09 -0500 Received: from DFLE113.ent.ti.com (10.64.6.34) by DFLE111.ent.ti.com (10.64.6.32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.23; Mon, 28 Apr 2025 02:21:08 -0500 Received: from lelvsmtp6.itg.ti.com (10.180.75.249) by DFLE113.ent.ti.com (10.64.6.34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.23 via Frontend Transport; Mon, 28 Apr 2025 02:21:07 -0500 Received: from uda0498651.dhcp.ti.com (uda0498651.dhcp.ti.com [172.24.227.7]) by lelvsmtp6.itg.ti.com (8.15.2/8.15.2) with ESMTP id 53S7KdMc068873; Mon, 28 Apr 2025 02:21:04 -0500 From: Sai Sree Kartheek Adivi To: , , , , , , , , , , , , , , , Subject: [PATCH 3/8] drivers: dma: ti: Refactor TI K3 UDMA driver Date: Mon, 28 Apr 2025 12:50:27 +0530 Message-ID: <20250428072032.946008-4-s-adivi@ti.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250428072032.946008-1-s-adivi@ti.com> References: <20250428072032.946008-1-s-adivi@ti.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-C2ProcessedOrg: 333ef613-75bf-4e12-a4b1-8e3623f5dcea Content-Type: text/plain; charset="utf-8" Refactors and split the driver into common and device specific parts. There are no functional changes. Signed-off-by: Sai Sree Kartheek Adivi --- drivers/dma/ti/Makefile | 2 +- drivers/dma/ti/k3-udma-common.c | 2909 ++++++++++++++++++++++++ drivers/dma/ti/k3-udma.c | 3751 ++----------------------------- drivers/dma/ti/k3-udma.h | 548 ++++- 4 files changed, 3700 insertions(+), 3510 deletions(-) create mode 100644 drivers/dma/ti/k3-udma-common.c diff --git a/drivers/dma/ti/Makefile b/drivers/dma/ti/Makefile index d376c117cecf6..257e8141d7fe0 100644 --- a/drivers/dma/ti/Makefile +++ b/drivers/dma/ti/Makefile @@ -2,7 +2,7 @@ obj-$(CONFIG_TI_CPPI41) +=3D cppi41.o obj-$(CONFIG_TI_EDMA) +=3D edma.o obj-$(CONFIG_DMA_OMAP) +=3D omap-dma.o -obj-$(CONFIG_TI_K3_UDMA) +=3D k3-udma.o +obj-$(CONFIG_TI_K3_UDMA) +=3D k3-udma.o k3-udma-common.o obj-$(CONFIG_TI_K3_UDMA_GLUE_LAYER) +=3D k3-udma-glue.o k3-psil-lib-objs :=3D k3-psil.o \ k3-psil-am654.o \ diff --git a/drivers/dma/ti/k3-udma-common.c b/drivers/dma/ti/k3-udma-commo= n.c new file mode 100644 index 0000000000000..078b018b22830 --- /dev/null +++ b/drivers/dma/ti/k3-udma-common.c @@ -0,0 +1,2909 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2025 Texas Instruments Incorporated - http://www.ti.com + * Author: Peter Ujfalusi + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../virt-dma.h" +#include "k3-udma.h" +#include "k3-psil-priv.h" + +static const char * const range_names[] =3D { + [RM_RANGE_BCHAN] =3D "ti,sci-rm-range-bchan", + [RM_RANGE_TCHAN] =3D "ti,sci-rm-range-tchan", + [RM_RANGE_RCHAN] =3D "ti,sci-rm-range-rchan", + [RM_RANGE_RFLOW] =3D "ti,sci-rm-range-rflow", + [RM_RANGE_TFLOW] =3D "ti,sci-rm-range-tflow", +}; + +void k3_configure_chan_coherency(struct dma_chan *chan, u32 asel) +{ + struct device *chan_dev =3D &chan->dev->device; + + if (asel =3D=3D 0) { + /* No special handling for the channel */ + chan->dev->chan_dma_dev =3D false; + + chan_dev->dma_coherent =3D false; + chan_dev->dma_parms =3D NULL; + } else if (asel =3D=3D 14 || asel =3D=3D 15) { + chan->dev->chan_dma_dev =3D true; + + chan_dev->dma_coherent =3D true; + dma_coerce_mask_and_coherent(chan_dev, DMA_BIT_MASK(48)); + chan_dev->dma_parms =3D chan_dev->parent->dma_parms; + } else { + dev_warn(chan->device->dev, "Invalid ASEL value: %u\n", asel); + + chan_dev->dma_coherent =3D false; + chan_dev->dma_parms =3D NULL; + } +} + +u8 udma_get_chan_tpl_index(struct udma_tpl *tpl_map, int chan_id) +{ + int i; + + for (i =3D 0; i < tpl_map->levels; i++) { + if (chan_id >=3D tpl_map->start_idx[i]) + return i; + } + + return 0; +} + +void udma_reset_uchan(struct udma_chan *uc) +{ + memset(&uc->config, 0, sizeof(uc->config)); + uc->config.remote_thread_id =3D -1; + uc->config.mapped_channel_id =3D -1; + uc->config.default_flow_id =3D -1; + uc->state =3D UDMA_CHAN_IS_IDLE; +} + +void udma_dump_chan_stdata(struct udma_chan *uc) +{ + struct device *dev =3D uc->ud->dev; + u32 offset; + int i; + + if (uc->config.dir =3D=3D DMA_MEM_TO_DEV || uc->config.dir =3D=3D DMA_MEM= _TO_MEM) { + dev_dbg(dev, "TCHAN State data:\n"); + for (i =3D 0; i < 32; i++) { + offset =3D UDMA_CHAN_RT_STDATA_REG + i * 4; + dev_dbg(dev, "TRT_STDATA[%02d]: 0x%08x\n", i, + udma_tchanrt_read(uc, offset)); + } + } + + if (uc->config.dir =3D=3D DMA_DEV_TO_MEM || uc->config.dir =3D=3D DMA_MEM= _TO_MEM) { + dev_dbg(dev, "RCHAN State data:\n"); + for (i =3D 0; i < 32; i++) { + offset =3D UDMA_CHAN_RT_STDATA_REG + i * 4; + dev_dbg(dev, "RRT_STDATA[%02d]: 0x%08x\n", i, + udma_rchanrt_read(uc, offset)); + } + } +} + +struct udma_desc *udma_udma_desc_from_paddr(struct udma_chan *uc, + dma_addr_t paddr) +{ + struct udma_desc *d =3D uc->terminated_desc; + + if (d) { + dma_addr_t desc_paddr =3D udma_curr_cppi5_desc_paddr(d, + d->desc_idx); + + if (desc_paddr !=3D paddr) + d =3D NULL; + } + + if (!d) { + d =3D uc->desc; + if (d) { + dma_addr_t desc_paddr =3D udma_curr_cppi5_desc_paddr(d, + d->desc_idx); + + if (desc_paddr !=3D paddr) + d =3D NULL; + } + } + + return d; +} + +void udma_free_hwdesc(struct udma_chan *uc, struct udma_desc *d) +{ + if (uc->use_dma_pool) { + int i; + + for (i =3D 0; i < d->hwdesc_count; i++) { + if (!d->hwdesc[i].cppi5_desc_vaddr) + continue; + + dma_pool_free(uc->hdesc_pool, + d->hwdesc[i].cppi5_desc_vaddr, + d->hwdesc[i].cppi5_desc_paddr); + + d->hwdesc[i].cppi5_desc_vaddr =3D NULL; + } + } else if (d->hwdesc[0].cppi5_desc_vaddr) { + dma_free_coherent(uc->dma_dev, d->hwdesc[0].cppi5_desc_size, + d->hwdesc[0].cppi5_desc_vaddr, + d->hwdesc[0].cppi5_desc_paddr); + + d->hwdesc[0].cppi5_desc_vaddr =3D NULL; + } +} + +void udma_purge_desc_work(struct work_struct *work) +{ + struct udma_dev *ud =3D container_of(work, typeof(*ud), purge_work); + struct virt_dma_desc *vd, *_vd; + unsigned long flags; + LIST_HEAD(head); + + spin_lock_irqsave(&ud->lock, flags); + list_splice_tail_init(&ud->desc_to_purge, &head); + spin_unlock_irqrestore(&ud->lock, flags); + + list_for_each_entry_safe(vd, _vd, &head, node) { + struct udma_chan *uc =3D to_udma_chan(vd->tx.chan); + struct udma_desc *d =3D to_udma_desc(&vd->tx); + + udma_free_hwdesc(uc, d); + list_del(&vd->node); + kfree(d); + } + + /* If more to purge, schedule the work again */ + if (!list_empty(&ud->desc_to_purge)) + schedule_work(&ud->purge_work); +} + +void udma_desc_free(struct virt_dma_desc *vd) +{ + struct udma_dev *ud =3D to_udma_dev(vd->tx.chan->device); + struct udma_chan *uc =3D to_udma_chan(vd->tx.chan); + struct udma_desc *d =3D to_udma_desc(&vd->tx); + unsigned long flags; + + if (uc->terminated_desc =3D=3D d) + uc->terminated_desc =3D NULL; + + if (uc->use_dma_pool) { + udma_free_hwdesc(uc, d); + kfree(d); + return; + } + + spin_lock_irqsave(&ud->lock, flags); + list_add_tail(&vd->node, &ud->desc_to_purge); + spin_unlock_irqrestore(&ud->lock, flags); + + schedule_work(&ud->purge_work); +} + +bool udma_is_chan_running(struct udma_chan *uc) +{ + u32 trt_ctl =3D 0; + u32 rrt_ctl =3D 0; + + if (uc->tchan) + trt_ctl =3D udma_tchanrt_read(uc, UDMA_CHAN_RT_CTL_REG); + if (uc->rchan) + rrt_ctl =3D udma_rchanrt_read(uc, UDMA_CHAN_RT_CTL_REG); + + if (trt_ctl & UDMA_CHAN_RT_CTL_EN || rrt_ctl & UDMA_CHAN_RT_CTL_EN) + return true; + + return false; +} + +void udma_reset_rings(struct udma_chan *uc) +{ + struct k3_ring *ring1 =3D NULL; + struct k3_ring *ring2 =3D NULL; + + switch (uc->config.dir) { + case DMA_DEV_TO_MEM: + if (uc->rchan) { + ring1 =3D uc->rflow->fd_ring; + ring2 =3D uc->rflow->r_ring; + } + break; + case DMA_MEM_TO_DEV: + case DMA_MEM_TO_MEM: + if (uc->tchan) { + ring1 =3D uc->tchan->t_ring; + ring2 =3D uc->tchan->tc_ring; + } + break; + default: + break; + } + + if (ring1) + k3_ringacc_ring_reset_dma(ring1, + k3_ringacc_ring_get_occ(ring1)); + if (ring2) + k3_ringacc_ring_reset(ring2); + + /* make sure we are not leaking memory by stalled descriptor */ + if (uc->terminated_desc) { + udma_desc_free(&uc->terminated_desc->vd); + uc->terminated_desc =3D NULL; + } +} + +int udma_push_to_ring(struct udma_chan *uc, int idx) +{ + struct udma_desc *d =3D uc->desc; + struct k3_ring *ring =3D NULL; + dma_addr_t paddr; + + switch (uc->config.dir) { + case DMA_DEV_TO_MEM: + ring =3D uc->rflow->fd_ring; + break; + case DMA_MEM_TO_DEV: + case DMA_MEM_TO_MEM: + ring =3D uc->tchan->t_ring; + break; + default: + return -EINVAL; + } + + /* RX flush packet: idx =3D=3D -1 is only passed in case of DEV_TO_MEM */ + if (idx =3D=3D -1) { + paddr =3D udma_get_rx_flush_hwdesc_paddr(uc); + } else { + paddr =3D udma_curr_cppi5_desc_paddr(d, idx); + + wmb(); /* Ensure that writes are not moved over this point */ + } + + return k3_ringacc_ring_push(ring, &paddr); +} + +bool udma_desc_is_rx_flush(struct udma_chan *uc, dma_addr_t addr) +{ + if (uc->config.dir !=3D DMA_DEV_TO_MEM) + return false; + + if (addr =3D=3D udma_get_rx_flush_hwdesc_paddr(uc)) + return true; + + return false; +} + +int udma_pop_from_ring(struct udma_chan *uc, dma_addr_t *addr) +{ + struct k3_ring *ring =3D NULL; + int ret; + + switch (uc->config.dir) { + case DMA_DEV_TO_MEM: + ring =3D uc->rflow->r_ring; + break; + case DMA_MEM_TO_DEV: + case DMA_MEM_TO_MEM: + ring =3D uc->tchan->tc_ring; + break; + default: + return -ENOENT; + } + + ret =3D k3_ringacc_ring_pop(ring, addr); + if (ret) + return ret; + + rmb(); /* Ensure that reads are not moved before this point */ + + /* Teardown completion */ + if (cppi5_desc_is_tdcm(*addr)) + return 0; + + /* Check for flush descriptor */ + if (udma_desc_is_rx_flush(uc, *addr)) + return -ENOENT; + + return 0; +} + +void udma_start_desc(struct udma_chan *uc) +{ + struct udma_chan_config *ucc =3D &uc->config; + + if (uc->ud->match_data->type =3D=3D DMA_TYPE_UDMA && ucc->pkt_mode && + (uc->cyclic || ucc->dir =3D=3D DMA_DEV_TO_MEM)) { + int i; + + /* + * UDMA only: Push all descriptors to ring for packet mode + * cyclic or RX + * PKTDMA supports pre-linked descriptor and cyclic is not + * supported + */ + for (i =3D 0; i < uc->desc->sglen; i++) + udma_push_to_ring(uc, i); + } else { + udma_push_to_ring(uc, 0); + } +} + +bool udma_chan_needs_reconfiguration(struct udma_chan *uc) +{ + /* Only PDMAs have staticTR */ + if (uc->config.ep_type =3D=3D PSIL_EP_NATIVE) + return false; + + /* Check if the staticTR configuration has changed for TX */ + if (memcmp(&uc->static_tr, &uc->desc->static_tr, sizeof(uc->static_tr))) + return true; + + return false; +} + +void udma_cyclic_packet_elapsed(struct udma_chan *uc) +{ + struct udma_desc *d =3D uc->desc; + struct cppi5_host_desc_t *h_desc; + + h_desc =3D d->hwdesc[d->desc_idx].cppi5_desc_vaddr; + cppi5_hdesc_reset_to_original(h_desc); + udma_push_to_ring(uc, d->desc_idx); + d->desc_idx =3D (d->desc_idx + 1) % d->sglen; +} + +void udma_check_tx_completion(struct work_struct *work) +{ + struct udma_chan *uc =3D container_of(work, typeof(*uc), + tx_drain.work.work); + struct udma_dev *ud =3D uc->ud; + bool desc_done =3D true; + u32 residue_diff; + ktime_t time_diff; + unsigned long delay; + + while (1) { + if (uc->desc) { + /* Get previous residue and time stamp */ + residue_diff =3D uc->tx_drain.residue; + time_diff =3D uc->tx_drain.tstamp; + /* + * Get current residue and time stamp or see if + * transfer is complete + */ + desc_done =3D ud->udma_is_desc_really_done(uc, uc->desc); + } + + if (!desc_done) { + /* + * Find the time delta and residue delta w.r.t + * previous poll + */ + time_diff =3D ktime_sub(uc->tx_drain.tstamp, + time_diff) + 1; + residue_diff -=3D uc->tx_drain.residue; + if (residue_diff) { + /* + * Try to guess when we should check + * next time by calculating rate at + * which data is being drained at the + * peer device + */ + delay =3D (time_diff / residue_diff) * + uc->tx_drain.residue; + } else { + /* No progress, check again in 1 second */ + schedule_delayed_work(&uc->tx_drain.work, HZ); + break; + } + + usleep_range(ktime_to_us(delay), + ktime_to_us(delay) + 10); + continue; + } + + if (uc->desc) { + struct udma_desc *d =3D uc->desc; + + ud->udma_decrement_byte_counters(uc, d->residue); + ud->udma_start(uc); + vchan_cookie_complete(&d->vd); + break; + } + + break; + } +} + +/** + * __udma_alloc_gp_rflow_range - alloc range of GP RX flows + * @ud: UDMA device + * @from: Start the search from this flow id number + * @cnt: Number of consecutive flow ids to allocate + * + * Allocate range of RX flow ids for future use, those flows can be reques= ted + * only using explicit flow id number. if @from is set to -1 it will try t= o find + * first free range. if @from is positive value it will force allocation o= nly + * of the specified range of flows. + * + * Returns -ENOMEM if can't find free range. + * -EEXIST if requested range is busy. + * -EINVAL if wrong input values passed. + * Returns flow id on success. + */ +int __udma_alloc_gp_rflow_range(struct udma_dev *ud, int from, int cnt) +{ + int start, tmp_from; + DECLARE_BITMAP(tmp, K3_UDMA_MAX_RFLOWS); + + tmp_from =3D from; + if (tmp_from < 0) + tmp_from =3D ud->rchan_cnt; + /* default flows can't be allocated and accessible only by id */ + if (tmp_from < ud->rchan_cnt) + return -EINVAL; + + if (tmp_from + cnt > ud->rflow_cnt) + return -EINVAL; + + bitmap_or(tmp, ud->rflow_gp_map, ud->rflow_gp_map_allocated, + ud->rflow_cnt); + + start =3D bitmap_find_next_zero_area(tmp, + ud->rflow_cnt, + tmp_from, cnt, 0); + if (start >=3D ud->rflow_cnt) + return -ENOMEM; + + if (from >=3D 0 && start !=3D from) + return -EEXIST; + + bitmap_set(ud->rflow_gp_map_allocated, start, cnt); + return start; +} + +int __udma_free_gp_rflow_range(struct udma_dev *ud, int from, int cnt) +{ + if (from < ud->rchan_cnt) + return -EINVAL; + if (from + cnt > ud->rflow_cnt) + return -EINVAL; + + bitmap_clear(ud->rflow_gp_map_allocated, from, cnt); + return 0; +} + +struct udma_rflow *__udma_get_rflow(struct udma_dev *ud, int id) +{ + /* + * Attempt to request rflow by ID can be made for any rflow + * if not in use with assumption that caller knows what's doing. + * TI-SCI FW will perform additional permission check ant way, it's + * safe + */ + + if (id < 0 || id >=3D ud->rflow_cnt) + return ERR_PTR(-ENOENT); + + if (test_bit(id, ud->rflow_in_use)) + return ERR_PTR(-ENOENT); + + if (ud->rflow_gp_map) { + /* GP rflow has to be allocated first */ + if (!test_bit(id, ud->rflow_gp_map) && + !test_bit(id, ud->rflow_gp_map_allocated)) + return ERR_PTR(-EINVAL); + } + + dev_dbg(ud->dev, "get rflow%d\n", id); + set_bit(id, ud->rflow_in_use); + return &ud->rflows[id]; +} + +void __udma_put_rflow(struct udma_dev *ud, struct udma_rflow *rflow) +{ + if (!test_bit(rflow->id, ud->rflow_in_use)) { + dev_err(ud->dev, "attempt to put unused rflow%d\n", rflow->id); + return; + } + + dev_dbg(ud->dev, "put rflow%d\n", rflow->id); + clear_bit(rflow->id, ud->rflow_in_use); +} + +#define UDMA_RESERVE_RESOURCE(res) \ +struct udma_##res *__udma_reserve_##res(struct udma_dev *ud, \ + enum udma_tp_level tpl, \ + int id) \ +{ \ + if (id >=3D 0) { \ + if (test_bit(id, ud->res##_map)) { \ + dev_err(ud->dev, "res##%d is in use\n", id); \ + return ERR_PTR(-ENOENT); \ + } \ + } else { \ + int start; \ + \ + if (tpl >=3D ud->res##_tpl.levels) \ + tpl =3D ud->res##_tpl.levels - 1; \ + \ + start =3D ud->res##_tpl.start_idx[tpl]; \ + \ + id =3D find_next_zero_bit(ud->res##_map, ud->res##_cnt, \ + start); \ + if (id =3D=3D ud->res##_cnt) { \ + return ERR_PTR(-ENOENT); \ + } \ + } \ + \ + set_bit(id, ud->res##_map); \ + return &ud->res##s[id]; \ +} + +UDMA_RESERVE_RESOURCE(bchan); +UDMA_RESERVE_RESOURCE(tchan); +UDMA_RESERVE_RESOURCE(rchan); + +int udma_get_tchan(struct udma_chan *uc) +{ + struct udma_dev *ud =3D uc->ud; + int ret; + + if (uc->tchan) { + dev_dbg(ud->dev, "chan%d: already have tchan%d allocated\n", + uc->id, uc->tchan->id); + return 0; + } + + /* + * mapped_channel_id is -1 for UDMA, BCDMA and PKTDMA unmapped channels. + * For PKTDMA mapped channels it is configured to a channel which must + * be used to service the peripheral. + */ + uc->tchan =3D __udma_reserve_tchan(ud, uc->config.channel_tpl, + uc->config.mapped_channel_id); + if (IS_ERR(uc->tchan)) { + ret =3D PTR_ERR(uc->tchan); + uc->tchan =3D NULL; + return ret; + } + + if (ud->tflow_cnt) { + int tflow_id; + + /* Only PKTDMA have support for tx flows */ + if (uc->config.default_flow_id >=3D 0) + tflow_id =3D uc->config.default_flow_id; + else + tflow_id =3D uc->tchan->id; + + if (test_bit(tflow_id, ud->tflow_map)) { + dev_err(ud->dev, "tflow%d is in use\n", tflow_id); + clear_bit(uc->tchan->id, ud->tchan_map); + uc->tchan =3D NULL; + return -ENOENT; + } + + uc->tchan->tflow_id =3D tflow_id; + set_bit(tflow_id, ud->tflow_map); + } else { + uc->tchan->tflow_id =3D -1; + } + + return 0; +} + +int udma_get_rchan(struct udma_chan *uc) +{ + struct udma_dev *ud =3D uc->ud; + int ret; + + if (uc->rchan) { + dev_dbg(ud->dev, "chan%d: already have rchan%d allocated\n", + uc->id, uc->rchan->id); + return 0; + } + + /* + * mapped_channel_id is -1 for UDMA, BCDMA and PKTDMA unmapped channels. + * For PKTDMA mapped channels it is configured to a channel which must + * be used to service the peripheral. + */ + uc->rchan =3D __udma_reserve_rchan(ud, uc->config.channel_tpl, + uc->config.mapped_channel_id); + if (IS_ERR(uc->rchan)) { + ret =3D PTR_ERR(uc->rchan); + uc->rchan =3D NULL; + return ret; + } + + return 0; +} + +int udma_get_chan_pair(struct udma_chan *uc) +{ + struct udma_dev *ud =3D uc->ud; + int chan_id, end; + + if ((uc->tchan && uc->rchan) && uc->tchan->id =3D=3D uc->rchan->id) { + dev_info(ud->dev, "chan%d: already have %d pair allocated\n", + uc->id, uc->tchan->id); + return 0; + } + + if (uc->tchan) { + dev_err(ud->dev, "chan%d: already have tchan%d allocated\n", + uc->id, uc->tchan->id); + return -EBUSY; + } else if (uc->rchan) { + dev_err(ud->dev, "chan%d: already have rchan%d allocated\n", + uc->id, uc->rchan->id); + return -EBUSY; + } + + /* Can be optimized, but let's have it like this for now */ + end =3D min(ud->tchan_cnt, ud->rchan_cnt); + /* + * Try to use the highest TPL channel pair for MEM_TO_MEM channels + * Note: in UDMAP the channel TPL is symmetric between tchan and rchan + */ + chan_id =3D ud->tchan_tpl.start_idx[ud->tchan_tpl.levels - 1]; + for (; chan_id < end; chan_id++) { + if (!test_bit(chan_id, ud->tchan_map) && + !test_bit(chan_id, ud->rchan_map)) + break; + } + + if (chan_id =3D=3D end) + return -ENOENT; + + set_bit(chan_id, ud->tchan_map); + set_bit(chan_id, ud->rchan_map); + uc->tchan =3D &ud->tchans[chan_id]; + uc->rchan =3D &ud->rchans[chan_id]; + + /* UDMA does not use tx flows */ + uc->tchan->tflow_id =3D -1; + + return 0; +} + +int udma_get_rflow(struct udma_chan *uc, int flow_id) +{ + struct udma_dev *ud =3D uc->ud; + int ret; + + if (!uc->rchan) { + dev_err(ud->dev, "chan%d: does not have rchan??\n", uc->id); + return -EINVAL; + } + + if (uc->rflow) { + dev_dbg(ud->dev, "chan%d: already have rflow%d allocated\n", + uc->id, uc->rflow->id); + return 0; + } + + uc->rflow =3D __udma_get_rflow(ud, flow_id); + if (IS_ERR(uc->rflow)) { + ret =3D PTR_ERR(uc->rflow); + uc->rflow =3D NULL; + return ret; + } + + return 0; +} + +void bcdma_put_bchan(struct udma_chan *uc) +{ + struct udma_dev *ud =3D uc->ud; + + if (uc->bchan) { + dev_dbg(ud->dev, "chan%d: put bchan%d\n", uc->id, + uc->bchan->id); + clear_bit(uc->bchan->id, ud->bchan_map); + uc->bchan =3D NULL; + uc->tchan =3D NULL; + } +} + +void udma_put_rchan(struct udma_chan *uc) +{ + struct udma_dev *ud =3D uc->ud; + + if (uc->rchan) { + dev_dbg(ud->dev, "chan%d: put rchan%d\n", uc->id, + uc->rchan->id); + clear_bit(uc->rchan->id, ud->rchan_map); + uc->rchan =3D NULL; + } +} + +void udma_put_tchan(struct udma_chan *uc) +{ + struct udma_dev *ud =3D uc->ud; + + if (uc->tchan) { + dev_dbg(ud->dev, "chan%d: put tchan%d\n", uc->id, + uc->tchan->id); + clear_bit(uc->tchan->id, ud->tchan_map); + + if (uc->tchan->tflow_id >=3D 0) + clear_bit(uc->tchan->tflow_id, ud->tflow_map); + + uc->tchan =3D NULL; + } +} + +void udma_put_rflow(struct udma_chan *uc) +{ + struct udma_dev *ud =3D uc->ud; + + if (uc->rflow) { + dev_dbg(ud->dev, "chan%d: put rflow%d\n", uc->id, + uc->rflow->id); + __udma_put_rflow(ud, uc->rflow); + uc->rflow =3D NULL; + } +} + +void bcdma_free_bchan_resources(struct udma_chan *uc) +{ + if (!uc->bchan) + return; + + k3_ringacc_ring_free(uc->bchan->tc_ring); + k3_ringacc_ring_free(uc->bchan->t_ring); + uc->bchan->tc_ring =3D NULL; + uc->bchan->t_ring =3D NULL; + k3_configure_chan_coherency(&uc->vc.chan, 0); + + bcdma_put_bchan(uc); +} + +void udma_free_tx_resources(struct udma_chan *uc) +{ + if (!uc->tchan) + return; + + k3_ringacc_ring_free(uc->tchan->t_ring); + k3_ringacc_ring_free(uc->tchan->tc_ring); + uc->tchan->t_ring =3D NULL; + uc->tchan->tc_ring =3D NULL; + + udma_put_tchan(uc); +} + +void udma_free_rx_resources(struct udma_chan *uc) +{ + if (!uc->rchan) + return; + + if (uc->rflow) { + struct udma_rflow *rflow =3D uc->rflow; + + k3_ringacc_ring_free(rflow->fd_ring); + k3_ringacc_ring_free(rflow->r_ring); + rflow->fd_ring =3D NULL; + rflow->r_ring =3D NULL; + + udma_put_rflow(uc); + } + + udma_put_rchan(uc); +} + +int udma_slave_config(struct dma_chan *chan, + struct dma_slave_config *cfg) +{ + struct udma_chan *uc =3D to_udma_chan(chan); + + memcpy(&uc->cfg, cfg, sizeof(uc->cfg)); + + return 0; +} + +struct udma_desc *udma_alloc_tr_desc(struct udma_chan *uc, + size_t tr_size, int tr_count, + enum dma_transfer_direction dir) +{ + struct udma_hwdesc *hwdesc; + struct cppi5_desc_hdr_t *tr_desc; + struct udma_desc *d; + u32 reload_count =3D 0; + u32 ring_id; + + switch (tr_size) { + case 16: + case 32: + case 64: + case 128: + break; + default: + dev_err(uc->ud->dev, "Unsupported TR size of %zu\n", tr_size); + return NULL; + } + + /* We have only one descriptor containing multiple TRs */ + d =3D kzalloc(sizeof(*d) + sizeof(d->hwdesc[0]), GFP_NOWAIT); + if (!d) + return NULL; + + d->sglen =3D tr_count; + + d->hwdesc_count =3D 1; + hwdesc =3D &d->hwdesc[0]; + + /* Allocate memory for DMA ring descriptor */ + if (uc->use_dma_pool) { + hwdesc->cppi5_desc_size =3D uc->config.hdesc_size; + hwdesc->cppi5_desc_vaddr =3D dma_pool_zalloc(uc->hdesc_pool, + GFP_NOWAIT, + &hwdesc->cppi5_desc_paddr); + } else { + hwdesc->cppi5_desc_size =3D cppi5_trdesc_calc_size(tr_size, + tr_count); + hwdesc->cppi5_desc_size =3D ALIGN(hwdesc->cppi5_desc_size, + uc->ud->desc_align); + hwdesc->cppi5_desc_vaddr =3D dma_alloc_coherent(uc->ud->dev, + hwdesc->cppi5_desc_size, + &hwdesc->cppi5_desc_paddr, + GFP_NOWAIT); + } + + if (!hwdesc->cppi5_desc_vaddr) { + kfree(d); + return NULL; + } + + /* Start of the TR req records */ + hwdesc->tr_req_base =3D hwdesc->cppi5_desc_vaddr + tr_size; + /* Start address of the TR response array */ + hwdesc->tr_resp_base =3D hwdesc->tr_req_base + tr_size * tr_count; + + tr_desc =3D hwdesc->cppi5_desc_vaddr; + + if (uc->cyclic) + reload_count =3D CPPI5_INFO0_TRDESC_RLDCNT_INFINITE; + + if (dir =3D=3D DMA_DEV_TO_MEM) + ring_id =3D k3_ringacc_get_ring_id(uc->rflow->r_ring); + else + ring_id =3D k3_ringacc_get_ring_id(uc->tchan->tc_ring); + + cppi5_trdesc_init(tr_desc, tr_count, tr_size, 0, reload_count); + cppi5_desc_set_pktids(tr_desc, uc->id, + CPPI5_INFO1_DESC_FLOWID_DEFAULT); + cppi5_desc_set_retpolicy(tr_desc, 0, ring_id); + + return d; +} + +/** + * udma_get_tr_counters - calculate TR counters for a given length + * @len: Length of the trasnfer + * @align_to: Preferred alignment + * @tr0_cnt0: First TR icnt0 + * @tr0_cnt1: First TR icnt1 + * @tr1_cnt0: Second (if used) TR icnt0 + * + * For len < SZ_64K only one TR is enough, tr1_cnt0 is not updated + * For len >=3D SZ_64K two TRs are used in a simple way: + * First TR: SZ_64K-alignment blocks (tr0_cnt0, tr0_cnt1) + * Second TR: the remaining length (tr1_cnt0) + * + * Returns the number of TRs the length needs (1 or 2) + * -EINVAL if the length can not be supported + */ +int udma_get_tr_counters(size_t len, unsigned long align_to, + u16 *tr0_cnt0, u16 *tr0_cnt1, u16 *tr1_cnt0) +{ + if (len < SZ_64K) { + *tr0_cnt0 =3D len; + *tr0_cnt1 =3D 1; + + return 1; + } + + if (align_to > 3) + align_to =3D 3; + +realign: + *tr0_cnt0 =3D SZ_64K - BIT(align_to); + if (len / *tr0_cnt0 >=3D SZ_64K) { + if (align_to) { + align_to--; + goto realign; + } + return -EINVAL; + } + + *tr0_cnt1 =3D len / *tr0_cnt0; + *tr1_cnt0 =3D len % *tr0_cnt0; + + return 2; +} + +struct udma_desc * +udma_prep_slave_sg_tr(struct udma_chan *uc, struct scatterlist *sgl, + unsigned int sglen, enum dma_transfer_direction dir, + unsigned long tx_flags, void *context) +{ + struct scatterlist *sgent; + struct udma_desc *d; + struct cppi5_tr_type1_t *tr_req =3D NULL; + u16 tr0_cnt0, tr0_cnt1, tr1_cnt0; + unsigned int i; + size_t tr_size; + int num_tr =3D 0; + int tr_idx =3D 0; + u64 asel; + + /* estimate the number of TRs we will need */ + for_each_sg(sgl, sgent, sglen, i) { + if (sg_dma_len(sgent) < SZ_64K) + num_tr++; + else + num_tr +=3D 2; + } + + /* Now allocate and setup the descriptor. */ + tr_size =3D sizeof(struct cppi5_tr_type1_t); + d =3D udma_alloc_tr_desc(uc, tr_size, num_tr, dir); + if (!d) + return NULL; + + d->sglen =3D sglen; + + if (uc->ud->match_data->type =3D=3D DMA_TYPE_UDMA) + asel =3D 0; + else + asel =3D (u64)uc->config.asel << K3_ADDRESS_ASEL_SHIFT; + + tr_req =3D d->hwdesc[0].tr_req_base; + for_each_sg(sgl, sgent, sglen, i) { + dma_addr_t sg_addr =3D sg_dma_address(sgent); + + num_tr =3D udma_get_tr_counters(sg_dma_len(sgent), __ffs(sg_addr), + &tr0_cnt0, &tr0_cnt1, &tr1_cnt0); + if (num_tr < 0) { + dev_err(uc->ud->dev, "size %u is not supported\n", + sg_dma_len(sgent)); + udma_free_hwdesc(uc, d); + kfree(d); + return NULL; + } + + cppi5_tr_init(&tr_req[tr_idx].flags, CPPI5_TR_TYPE1, false, + false, CPPI5_TR_EVENT_SIZE_COMPLETION, 0); + cppi5_tr_csf_set(&tr_req[tr_idx].flags, CPPI5_TR_CSF_SUPR_EVT); + + sg_addr |=3D asel; + tr_req[tr_idx].addr =3D sg_addr; + tr_req[tr_idx].icnt0 =3D tr0_cnt0; + tr_req[tr_idx].icnt1 =3D tr0_cnt1; + tr_req[tr_idx].dim1 =3D tr0_cnt0; + tr_idx++; + + if (num_tr =3D=3D 2) { + cppi5_tr_init(&tr_req[tr_idx].flags, CPPI5_TR_TYPE1, + false, false, + CPPI5_TR_EVENT_SIZE_COMPLETION, 0); + cppi5_tr_csf_set(&tr_req[tr_idx].flags, + CPPI5_TR_CSF_SUPR_EVT); + + tr_req[tr_idx].addr =3D sg_addr + tr0_cnt1 * tr0_cnt0; + tr_req[tr_idx].icnt0 =3D tr1_cnt0; + tr_req[tr_idx].icnt1 =3D 1; + tr_req[tr_idx].dim1 =3D tr1_cnt0; + tr_idx++; + } + + d->residue +=3D sg_dma_len(sgent); + } + + cppi5_tr_csf_set(&tr_req[tr_idx - 1].flags, + CPPI5_TR_CSF_SUPR_EVT | CPPI5_TR_CSF_EOP); + + return d; +} + +struct udma_desc * +udma_prep_slave_sg_triggered_tr(struct udma_chan *uc, struct scatterlist *= sgl, + unsigned int sglen, + enum dma_transfer_direction dir, + unsigned long tx_flags, void *context) +{ + struct scatterlist *sgent; + struct cppi5_tr_type15_t *tr_req =3D NULL; + enum dma_slave_buswidth dev_width; + u32 csf =3D CPPI5_TR_CSF_SUPR_EVT; + u16 tr_cnt0, tr_cnt1; + dma_addr_t dev_addr; + struct udma_desc *d; + unsigned int i; + size_t tr_size, sg_len; + int num_tr =3D 0; + int tr_idx =3D 0; + u32 burst, trigger_size, port_window; + u64 asel; + + if (dir =3D=3D DMA_DEV_TO_MEM) { + dev_addr =3D uc->cfg.src_addr; + dev_width =3D uc->cfg.src_addr_width; + burst =3D uc->cfg.src_maxburst; + port_window =3D uc->cfg.src_port_window_size; + } else if (dir =3D=3D DMA_MEM_TO_DEV) { + dev_addr =3D uc->cfg.dst_addr; + dev_width =3D uc->cfg.dst_addr_width; + burst =3D uc->cfg.dst_maxburst; + port_window =3D uc->cfg.dst_port_window_size; + } else { + dev_err(uc->ud->dev, "%s: bad direction?\n", __func__); + return NULL; + } + + if (!burst) + burst =3D 1; + + if (port_window) { + if (port_window !=3D burst) { + dev_err(uc->ud->dev, + "The burst must be equal to port_window\n"); + return NULL; + } + + tr_cnt0 =3D dev_width * port_window; + tr_cnt1 =3D 1; + } else { + tr_cnt0 =3D dev_width; + tr_cnt1 =3D burst; + } + trigger_size =3D tr_cnt0 * tr_cnt1; + + /* estimate the number of TRs we will need */ + for_each_sg(sgl, sgent, sglen, i) { + sg_len =3D sg_dma_len(sgent); + + if (sg_len % trigger_size) { + dev_err(uc->ud->dev, + "Not aligned SG entry (%zu for %u)\n", sg_len, + trigger_size); + return NULL; + } + + if (sg_len / trigger_size < SZ_64K) + num_tr++; + else + num_tr +=3D 2; + } + + /* Now allocate and setup the descriptor. */ + tr_size =3D sizeof(struct cppi5_tr_type15_t); + d =3D udma_alloc_tr_desc(uc, tr_size, num_tr, dir); + if (!d) + return NULL; + + d->sglen =3D sglen; + + if (uc->ud->match_data->type =3D=3D DMA_TYPE_UDMA) { + asel =3D 0; + csf |=3D CPPI5_TR_CSF_EOL_ICNT0; + } else { + asel =3D (u64)uc->config.asel << K3_ADDRESS_ASEL_SHIFT; + dev_addr |=3D asel; + } + + tr_req =3D d->hwdesc[0].tr_req_base; + for_each_sg(sgl, sgent, sglen, i) { + u16 tr0_cnt2, tr0_cnt3, tr1_cnt2; + dma_addr_t sg_addr =3D sg_dma_address(sgent); + + sg_len =3D sg_dma_len(sgent); + num_tr =3D udma_get_tr_counters(sg_len / trigger_size, 0, + &tr0_cnt2, &tr0_cnt3, &tr1_cnt2); + if (num_tr < 0) { + dev_err(uc->ud->dev, "size %zu is not supported\n", + sg_len); + udma_free_hwdesc(uc, d); + kfree(d); + return NULL; + } + + cppi5_tr_init(&tr_req[tr_idx].flags, CPPI5_TR_TYPE15, false, + true, CPPI5_TR_EVENT_SIZE_COMPLETION, 0); + cppi5_tr_csf_set(&tr_req[tr_idx].flags, csf); + cppi5_tr_set_trigger(&tr_req[tr_idx].flags, + uc->config.tr_trigger_type, + CPPI5_TR_TRIGGER_TYPE_ICNT2_DEC, 0, 0); + + sg_addr |=3D asel; + if (dir =3D=3D DMA_DEV_TO_MEM) { + tr_req[tr_idx].addr =3D dev_addr; + tr_req[tr_idx].icnt0 =3D tr_cnt0; + tr_req[tr_idx].icnt1 =3D tr_cnt1; + tr_req[tr_idx].icnt2 =3D tr0_cnt2; + tr_req[tr_idx].icnt3 =3D tr0_cnt3; + tr_req[tr_idx].dim1 =3D (-1) * tr_cnt0; + + tr_req[tr_idx].daddr =3D sg_addr; + tr_req[tr_idx].dicnt0 =3D tr_cnt0; + tr_req[tr_idx].dicnt1 =3D tr_cnt1; + tr_req[tr_idx].dicnt2 =3D tr0_cnt2; + tr_req[tr_idx].dicnt3 =3D tr0_cnt3; + tr_req[tr_idx].ddim1 =3D tr_cnt0; + tr_req[tr_idx].ddim2 =3D trigger_size; + tr_req[tr_idx].ddim3 =3D trigger_size * tr0_cnt2; + } else { + tr_req[tr_idx].addr =3D sg_addr; + tr_req[tr_idx].icnt0 =3D tr_cnt0; + tr_req[tr_idx].icnt1 =3D tr_cnt1; + tr_req[tr_idx].icnt2 =3D tr0_cnt2; + tr_req[tr_idx].icnt3 =3D tr0_cnt3; + tr_req[tr_idx].dim1 =3D tr_cnt0; + tr_req[tr_idx].dim2 =3D trigger_size; + tr_req[tr_idx].dim3 =3D trigger_size * tr0_cnt2; + + tr_req[tr_idx].daddr =3D dev_addr; + tr_req[tr_idx].dicnt0 =3D tr_cnt0; + tr_req[tr_idx].dicnt1 =3D tr_cnt1; + tr_req[tr_idx].dicnt2 =3D tr0_cnt2; + tr_req[tr_idx].dicnt3 =3D tr0_cnt3; + tr_req[tr_idx].ddim1 =3D (-1) * tr_cnt0; + } + + tr_idx++; + + if (num_tr =3D=3D 2) { + cppi5_tr_init(&tr_req[tr_idx].flags, CPPI5_TR_TYPE15, + false, true, + CPPI5_TR_EVENT_SIZE_COMPLETION, 0); + cppi5_tr_csf_set(&tr_req[tr_idx].flags, csf); + cppi5_tr_set_trigger(&tr_req[tr_idx].flags, + uc->config.tr_trigger_type, + CPPI5_TR_TRIGGER_TYPE_ICNT2_DEC, + 0, 0); + + sg_addr +=3D trigger_size * tr0_cnt2 * tr0_cnt3; + if (dir =3D=3D DMA_DEV_TO_MEM) { + tr_req[tr_idx].addr =3D dev_addr; + tr_req[tr_idx].icnt0 =3D tr_cnt0; + tr_req[tr_idx].icnt1 =3D tr_cnt1; + tr_req[tr_idx].icnt2 =3D tr1_cnt2; + tr_req[tr_idx].icnt3 =3D 1; + tr_req[tr_idx].dim1 =3D (-1) * tr_cnt0; + + tr_req[tr_idx].daddr =3D sg_addr; + tr_req[tr_idx].dicnt0 =3D tr_cnt0; + tr_req[tr_idx].dicnt1 =3D tr_cnt1; + tr_req[tr_idx].dicnt2 =3D tr1_cnt2; + tr_req[tr_idx].dicnt3 =3D 1; + tr_req[tr_idx].ddim1 =3D tr_cnt0; + tr_req[tr_idx].ddim2 =3D trigger_size; + } else { + tr_req[tr_idx].addr =3D sg_addr; + tr_req[tr_idx].icnt0 =3D tr_cnt0; + tr_req[tr_idx].icnt1 =3D tr_cnt1; + tr_req[tr_idx].icnt2 =3D tr1_cnt2; + tr_req[tr_idx].icnt3 =3D 1; + tr_req[tr_idx].dim1 =3D tr_cnt0; + tr_req[tr_idx].dim2 =3D trigger_size; + + tr_req[tr_idx].daddr =3D dev_addr; + tr_req[tr_idx].dicnt0 =3D tr_cnt0; + tr_req[tr_idx].dicnt1 =3D tr_cnt1; + tr_req[tr_idx].dicnt2 =3D tr1_cnt2; + tr_req[tr_idx].dicnt3 =3D 1; + tr_req[tr_idx].ddim1 =3D (-1) * tr_cnt0; + } + tr_idx++; + } + + d->residue +=3D sg_len; + } + + cppi5_tr_csf_set(&tr_req[tr_idx - 1].flags, csf | CPPI5_TR_CSF_EOP); + + return d; +} + +int udma_configure_statictr(struct udma_chan *uc, struct udma_desc *d, + enum dma_slave_buswidth dev_width, + u16 elcnt) +{ + if (uc->config.ep_type !=3D PSIL_EP_PDMA_XY) + return 0; + + /* Bus width translates to the element size (ES) */ + switch (dev_width) { + case DMA_SLAVE_BUSWIDTH_1_BYTE: + d->static_tr.elsize =3D 0; + break; + case DMA_SLAVE_BUSWIDTH_2_BYTES: + d->static_tr.elsize =3D 1; + break; + case DMA_SLAVE_BUSWIDTH_3_BYTES: + d->static_tr.elsize =3D 2; + break; + case DMA_SLAVE_BUSWIDTH_4_BYTES: + d->static_tr.elsize =3D 3; + break; + case DMA_SLAVE_BUSWIDTH_8_BYTES: + d->static_tr.elsize =3D 4; + break; + default: /* not reached */ + return -EINVAL; + } + + d->static_tr.elcnt =3D elcnt; + + if (uc->config.pkt_mode || !uc->cyclic) { + /* + * PDMA must close the packet when the channel is in packet mode. + * For TR mode when the channel is not cyclic we also need PDMA + * to close the packet otherwise the transfer will stall because + * PDMA holds on the data it has received from the peripheral. + */ + unsigned int div =3D dev_width * elcnt; + + if (uc->cyclic) + d->static_tr.bstcnt =3D d->residue / d->sglen / div; + else + d->static_tr.bstcnt =3D d->residue / div; + } else if (uc->ud->match_data->type =3D=3D DMA_TYPE_BCDMA && + uc->config.dir =3D=3D DMA_DEV_TO_MEM && + uc->cyclic) { + /* + * For cyclic mode with BCDMA we have to set EOP in each TR to + * prevent short packet errors seen on channel teardown. So the + * PDMA must close the packet after every TR transfer by setting + * burst count equal to the number of bytes transferred. + */ + struct cppi5_tr_type1_t *tr_req =3D d->hwdesc[0].tr_req_base; + + d->static_tr.bstcnt =3D + (tr_req->icnt0 * tr_req->icnt1) / dev_width; + } else { + d->static_tr.bstcnt =3D 0; + } + + if (uc->config.dir =3D=3D DMA_DEV_TO_MEM && + d->static_tr.bstcnt > uc->ud->match_data->statictr_z_mask) + return -EINVAL; + + return 0; +} + +struct udma_desc * +udma_prep_slave_sg_pkt(struct udma_chan *uc, struct scatterlist *sgl, + unsigned int sglen, enum dma_transfer_direction dir, + unsigned long tx_flags, void *context) +{ + struct scatterlist *sgent; + struct cppi5_host_desc_t *h_desc =3D NULL; + struct udma_desc *d; + u32 ring_id; + unsigned int i; + u64 asel; + + d =3D kzalloc(struct_size(d, hwdesc, sglen), GFP_NOWAIT); + if (!d) + return NULL; + + d->sglen =3D sglen; + d->hwdesc_count =3D sglen; + + if (dir =3D=3D DMA_DEV_TO_MEM) + ring_id =3D k3_ringacc_get_ring_id(uc->rflow->r_ring); + else + ring_id =3D k3_ringacc_get_ring_id(uc->tchan->tc_ring); + + if (uc->ud->match_data->type =3D=3D DMA_TYPE_UDMA) + asel =3D 0; + else + asel =3D (u64)uc->config.asel << K3_ADDRESS_ASEL_SHIFT; + + for_each_sg(sgl, sgent, sglen, i) { + struct udma_hwdesc *hwdesc =3D &d->hwdesc[i]; + dma_addr_t sg_addr =3D sg_dma_address(sgent); + struct cppi5_host_desc_t *desc; + size_t sg_len =3D sg_dma_len(sgent); + + hwdesc->cppi5_desc_vaddr =3D dma_pool_zalloc(uc->hdesc_pool, + GFP_NOWAIT, + &hwdesc->cppi5_desc_paddr); + if (!hwdesc->cppi5_desc_vaddr) { + dev_err(uc->ud->dev, + "descriptor%d allocation failed\n", i); + + udma_free_hwdesc(uc, d); + kfree(d); + return NULL; + } + + d->residue +=3D sg_len; + hwdesc->cppi5_desc_size =3D uc->config.hdesc_size; + desc =3D hwdesc->cppi5_desc_vaddr; + + if (i =3D=3D 0) { + cppi5_hdesc_init(desc, 0, 0); + /* Flow and Packed ID */ + cppi5_desc_set_pktids(&desc->hdr, uc->id, + CPPI5_INFO1_DESC_FLOWID_DEFAULT); + cppi5_desc_set_retpolicy(&desc->hdr, 0, ring_id); + } else { + cppi5_hdesc_reset_hbdesc(desc); + cppi5_desc_set_retpolicy(&desc->hdr, 0, 0xffff); + } + + /* attach the sg buffer to the descriptor */ + sg_addr |=3D asel; + cppi5_hdesc_attach_buf(desc, sg_addr, sg_len, sg_addr, sg_len); + + /* Attach link as host buffer descriptor */ + if (h_desc) + cppi5_hdesc_link_hbdesc(h_desc, + hwdesc->cppi5_desc_paddr | asel); + + if (uc->ud->match_data->type =3D=3D DMA_TYPE_PKTDMA || + dir =3D=3D DMA_MEM_TO_DEV) + h_desc =3D desc; + } + + if (d->residue >=3D SZ_4M) { + dev_err(uc->ud->dev, + "%s: Transfer size %u is over the supported 4M range\n", + __func__, d->residue); + udma_free_hwdesc(uc, d); + kfree(d); + return NULL; + } + + h_desc =3D d->hwdesc[0].cppi5_desc_vaddr; + cppi5_hdesc_set_pktlen(h_desc, d->residue); + + return d; +} + +int udma_attach_metadata(struct dma_async_tx_descriptor *desc, + void *data, size_t len) +{ + struct udma_desc *d =3D to_udma_desc(desc); + struct udma_chan *uc =3D to_udma_chan(desc->chan); + struct cppi5_host_desc_t *h_desc; + u32 psd_size =3D len; + u32 flags =3D 0; + + if (!uc->config.pkt_mode || !uc->config.metadata_size) + return -EOPNOTSUPP; + + if (!data || len > uc->config.metadata_size) + return -EINVAL; + + if (uc->config.needs_epib && len < CPPI5_INFO0_HDESC_EPIB_SIZE) + return -EINVAL; + + h_desc =3D d->hwdesc[0].cppi5_desc_vaddr; + if (d->dir =3D=3D DMA_MEM_TO_DEV) + memcpy(h_desc->epib, data, len); + + if (uc->config.needs_epib) + psd_size -=3D CPPI5_INFO0_HDESC_EPIB_SIZE; + + d->metadata =3D data; + d->metadata_size =3D len; + if (uc->config.needs_epib) + flags |=3D CPPI5_INFO0_HDESC_EPIB_PRESENT; + + cppi5_hdesc_update_flags(h_desc, flags); + cppi5_hdesc_update_psdata_size(h_desc, psd_size); + + return 0; +} + +void *udma_get_metadata_ptr(struct dma_async_tx_descriptor *desc, + size_t *payload_len, size_t *max_len) +{ + struct udma_desc *d =3D to_udma_desc(desc); + struct udma_chan *uc =3D to_udma_chan(desc->chan); + struct cppi5_host_desc_t *h_desc; + + if (!uc->config.pkt_mode || !uc->config.metadata_size) + return ERR_PTR(-EOPNOTSUPP); + + h_desc =3D d->hwdesc[0].cppi5_desc_vaddr; + + *max_len =3D uc->config.metadata_size; + + *payload_len =3D cppi5_hdesc_epib_present(&h_desc->hdr) ? + CPPI5_INFO0_HDESC_EPIB_SIZE : 0; + *payload_len +=3D cppi5_hdesc_get_psdata_size(h_desc); + + return h_desc->epib; +} + +int udma_set_metadata_len(struct dma_async_tx_descriptor *desc, + size_t payload_len) +{ + struct udma_desc *d =3D to_udma_desc(desc); + struct udma_chan *uc =3D to_udma_chan(desc->chan); + struct cppi5_host_desc_t *h_desc; + u32 psd_size =3D payload_len; + u32 flags =3D 0; + + if (!uc->config.pkt_mode || !uc->config.metadata_size) + return -EOPNOTSUPP; + + if (payload_len > uc->config.metadata_size) + return -EINVAL; + + if (uc->config.needs_epib && payload_len < CPPI5_INFO0_HDESC_EPIB_SIZE) + return -EINVAL; + + h_desc =3D d->hwdesc[0].cppi5_desc_vaddr; + + if (uc->config.needs_epib) { + psd_size -=3D CPPI5_INFO0_HDESC_EPIB_SIZE; + flags |=3D CPPI5_INFO0_HDESC_EPIB_PRESENT; + } + + cppi5_hdesc_update_flags(h_desc, flags); + cppi5_hdesc_update_psdata_size(h_desc, psd_size); + + return 0; +} + +struct dma_descriptor_metadata_ops metadata_ops =3D { + .attach =3D udma_attach_metadata, + .get_ptr =3D udma_get_metadata_ptr, + .set_len =3D udma_set_metadata_len, +}; + +struct dma_async_tx_descriptor * +udma_prep_slave_sg(struct dma_chan *chan, struct scatterlist *sgl, + unsigned int sglen, enum dma_transfer_direction dir, + unsigned long tx_flags, void *context) +{ + struct udma_chan *uc =3D to_udma_chan(chan); + enum dma_slave_buswidth dev_width; + struct udma_desc *d; + u32 burst; + + if (dir !=3D uc->config.dir && + (uc->config.dir =3D=3D DMA_MEM_TO_MEM && !uc->config.tr_trigger_type)= ) { + dev_err(chan->device->dev, + "%s: chan%d is for %s, not supporting %s\n", + __func__, uc->id, + dmaengine_get_direction_text(uc->config.dir), + dmaengine_get_direction_text(dir)); + return NULL; + } + + if (dir =3D=3D DMA_DEV_TO_MEM) { + dev_width =3D uc->cfg.src_addr_width; + burst =3D uc->cfg.src_maxburst; + } else if (dir =3D=3D DMA_MEM_TO_DEV) { + dev_width =3D uc->cfg.dst_addr_width; + burst =3D uc->cfg.dst_maxburst; + } else { + dev_err(chan->device->dev, "%s: bad direction?\n", __func__); + return NULL; + } + + if (!burst) + burst =3D 1; + + uc->config.tx_flags =3D tx_flags; + + if (uc->config.pkt_mode) + d =3D udma_prep_slave_sg_pkt(uc, sgl, sglen, dir, tx_flags, + context); + else if (is_slave_direction(uc->config.dir)) + d =3D udma_prep_slave_sg_tr(uc, sgl, sglen, dir, tx_flags, + context); + else + d =3D udma_prep_slave_sg_triggered_tr(uc, sgl, sglen, dir, + tx_flags, context); + + if (!d) + return NULL; + + d->dir =3D dir; + d->desc_idx =3D 0; + d->tr_idx =3D 0; + + /* static TR for remote PDMA */ + if (udma_configure_statictr(uc, d, dev_width, burst)) { + dev_err(uc->ud->dev, + "%s: StaticTR Z is limited to maximum %u (%u)\n", + __func__, uc->ud->match_data->statictr_z_mask, + d->static_tr.bstcnt); + + udma_free_hwdesc(uc, d); + kfree(d); + return NULL; + } + + if (uc->config.metadata_size) + d->vd.tx.metadata_ops =3D &metadata_ops; + + return vchan_tx_prep(&uc->vc, &d->vd, tx_flags); +} + +struct udma_desc * +udma_prep_dma_cyclic_tr(struct udma_chan *uc, dma_addr_t buf_addr, + size_t buf_len, size_t period_len, + enum dma_transfer_direction dir, unsigned long flags) +{ + struct udma_desc *d; + size_t tr_size, period_addr; + struct cppi5_tr_type1_t *tr_req; + unsigned int periods =3D buf_len / period_len; + u16 tr0_cnt0, tr0_cnt1, tr1_cnt0; + unsigned int i; + int num_tr; + u32 period_csf =3D 0; + + num_tr =3D udma_get_tr_counters(period_len, __ffs(buf_addr), &tr0_cnt0, + &tr0_cnt1, &tr1_cnt0); + if (num_tr < 0) { + dev_err(uc->ud->dev, "size %zu is not supported\n", + period_len); + return NULL; + } + + /* Now allocate and setup the descriptor. */ + tr_size =3D sizeof(struct cppi5_tr_type1_t); + d =3D udma_alloc_tr_desc(uc, tr_size, periods * num_tr, dir); + if (!d) + return NULL; + + tr_req =3D d->hwdesc[0].tr_req_base; + if (uc->ud->match_data->type =3D=3D DMA_TYPE_UDMA) + period_addr =3D buf_addr; + else + period_addr =3D buf_addr | + ((u64)uc->config.asel << K3_ADDRESS_ASEL_SHIFT); + + /* + * For BCDMA <-> PDMA transfers, the EOP flag needs to be set on the + * last TR of a descriptor, to mark the packet as complete. + * This is required for getting the teardown completion message in case + * of TX, and to avoid short-packet error in case of RX. + * + * As we are in cyclic mode, we do not know which period might be the + * last one, so set the flag for each period. + */ + if (uc->config.ep_type =3D=3D PSIL_EP_PDMA_XY && + uc->ud->match_data->type =3D=3D DMA_TYPE_BCDMA) { + period_csf =3D CPPI5_TR_CSF_EOP; + } + + for (i =3D 0; i < periods; i++) { + int tr_idx =3D i * num_tr; + + cppi5_tr_init(&tr_req[tr_idx].flags, CPPI5_TR_TYPE1, false, + false, CPPI5_TR_EVENT_SIZE_COMPLETION, 0); + + tr_req[tr_idx].addr =3D period_addr; + tr_req[tr_idx].icnt0 =3D tr0_cnt0; + tr_req[tr_idx].icnt1 =3D tr0_cnt1; + tr_req[tr_idx].dim1 =3D tr0_cnt0; + + if (num_tr =3D=3D 2) { + cppi5_tr_csf_set(&tr_req[tr_idx].flags, + CPPI5_TR_CSF_SUPR_EVT); + tr_idx++; + + cppi5_tr_init(&tr_req[tr_idx].flags, CPPI5_TR_TYPE1, + false, false, + CPPI5_TR_EVENT_SIZE_COMPLETION, 0); + + tr_req[tr_idx].addr =3D period_addr + tr0_cnt1 * tr0_cnt0; + tr_req[tr_idx].icnt0 =3D tr1_cnt0; + tr_req[tr_idx].icnt1 =3D 1; + tr_req[tr_idx].dim1 =3D tr1_cnt0; + } + + if (!(flags & DMA_PREP_INTERRUPT)) + period_csf |=3D CPPI5_TR_CSF_SUPR_EVT; + + if (period_csf) + cppi5_tr_csf_set(&tr_req[tr_idx].flags, period_csf); + + period_addr +=3D period_len; + } + + return d; +} + +struct udma_desc * +udma_prep_dma_cyclic_pkt(struct udma_chan *uc, dma_addr_t buf_addr, + size_t buf_len, size_t period_len, + enum dma_transfer_direction dir, unsigned long flags) +{ + struct udma_desc *d; + u32 ring_id; + int i; + int periods =3D buf_len / period_len; + + if (periods > (K3_UDMA_DEFAULT_RING_SIZE - 1)) + return NULL; + + if (period_len >=3D SZ_4M) + return NULL; + + d =3D kzalloc(struct_size(d, hwdesc, periods), GFP_NOWAIT); + if (!d) + return NULL; + + d->hwdesc_count =3D periods; + + /* TODO: re-check this... */ + if (dir =3D=3D DMA_DEV_TO_MEM) + ring_id =3D k3_ringacc_get_ring_id(uc->rflow->r_ring); + else + ring_id =3D k3_ringacc_get_ring_id(uc->tchan->tc_ring); + + if (uc->ud->match_data->type !=3D DMA_TYPE_UDMA) + buf_addr |=3D (u64)uc->config.asel << K3_ADDRESS_ASEL_SHIFT; + + for (i =3D 0; i < periods; i++) { + struct udma_hwdesc *hwdesc =3D &d->hwdesc[i]; + dma_addr_t period_addr =3D buf_addr + (period_len * i); + struct cppi5_host_desc_t *h_desc; + + hwdesc->cppi5_desc_vaddr =3D dma_pool_zalloc(uc->hdesc_pool, + GFP_NOWAIT, + &hwdesc->cppi5_desc_paddr); + if (!hwdesc->cppi5_desc_vaddr) { + dev_err(uc->ud->dev, + "descriptor%d allocation failed\n", i); + + udma_free_hwdesc(uc, d); + kfree(d); + return NULL; + } + + hwdesc->cppi5_desc_size =3D uc->config.hdesc_size; + h_desc =3D hwdesc->cppi5_desc_vaddr; + + cppi5_hdesc_init(h_desc, 0, 0); + cppi5_hdesc_set_pktlen(h_desc, period_len); + + /* Flow and Packed ID */ + cppi5_desc_set_pktids(&h_desc->hdr, uc->id, + CPPI5_INFO1_DESC_FLOWID_DEFAULT); + cppi5_desc_set_retpolicy(&h_desc->hdr, 0, ring_id); + + /* attach each period to a new descriptor */ + cppi5_hdesc_attach_buf(h_desc, + period_addr, period_len, + period_addr, period_len); + } + + return d; +} + +struct dma_async_tx_descriptor * +udma_prep_dma_cyclic(struct dma_chan *chan, dma_addr_t buf_addr, size_t bu= f_len, + size_t period_len, enum dma_transfer_direction dir, + unsigned long flags) +{ + struct udma_chan *uc =3D to_udma_chan(chan); + enum dma_slave_buswidth dev_width; + struct udma_desc *d; + u32 burst; + + if (dir !=3D uc->config.dir) { + dev_err(chan->device->dev, + "%s: chan%d is for %s, not supporting %s\n", + __func__, uc->id, + dmaengine_get_direction_text(uc->config.dir), + dmaengine_get_direction_text(dir)); + return NULL; + } + + uc->cyclic =3D true; + + if (dir =3D=3D DMA_DEV_TO_MEM) { + dev_width =3D uc->cfg.src_addr_width; + burst =3D uc->cfg.src_maxburst; + } else if (dir =3D=3D DMA_MEM_TO_DEV) { + dev_width =3D uc->cfg.dst_addr_width; + burst =3D uc->cfg.dst_maxburst; + } else { + dev_err(uc->ud->dev, "%s: bad direction?\n", __func__); + return NULL; + } + + if (!burst) + burst =3D 1; + + if (uc->config.pkt_mode) + d =3D udma_prep_dma_cyclic_pkt(uc, buf_addr, buf_len, period_len, + dir, flags); + else + d =3D udma_prep_dma_cyclic_tr(uc, buf_addr, buf_len, period_len, + dir, flags); + + if (!d) + return NULL; + + d->sglen =3D buf_len / period_len; + + d->dir =3D dir; + d->residue =3D buf_len; + + /* static TR for remote PDMA */ + if (udma_configure_statictr(uc, d, dev_width, burst)) { + dev_err(uc->ud->dev, + "%s: StaticTR Z is limited to maximum %u (%u)\n", + __func__, uc->ud->match_data->statictr_z_mask, + d->static_tr.bstcnt); + + udma_free_hwdesc(uc, d); + kfree(d); + return NULL; + } + + if (uc->config.metadata_size) + d->vd.tx.metadata_ops =3D &metadata_ops; + + return vchan_tx_prep(&uc->vc, &d->vd, flags); +} + +struct dma_async_tx_descriptor * +udma_prep_dma_memcpy(struct dma_chan *chan, dma_addr_t dest, dma_addr_t sr= c, + size_t len, unsigned long tx_flags) +{ + struct udma_chan *uc =3D to_udma_chan(chan); + struct udma_desc *d; + struct cppi5_tr_type15_t *tr_req; + int num_tr; + size_t tr_size =3D sizeof(struct cppi5_tr_type15_t); + u16 tr0_cnt0, tr0_cnt1, tr1_cnt0; + u32 csf =3D CPPI5_TR_CSF_SUPR_EVT; + + if (uc->config.dir !=3D DMA_MEM_TO_MEM) { + dev_err(chan->device->dev, + "%s: chan%d is for %s, not supporting %s\n", + __func__, uc->id, + dmaengine_get_direction_text(uc->config.dir), + dmaengine_get_direction_text(DMA_MEM_TO_MEM)); + return NULL; + } + + num_tr =3D udma_get_tr_counters(len, __ffs(src | dest), &tr0_cnt0, + &tr0_cnt1, &tr1_cnt0); + if (num_tr < 0) { + dev_err(uc->ud->dev, "size %zu is not supported\n", + len); + return NULL; + } + + d =3D udma_alloc_tr_desc(uc, tr_size, num_tr, DMA_MEM_TO_MEM); + if (!d) + return NULL; + + d->dir =3D DMA_MEM_TO_MEM; + d->desc_idx =3D 0; + d->tr_idx =3D 0; + d->residue =3D len; + + if (uc->ud->match_data->type !=3D DMA_TYPE_UDMA) { + src |=3D (u64)uc->ud->asel << K3_ADDRESS_ASEL_SHIFT; + dest |=3D (u64)uc->ud->asel << K3_ADDRESS_ASEL_SHIFT; + } else { + csf |=3D CPPI5_TR_CSF_EOL_ICNT0; + } + + tr_req =3D d->hwdesc[0].tr_req_base; + + cppi5_tr_init(&tr_req[0].flags, CPPI5_TR_TYPE15, false, true, + CPPI5_TR_EVENT_SIZE_COMPLETION, 0); + cppi5_tr_csf_set(&tr_req[0].flags, csf); + + tr_req[0].addr =3D src; + tr_req[0].icnt0 =3D tr0_cnt0; + tr_req[0].icnt1 =3D tr0_cnt1; + tr_req[0].icnt2 =3D 1; + tr_req[0].icnt3 =3D 1; + tr_req[0].dim1 =3D tr0_cnt0; + + tr_req[0].daddr =3D dest; + tr_req[0].dicnt0 =3D tr0_cnt0; + tr_req[0].dicnt1 =3D tr0_cnt1; + tr_req[0].dicnt2 =3D 1; + tr_req[0].dicnt3 =3D 1; + tr_req[0].ddim1 =3D tr0_cnt0; + + if (num_tr =3D=3D 2) { + cppi5_tr_init(&tr_req[1].flags, CPPI5_TR_TYPE15, false, true, + CPPI5_TR_EVENT_SIZE_COMPLETION, 0); + cppi5_tr_csf_set(&tr_req[1].flags, csf); + + tr_req[1].addr =3D src + tr0_cnt1 * tr0_cnt0; + tr_req[1].icnt0 =3D tr1_cnt0; + tr_req[1].icnt1 =3D 1; + tr_req[1].icnt2 =3D 1; + tr_req[1].icnt3 =3D 1; + + tr_req[1].daddr =3D dest + tr0_cnt1 * tr0_cnt0; + tr_req[1].dicnt0 =3D tr1_cnt0; + tr_req[1].dicnt1 =3D 1; + tr_req[1].dicnt2 =3D 1; + tr_req[1].dicnt3 =3D 1; + } + + cppi5_tr_csf_set(&tr_req[num_tr - 1].flags, csf | CPPI5_TR_CSF_EOP); + + if (uc->config.metadata_size) + d->vd.tx.metadata_ops =3D &metadata_ops; + + return vchan_tx_prep(&uc->vc, &d->vd, tx_flags); +} + +void udma_issue_pending(struct dma_chan *chan) +{ + struct udma_chan *uc =3D to_udma_chan(chan); + struct udma_dev *ud =3D to_udma_dev(chan->device); + unsigned long flags; + + spin_lock_irqsave(&uc->vc.lock, flags); + + /* If we have something pending and no active descriptor, then */ + if (vchan_issue_pending(&uc->vc) && !uc->desc) { + /* + * start a descriptor if the channel is NOT [marked as + * terminating _and_ it is still running (teardown has not + * completed yet)]. + */ + if (!(uc->state =3D=3D UDMA_CHAN_IS_TERMINATING && + udma_is_chan_running(uc))) + ud->udma_start(uc); + } + + spin_unlock_irqrestore(&uc->vc.lock, flags); +} + +int udma_terminate_all(struct dma_chan *chan) +{ + struct udma_chan *uc =3D to_udma_chan(chan); + struct udma_dev *ud =3D to_udma_dev(chan->device); + unsigned long flags; + LIST_HEAD(head); + + spin_lock_irqsave(&uc->vc.lock, flags); + + if (udma_is_chan_running(uc)) + ud->udma_stop(uc); + + if (uc->desc) { + uc->terminated_desc =3D uc->desc; + uc->desc =3D NULL; + uc->terminated_desc->terminated =3D true; + cancel_delayed_work(&uc->tx_drain.work); + } + + uc->paused =3D false; + + vchan_get_all_descriptors(&uc->vc, &head); + spin_unlock_irqrestore(&uc->vc.lock, flags); + vchan_dma_desc_free_list(&uc->vc, &head); + + return 0; +} + +void udma_synchronize(struct dma_chan *chan) +{ + struct udma_chan *uc =3D to_udma_chan(chan); + struct udma_dev *ud =3D to_udma_dev(chan->device); + unsigned long timeout =3D msecs_to_jiffies(1000); + + vchan_synchronize(&uc->vc); + + if (uc->state =3D=3D UDMA_CHAN_IS_TERMINATING) { + timeout =3D wait_for_completion_timeout(&uc->teardown_completed, + timeout); + if (!timeout) { + dev_warn(uc->ud->dev, "chan%d teardown timeout!\n", + uc->id); + udma_dump_chan_stdata(uc); + ud->udma_reset_chan(uc, true); + } + } + + ud->udma_reset_chan(uc, false); + if (udma_is_chan_running(uc)) + dev_warn(uc->ud->dev, "chan%d refused to stop!\n", uc->id); + + cancel_delayed_work_sync(&uc->tx_drain.work); + udma_reset_rings(uc); +} + +void udma_desc_pre_callback(struct virt_dma_chan *vc, + struct virt_dma_desc *vd, + struct dmaengine_result *result) +{ + struct udma_chan *uc =3D to_udma_chan(&vc->chan); + struct udma_desc *d; + u8 status; + + if (!vd) + return; + + d =3D to_udma_desc(&vd->tx); + + if (d->metadata_size) + udma_fetch_epib(uc, d); + + if (result) { + void *desc_vaddr =3D udma_curr_cppi5_desc_vaddr(d, d->desc_idx); + + if (cppi5_desc_get_type(desc_vaddr) =3D=3D + CPPI5_INFO0_DESC_TYPE_VAL_HOST) { + /* Provide residue information for the client */ + result->residue =3D d->residue - + cppi5_hdesc_get_pktlen(desc_vaddr); + if (result->residue) + result->result =3D DMA_TRANS_ABORTED; + else + result->result =3D DMA_TRANS_NOERROR; + } else { + result->residue =3D 0; + /* Propagate TR Response errors to the client */ + status =3D d->hwdesc[0].tr_resp_base->status; + if (status) + result->result =3D DMA_TRANS_ABORTED; + else + result->result =3D DMA_TRANS_NOERROR; + } + } +} + +/* + * This tasklet handles the completion of a DMA descriptor by + * calling its callback and freeing it. + */ +void udma_vchan_complete(struct tasklet_struct *t) +{ + struct virt_dma_chan *vc =3D from_tasklet(vc, t, task); + struct virt_dma_desc *vd, *_vd; + struct dmaengine_desc_callback cb; + LIST_HEAD(head); + + spin_lock_irq(&vc->lock); + list_splice_tail_init(&vc->desc_completed, &head); + vd =3D vc->cyclic; + if (vd) { + vc->cyclic =3D NULL; + dmaengine_desc_get_callback(&vd->tx, &cb); + } else { + memset(&cb, 0, sizeof(cb)); + } + spin_unlock_irq(&vc->lock); + + udma_desc_pre_callback(vc, vd, NULL); + dmaengine_desc_callback_invoke(&cb, NULL); + + list_for_each_entry_safe(vd, _vd, &head, node) { + struct dmaengine_result result; + + dmaengine_desc_get_callback(&vd->tx, &cb); + + list_del(&vd->node); + + udma_desc_pre_callback(vc, vd, &result); + dmaengine_desc_callback_invoke(&cb, &result); + + vchan_vdesc_fini(vd); + } +} + +void udma_free_chan_resources(struct dma_chan *chan) +{ + struct udma_chan *uc =3D to_udma_chan(chan); + struct udma_dev *ud =3D to_udma_dev(chan->device); + + udma_terminate_all(chan); + if (uc->terminated_desc) { + ud->udma_reset_chan(uc, false); + udma_reset_rings(uc); + } + + cancel_delayed_work_sync(&uc->tx_drain.work); + + if (uc->irq_num_ring > 0) { + free_irq(uc->irq_num_ring, uc); + + uc->irq_num_ring =3D 0; + } + if (uc->irq_num_udma > 0) { + free_irq(uc->irq_num_udma, uc); + + uc->irq_num_udma =3D 0; + } + + /* Release PSI-L pairing */ + if (uc->psil_paired) { + navss_psil_unpair(ud, uc->config.src_thread, + uc->config.dst_thread); + uc->psil_paired =3D false; + } + + vchan_free_chan_resources(&uc->vc); + tasklet_kill(&uc->vc.task); + + bcdma_free_bchan_resources(uc); + udma_free_tx_resources(uc); + udma_free_rx_resources(uc); + udma_reset_uchan(uc); + + if (uc->use_dma_pool) { + dma_pool_destroy(uc->hdesc_pool); + uc->use_dma_pool =3D false; + } +} + +int setup_resources(struct udma_dev *ud) +{ + struct device *dev =3D ud->dev; + int ch_count, ret; + + switch (ud->match_data->type) { + case DMA_TYPE_UDMA: + ret =3D udma_setup_resources(ud); + break; + case DMA_TYPE_BCDMA: + ret =3D bcdma_setup_resources(ud); + break; + case DMA_TYPE_PKTDMA: + ret =3D pktdma_setup_resources(ud); + break; + default: + return -EINVAL; + } + + if (ret) + return ret; + + ch_count =3D ud->bchan_cnt + ud->tchan_cnt + ud->rchan_cnt; + if (ud->bchan_cnt) + ch_count -=3D bitmap_weight(ud->bchan_map, ud->bchan_cnt); + ch_count -=3D bitmap_weight(ud->tchan_map, ud->tchan_cnt); + ch_count -=3D bitmap_weight(ud->rchan_map, ud->rchan_cnt); + if (!ch_count) + return -ENODEV; + + ud->channels =3D devm_kcalloc(dev, ch_count, sizeof(*ud->channels), + GFP_KERNEL); + if (!ud->channels) + return -ENOMEM; + + switch (ud->match_data->type) { + case DMA_TYPE_UDMA: + dev_info(dev, + "Channels: %d (tchan: %u, rchan: %u, gp-rflow: %u)\n", + ch_count, + ud->tchan_cnt - bitmap_weight(ud->tchan_map, + ud->tchan_cnt), + ud->rchan_cnt - bitmap_weight(ud->rchan_map, + ud->rchan_cnt), + ud->rflow_cnt - bitmap_weight(ud->rflow_gp_map, + ud->rflow_cnt)); + break; + case DMA_TYPE_BCDMA: + dev_info(dev, + "Channels: %d (bchan: %u, tchan: %u, rchan: %u)\n", + ch_count, + ud->bchan_cnt - bitmap_weight(ud->bchan_map, + ud->bchan_cnt), + ud->tchan_cnt - bitmap_weight(ud->tchan_map, + ud->tchan_cnt), + ud->rchan_cnt - bitmap_weight(ud->rchan_map, + ud->rchan_cnt)); + break; + case DMA_TYPE_PKTDMA: + dev_info(dev, + "Channels: %d (tchan: %u, rchan: %u)\n", + ch_count, + ud->tchan_cnt - bitmap_weight(ud->tchan_map, + ud->tchan_cnt), + ud->rchan_cnt - bitmap_weight(ud->rchan_map, + ud->rchan_cnt)); + break; + default: + break; + } + + return ch_count; +} + +void udma_mark_resource_ranges(struct udma_dev *ud, unsigned long *map, + struct ti_sci_resource_desc *rm_desc, + char *name) +{ + bitmap_clear(map, rm_desc->start, rm_desc->num); + bitmap_clear(map, rm_desc->start_sec, rm_desc->num_sec); + dev_dbg(ud->dev, "ti_sci resource range for %s: %d:%d | %d:%d\n", name, + rm_desc->start, rm_desc->num, rm_desc->start_sec, + rm_desc->num_sec); +} + +int udma_setup_resources(struct udma_dev *ud) +{ + int ret, i, j; + struct device *dev =3D ud->dev; + struct ti_sci_resource *rm_res, irq_res; + struct udma_tisci_rm *tisci_rm =3D &ud->tisci_rm; + u32 cap3; + + /* Set up the throughput level start indexes */ + cap3 =3D udma_read(ud->mmrs[MMR_GCFG], 0x2c); + if (of_device_is_compatible(dev->of_node, + "ti,am654-navss-main-udmap")) { + ud->tchan_tpl.levels =3D 2; + ud->tchan_tpl.start_idx[0] =3D 8; + } else if (of_device_is_compatible(dev->of_node, + "ti,am654-navss-mcu-udmap")) { + ud->tchan_tpl.levels =3D 2; + ud->tchan_tpl.start_idx[0] =3D 2; + } else if (UDMA_CAP3_UCHAN_CNT(cap3)) { + ud->tchan_tpl.levels =3D 3; + ud->tchan_tpl.start_idx[1] =3D UDMA_CAP3_UCHAN_CNT(cap3); + ud->tchan_tpl.start_idx[0] =3D UDMA_CAP3_HCHAN_CNT(cap3); + } else if (UDMA_CAP3_HCHAN_CNT(cap3)) { + ud->tchan_tpl.levels =3D 2; + ud->tchan_tpl.start_idx[0] =3D UDMA_CAP3_HCHAN_CNT(cap3); + } else { + ud->tchan_tpl.levels =3D 1; + } + + ud->rchan_tpl.levels =3D ud->tchan_tpl.levels; + ud->rchan_tpl.start_idx[0] =3D ud->tchan_tpl.start_idx[0]; + ud->rchan_tpl.start_idx[1] =3D ud->tchan_tpl.start_idx[1]; + + ud->tchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->tchan_cnt), + sizeof(unsigned long), GFP_KERNEL); + ud->tchans =3D devm_kcalloc(dev, ud->tchan_cnt, sizeof(*ud->tchans), + GFP_KERNEL); + ud->rchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->rchan_cnt), + sizeof(unsigned long), GFP_KERNEL); + ud->rchans =3D devm_kcalloc(dev, ud->rchan_cnt, sizeof(*ud->rchans), + GFP_KERNEL); + ud->rflow_gp_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->rflow_cnt), + sizeof(unsigned long), + GFP_KERNEL); + ud->rflow_gp_map_allocated =3D devm_kcalloc(dev, + BITS_TO_LONGS(ud->rflow_cnt), + sizeof(unsigned long), + GFP_KERNEL); + ud->rflow_in_use =3D devm_kcalloc(dev, BITS_TO_LONGS(ud->rflow_cnt), + sizeof(unsigned long), + GFP_KERNEL); + ud->rflows =3D devm_kcalloc(dev, ud->rflow_cnt, sizeof(*ud->rflows), + GFP_KERNEL); + + if (!ud->tchan_map || !ud->rchan_map || !ud->rflow_gp_map || + !ud->rflow_gp_map_allocated || !ud->tchans || !ud->rchans || + !ud->rflows || !ud->rflow_in_use) + return -ENOMEM; + + /* + * RX flows with the same Ids as RX channels are reserved to be used + * as default flows if remote HW can't generate flow_ids. Those + * RX flows can be requested only explicitly by id. + */ + bitmap_set(ud->rflow_gp_map_allocated, 0, ud->rchan_cnt); + + /* by default no GP rflows are assigned to Linux */ + bitmap_set(ud->rflow_gp_map, 0, ud->rflow_cnt); + + /* Get resource ranges from tisci */ + for (i =3D 0; i < RM_RANGE_LAST; i++) { + if (i =3D=3D RM_RANGE_BCHAN || i =3D=3D RM_RANGE_TFLOW) + continue; + + tisci_rm->rm_ranges[i] =3D + devm_ti_sci_get_of_resource(tisci_rm->tisci, dev, + tisci_rm->tisci_dev_id, + (char *)range_names[i]); + } + + /* tchan ranges */ + rm_res =3D tisci_rm->rm_ranges[RM_RANGE_TCHAN]; + if (IS_ERR(rm_res)) { + bitmap_zero(ud->tchan_map, ud->tchan_cnt); + irq_res.sets =3D 1; + } else { + bitmap_fill(ud->tchan_map, ud->tchan_cnt); + for (i =3D 0; i < rm_res->sets; i++) + udma_mark_resource_ranges(ud, ud->tchan_map, + &rm_res->desc[i], "tchan"); + irq_res.sets =3D rm_res->sets; + } + + /* rchan and matching default flow ranges */ + rm_res =3D tisci_rm->rm_ranges[RM_RANGE_RCHAN]; + if (IS_ERR(rm_res)) { + bitmap_zero(ud->rchan_map, ud->rchan_cnt); + irq_res.sets++; + } else { + bitmap_fill(ud->rchan_map, ud->rchan_cnt); + for (i =3D 0; i < rm_res->sets; i++) + udma_mark_resource_ranges(ud, ud->rchan_map, + &rm_res->desc[i], "rchan"); + irq_res.sets +=3D rm_res->sets; + } + + irq_res.desc =3D kcalloc(irq_res.sets, sizeof(*irq_res.desc), GFP_KERNEL); + if (!irq_res.desc) + return -ENOMEM; + rm_res =3D tisci_rm->rm_ranges[RM_RANGE_TCHAN]; + if (IS_ERR(rm_res)) { + irq_res.desc[0].start =3D 0; + irq_res.desc[0].num =3D ud->tchan_cnt; + i =3D 1; + } else { + for (i =3D 0; i < rm_res->sets; i++) { + irq_res.desc[i].start =3D rm_res->desc[i].start; + irq_res.desc[i].num =3D rm_res->desc[i].num; + irq_res.desc[i].start_sec =3D rm_res->desc[i].start_sec; + irq_res.desc[i].num_sec =3D rm_res->desc[i].num_sec; + } + } + rm_res =3D tisci_rm->rm_ranges[RM_RANGE_RCHAN]; + if (IS_ERR(rm_res)) { + irq_res.desc[i].start =3D 0; + irq_res.desc[i].num =3D ud->rchan_cnt; + } else { + for (j =3D 0; j < rm_res->sets; j++, i++) { + if (rm_res->desc[j].num) { + irq_res.desc[i].start =3D rm_res->desc[j].start + + ud->soc_data->oes.udma_rchan; + irq_res.desc[i].num =3D rm_res->desc[j].num; + } + if (rm_res->desc[j].num_sec) { + irq_res.desc[i].start_sec =3D rm_res->desc[j].start_sec + + ud->soc_data->oes.udma_rchan; + irq_res.desc[i].num_sec =3D rm_res->desc[j].num_sec; + } + } + } + ret =3D ti_sci_inta_msi_domain_alloc_irqs(ud->dev, &irq_res); + kfree(irq_res.desc); + if (ret) { + dev_err(ud->dev, "Failed to allocate MSI interrupts\n"); + return ret; + } + + /* GP rflow ranges */ + rm_res =3D tisci_rm->rm_ranges[RM_RANGE_RFLOW]; + if (IS_ERR(rm_res)) { + /* all gp flows are assigned exclusively to Linux */ + bitmap_clear(ud->rflow_gp_map, ud->rchan_cnt, + ud->rflow_cnt - ud->rchan_cnt); + } else { + for (i =3D 0; i < rm_res->sets; i++) + udma_mark_resource_ranges(ud, ud->rflow_gp_map, + &rm_res->desc[i], "gp-rflow"); + } + + return 0; +} + +int bcdma_setup_resources(struct udma_dev *ud) +{ + int ret, i, j; + struct device *dev =3D ud->dev; + struct ti_sci_resource *rm_res, irq_res; + struct udma_tisci_rm *tisci_rm =3D &ud->tisci_rm; + const struct udma_oes_offsets *oes =3D &ud->soc_data->oes; + u32 cap; + + /* Set up the throughput level start indexes */ + cap =3D udma_read(ud->mmrs[MMR_GCFG], 0x2c); + if (BCDMA_CAP3_UBCHAN_CNT(cap)) { + ud->bchan_tpl.levels =3D 3; + ud->bchan_tpl.start_idx[1] =3D BCDMA_CAP3_UBCHAN_CNT(cap); + ud->bchan_tpl.start_idx[0] =3D BCDMA_CAP3_HBCHAN_CNT(cap); + } else if (BCDMA_CAP3_HBCHAN_CNT(cap)) { + ud->bchan_tpl.levels =3D 2; + ud->bchan_tpl.start_idx[0] =3D BCDMA_CAP3_HBCHAN_CNT(cap); + } else { + ud->bchan_tpl.levels =3D 1; + } + + cap =3D udma_read(ud->mmrs[MMR_GCFG], 0x30); + if (BCDMA_CAP4_URCHAN_CNT(cap)) { + ud->rchan_tpl.levels =3D 3; + ud->rchan_tpl.start_idx[1] =3D BCDMA_CAP4_URCHAN_CNT(cap); + ud->rchan_tpl.start_idx[0] =3D BCDMA_CAP4_HRCHAN_CNT(cap); + } else if (BCDMA_CAP4_HRCHAN_CNT(cap)) { + ud->rchan_tpl.levels =3D 2; + ud->rchan_tpl.start_idx[0] =3D BCDMA_CAP4_HRCHAN_CNT(cap); + } else { + ud->rchan_tpl.levels =3D 1; + } + + if (BCDMA_CAP4_UTCHAN_CNT(cap)) { + ud->tchan_tpl.levels =3D 3; + ud->tchan_tpl.start_idx[1] =3D BCDMA_CAP4_UTCHAN_CNT(cap); + ud->tchan_tpl.start_idx[0] =3D BCDMA_CAP4_HTCHAN_CNT(cap); + } else if (BCDMA_CAP4_HTCHAN_CNT(cap)) { + ud->tchan_tpl.levels =3D 2; + ud->tchan_tpl.start_idx[0] =3D BCDMA_CAP4_HTCHAN_CNT(cap); + } else { + ud->tchan_tpl.levels =3D 1; + } + + ud->bchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->bchan_cnt), + sizeof(unsigned long), GFP_KERNEL); + ud->bchans =3D devm_kcalloc(dev, ud->bchan_cnt, sizeof(*ud->bchans), + GFP_KERNEL); + ud->tchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->tchan_cnt), + sizeof(unsigned long), GFP_KERNEL); + ud->tchans =3D devm_kcalloc(dev, ud->tchan_cnt, sizeof(*ud->tchans), + GFP_KERNEL); + ud->rchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->rchan_cnt), + sizeof(unsigned long), GFP_KERNEL); + ud->rchans =3D devm_kcalloc(dev, ud->rchan_cnt, sizeof(*ud->rchans), + GFP_KERNEL); + /* BCDMA do not really have flows, but the driver expect it */ + ud->rflow_in_use =3D devm_kcalloc(dev, BITS_TO_LONGS(ud->rchan_cnt), + sizeof(unsigned long), + GFP_KERNEL); + ud->rflows =3D devm_kcalloc(dev, ud->rchan_cnt, sizeof(*ud->rflows), + GFP_KERNEL); + + if (!ud->bchan_map || !ud->tchan_map || !ud->rchan_map || + !ud->rflow_in_use || !ud->bchans || !ud->tchans || !ud->rchans || + !ud->rflows) + return -ENOMEM; + + /* Get resource ranges from tisci */ + for (i =3D 0; i < RM_RANGE_LAST; i++) { + if (i =3D=3D RM_RANGE_RFLOW || i =3D=3D RM_RANGE_TFLOW) + continue; + if (i =3D=3D RM_RANGE_BCHAN && ud->bchan_cnt =3D=3D 0) + continue; + if (i =3D=3D RM_RANGE_TCHAN && ud->tchan_cnt =3D=3D 0) + continue; + if (i =3D=3D RM_RANGE_RCHAN && ud->rchan_cnt =3D=3D 0) + continue; + + tisci_rm->rm_ranges[i] =3D + devm_ti_sci_get_of_resource(tisci_rm->tisci, dev, + tisci_rm->tisci_dev_id, + (char *)range_names[i]); + } + + irq_res.sets =3D 0; + + /* bchan ranges */ + if (ud->bchan_cnt) { + rm_res =3D tisci_rm->rm_ranges[RM_RANGE_BCHAN]; + if (IS_ERR(rm_res)) { + bitmap_zero(ud->bchan_map, ud->bchan_cnt); + irq_res.sets++; + } else { + bitmap_fill(ud->bchan_map, ud->bchan_cnt); + for (i =3D 0; i < rm_res->sets; i++) + udma_mark_resource_ranges(ud, ud->bchan_map, + &rm_res->desc[i], + "bchan"); + irq_res.sets +=3D rm_res->sets; + } + } + + /* tchan ranges */ + if (ud->tchan_cnt) { + rm_res =3D tisci_rm->rm_ranges[RM_RANGE_TCHAN]; + if (IS_ERR(rm_res)) { + bitmap_zero(ud->tchan_map, ud->tchan_cnt); + irq_res.sets +=3D 2; + } else { + bitmap_fill(ud->tchan_map, ud->tchan_cnt); + for (i =3D 0; i < rm_res->sets; i++) + udma_mark_resource_ranges(ud, ud->tchan_map, + &rm_res->desc[i], + "tchan"); + irq_res.sets +=3D rm_res->sets * 2; + } + } + + /* rchan ranges */ + if (ud->rchan_cnt) { + rm_res =3D tisci_rm->rm_ranges[RM_RANGE_RCHAN]; + if (IS_ERR(rm_res)) { + bitmap_zero(ud->rchan_map, ud->rchan_cnt); + irq_res.sets +=3D 2; + } else { + bitmap_fill(ud->rchan_map, ud->rchan_cnt); + for (i =3D 0; i < rm_res->sets; i++) + udma_mark_resource_ranges(ud, ud->rchan_map, + &rm_res->desc[i], + "rchan"); + irq_res.sets +=3D rm_res->sets * 2; + } + } + + irq_res.desc =3D kcalloc(irq_res.sets, sizeof(*irq_res.desc), GFP_KERNEL); + if (!irq_res.desc) + return -ENOMEM; + if (ud->bchan_cnt) { + rm_res =3D tisci_rm->rm_ranges[RM_RANGE_BCHAN]; + if (IS_ERR(rm_res)) { + irq_res.desc[0].start =3D oes->bcdma_bchan_ring; + irq_res.desc[0].num =3D ud->bchan_cnt; + i =3D 1; + } else { + for (i =3D 0; i < rm_res->sets; i++) { + irq_res.desc[i].start =3D rm_res->desc[i].start + + oes->bcdma_bchan_ring; + irq_res.desc[i].num =3D rm_res->desc[i].num; + + if (rm_res->desc[i].num_sec) { + irq_res.desc[i].start_sec =3D rm_res->desc[i].start_sec + + oes->bcdma_bchan_ring; + irq_res.desc[i].num_sec =3D rm_res->desc[i].num_sec; + } + } + } + } else { + i =3D 0; + } + + if (ud->tchan_cnt) { + rm_res =3D tisci_rm->rm_ranges[RM_RANGE_TCHAN]; + if (IS_ERR(rm_res)) { + irq_res.desc[i].start =3D oes->bcdma_tchan_data; + irq_res.desc[i].num =3D ud->tchan_cnt; + irq_res.desc[i + 1].start =3D oes->bcdma_tchan_ring; + irq_res.desc[i + 1].num =3D ud->tchan_cnt; + i +=3D 2; + } else { + for (j =3D 0; j < rm_res->sets; j++, i +=3D 2) { + irq_res.desc[i].start =3D rm_res->desc[j].start + + oes->bcdma_tchan_data; + irq_res.desc[i].num =3D rm_res->desc[j].num; + + irq_res.desc[i + 1].start =3D rm_res->desc[j].start + + oes->bcdma_tchan_ring; + irq_res.desc[i + 1].num =3D rm_res->desc[j].num; + + if (rm_res->desc[j].num_sec) { + irq_res.desc[i].start_sec =3D rm_res->desc[j].start_sec + + oes->bcdma_tchan_data; + irq_res.desc[i].num_sec =3D rm_res->desc[j].num_sec; + irq_res.desc[i + 1].start_sec =3D rm_res->desc[j].start_sec + + oes->bcdma_tchan_ring; + irq_res.desc[i + 1].num_sec =3D rm_res->desc[j].num_sec; + } + } + } + } + if (ud->rchan_cnt) { + rm_res =3D tisci_rm->rm_ranges[RM_RANGE_RCHAN]; + if (IS_ERR(rm_res)) { + irq_res.desc[i].start =3D oes->bcdma_rchan_data; + irq_res.desc[i].num =3D ud->rchan_cnt; + irq_res.desc[i + 1].start =3D oes->bcdma_rchan_ring; + irq_res.desc[i + 1].num =3D ud->rchan_cnt; + i +=3D 2; + } else { + for (j =3D 0; j < rm_res->sets; j++, i +=3D 2) { + irq_res.desc[i].start =3D rm_res->desc[j].start + + oes->bcdma_rchan_data; + irq_res.desc[i].num =3D rm_res->desc[j].num; + + irq_res.desc[i + 1].start =3D rm_res->desc[j].start + + oes->bcdma_rchan_ring; + irq_res.desc[i + 1].num =3D rm_res->desc[j].num; + + if (rm_res->desc[j].num_sec) { + irq_res.desc[i].start_sec =3D rm_res->desc[j].start_sec + + oes->bcdma_rchan_data; + irq_res.desc[i].num_sec =3D rm_res->desc[j].num_sec; + irq_res.desc[i + 1].start_sec =3D rm_res->desc[j].start_sec + + oes->bcdma_rchan_ring; + irq_res.desc[i + 1].num_sec =3D rm_res->desc[j].num_sec; + } + } + } + } + + ret =3D ti_sci_inta_msi_domain_alloc_irqs(ud->dev, &irq_res); + kfree(irq_res.desc); + if (ret) { + dev_err(ud->dev, "Failed to allocate MSI interrupts\n"); + return ret; + } + + return 0; +} + +int pktdma_setup_resources(struct udma_dev *ud) +{ + int ret, i, j; + struct device *dev =3D ud->dev; + struct ti_sci_resource *rm_res, irq_res; + struct udma_tisci_rm *tisci_rm =3D &ud->tisci_rm; + const struct udma_oes_offsets *oes =3D &ud->soc_data->oes; + u32 cap3; + + /* Set up the throughput level start indexes */ + cap3 =3D udma_read(ud->mmrs[MMR_GCFG], 0x2c); + if (UDMA_CAP3_UCHAN_CNT(cap3)) { + ud->tchan_tpl.levels =3D 3; + ud->tchan_tpl.start_idx[1] =3D UDMA_CAP3_UCHAN_CNT(cap3); + ud->tchan_tpl.start_idx[0] =3D UDMA_CAP3_HCHAN_CNT(cap3); + } else if (UDMA_CAP3_HCHAN_CNT(cap3)) { + ud->tchan_tpl.levels =3D 2; + ud->tchan_tpl.start_idx[0] =3D UDMA_CAP3_HCHAN_CNT(cap3); + } else { + ud->tchan_tpl.levels =3D 1; + } + + ud->rchan_tpl.levels =3D ud->tchan_tpl.levels; + ud->rchan_tpl.start_idx[0] =3D ud->tchan_tpl.start_idx[0]; + ud->rchan_tpl.start_idx[1] =3D ud->tchan_tpl.start_idx[1]; + + ud->tchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->tchan_cnt), + sizeof(unsigned long), GFP_KERNEL); + ud->tchans =3D devm_kcalloc(dev, ud->tchan_cnt, sizeof(*ud->tchans), + GFP_KERNEL); + ud->rchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->rchan_cnt), + sizeof(unsigned long), GFP_KERNEL); + ud->rchans =3D devm_kcalloc(dev, ud->rchan_cnt, sizeof(*ud->rchans), + GFP_KERNEL); + ud->rflow_in_use =3D devm_kcalloc(dev, BITS_TO_LONGS(ud->rflow_cnt), + sizeof(unsigned long), + GFP_KERNEL); + ud->rflows =3D devm_kcalloc(dev, ud->rflow_cnt, sizeof(*ud->rflows), + GFP_KERNEL); + ud->tflow_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->tflow_cnt), + sizeof(unsigned long), GFP_KERNEL); + + if (!ud->tchan_map || !ud->rchan_map || !ud->tflow_map || !ud->tchans || + !ud->rchans || !ud->rflows || !ud->rflow_in_use) + return -ENOMEM; + + /* Get resource ranges from tisci */ + for (i =3D 0; i < RM_RANGE_LAST; i++) { + if (i =3D=3D RM_RANGE_BCHAN) + continue; + + tisci_rm->rm_ranges[i] =3D + devm_ti_sci_get_of_resource(tisci_rm->tisci, dev, + tisci_rm->tisci_dev_id, + (char *)range_names[i]); + } + + /* tchan ranges */ + rm_res =3D tisci_rm->rm_ranges[RM_RANGE_TCHAN]; + if (IS_ERR(rm_res)) { + bitmap_zero(ud->tchan_map, ud->tchan_cnt); + } else { + bitmap_fill(ud->tchan_map, ud->tchan_cnt); + for (i =3D 0; i < rm_res->sets; i++) + udma_mark_resource_ranges(ud, ud->tchan_map, + &rm_res->desc[i], "tchan"); + } + + /* rchan ranges */ + rm_res =3D tisci_rm->rm_ranges[RM_RANGE_RCHAN]; + if (IS_ERR(rm_res)) { + bitmap_zero(ud->rchan_map, ud->rchan_cnt); + } else { + bitmap_fill(ud->rchan_map, ud->rchan_cnt); + for (i =3D 0; i < rm_res->sets; i++) + udma_mark_resource_ranges(ud, ud->rchan_map, + &rm_res->desc[i], "rchan"); + } + + /* rflow ranges */ + rm_res =3D tisci_rm->rm_ranges[RM_RANGE_RFLOW]; + if (IS_ERR(rm_res)) { + /* all rflows are assigned exclusively to Linux */ + bitmap_zero(ud->rflow_in_use, ud->rflow_cnt); + irq_res.sets =3D 1; + } else { + bitmap_fill(ud->rflow_in_use, ud->rflow_cnt); + for (i =3D 0; i < rm_res->sets; i++) + udma_mark_resource_ranges(ud, ud->rflow_in_use, + &rm_res->desc[i], "rflow"); + irq_res.sets =3D rm_res->sets; + } + + /* tflow ranges */ + rm_res =3D tisci_rm->rm_ranges[RM_RANGE_TFLOW]; + if (IS_ERR(rm_res)) { + /* all tflows are assigned exclusively to Linux */ + bitmap_zero(ud->tflow_map, ud->tflow_cnt); + irq_res.sets++; + } else { + bitmap_fill(ud->tflow_map, ud->tflow_cnt); + for (i =3D 0; i < rm_res->sets; i++) + udma_mark_resource_ranges(ud, ud->tflow_map, + &rm_res->desc[i], "tflow"); + irq_res.sets +=3D rm_res->sets; + } + + irq_res.desc =3D kcalloc(irq_res.sets, sizeof(*irq_res.desc), GFP_KERNEL); + if (!irq_res.desc) + return -ENOMEM; + rm_res =3D tisci_rm->rm_ranges[RM_RANGE_TFLOW]; + if (IS_ERR(rm_res)) { + irq_res.desc[0].start =3D oes->pktdma_tchan_flow; + irq_res.desc[0].num =3D ud->tflow_cnt; + i =3D 1; + } else { + for (i =3D 0; i < rm_res->sets; i++) { + irq_res.desc[i].start =3D rm_res->desc[i].start + + oes->pktdma_tchan_flow; + irq_res.desc[i].num =3D rm_res->desc[i].num; + + if (rm_res->desc[i].num_sec) { + irq_res.desc[i].start_sec =3D rm_res->desc[i].start_sec + + oes->pktdma_tchan_flow; + irq_res.desc[i].num_sec =3D rm_res->desc[i].num_sec; + } + } + } + rm_res =3D tisci_rm->rm_ranges[RM_RANGE_RFLOW]; + if (IS_ERR(rm_res)) { + irq_res.desc[i].start =3D oes->pktdma_rchan_flow; + irq_res.desc[i].num =3D ud->rflow_cnt; + } else { + for (j =3D 0; j < rm_res->sets; j++, i++) { + irq_res.desc[i].start =3D rm_res->desc[j].start + + oes->pktdma_rchan_flow; + irq_res.desc[i].num =3D rm_res->desc[j].num; + + if (rm_res->desc[j].num_sec) { + irq_res.desc[i].start_sec =3D rm_res->desc[j].start_sec + + oes->pktdma_rchan_flow; + irq_res.desc[i].num_sec =3D rm_res->desc[j].num_sec; + } + } + } + ret =3D ti_sci_inta_msi_domain_alloc_irqs(ud->dev, &irq_res); + kfree(irq_res.desc); + if (ret) { + dev_err(ud->dev, "Failed to allocate MSI interrupts\n"); + return ret; + } + + return 0; +} + +int udma_setup_rx_flush(struct udma_dev *ud) +{ + struct udma_rx_flush *rx_flush =3D &ud->rx_flush; + struct cppi5_desc_hdr_t *tr_desc; + struct cppi5_tr_type1_t *tr_req; + struct cppi5_host_desc_t *desc; + struct device *dev =3D ud->dev; + struct udma_hwdesc *hwdesc; + size_t tr_size; + + /* Allocate 1K buffer for discarded data on RX channel teardown */ + rx_flush->buffer_size =3D SZ_1K; + rx_flush->buffer_vaddr =3D devm_kzalloc(dev, rx_flush->buffer_size, + GFP_KERNEL); + if (!rx_flush->buffer_vaddr) + return -ENOMEM; + + rx_flush->buffer_paddr =3D dma_map_single(dev, rx_flush->buffer_vaddr, + rx_flush->buffer_size, + DMA_TO_DEVICE); + if (dma_mapping_error(dev, rx_flush->buffer_paddr)) + return -ENOMEM; + + /* Set up descriptor to be used for TR mode */ + hwdesc =3D &rx_flush->hwdescs[0]; + tr_size =3D sizeof(struct cppi5_tr_type1_t); + hwdesc->cppi5_desc_size =3D cppi5_trdesc_calc_size(tr_size, 1); + hwdesc->cppi5_desc_size =3D ALIGN(hwdesc->cppi5_desc_size, + ud->desc_align); + + hwdesc->cppi5_desc_vaddr =3D devm_kzalloc(dev, hwdesc->cppi5_desc_size, + GFP_KERNEL); + if (!hwdesc->cppi5_desc_vaddr) + return -ENOMEM; + + hwdesc->cppi5_desc_paddr =3D dma_map_single(dev, hwdesc->cppi5_desc_vaddr, + hwdesc->cppi5_desc_size, + DMA_TO_DEVICE); + if (dma_mapping_error(dev, hwdesc->cppi5_desc_paddr)) + return -ENOMEM; + + /* Start of the TR req records */ + hwdesc->tr_req_base =3D hwdesc->cppi5_desc_vaddr + tr_size; + /* Start address of the TR response array */ + hwdesc->tr_resp_base =3D hwdesc->tr_req_base + tr_size; + + tr_desc =3D hwdesc->cppi5_desc_vaddr; + cppi5_trdesc_init(tr_desc, 1, tr_size, 0, 0); + cppi5_desc_set_pktids(tr_desc, 0, CPPI5_INFO1_DESC_FLOWID_DEFAULT); + cppi5_desc_set_retpolicy(tr_desc, 0, 0); + + tr_req =3D hwdesc->tr_req_base; + cppi5_tr_init(&tr_req->flags, CPPI5_TR_TYPE1, false, false, + CPPI5_TR_EVENT_SIZE_COMPLETION, 0); + cppi5_tr_csf_set(&tr_req->flags, CPPI5_TR_CSF_SUPR_EVT); + + tr_req->addr =3D rx_flush->buffer_paddr; + tr_req->icnt0 =3D rx_flush->buffer_size; + tr_req->icnt1 =3D 1; + + dma_sync_single_for_device(dev, hwdesc->cppi5_desc_paddr, + hwdesc->cppi5_desc_size, DMA_TO_DEVICE); + + /* Set up descriptor to be used for packet mode */ + hwdesc =3D &rx_flush->hwdescs[1]; + hwdesc->cppi5_desc_size =3D ALIGN(sizeof(struct cppi5_host_desc_t) + + CPPI5_INFO0_HDESC_EPIB_SIZE + + CPPI5_INFO0_HDESC_PSDATA_MAX_SIZE, + ud->desc_align); + + hwdesc->cppi5_desc_vaddr =3D devm_kzalloc(dev, hwdesc->cppi5_desc_size, + GFP_KERNEL); + if (!hwdesc->cppi5_desc_vaddr) + return -ENOMEM; + + hwdesc->cppi5_desc_paddr =3D dma_map_single(dev, hwdesc->cppi5_desc_vaddr, + hwdesc->cppi5_desc_size, + DMA_TO_DEVICE); + if (dma_mapping_error(dev, hwdesc->cppi5_desc_paddr)) + return -ENOMEM; + + desc =3D hwdesc->cppi5_desc_vaddr; + cppi5_hdesc_init(desc, 0, 0); + cppi5_desc_set_pktids(&desc->hdr, 0, CPPI5_INFO1_DESC_FLOWID_DEFAULT); + cppi5_desc_set_retpolicy(&desc->hdr, 0, 0); + + cppi5_hdesc_attach_buf(desc, + rx_flush->buffer_paddr, rx_flush->buffer_size, + rx_flush->buffer_paddr, rx_flush->buffer_size); + + dma_sync_single_for_device(dev, hwdesc->cppi5_desc_paddr, + hwdesc->cppi5_desc_size, DMA_TO_DEVICE); + return 0; +} + +#ifdef CONFIG_DEBUG_FS +void udma_dbg_summary_show_chan(struct seq_file *s, + struct dma_chan *chan) +{ + struct udma_chan *uc =3D to_udma_chan(chan); + struct udma_chan_config *ucc =3D &uc->config; + + seq_printf(s, " %-13s| %s", dma_chan_name(chan), + chan->dbg_client_name ?: "in-use"); + if (ucc->tr_trigger_type) + seq_puts(s, " (triggered, "); + else + seq_printf(s, " (%s, ", + dmaengine_get_direction_text(uc->config.dir)); + + switch (uc->config.dir) { + case DMA_MEM_TO_MEM: + if (uc->ud->match_data->type =3D=3D DMA_TYPE_BCDMA) { + seq_printf(s, "bchan%d)\n", uc->bchan->id); + return; + } + + seq_printf(s, "chan%d pair [0x%04x -> 0x%04x], ", uc->tchan->id, + ucc->src_thread, ucc->dst_thread); + break; + case DMA_DEV_TO_MEM: + seq_printf(s, "rchan%d [0x%04x -> 0x%04x], ", uc->rchan->id, + ucc->src_thread, ucc->dst_thread); + if (uc->ud->match_data->type =3D=3D DMA_TYPE_PKTDMA) + seq_printf(s, "rflow%d, ", uc->rflow->id); + break; + case DMA_MEM_TO_DEV: + seq_printf(s, "tchan%d [0x%04x -> 0x%04x], ", uc->tchan->id, + ucc->src_thread, ucc->dst_thread); + if (uc->ud->match_data->type =3D=3D DMA_TYPE_PKTDMA) + seq_printf(s, "tflow%d, ", uc->tchan->tflow_id); + break; + default: + seq_puts(s, ")\n"); + return; + } + + if (ucc->ep_type =3D=3D PSIL_EP_NATIVE) { + seq_puts(s, "PSI-L Native"); + if (ucc->metadata_size) { + seq_printf(s, "[%s", ucc->needs_epib ? " EPIB" : ""); + if (ucc->psd_size) + seq_printf(s, " PSDsize:%u", ucc->psd_size); + seq_puts(s, " ]"); + } + } else { + seq_puts(s, "PDMA"); + if (ucc->enable_acc32 || ucc->enable_burst) + seq_printf(s, "[%s%s ]", + ucc->enable_acc32 ? " ACC32" : "", + ucc->enable_burst ? " BURST" : ""); + } + + seq_printf(s, ", %s)\n", ucc->pkt_mode ? "Packet mode" : "TR mode"); +} + +void udma_dbg_summary_show(struct seq_file *s, + struct dma_device *dma_dev) +{ + struct dma_chan *chan; + + list_for_each_entry(chan, &dma_dev->channels, device_node) { + if (chan->client_count) + udma_dbg_summary_show_chan(s, chan); + } +} +#endif /* CONFIG_DEBUG_FS */ + +enum dmaengine_alignment udma_get_copy_align(struct udma_dev *ud) +{ + const struct udma_match_data *match_data =3D ud->match_data; + u8 tpl; + + if (!match_data->enable_memcpy_support) + return DMAENGINE_ALIGN_8_BYTES; + + /* Get the highest TPL level the device supports for memcpy */ + if (ud->bchan_cnt) + tpl =3D udma_get_chan_tpl_index(&ud->bchan_tpl, 0); + else if (ud->tchan_cnt) + tpl =3D udma_get_chan_tpl_index(&ud->tchan_tpl, 0); + else + return DMAENGINE_ALIGN_8_BYTES; + + switch (match_data->burst_size[tpl]) { + case TI_SCI_RM_UDMAP_CHAN_BURST_SIZE_256_BYTES: + return DMAENGINE_ALIGN_256_BYTES; + case TI_SCI_RM_UDMAP_CHAN_BURST_SIZE_128_BYTES: + return DMAENGINE_ALIGN_128_BYTES; + case TI_SCI_RM_UDMAP_CHAN_BURST_SIZE_64_BYTES: + fallthrough; + default: + return DMAENGINE_ALIGN_64_BYTES; + } +} + +/* Private interfaces to UDMA */ +#include "k3-udma-private.c" diff --git a/drivers/dma/ti/k3-udma.c b/drivers/dma/ti/k3-udma.c index b223a7aacb0cf..4bea821bf1262 100644 --- a/drivers/dma/ti/k3-udma.c +++ b/drivers/dma/ti/k3-udma.c @@ -33,43 +33,6 @@ #include "k3-udma.h" #include "k3-psil-priv.h" =20 -struct udma_static_tr { - u8 elsize; /* RPSTR0 */ - u16 elcnt; /* RPSTR0 */ - u16 bstcnt; /* RPSTR1 */ -}; - -#define K3_UDMA_MAX_RFLOWS 1024 -#define K3_UDMA_DEFAULT_RING_SIZE 16 - -/* How SRC/DST tag should be updated by UDMA in the descriptor's Word 3 */ -#define UDMA_RFLOW_SRCTAG_NONE 0 -#define UDMA_RFLOW_SRCTAG_CFG_TAG 1 -#define UDMA_RFLOW_SRCTAG_FLOW_ID 2 -#define UDMA_RFLOW_SRCTAG_SRC_TAG 4 - -#define UDMA_RFLOW_DSTTAG_NONE 0 -#define UDMA_RFLOW_DSTTAG_CFG_TAG 1 -#define UDMA_RFLOW_DSTTAG_FLOW_ID 2 -#define UDMA_RFLOW_DSTTAG_DST_TAG_LO 4 -#define UDMA_RFLOW_DSTTAG_DST_TAG_HI 5 - -struct udma_chan; - -enum k3_dma_type { - DMA_TYPE_UDMA =3D 0, - DMA_TYPE_BCDMA, - DMA_TYPE_PKTDMA, -}; - -enum udma_mmr { - MMR_GCFG =3D 0, - MMR_BCHANRT, - MMR_RCHANRT, - MMR_TCHANRT, - MMR_LAST, -}; - static const char * const mmr_names[] =3D { [MMR_GCFG] =3D "gcfg", [MMR_BCHANRT] =3D "bchanrt", @@ -77,329 +40,7 @@ static const char * const mmr_names[] =3D { [MMR_TCHANRT] =3D "tchanrt", }; =20 -struct udma_tchan { - void __iomem *reg_rt; - - int id; - struct k3_ring *t_ring; /* Transmit ring */ - struct k3_ring *tc_ring; /* Transmit Completion ring */ - int tflow_id; /* applicable only for PKTDMA */ - -}; - -#define udma_bchan udma_tchan - -struct udma_rflow { - int id; - struct k3_ring *fd_ring; /* Free Descriptor ring */ - struct k3_ring *r_ring; /* Receive ring */ -}; - -struct udma_rchan { - void __iomem *reg_rt; - - int id; -}; - -struct udma_oes_offsets { - /* K3 UDMA Output Event Offset */ - u32 udma_rchan; - - /* BCDMA Output Event Offsets */ - u32 bcdma_bchan_data; - u32 bcdma_bchan_ring; - u32 bcdma_tchan_data; - u32 bcdma_tchan_ring; - u32 bcdma_rchan_data; - u32 bcdma_rchan_ring; - - /* PKTDMA Output Event Offsets */ - u32 pktdma_tchan_flow; - u32 pktdma_rchan_flow; -}; - -#define UDMA_FLAG_PDMA_ACC32 BIT(0) -#define UDMA_FLAG_PDMA_BURST BIT(1) -#define UDMA_FLAG_TDTYPE BIT(2) -#define UDMA_FLAG_BURST_SIZE BIT(3) -#define UDMA_FLAGS_J7_CLASS (UDMA_FLAG_PDMA_ACC32 | \ - UDMA_FLAG_PDMA_BURST | \ - UDMA_FLAG_TDTYPE | \ - UDMA_FLAG_BURST_SIZE) - -struct udma_match_data { - enum k3_dma_type type; - u32 psil_base; - bool enable_memcpy_support; - u32 flags; - u32 statictr_z_mask; - u8 burst_size[3]; - struct udma_soc_data *soc_data; -}; - -struct udma_soc_data { - struct udma_oes_offsets oes; - u32 bcdma_trigger_event_offset; -}; - -struct udma_hwdesc { - size_t cppi5_desc_size; - void *cppi5_desc_vaddr; - dma_addr_t cppi5_desc_paddr; - - /* TR descriptor internal pointers */ - void *tr_req_base; - struct cppi5_tr_resp_t *tr_resp_base; -}; - -struct udma_rx_flush { - struct udma_hwdesc hwdescs[2]; - - size_t buffer_size; - void *buffer_vaddr; - dma_addr_t buffer_paddr; -}; - -struct udma_tpl { - u8 levels; - u32 start_idx[3]; -}; - -struct udma_dev { - struct dma_device ddev; - struct device *dev; - void __iomem *mmrs[MMR_LAST]; - const struct udma_match_data *match_data; - const struct udma_soc_data *soc_data; - - struct udma_tpl bchan_tpl; - struct udma_tpl tchan_tpl; - struct udma_tpl rchan_tpl; - - size_t desc_align; /* alignment to use for descriptors */ - - struct udma_tisci_rm tisci_rm; - - struct k3_ringacc *ringacc; - - struct work_struct purge_work; - struct list_head desc_to_purge; - spinlock_t lock; - - struct udma_rx_flush rx_flush; - - int bchan_cnt; - int tchan_cnt; - int echan_cnt; - int rchan_cnt; - int rflow_cnt; - int tflow_cnt; - unsigned long *bchan_map; - unsigned long *tchan_map; - unsigned long *rchan_map; - unsigned long *rflow_gp_map; - unsigned long *rflow_gp_map_allocated; - unsigned long *rflow_in_use; - unsigned long *tflow_map; - - struct udma_bchan *bchans; - struct udma_tchan *tchans; - struct udma_rchan *rchans; - struct udma_rflow *rflows; - - struct udma_chan *channels; - u32 psil_base; - u32 atype; - u32 asel; -}; - -struct udma_desc { - struct virt_dma_desc vd; - - bool terminated; - - enum dma_transfer_direction dir; - - struct udma_static_tr static_tr; - u32 residue; - - unsigned int sglen; - unsigned int desc_idx; /* Only used for cyclic in packet mode */ - unsigned int tr_idx; - - u32 metadata_size; - void *metadata; /* pointer to provided metadata buffer (EPIP, PSdata) */ - - unsigned int hwdesc_count; - struct udma_hwdesc hwdesc[]; -}; - -enum udma_chan_state { - UDMA_CHAN_IS_IDLE =3D 0, /* not active, no teardown is in progress */ - UDMA_CHAN_IS_ACTIVE, /* Normal operation */ - UDMA_CHAN_IS_TERMINATING, /* channel is being terminated */ -}; - -struct udma_tx_drain { - struct delayed_work work; - ktime_t tstamp; - u32 residue; -}; - -struct udma_chan_config { - bool pkt_mode; /* TR or packet */ - bool needs_epib; /* EPIB is needed for the communication or not */ - u32 psd_size; /* size of Protocol Specific Data */ - u32 metadata_size; /* (needs_epib ? 16:0) + psd_size */ - u32 hdesc_size; /* Size of a packet descriptor in packet mode */ - bool notdpkt; /* Suppress sending TDC packet */ - int remote_thread_id; - u32 atype; - u32 asel; - u32 src_thread; - u32 dst_thread; - enum psil_endpoint_type ep_type; - bool enable_acc32; - bool enable_burst; - enum udma_tp_level channel_tpl; /* Channel Throughput Level */ - - u32 tr_trigger_type; - unsigned long tx_flags; - - /* PKDMA mapped channel */ - int mapped_channel_id; - /* PKTDMA default tflow or rflow for mapped channel */ - int default_flow_id; - - enum dma_transfer_direction dir; -}; - -struct udma_chan { - struct virt_dma_chan vc; - struct dma_slave_config cfg; - struct udma_dev *ud; - struct device *dma_dev; - struct udma_desc *desc; - struct udma_desc *terminated_desc; - struct udma_static_tr static_tr; - char *name; - - struct udma_bchan *bchan; - struct udma_tchan *tchan; - struct udma_rchan *rchan; - struct udma_rflow *rflow; - - bool psil_paired; - - int irq_num_ring; - int irq_num_udma; - - bool cyclic; - bool paused; - - enum udma_chan_state state; - struct completion teardown_completed; - - struct udma_tx_drain tx_drain; - - /* Channel configuration parameters */ - struct udma_chan_config config; - /* Channel configuration parameters (backup) */ - struct udma_chan_config backup_config; - - /* dmapool for packet mode descriptors */ - bool use_dma_pool; - struct dma_pool *hdesc_pool; - - u32 id; -}; - -static inline struct udma_dev *to_udma_dev(struct dma_device *d) -{ - return container_of(d, struct udma_dev, ddev); -} - -static inline struct udma_chan *to_udma_chan(struct dma_chan *c) -{ - return container_of(c, struct udma_chan, vc.chan); -} - -static inline struct udma_desc *to_udma_desc(struct dma_async_tx_descripto= r *t) -{ - return container_of(t, struct udma_desc, vd.tx); -} - -/* Generic register access functions */ -static inline u32 udma_read(void __iomem *base, int reg) -{ - return readl(base + reg); -} - -static inline void udma_write(void __iomem *base, int reg, u32 val) -{ - writel(val, base + reg); -} - -static inline void udma_update_bits(void __iomem *base, int reg, - u32 mask, u32 val) -{ - u32 tmp, orig; - - orig =3D readl(base + reg); - tmp =3D orig & ~mask; - tmp |=3D (val & mask); - - if (tmp !=3D orig) - writel(tmp, base + reg); -} - -/* TCHANRT */ -static inline u32 udma_tchanrt_read(struct udma_chan *uc, int reg) -{ - if (!uc->tchan) - return 0; - return udma_read(uc->tchan->reg_rt, reg); -} - -static inline void udma_tchanrt_write(struct udma_chan *uc, int reg, u32 v= al) -{ - if (!uc->tchan) - return; - udma_write(uc->tchan->reg_rt, reg, val); -} - -static inline void udma_tchanrt_update_bits(struct udma_chan *uc, int reg, - u32 mask, u32 val) -{ - if (!uc->tchan) - return; - udma_update_bits(uc->tchan->reg_rt, reg, mask, val); -} - -/* RCHANRT */ -static inline u32 udma_rchanrt_read(struct udma_chan *uc, int reg) -{ - if (!uc->rchan) - return 0; - return udma_read(uc->rchan->reg_rt, reg); -} - -static inline void udma_rchanrt_write(struct udma_chan *uc, int reg, u32 v= al) -{ - if (!uc->rchan) - return; - udma_write(uc->rchan->reg_rt, reg, val); -} - -static inline void udma_rchanrt_update_bits(struct udma_chan *uc, int reg, - u32 mask, u32 val) -{ - if (!uc->rchan) - return; - udma_update_bits(uc->rchan->reg_rt, reg, mask, val); -} - -static int navss_psil_pair(struct udma_dev *ud, u32 src_thread, u32 dst_th= read) +int navss_psil_pair(struct udma_dev *ud, u32 src_thread, u32 dst_thread) { struct udma_tisci_rm *tisci_rm =3D &ud->tisci_rm; =20 @@ -409,7 +50,7 @@ static int navss_psil_pair(struct udma_dev *ud, u32 src_= thread, u32 dst_thread) src_thread, dst_thread); } =20 -static int navss_psil_unpair(struct udma_dev *ud, u32 src_thread, +int navss_psil_unpair(struct udma_dev *ud, u32 src_thread, u32 dst_thread) { struct udma_tisci_rm *tisci_rm =3D &ud->tisci_rm; @@ -420,202 +61,6 @@ static int navss_psil_unpair(struct udma_dev *ud, u32 = src_thread, src_thread, dst_thread); } =20 -static void k3_configure_chan_coherency(struct dma_chan *chan, u32 asel) -{ - struct device *chan_dev =3D &chan->dev->device; - - if (asel =3D=3D 0) { - /* No special handling for the channel */ - chan->dev->chan_dma_dev =3D false; - - chan_dev->dma_coherent =3D false; - chan_dev->dma_parms =3D NULL; - } else if (asel =3D=3D 14 || asel =3D=3D 15) { - chan->dev->chan_dma_dev =3D true; - - chan_dev->dma_coherent =3D true; - dma_coerce_mask_and_coherent(chan_dev, DMA_BIT_MASK(48)); - chan_dev->dma_parms =3D chan_dev->parent->dma_parms; - } else { - dev_warn(chan->device->dev, "Invalid ASEL value: %u\n", asel); - - chan_dev->dma_coherent =3D false; - chan_dev->dma_parms =3D NULL; - } -} - -static u8 udma_get_chan_tpl_index(struct udma_tpl *tpl_map, int chan_id) -{ - int i; - - for (i =3D 0; i < tpl_map->levels; i++) { - if (chan_id >=3D tpl_map->start_idx[i]) - return i; - } - - return 0; -} - -static void udma_reset_uchan(struct udma_chan *uc) -{ - memset(&uc->config, 0, sizeof(uc->config)); - uc->config.remote_thread_id =3D -1; - uc->config.mapped_channel_id =3D -1; - uc->config.default_flow_id =3D -1; - uc->state =3D UDMA_CHAN_IS_IDLE; -} - -static void udma_dump_chan_stdata(struct udma_chan *uc) -{ - struct device *dev =3D uc->ud->dev; - u32 offset; - int i; - - if (uc->config.dir =3D=3D DMA_MEM_TO_DEV || uc->config.dir =3D=3D DMA_MEM= _TO_MEM) { - dev_dbg(dev, "TCHAN State data:\n"); - for (i =3D 0; i < 32; i++) { - offset =3D UDMA_CHAN_RT_STDATA_REG + i * 4; - dev_dbg(dev, "TRT_STDATA[%02d]: 0x%08x\n", i, - udma_tchanrt_read(uc, offset)); - } - } - - if (uc->config.dir =3D=3D DMA_DEV_TO_MEM || uc->config.dir =3D=3D DMA_MEM= _TO_MEM) { - dev_dbg(dev, "RCHAN State data:\n"); - for (i =3D 0; i < 32; i++) { - offset =3D UDMA_CHAN_RT_STDATA_REG + i * 4; - dev_dbg(dev, "RRT_STDATA[%02d]: 0x%08x\n", i, - udma_rchanrt_read(uc, offset)); - } - } -} - -static inline dma_addr_t udma_curr_cppi5_desc_paddr(struct udma_desc *d, - int idx) -{ - return d->hwdesc[idx].cppi5_desc_paddr; -} - -static inline void *udma_curr_cppi5_desc_vaddr(struct udma_desc *d, int id= x) -{ - return d->hwdesc[idx].cppi5_desc_vaddr; -} - -static struct udma_desc *udma_udma_desc_from_paddr(struct udma_chan *uc, - dma_addr_t paddr) -{ - struct udma_desc *d =3D uc->terminated_desc; - - if (d) { - dma_addr_t desc_paddr =3D udma_curr_cppi5_desc_paddr(d, - d->desc_idx); - - if (desc_paddr !=3D paddr) - d =3D NULL; - } - - if (!d) { - d =3D uc->desc; - if (d) { - dma_addr_t desc_paddr =3D udma_curr_cppi5_desc_paddr(d, - d->desc_idx); - - if (desc_paddr !=3D paddr) - d =3D NULL; - } - } - - return d; -} - -static void udma_free_hwdesc(struct udma_chan *uc, struct udma_desc *d) -{ - if (uc->use_dma_pool) { - int i; - - for (i =3D 0; i < d->hwdesc_count; i++) { - if (!d->hwdesc[i].cppi5_desc_vaddr) - continue; - - dma_pool_free(uc->hdesc_pool, - d->hwdesc[i].cppi5_desc_vaddr, - d->hwdesc[i].cppi5_desc_paddr); - - d->hwdesc[i].cppi5_desc_vaddr =3D NULL; - } - } else if (d->hwdesc[0].cppi5_desc_vaddr) { - dma_free_coherent(uc->dma_dev, d->hwdesc[0].cppi5_desc_size, - d->hwdesc[0].cppi5_desc_vaddr, - d->hwdesc[0].cppi5_desc_paddr); - - d->hwdesc[0].cppi5_desc_vaddr =3D NULL; - } -} - -static void udma_purge_desc_work(struct work_struct *work) -{ - struct udma_dev *ud =3D container_of(work, typeof(*ud), purge_work); - struct virt_dma_desc *vd, *_vd; - unsigned long flags; - LIST_HEAD(head); - - spin_lock_irqsave(&ud->lock, flags); - list_splice_tail_init(&ud->desc_to_purge, &head); - spin_unlock_irqrestore(&ud->lock, flags); - - list_for_each_entry_safe(vd, _vd, &head, node) { - struct udma_chan *uc =3D to_udma_chan(vd->tx.chan); - struct udma_desc *d =3D to_udma_desc(&vd->tx); - - udma_free_hwdesc(uc, d); - list_del(&vd->node); - kfree(d); - } - - /* If more to purge, schedule the work again */ - if (!list_empty(&ud->desc_to_purge)) - schedule_work(&ud->purge_work); -} - -static void udma_desc_free(struct virt_dma_desc *vd) -{ - struct udma_dev *ud =3D to_udma_dev(vd->tx.chan->device); - struct udma_chan *uc =3D to_udma_chan(vd->tx.chan); - struct udma_desc *d =3D to_udma_desc(&vd->tx); - unsigned long flags; - - if (uc->terminated_desc =3D=3D d) - uc->terminated_desc =3D NULL; - - if (uc->use_dma_pool) { - udma_free_hwdesc(uc, d); - kfree(d); - return; - } - - spin_lock_irqsave(&ud->lock, flags); - list_add_tail(&vd->node, &ud->desc_to_purge); - spin_unlock_irqrestore(&ud->lock, flags); - - schedule_work(&ud->purge_work); -} - -static bool udma_is_chan_running(struct udma_chan *uc) -{ - u32 trt_ctl =3D 0; - u32 rrt_ctl =3D 0; - - if (uc->tchan) - trt_ctl =3D udma_tchanrt_read(uc, UDMA_CHAN_RT_CTL_REG); - if (uc->rchan) - rrt_ctl =3D udma_rchanrt_read(uc, UDMA_CHAN_RT_CTL_REG); - - if (trt_ctl & UDMA_CHAN_RT_CTL_EN || rrt_ctl & UDMA_CHAN_RT_CTL_EN) - return true; - - return false; -} - static bool udma_is_chan_paused(struct udma_chan *uc) { u32 val, pause_mask; @@ -643,189 +88,73 @@ static bool udma_is_chan_paused(struct udma_chan *uc) return false; } =20 -static inline dma_addr_t udma_get_rx_flush_hwdesc_paddr(struct udma_chan *= uc) +static void udma_decrement_byte_counters(struct udma_chan *uc, u32 val) { - return uc->ud->rx_flush.hwdescs[uc->config.pkt_mode].cppi5_desc_paddr; + if (uc->desc->dir =3D=3D DMA_DEV_TO_MEM) { + udma_rchanrt_write(uc, UDMA_CHAN_RT_BCNT_REG, val); + udma_rchanrt_write(uc, UDMA_CHAN_RT_SBCNT_REG, val); + if (uc->config.ep_type !=3D PSIL_EP_NATIVE) + udma_rchanrt_write(uc, UDMA_CHAN_RT_PEER_BCNT_REG, val); + } else { + udma_tchanrt_write(uc, UDMA_CHAN_RT_BCNT_REG, val); + udma_tchanrt_write(uc, UDMA_CHAN_RT_SBCNT_REG, val); + if (!uc->bchan && uc->config.ep_type !=3D PSIL_EP_NATIVE) + udma_tchanrt_write(uc, UDMA_CHAN_RT_PEER_BCNT_REG, val); + } } =20 -static int udma_push_to_ring(struct udma_chan *uc, int idx) +static void udma_reset_counters(struct udma_chan *uc) { - struct udma_desc *d =3D uc->desc; - struct k3_ring *ring =3D NULL; - dma_addr_t paddr; + u32 val; =20 - switch (uc->config.dir) { - case DMA_DEV_TO_MEM: - ring =3D uc->rflow->fd_ring; - break; - case DMA_MEM_TO_DEV: - case DMA_MEM_TO_MEM: - ring =3D uc->tchan->t_ring; - break; - default: - return -EINVAL; - } + if (uc->tchan) { + val =3D udma_tchanrt_read(uc, UDMA_CHAN_RT_BCNT_REG); + udma_tchanrt_write(uc, UDMA_CHAN_RT_BCNT_REG, val); =20 - /* RX flush packet: idx =3D=3D -1 is only passed in case of DEV_TO_MEM */ - if (idx =3D=3D -1) { - paddr =3D udma_get_rx_flush_hwdesc_paddr(uc); - } else { - paddr =3D udma_curr_cppi5_desc_paddr(d, idx); + val =3D udma_tchanrt_read(uc, UDMA_CHAN_RT_SBCNT_REG); + udma_tchanrt_write(uc, UDMA_CHAN_RT_SBCNT_REG, val); + + val =3D udma_tchanrt_read(uc, UDMA_CHAN_RT_PCNT_REG); + udma_tchanrt_write(uc, UDMA_CHAN_RT_PCNT_REG, val); =20 - wmb(); /* Ensure that writes are not moved over this point */ + if (!uc->bchan) { + val =3D udma_tchanrt_read(uc, UDMA_CHAN_RT_PEER_BCNT_REG); + udma_tchanrt_write(uc, UDMA_CHAN_RT_PEER_BCNT_REG, val); + } } =20 - return k3_ringacc_ring_push(ring, &paddr); -} + if (uc->rchan) { + val =3D udma_rchanrt_read(uc, UDMA_CHAN_RT_BCNT_REG); + udma_rchanrt_write(uc, UDMA_CHAN_RT_BCNT_REG, val); =20 -static bool udma_desc_is_rx_flush(struct udma_chan *uc, dma_addr_t addr) -{ - if (uc->config.dir !=3D DMA_DEV_TO_MEM) - return false; + val =3D udma_rchanrt_read(uc, UDMA_CHAN_RT_SBCNT_REG); + udma_rchanrt_write(uc, UDMA_CHAN_RT_SBCNT_REG, val); =20 - if (addr =3D=3D udma_get_rx_flush_hwdesc_paddr(uc)) - return true; + val =3D udma_rchanrt_read(uc, UDMA_CHAN_RT_PCNT_REG); + udma_rchanrt_write(uc, UDMA_CHAN_RT_PCNT_REG, val); =20 - return false; + val =3D udma_rchanrt_read(uc, UDMA_CHAN_RT_PEER_BCNT_REG); + udma_rchanrt_write(uc, UDMA_CHAN_RT_PEER_BCNT_REG, val); + } } =20 -static int udma_pop_from_ring(struct udma_chan *uc, dma_addr_t *addr) +static int udma_reset_chan(struct udma_chan *uc, bool hard) { - struct k3_ring *ring =3D NULL; - int ret; - switch (uc->config.dir) { case DMA_DEV_TO_MEM: - ring =3D uc->rflow->r_ring; + udma_rchanrt_write(uc, UDMA_CHAN_RT_PEER_RT_EN_REG, 0); + udma_rchanrt_write(uc, UDMA_CHAN_RT_CTL_REG, 0); break; case DMA_MEM_TO_DEV: + udma_tchanrt_write(uc, UDMA_CHAN_RT_CTL_REG, 0); + udma_tchanrt_write(uc, UDMA_CHAN_RT_PEER_RT_EN_REG, 0); + break; case DMA_MEM_TO_MEM: - ring =3D uc->tchan->tc_ring; + udma_rchanrt_write(uc, UDMA_CHAN_RT_CTL_REG, 0); + udma_tchanrt_write(uc, UDMA_CHAN_RT_CTL_REG, 0); break; default: - return -ENOENT; - } - - ret =3D k3_ringacc_ring_pop(ring, addr); - if (ret) - return ret; - - rmb(); /* Ensure that reads are not moved before this point */ - - /* Teardown completion */ - if (cppi5_desc_is_tdcm(*addr)) - return 0; - - /* Check for flush descriptor */ - if (udma_desc_is_rx_flush(uc, *addr)) - return -ENOENT; - - return 0; -} - -static void udma_reset_rings(struct udma_chan *uc) -{ - struct k3_ring *ring1 =3D NULL; - struct k3_ring *ring2 =3D NULL; - - switch (uc->config.dir) { - case DMA_DEV_TO_MEM: - if (uc->rchan) { - ring1 =3D uc->rflow->fd_ring; - ring2 =3D uc->rflow->r_ring; - } - break; - case DMA_MEM_TO_DEV: - case DMA_MEM_TO_MEM: - if (uc->tchan) { - ring1 =3D uc->tchan->t_ring; - ring2 =3D uc->tchan->tc_ring; - } - break; - default: - break; - } - - if (ring1) - k3_ringacc_ring_reset_dma(ring1, - k3_ringacc_ring_get_occ(ring1)); - if (ring2) - k3_ringacc_ring_reset(ring2); - - /* make sure we are not leaking memory by stalled descriptor */ - if (uc->terminated_desc) { - udma_desc_free(&uc->terminated_desc->vd); - uc->terminated_desc =3D NULL; - } -} - -static void udma_decrement_byte_counters(struct udma_chan *uc, u32 val) -{ - if (uc->desc->dir =3D=3D DMA_DEV_TO_MEM) { - udma_rchanrt_write(uc, UDMA_CHAN_RT_BCNT_REG, val); - udma_rchanrt_write(uc, UDMA_CHAN_RT_SBCNT_REG, val); - if (uc->config.ep_type !=3D PSIL_EP_NATIVE) - udma_rchanrt_write(uc, UDMA_CHAN_RT_PEER_BCNT_REG, val); - } else { - udma_tchanrt_write(uc, UDMA_CHAN_RT_BCNT_REG, val); - udma_tchanrt_write(uc, UDMA_CHAN_RT_SBCNT_REG, val); - if (!uc->bchan && uc->config.ep_type !=3D PSIL_EP_NATIVE) - udma_tchanrt_write(uc, UDMA_CHAN_RT_PEER_BCNT_REG, val); - } -} - -static void udma_reset_counters(struct udma_chan *uc) -{ - u32 val; - - if (uc->tchan) { - val =3D udma_tchanrt_read(uc, UDMA_CHAN_RT_BCNT_REG); - udma_tchanrt_write(uc, UDMA_CHAN_RT_BCNT_REG, val); - - val =3D udma_tchanrt_read(uc, UDMA_CHAN_RT_SBCNT_REG); - udma_tchanrt_write(uc, UDMA_CHAN_RT_SBCNT_REG, val); - - val =3D udma_tchanrt_read(uc, UDMA_CHAN_RT_PCNT_REG); - udma_tchanrt_write(uc, UDMA_CHAN_RT_PCNT_REG, val); - - if (!uc->bchan) { - val =3D udma_tchanrt_read(uc, UDMA_CHAN_RT_PEER_BCNT_REG); - udma_tchanrt_write(uc, UDMA_CHAN_RT_PEER_BCNT_REG, val); - } - } - - if (uc->rchan) { - val =3D udma_rchanrt_read(uc, UDMA_CHAN_RT_BCNT_REG); - udma_rchanrt_write(uc, UDMA_CHAN_RT_BCNT_REG, val); - - val =3D udma_rchanrt_read(uc, UDMA_CHAN_RT_SBCNT_REG); - udma_rchanrt_write(uc, UDMA_CHAN_RT_SBCNT_REG, val); - - val =3D udma_rchanrt_read(uc, UDMA_CHAN_RT_PCNT_REG); - udma_rchanrt_write(uc, UDMA_CHAN_RT_PCNT_REG, val); - - val =3D udma_rchanrt_read(uc, UDMA_CHAN_RT_PEER_BCNT_REG); - udma_rchanrt_write(uc, UDMA_CHAN_RT_PEER_BCNT_REG, val); - } -} - -static int udma_reset_chan(struct udma_chan *uc, bool hard) -{ - switch (uc->config.dir) { - case DMA_DEV_TO_MEM: - udma_rchanrt_write(uc, UDMA_CHAN_RT_PEER_RT_EN_REG, 0); - udma_rchanrt_write(uc, UDMA_CHAN_RT_CTL_REG, 0); - break; - case DMA_MEM_TO_DEV: - udma_tchanrt_write(uc, UDMA_CHAN_RT_CTL_REG, 0); - udma_tchanrt_write(uc, UDMA_CHAN_RT_PEER_RT_EN_REG, 0); - break; - case DMA_MEM_TO_MEM: - udma_rchanrt_write(uc, UDMA_CHAN_RT_CTL_REG, 0); - udma_tchanrt_write(uc, UDMA_CHAN_RT_CTL_REG, 0); - break; - default: - return -EINVAL; + return -EINVAL; } =20 /* Reset all counters */ @@ -860,40 +189,6 @@ static int udma_reset_chan(struct udma_chan *uc, bool = hard) return 0; } =20 -static void udma_start_desc(struct udma_chan *uc) -{ - struct udma_chan_config *ucc =3D &uc->config; - - if (uc->ud->match_data->type =3D=3D DMA_TYPE_UDMA && ucc->pkt_mode && - (uc->cyclic || ucc->dir =3D=3D DMA_DEV_TO_MEM)) { - int i; - - /* - * UDMA only: Push all descriptors to ring for packet mode - * cyclic or RX - * PKTDMA supports pre-linked descriptor and cyclic is not - * supported - */ - for (i =3D 0; i < uc->desc->sglen; i++) - udma_push_to_ring(uc, i); - } else { - udma_push_to_ring(uc, 0); - } -} - -static bool udma_chan_needs_reconfiguration(struct udma_chan *uc) -{ - /* Only PDMAs have staticTR */ - if (uc->config.ep_type =3D=3D PSIL_EP_NATIVE) - return false; - - /* Check if the staticTR configuration has changed for TX */ - if (memcmp(&uc->static_tr, &uc->desc->static_tr, sizeof(uc->static_tr))) - return true; - - return false; -} - static int udma_start(struct udma_chan *uc) { struct virt_dma_desc *vd =3D vchan_next_desc(&uc->vc); @@ -1038,24 +333,6 @@ static int udma_stop(struct udma_chan *uc) return 0; } =20 -static void udma_cyclic_packet_elapsed(struct udma_chan *uc) -{ - struct udma_desc *d =3D uc->desc; - struct cppi5_host_desc_t *h_desc; - - h_desc =3D d->hwdesc[d->desc_idx].cppi5_desc_vaddr; - cppi5_hdesc_reset_to_original(h_desc); - udma_push_to_ring(uc, d->desc_idx); - d->desc_idx =3D (d->desc_idx + 1) % d->sglen; -} - -static inline void udma_fetch_epib(struct udma_chan *uc, struct udma_desc = *d) -{ - struct cppi5_host_desc_t *h_desc =3D d->hwdesc[0].cppi5_desc_vaddr; - - memcpy(d->metadata, h_desc->epib, d->metadata_size); -} - static bool udma_is_desc_really_done(struct udma_chan *uc, struct udma_des= c *d) { u32 peer_bcnt, bcnt; @@ -1083,68 +360,6 @@ static bool udma_is_desc_really_done(struct udma_chan= *uc, struct udma_desc *d) return true; } =20 -static void udma_check_tx_completion(struct work_struct *work) -{ - struct udma_chan *uc =3D container_of(work, typeof(*uc), - tx_drain.work.work); - bool desc_done =3D true; - u32 residue_diff; - ktime_t time_diff; - unsigned long delay; - - while (1) { - if (uc->desc) { - /* Get previous residue and time stamp */ - residue_diff =3D uc->tx_drain.residue; - time_diff =3D uc->tx_drain.tstamp; - /* - * Get current residue and time stamp or see if - * transfer is complete - */ - desc_done =3D udma_is_desc_really_done(uc, uc->desc); - } - - if (!desc_done) { - /* - * Find the time delta and residue delta w.r.t - * previous poll - */ - time_diff =3D ktime_sub(uc->tx_drain.tstamp, - time_diff) + 1; - residue_diff -=3D uc->tx_drain.residue; - if (residue_diff) { - /* - * Try to guess when we should check - * next time by calculating rate at - * which data is being drained at the - * peer device - */ - delay =3D (time_diff / residue_diff) * - uc->tx_drain.residue; - } else { - /* No progress, check again in 1 second */ - schedule_delayed_work(&uc->tx_drain.work, HZ); - break; - } - - usleep_range(ktime_to_us(delay), - ktime_to_us(delay) + 10); - continue; - } - - if (uc->desc) { - struct udma_desc *d =3D uc->desc; - - udma_decrement_byte_counters(uc, d->residue); - udma_start(uc); - vchan_cookie_complete(&d->vd); - break; - } - - break; - } -} - static irqreturn_t udma_ring_irq_handler(int irq, void *data) { struct udma_chan *uc =3D data; @@ -1235,135 +450,6 @@ static irqreturn_t udma_udma_irq_handler(int irq, vo= id *data) return IRQ_HANDLED; } =20 -/** - * __udma_alloc_gp_rflow_range - alloc range of GP RX flows - * @ud: UDMA device - * @from: Start the search from this flow id number - * @cnt: Number of consecutive flow ids to allocate - * - * Allocate range of RX flow ids for future use, those flows can be reques= ted - * only using explicit flow id number. if @from is set to -1 it will try t= o find - * first free range. if @from is positive value it will force allocation o= nly - * of the specified range of flows. - * - * Returns -ENOMEM if can't find free range. - * -EEXIST if requested range is busy. - * -EINVAL if wrong input values passed. - * Returns flow id on success. - */ -static int __udma_alloc_gp_rflow_range(struct udma_dev *ud, int from, int = cnt) -{ - int start, tmp_from; - DECLARE_BITMAP(tmp, K3_UDMA_MAX_RFLOWS); - - tmp_from =3D from; - if (tmp_from < 0) - tmp_from =3D ud->rchan_cnt; - /* default flows can't be allocated and accessible only by id */ - if (tmp_from < ud->rchan_cnt) - return -EINVAL; - - if (tmp_from + cnt > ud->rflow_cnt) - return -EINVAL; - - bitmap_or(tmp, ud->rflow_gp_map, ud->rflow_gp_map_allocated, - ud->rflow_cnt); - - start =3D bitmap_find_next_zero_area(tmp, - ud->rflow_cnt, - tmp_from, cnt, 0); - if (start >=3D ud->rflow_cnt) - return -ENOMEM; - - if (from >=3D 0 && start !=3D from) - return -EEXIST; - - bitmap_set(ud->rflow_gp_map_allocated, start, cnt); - return start; -} - -static int __udma_free_gp_rflow_range(struct udma_dev *ud, int from, int c= nt) -{ - if (from < ud->rchan_cnt) - return -EINVAL; - if (from + cnt > ud->rflow_cnt) - return -EINVAL; - - bitmap_clear(ud->rflow_gp_map_allocated, from, cnt); - return 0; -} - -static struct udma_rflow *__udma_get_rflow(struct udma_dev *ud, int id) -{ - /* - * Attempt to request rflow by ID can be made for any rflow - * if not in use with assumption that caller knows what's doing. - * TI-SCI FW will perform additional permission check ant way, it's - * safe - */ - - if (id < 0 || id >=3D ud->rflow_cnt) - return ERR_PTR(-ENOENT); - - if (test_bit(id, ud->rflow_in_use)) - return ERR_PTR(-ENOENT); - - if (ud->rflow_gp_map) { - /* GP rflow has to be allocated first */ - if (!test_bit(id, ud->rflow_gp_map) && - !test_bit(id, ud->rflow_gp_map_allocated)) - return ERR_PTR(-EINVAL); - } - - dev_dbg(ud->dev, "get rflow%d\n", id); - set_bit(id, ud->rflow_in_use); - return &ud->rflows[id]; -} - -static void __udma_put_rflow(struct udma_dev *ud, struct udma_rflow *rflow) -{ - if (!test_bit(rflow->id, ud->rflow_in_use)) { - dev_err(ud->dev, "attempt to put unused rflow%d\n", rflow->id); - return; - } - - dev_dbg(ud->dev, "put rflow%d\n", rflow->id); - clear_bit(rflow->id, ud->rflow_in_use); -} - -#define UDMA_RESERVE_RESOURCE(res) \ -static struct udma_##res *__udma_reserve_##res(struct udma_dev *ud, \ - enum udma_tp_level tpl, \ - int id) \ -{ \ - if (id >=3D 0) { \ - if (test_bit(id, ud->res##_map)) { \ - dev_err(ud->dev, "res##%d is in use\n", id); \ - return ERR_PTR(-ENOENT); \ - } \ - } else { \ - int start; \ - \ - if (tpl >=3D ud->res##_tpl.levels) \ - tpl =3D ud->res##_tpl.levels - 1; \ - \ - start =3D ud->res##_tpl.start_idx[tpl]; \ - \ - id =3D find_next_zero_bit(ud->res##_map, ud->res##_cnt, \ - start); \ - if (id =3D=3D ud->res##_cnt) { \ - return ERR_PTR(-ENOENT); \ - } \ - } \ - \ - set_bit(id, ud->res##_map); \ - return &ud->res##s[id]; \ -} - -UDMA_RESERVE_RESOURCE(bchan); -UDMA_RESERVE_RESOURCE(tchan); -UDMA_RESERVE_RESOURCE(rchan); - static int bcdma_get_bchan(struct udma_chan *uc) { struct udma_dev *ud =3D uc->ud; @@ -1397,297 +483,67 @@ static int bcdma_get_bchan(struct udma_chan *uc) return 0; } =20 -static int udma_get_tchan(struct udma_chan *uc) +static int bcdma_alloc_bchan_resources(struct udma_chan *uc) { + struct k3_ring_cfg ring_cfg; struct udma_dev *ud =3D uc->ud; int ret; =20 - if (uc->tchan) { - dev_dbg(ud->dev, "chan%d: already have tchan%d allocated\n", - uc->id, uc->tchan->id); - return 0; - } - - /* - * mapped_channel_id is -1 for UDMA, BCDMA and PKTDMA unmapped channels. - * For PKTDMA mapped channels it is configured to a channel which must - * be used to service the peripheral. - */ - uc->tchan =3D __udma_reserve_tchan(ud, uc->config.channel_tpl, - uc->config.mapped_channel_id); - if (IS_ERR(uc->tchan)) { - ret =3D PTR_ERR(uc->tchan); - uc->tchan =3D NULL; + ret =3D bcdma_get_bchan(uc); + if (ret) return ret; - } =20 - if (ud->tflow_cnt) { - int tflow_id; + ret =3D k3_ringacc_request_rings_pair(ud->ringacc, uc->bchan->id, -1, + &uc->bchan->t_ring, + &uc->bchan->tc_ring); + if (ret) { + ret =3D -EBUSY; + goto err_ring; + } =20 - /* Only PKTDMA have support for tx flows */ - if (uc->config.default_flow_id >=3D 0) - tflow_id =3D uc->config.default_flow_id; - else - tflow_id =3D uc->tchan->id; + memset(&ring_cfg, 0, sizeof(ring_cfg)); + ring_cfg.size =3D K3_UDMA_DEFAULT_RING_SIZE; + ring_cfg.elm_size =3D K3_RINGACC_RING_ELSIZE_8; + ring_cfg.mode =3D K3_RINGACC_RING_MODE_RING; =20 - if (test_bit(tflow_id, ud->tflow_map)) { - dev_err(ud->dev, "tflow%d is in use\n", tflow_id); - clear_bit(uc->tchan->id, ud->tchan_map); - uc->tchan =3D NULL; - return -ENOENT; - } + k3_configure_chan_coherency(&uc->vc.chan, ud->asel); + ring_cfg.asel =3D ud->asel; + ring_cfg.dma_dev =3D dmaengine_get_dma_device(&uc->vc.chan); =20 - uc->tchan->tflow_id =3D tflow_id; - set_bit(tflow_id, ud->tflow_map); - } else { - uc->tchan->tflow_id =3D -1; - } + ret =3D k3_ringacc_ring_cfg(uc->bchan->t_ring, &ring_cfg); + if (ret) + goto err_ringcfg; =20 return 0; -} - -static int udma_get_rchan(struct udma_chan *uc) -{ - struct udma_dev *ud =3D uc->ud; - int ret; - - if (uc->rchan) { - dev_dbg(ud->dev, "chan%d: already have rchan%d allocated\n", - uc->id, uc->rchan->id); - return 0; - } =20 - /* - * mapped_channel_id is -1 for UDMA, BCDMA and PKTDMA unmapped channels. - * For PKTDMA mapped channels it is configured to a channel which must - * be used to service the peripheral. - */ - uc->rchan =3D __udma_reserve_rchan(ud, uc->config.channel_tpl, - uc->config.mapped_channel_id); - if (IS_ERR(uc->rchan)) { - ret =3D PTR_ERR(uc->rchan); - uc->rchan =3D NULL; - return ret; - } +err_ringcfg: + k3_ringacc_ring_free(uc->bchan->tc_ring); + uc->bchan->tc_ring =3D NULL; + k3_ringacc_ring_free(uc->bchan->t_ring); + uc->bchan->t_ring =3D NULL; + k3_configure_chan_coherency(&uc->vc.chan, 0); +err_ring: + bcdma_put_bchan(uc); =20 - return 0; + return ret; } =20 -static int udma_get_chan_pair(struct udma_chan *uc) +static int udma_alloc_tx_resources(struct udma_chan *uc) { + struct k3_ring_cfg ring_cfg; struct udma_dev *ud =3D uc->ud; - int chan_id, end; + struct udma_tchan *tchan; + int ring_idx, ret; =20 - if ((uc->tchan && uc->rchan) && uc->tchan->id =3D=3D uc->rchan->id) { - dev_info(ud->dev, "chan%d: already have %d pair allocated\n", - uc->id, uc->tchan->id); - return 0; - } + ret =3D udma_get_tchan(uc); + if (ret) + return ret; =20 - if (uc->tchan) { - dev_err(ud->dev, "chan%d: already have tchan%d allocated\n", - uc->id, uc->tchan->id); - return -EBUSY; - } else if (uc->rchan) { - dev_err(ud->dev, "chan%d: already have rchan%d allocated\n", - uc->id, uc->rchan->id); - return -EBUSY; - } - - /* Can be optimized, but let's have it like this for now */ - end =3D min(ud->tchan_cnt, ud->rchan_cnt); - /* - * Try to use the highest TPL channel pair for MEM_TO_MEM channels - * Note: in UDMAP the channel TPL is symmetric between tchan and rchan - */ - chan_id =3D ud->tchan_tpl.start_idx[ud->tchan_tpl.levels - 1]; - for (; chan_id < end; chan_id++) { - if (!test_bit(chan_id, ud->tchan_map) && - !test_bit(chan_id, ud->rchan_map)) - break; - } - - if (chan_id =3D=3D end) - return -ENOENT; - - set_bit(chan_id, ud->tchan_map); - set_bit(chan_id, ud->rchan_map); - uc->tchan =3D &ud->tchans[chan_id]; - uc->rchan =3D &ud->rchans[chan_id]; - - /* UDMA does not use tx flows */ - uc->tchan->tflow_id =3D -1; - - return 0; -} - -static int udma_get_rflow(struct udma_chan *uc, int flow_id) -{ - struct udma_dev *ud =3D uc->ud; - int ret; - - if (!uc->rchan) { - dev_err(ud->dev, "chan%d: does not have rchan??\n", uc->id); - return -EINVAL; - } - - if (uc->rflow) { - dev_dbg(ud->dev, "chan%d: already have rflow%d allocated\n", - uc->id, uc->rflow->id); - return 0; - } - - uc->rflow =3D __udma_get_rflow(ud, flow_id); - if (IS_ERR(uc->rflow)) { - ret =3D PTR_ERR(uc->rflow); - uc->rflow =3D NULL; - return ret; - } - - return 0; -} - -static void bcdma_put_bchan(struct udma_chan *uc) -{ - struct udma_dev *ud =3D uc->ud; - - if (uc->bchan) { - dev_dbg(ud->dev, "chan%d: put bchan%d\n", uc->id, - uc->bchan->id); - clear_bit(uc->bchan->id, ud->bchan_map); - uc->bchan =3D NULL; - uc->tchan =3D NULL; - } -} - -static void udma_put_rchan(struct udma_chan *uc) -{ - struct udma_dev *ud =3D uc->ud; - - if (uc->rchan) { - dev_dbg(ud->dev, "chan%d: put rchan%d\n", uc->id, - uc->rchan->id); - clear_bit(uc->rchan->id, ud->rchan_map); - uc->rchan =3D NULL; - } -} - -static void udma_put_tchan(struct udma_chan *uc) -{ - struct udma_dev *ud =3D uc->ud; - - if (uc->tchan) { - dev_dbg(ud->dev, "chan%d: put tchan%d\n", uc->id, - uc->tchan->id); - clear_bit(uc->tchan->id, ud->tchan_map); - - if (uc->tchan->tflow_id >=3D 0) - clear_bit(uc->tchan->tflow_id, ud->tflow_map); - - uc->tchan =3D NULL; - } -} - -static void udma_put_rflow(struct udma_chan *uc) -{ - struct udma_dev *ud =3D uc->ud; - - if (uc->rflow) { - dev_dbg(ud->dev, "chan%d: put rflow%d\n", uc->id, - uc->rflow->id); - __udma_put_rflow(ud, uc->rflow); - uc->rflow =3D NULL; - } -} - -static void bcdma_free_bchan_resources(struct udma_chan *uc) -{ - if (!uc->bchan) - return; - - k3_ringacc_ring_free(uc->bchan->tc_ring); - k3_ringacc_ring_free(uc->bchan->t_ring); - uc->bchan->tc_ring =3D NULL; - uc->bchan->t_ring =3D NULL; - k3_configure_chan_coherency(&uc->vc.chan, 0); - - bcdma_put_bchan(uc); -} - -static int bcdma_alloc_bchan_resources(struct udma_chan *uc) -{ - struct k3_ring_cfg ring_cfg; - struct udma_dev *ud =3D uc->ud; - int ret; - - ret =3D bcdma_get_bchan(uc); - if (ret) - return ret; - - ret =3D k3_ringacc_request_rings_pair(ud->ringacc, uc->bchan->id, -1, - &uc->bchan->t_ring, - &uc->bchan->tc_ring); - if (ret) { - ret =3D -EBUSY; - goto err_ring; - } - - memset(&ring_cfg, 0, sizeof(ring_cfg)); - ring_cfg.size =3D K3_UDMA_DEFAULT_RING_SIZE; - ring_cfg.elm_size =3D K3_RINGACC_RING_ELSIZE_8; - ring_cfg.mode =3D K3_RINGACC_RING_MODE_RING; - - k3_configure_chan_coherency(&uc->vc.chan, ud->asel); - ring_cfg.asel =3D ud->asel; - ring_cfg.dma_dev =3D dmaengine_get_dma_device(&uc->vc.chan); - - ret =3D k3_ringacc_ring_cfg(uc->bchan->t_ring, &ring_cfg); - if (ret) - goto err_ringcfg; - - return 0; - -err_ringcfg: - k3_ringacc_ring_free(uc->bchan->tc_ring); - uc->bchan->tc_ring =3D NULL; - k3_ringacc_ring_free(uc->bchan->t_ring); - uc->bchan->t_ring =3D NULL; - k3_configure_chan_coherency(&uc->vc.chan, 0); -err_ring: - bcdma_put_bchan(uc); - - return ret; -} - -static void udma_free_tx_resources(struct udma_chan *uc) -{ - if (!uc->tchan) - return; - - k3_ringacc_ring_free(uc->tchan->t_ring); - k3_ringacc_ring_free(uc->tchan->tc_ring); - uc->tchan->t_ring =3D NULL; - uc->tchan->tc_ring =3D NULL; - - udma_put_tchan(uc); -} - -static int udma_alloc_tx_resources(struct udma_chan *uc) -{ - struct k3_ring_cfg ring_cfg; - struct udma_dev *ud =3D uc->ud; - struct udma_tchan *tchan; - int ring_idx, ret; - - ret =3D udma_get_tchan(uc); - if (ret) - return ret; - - tchan =3D uc->tchan; - if (tchan->tflow_id >=3D 0) - ring_idx =3D tchan->tflow_id; - else - ring_idx =3D ud->bchan_cnt + tchan->id; + tchan =3D uc->tchan; + if (tchan->tflow_id >=3D 0) + ring_idx =3D tchan->tflow_id; + else + ring_idx =3D ud->bchan_cnt + tchan->id; =20 ret =3D k3_ringacc_request_rings_pair(ud->ringacc, ring_idx, -1, &tchan->t_ring, @@ -1729,25 +585,6 @@ static int udma_alloc_tx_resources(struct udma_chan *= uc) return ret; } =20 -static void udma_free_rx_resources(struct udma_chan *uc) -{ - if (!uc->rchan) - return; - - if (uc->rflow) { - struct udma_rflow *rflow =3D uc->rflow; - - k3_ringacc_ring_free(rflow->fd_ring); - k3_ringacc_ring_free(rflow->r_ring); - rflow->fd_ring =3D NULL; - rflow->r_ring =3D NULL; - - udma_put_rflow(uc); - } - - udma_put_rchan(uc); -} - static int udma_alloc_rx_resources(struct udma_chan *uc) { struct udma_dev *ud =3D uc->ud; @@ -2559,1259 +1396,186 @@ static int bcdma_alloc_chan_resources(struct dma= _chan *chan) uc->psil_paired =3D false; err_res_free: bcdma_free_bchan_resources(uc); - udma_free_tx_resources(uc); - udma_free_rx_resources(uc); - - udma_reset_uchan(uc); - - if (uc->use_dma_pool) { - dma_pool_destroy(uc->hdesc_pool); - uc->use_dma_pool =3D false; - } - - return ret; -} - -static int bcdma_router_config(struct dma_chan *chan) -{ - struct k3_event_route_data *router_data =3D chan->route_data; - struct udma_chan *uc =3D to_udma_chan(chan); - u32 trigger_event; - - if (!uc->bchan) - return -EINVAL; - - if (uc->config.tr_trigger_type !=3D 1 && uc->config.tr_trigger_type !=3D = 2) - return -EINVAL; - - trigger_event =3D uc->ud->soc_data->bcdma_trigger_event_offset; - trigger_event +=3D (uc->bchan->id * 2) + uc->config.tr_trigger_type - 1; - - return router_data->set_event(router_data->priv, trigger_event); -} - -static int pktdma_alloc_chan_resources(struct dma_chan *chan) -{ - struct udma_chan *uc =3D to_udma_chan(chan); - struct udma_dev *ud =3D to_udma_dev(chan->device); - const struct udma_oes_offsets *oes =3D &ud->soc_data->oes; - u32 irq_ring_idx; - int ret; - - /* - * Make sure that the completion is in a known state: - * No teardown, the channel is idle - */ - reinit_completion(&uc->teardown_completed); - complete_all(&uc->teardown_completed); - uc->state =3D UDMA_CHAN_IS_IDLE; - - switch (uc->config.dir) { - case DMA_MEM_TO_DEV: - /* Slave transfer synchronized - mem to dev (TX) trasnfer */ - dev_dbg(uc->ud->dev, "%s: chan%d as MEM-to-DEV\n", __func__, - uc->id); - - ret =3D udma_alloc_tx_resources(uc); - if (ret) { - uc->config.remote_thread_id =3D -1; - return ret; - } - - uc->config.src_thread =3D ud->psil_base + uc->tchan->id; - uc->config.dst_thread =3D uc->config.remote_thread_id; - uc->config.dst_thread |=3D K3_PSIL_DST_THREAD_ID_OFFSET; - - irq_ring_idx =3D uc->tchan->tflow_id + oes->pktdma_tchan_flow; - - ret =3D pktdma_tisci_tx_channel_config(uc); - break; - case DMA_DEV_TO_MEM: - /* Slave transfer synchronized - dev to mem (RX) trasnfer */ - dev_dbg(uc->ud->dev, "%s: chan%d as DEV-to-MEM\n", __func__, - uc->id); - - ret =3D udma_alloc_rx_resources(uc); - if (ret) { - uc->config.remote_thread_id =3D -1; - return ret; - } - - uc->config.src_thread =3D uc->config.remote_thread_id; - uc->config.dst_thread =3D (ud->psil_base + uc->rchan->id) | - K3_PSIL_DST_THREAD_ID_OFFSET; - - irq_ring_idx =3D uc->rflow->id + oes->pktdma_rchan_flow; - - ret =3D pktdma_tisci_rx_channel_config(uc); - break; - default: - /* Can not happen */ - dev_err(uc->ud->dev, "%s: chan%d invalid direction (%u)\n", - __func__, uc->id, uc->config.dir); - return -EINVAL; - } - - /* check if the channel configuration was successful */ - if (ret) - goto err_res_free; - - if (udma_is_chan_running(uc)) { - dev_warn(ud->dev, "chan%d: is running!\n", uc->id); - udma_reset_chan(uc, false); - if (udma_is_chan_running(uc)) { - dev_err(ud->dev, "chan%d: won't stop!\n", uc->id); - ret =3D -EBUSY; - goto err_res_free; - } - } - - uc->dma_dev =3D dmaengine_get_dma_device(chan); - uc->hdesc_pool =3D dma_pool_create(uc->name, uc->dma_dev, - uc->config.hdesc_size, ud->desc_align, - 0); - if (!uc->hdesc_pool) { - dev_err(ud->ddev.dev, - "Descriptor pool allocation failed\n"); - uc->use_dma_pool =3D false; - ret =3D -ENOMEM; - goto err_res_free; - } - - uc->use_dma_pool =3D true; - - /* PSI-L pairing */ - ret =3D navss_psil_pair(ud, uc->config.src_thread, uc->config.dst_thread); - if (ret) { - dev_err(ud->dev, "PSI-L pairing failed: 0x%04x -> 0x%04x\n", - uc->config.src_thread, uc->config.dst_thread); - goto err_res_free; - } - - uc->psil_paired =3D true; - - uc->irq_num_ring =3D msi_get_virq(ud->dev, irq_ring_idx); - if (uc->irq_num_ring <=3D 0) { - dev_err(ud->dev, "Failed to get ring irq (index: %u)\n", - irq_ring_idx); - ret =3D -EINVAL; - goto err_psi_free; - } - - ret =3D request_irq(uc->irq_num_ring, udma_ring_irq_handler, - IRQF_TRIGGER_HIGH, uc->name, uc); - if (ret) { - dev_err(ud->dev, "chan%d: ring irq request failed\n", uc->id); - goto err_irq_free; - } - - uc->irq_num_udma =3D 0; - - udma_reset_rings(uc); - - INIT_DELAYED_WORK_ONSTACK(&uc->tx_drain.work, - udma_check_tx_completion); - - if (uc->tchan) - dev_dbg(ud->dev, - "chan%d: tchan%d, tflow%d, Remote thread: 0x%04x\n", - uc->id, uc->tchan->id, uc->tchan->tflow_id, - uc->config.remote_thread_id); - else if (uc->rchan) - dev_dbg(ud->dev, - "chan%d: rchan%d, rflow%d, Remote thread: 0x%04x\n", - uc->id, uc->rchan->id, uc->rflow->id, - uc->config.remote_thread_id); - return 0; - -err_irq_free: - uc->irq_num_ring =3D 0; -err_psi_free: - navss_psil_unpair(ud, uc->config.src_thread, uc->config.dst_thread); - uc->psil_paired =3D false; -err_res_free: - udma_free_tx_resources(uc); - udma_free_rx_resources(uc); - - udma_reset_uchan(uc); - - dma_pool_destroy(uc->hdesc_pool); - uc->use_dma_pool =3D false; - - return ret; -} - -static int udma_slave_config(struct dma_chan *chan, - struct dma_slave_config *cfg) -{ - struct udma_chan *uc =3D to_udma_chan(chan); - - memcpy(&uc->cfg, cfg, sizeof(uc->cfg)); - - return 0; -} - -static struct udma_desc *udma_alloc_tr_desc(struct udma_chan *uc, - size_t tr_size, int tr_count, - enum dma_transfer_direction dir) -{ - struct udma_hwdesc *hwdesc; - struct cppi5_desc_hdr_t *tr_desc; - struct udma_desc *d; - u32 reload_count =3D 0; - u32 ring_id; - - switch (tr_size) { - case 16: - case 32: - case 64: - case 128: - break; - default: - dev_err(uc->ud->dev, "Unsupported TR size of %zu\n", tr_size); - return NULL; - } - - /* We have only one descriptor containing multiple TRs */ - d =3D kzalloc(sizeof(*d) + sizeof(d->hwdesc[0]), GFP_NOWAIT); - if (!d) - return NULL; - - d->sglen =3D tr_count; - - d->hwdesc_count =3D 1; - hwdesc =3D &d->hwdesc[0]; - - /* Allocate memory for DMA ring descriptor */ - if (uc->use_dma_pool) { - hwdesc->cppi5_desc_size =3D uc->config.hdesc_size; - hwdesc->cppi5_desc_vaddr =3D dma_pool_zalloc(uc->hdesc_pool, - GFP_NOWAIT, - &hwdesc->cppi5_desc_paddr); - } else { - hwdesc->cppi5_desc_size =3D cppi5_trdesc_calc_size(tr_size, - tr_count); - hwdesc->cppi5_desc_size =3D ALIGN(hwdesc->cppi5_desc_size, - uc->ud->desc_align); - hwdesc->cppi5_desc_vaddr =3D dma_alloc_coherent(uc->ud->dev, - hwdesc->cppi5_desc_size, - &hwdesc->cppi5_desc_paddr, - GFP_NOWAIT); - } - - if (!hwdesc->cppi5_desc_vaddr) { - kfree(d); - return NULL; - } - - /* Start of the TR req records */ - hwdesc->tr_req_base =3D hwdesc->cppi5_desc_vaddr + tr_size; - /* Start address of the TR response array */ - hwdesc->tr_resp_base =3D hwdesc->tr_req_base + tr_size * tr_count; - - tr_desc =3D hwdesc->cppi5_desc_vaddr; - - if (uc->cyclic) - reload_count =3D CPPI5_INFO0_TRDESC_RLDCNT_INFINITE; - - if (dir =3D=3D DMA_DEV_TO_MEM) - ring_id =3D k3_ringacc_get_ring_id(uc->rflow->r_ring); - else - ring_id =3D k3_ringacc_get_ring_id(uc->tchan->tc_ring); - - cppi5_trdesc_init(tr_desc, tr_count, tr_size, 0, reload_count); - cppi5_desc_set_pktids(tr_desc, uc->id, - CPPI5_INFO1_DESC_FLOWID_DEFAULT); - cppi5_desc_set_retpolicy(tr_desc, 0, ring_id); - - return d; -} - -/** - * udma_get_tr_counters - calculate TR counters for a given length - * @len: Length of the trasnfer - * @align_to: Preferred alignment - * @tr0_cnt0: First TR icnt0 - * @tr0_cnt1: First TR icnt1 - * @tr1_cnt0: Second (if used) TR icnt0 - * - * For len < SZ_64K only one TR is enough, tr1_cnt0 is not updated - * For len >=3D SZ_64K two TRs are used in a simple way: - * First TR: SZ_64K-alignment blocks (tr0_cnt0, tr0_cnt1) - * Second TR: the remaining length (tr1_cnt0) - * - * Returns the number of TRs the length needs (1 or 2) - * -EINVAL if the length can not be supported - */ -static int udma_get_tr_counters(size_t len, unsigned long align_to, - u16 *tr0_cnt0, u16 *tr0_cnt1, u16 *tr1_cnt0) -{ - if (len < SZ_64K) { - *tr0_cnt0 =3D len; - *tr0_cnt1 =3D 1; - - return 1; - } - - if (align_to > 3) - align_to =3D 3; - -realign: - *tr0_cnt0 =3D SZ_64K - BIT(align_to); - if (len / *tr0_cnt0 >=3D SZ_64K) { - if (align_to) { - align_to--; - goto realign; - } - return -EINVAL; - } - - *tr0_cnt1 =3D len / *tr0_cnt0; - *tr1_cnt0 =3D len % *tr0_cnt0; - - return 2; -} - -static struct udma_desc * -udma_prep_slave_sg_tr(struct udma_chan *uc, struct scatterlist *sgl, - unsigned int sglen, enum dma_transfer_direction dir, - unsigned long tx_flags, void *context) -{ - struct scatterlist *sgent; - struct udma_desc *d; - struct cppi5_tr_type1_t *tr_req =3D NULL; - u16 tr0_cnt0, tr0_cnt1, tr1_cnt0; - unsigned int i; - size_t tr_size; - int num_tr =3D 0; - int tr_idx =3D 0; - u64 asel; - - /* estimate the number of TRs we will need */ - for_each_sg(sgl, sgent, sglen, i) { - if (sg_dma_len(sgent) < SZ_64K) - num_tr++; - else - num_tr +=3D 2; - } - - /* Now allocate and setup the descriptor. */ - tr_size =3D sizeof(struct cppi5_tr_type1_t); - d =3D udma_alloc_tr_desc(uc, tr_size, num_tr, dir); - if (!d) - return NULL; - - d->sglen =3D sglen; - - if (uc->ud->match_data->type =3D=3D DMA_TYPE_UDMA) - asel =3D 0; - else - asel =3D (u64)uc->config.asel << K3_ADDRESS_ASEL_SHIFT; - - tr_req =3D d->hwdesc[0].tr_req_base; - for_each_sg(sgl, sgent, sglen, i) { - dma_addr_t sg_addr =3D sg_dma_address(sgent); - - num_tr =3D udma_get_tr_counters(sg_dma_len(sgent), __ffs(sg_addr), - &tr0_cnt0, &tr0_cnt1, &tr1_cnt0); - if (num_tr < 0) { - dev_err(uc->ud->dev, "size %u is not supported\n", - sg_dma_len(sgent)); - udma_free_hwdesc(uc, d); - kfree(d); - return NULL; - } - - cppi5_tr_init(&tr_req[tr_idx].flags, CPPI5_TR_TYPE1, false, - false, CPPI5_TR_EVENT_SIZE_COMPLETION, 0); - cppi5_tr_csf_set(&tr_req[tr_idx].flags, CPPI5_TR_CSF_SUPR_EVT); - - sg_addr |=3D asel; - tr_req[tr_idx].addr =3D sg_addr; - tr_req[tr_idx].icnt0 =3D tr0_cnt0; - tr_req[tr_idx].icnt1 =3D tr0_cnt1; - tr_req[tr_idx].dim1 =3D tr0_cnt0; - tr_idx++; - - if (num_tr =3D=3D 2) { - cppi5_tr_init(&tr_req[tr_idx].flags, CPPI5_TR_TYPE1, - false, false, - CPPI5_TR_EVENT_SIZE_COMPLETION, 0); - cppi5_tr_csf_set(&tr_req[tr_idx].flags, - CPPI5_TR_CSF_SUPR_EVT); - - tr_req[tr_idx].addr =3D sg_addr + tr0_cnt1 * tr0_cnt0; - tr_req[tr_idx].icnt0 =3D tr1_cnt0; - tr_req[tr_idx].icnt1 =3D 1; - tr_req[tr_idx].dim1 =3D tr1_cnt0; - tr_idx++; - } - - d->residue +=3D sg_dma_len(sgent); - } - - cppi5_tr_csf_set(&tr_req[tr_idx - 1].flags, - CPPI5_TR_CSF_SUPR_EVT | CPPI5_TR_CSF_EOP); - - return d; -} - -static struct udma_desc * -udma_prep_slave_sg_triggered_tr(struct udma_chan *uc, struct scatterlist *= sgl, - unsigned int sglen, - enum dma_transfer_direction dir, - unsigned long tx_flags, void *context) -{ - struct scatterlist *sgent; - struct cppi5_tr_type15_t *tr_req =3D NULL; - enum dma_slave_buswidth dev_width; - u32 csf =3D CPPI5_TR_CSF_SUPR_EVT; - u16 tr_cnt0, tr_cnt1; - dma_addr_t dev_addr; - struct udma_desc *d; - unsigned int i; - size_t tr_size, sg_len; - int num_tr =3D 0; - int tr_idx =3D 0; - u32 burst, trigger_size, port_window; - u64 asel; - - if (dir =3D=3D DMA_DEV_TO_MEM) { - dev_addr =3D uc->cfg.src_addr; - dev_width =3D uc->cfg.src_addr_width; - burst =3D uc->cfg.src_maxburst; - port_window =3D uc->cfg.src_port_window_size; - } else if (dir =3D=3D DMA_MEM_TO_DEV) { - dev_addr =3D uc->cfg.dst_addr; - dev_width =3D uc->cfg.dst_addr_width; - burst =3D uc->cfg.dst_maxburst; - port_window =3D uc->cfg.dst_port_window_size; - } else { - dev_err(uc->ud->dev, "%s: bad direction?\n", __func__); - return NULL; - } - - if (!burst) - burst =3D 1; - - if (port_window) { - if (port_window !=3D burst) { - dev_err(uc->ud->dev, - "The burst must be equal to port_window\n"); - return NULL; - } - - tr_cnt0 =3D dev_width * port_window; - tr_cnt1 =3D 1; - } else { - tr_cnt0 =3D dev_width; - tr_cnt1 =3D burst; - } - trigger_size =3D tr_cnt0 * tr_cnt1; - - /* estimate the number of TRs we will need */ - for_each_sg(sgl, sgent, sglen, i) { - sg_len =3D sg_dma_len(sgent); - - if (sg_len % trigger_size) { - dev_err(uc->ud->dev, - "Not aligned SG entry (%zu for %u)\n", sg_len, - trigger_size); - return NULL; - } - - if (sg_len / trigger_size < SZ_64K) - num_tr++; - else - num_tr +=3D 2; - } - - /* Now allocate and setup the descriptor. */ - tr_size =3D sizeof(struct cppi5_tr_type15_t); - d =3D udma_alloc_tr_desc(uc, tr_size, num_tr, dir); - if (!d) - return NULL; - - d->sglen =3D sglen; - - if (uc->ud->match_data->type =3D=3D DMA_TYPE_UDMA) { - asel =3D 0; - csf |=3D CPPI5_TR_CSF_EOL_ICNT0; - } else { - asel =3D (u64)uc->config.asel << K3_ADDRESS_ASEL_SHIFT; - dev_addr |=3D asel; - } - - tr_req =3D d->hwdesc[0].tr_req_base; - for_each_sg(sgl, sgent, sglen, i) { - u16 tr0_cnt2, tr0_cnt3, tr1_cnt2; - dma_addr_t sg_addr =3D sg_dma_address(sgent); - - sg_len =3D sg_dma_len(sgent); - num_tr =3D udma_get_tr_counters(sg_len / trigger_size, 0, - &tr0_cnt2, &tr0_cnt3, &tr1_cnt2); - if (num_tr < 0) { - dev_err(uc->ud->dev, "size %zu is not supported\n", - sg_len); - udma_free_hwdesc(uc, d); - kfree(d); - return NULL; - } - - cppi5_tr_init(&tr_req[tr_idx].flags, CPPI5_TR_TYPE15, false, - true, CPPI5_TR_EVENT_SIZE_COMPLETION, 0); - cppi5_tr_csf_set(&tr_req[tr_idx].flags, csf); - cppi5_tr_set_trigger(&tr_req[tr_idx].flags, - uc->config.tr_trigger_type, - CPPI5_TR_TRIGGER_TYPE_ICNT2_DEC, 0, 0); - - sg_addr |=3D asel; - if (dir =3D=3D DMA_DEV_TO_MEM) { - tr_req[tr_idx].addr =3D dev_addr; - tr_req[tr_idx].icnt0 =3D tr_cnt0; - tr_req[tr_idx].icnt1 =3D tr_cnt1; - tr_req[tr_idx].icnt2 =3D tr0_cnt2; - tr_req[tr_idx].icnt3 =3D tr0_cnt3; - tr_req[tr_idx].dim1 =3D (-1) * tr_cnt0; - - tr_req[tr_idx].daddr =3D sg_addr; - tr_req[tr_idx].dicnt0 =3D tr_cnt0; - tr_req[tr_idx].dicnt1 =3D tr_cnt1; - tr_req[tr_idx].dicnt2 =3D tr0_cnt2; - tr_req[tr_idx].dicnt3 =3D tr0_cnt3; - tr_req[tr_idx].ddim1 =3D tr_cnt0; - tr_req[tr_idx].ddim2 =3D trigger_size; - tr_req[tr_idx].ddim3 =3D trigger_size * tr0_cnt2; - } else { - tr_req[tr_idx].addr =3D sg_addr; - tr_req[tr_idx].icnt0 =3D tr_cnt0; - tr_req[tr_idx].icnt1 =3D tr_cnt1; - tr_req[tr_idx].icnt2 =3D tr0_cnt2; - tr_req[tr_idx].icnt3 =3D tr0_cnt3; - tr_req[tr_idx].dim1 =3D tr_cnt0; - tr_req[tr_idx].dim2 =3D trigger_size; - tr_req[tr_idx].dim3 =3D trigger_size * tr0_cnt2; - - tr_req[tr_idx].daddr =3D dev_addr; - tr_req[tr_idx].dicnt0 =3D tr_cnt0; - tr_req[tr_idx].dicnt1 =3D tr_cnt1; - tr_req[tr_idx].dicnt2 =3D tr0_cnt2; - tr_req[tr_idx].dicnt3 =3D tr0_cnt3; - tr_req[tr_idx].ddim1 =3D (-1) * tr_cnt0; - } - - tr_idx++; - - if (num_tr =3D=3D 2) { - cppi5_tr_init(&tr_req[tr_idx].flags, CPPI5_TR_TYPE15, - false, true, - CPPI5_TR_EVENT_SIZE_COMPLETION, 0); - cppi5_tr_csf_set(&tr_req[tr_idx].flags, csf); - cppi5_tr_set_trigger(&tr_req[tr_idx].flags, - uc->config.tr_trigger_type, - CPPI5_TR_TRIGGER_TYPE_ICNT2_DEC, - 0, 0); - - sg_addr +=3D trigger_size * tr0_cnt2 * tr0_cnt3; - if (dir =3D=3D DMA_DEV_TO_MEM) { - tr_req[tr_idx].addr =3D dev_addr; - tr_req[tr_idx].icnt0 =3D tr_cnt0; - tr_req[tr_idx].icnt1 =3D tr_cnt1; - tr_req[tr_idx].icnt2 =3D tr1_cnt2; - tr_req[tr_idx].icnt3 =3D 1; - tr_req[tr_idx].dim1 =3D (-1) * tr_cnt0; - - tr_req[tr_idx].daddr =3D sg_addr; - tr_req[tr_idx].dicnt0 =3D tr_cnt0; - tr_req[tr_idx].dicnt1 =3D tr_cnt1; - tr_req[tr_idx].dicnt2 =3D tr1_cnt2; - tr_req[tr_idx].dicnt3 =3D 1; - tr_req[tr_idx].ddim1 =3D tr_cnt0; - tr_req[tr_idx].ddim2 =3D trigger_size; - } else { - tr_req[tr_idx].addr =3D sg_addr; - tr_req[tr_idx].icnt0 =3D tr_cnt0; - tr_req[tr_idx].icnt1 =3D tr_cnt1; - tr_req[tr_idx].icnt2 =3D tr1_cnt2; - tr_req[tr_idx].icnt3 =3D 1; - tr_req[tr_idx].dim1 =3D tr_cnt0; - tr_req[tr_idx].dim2 =3D trigger_size; - - tr_req[tr_idx].daddr =3D dev_addr; - tr_req[tr_idx].dicnt0 =3D tr_cnt0; - tr_req[tr_idx].dicnt1 =3D tr_cnt1; - tr_req[tr_idx].dicnt2 =3D tr1_cnt2; - tr_req[tr_idx].dicnt3 =3D 1; - tr_req[tr_idx].ddim1 =3D (-1) * tr_cnt0; - } - tr_idx++; - } - - d->residue +=3D sg_len; - } - - cppi5_tr_csf_set(&tr_req[tr_idx - 1].flags, csf | CPPI5_TR_CSF_EOP); - - return d; -} - -static int udma_configure_statictr(struct udma_chan *uc, struct udma_desc = *d, - enum dma_slave_buswidth dev_width, - u16 elcnt) -{ - if (uc->config.ep_type !=3D PSIL_EP_PDMA_XY) - return 0; - - /* Bus width translates to the element size (ES) */ - switch (dev_width) { - case DMA_SLAVE_BUSWIDTH_1_BYTE: - d->static_tr.elsize =3D 0; - break; - case DMA_SLAVE_BUSWIDTH_2_BYTES: - d->static_tr.elsize =3D 1; - break; - case DMA_SLAVE_BUSWIDTH_3_BYTES: - d->static_tr.elsize =3D 2; - break; - case DMA_SLAVE_BUSWIDTH_4_BYTES: - d->static_tr.elsize =3D 3; - break; - case DMA_SLAVE_BUSWIDTH_8_BYTES: - d->static_tr.elsize =3D 4; - break; - default: /* not reached */ - return -EINVAL; - } - - d->static_tr.elcnt =3D elcnt; - - if (uc->config.pkt_mode || !uc->cyclic) { - /* - * PDMA must close the packet when the channel is in packet mode. - * For TR mode when the channel is not cyclic we also need PDMA - * to close the packet otherwise the transfer will stall because - * PDMA holds on the data it has received from the peripheral. - */ - unsigned int div =3D dev_width * elcnt; - - if (uc->cyclic) - d->static_tr.bstcnt =3D d->residue / d->sglen / div; - else - d->static_tr.bstcnt =3D d->residue / div; - } else if (uc->ud->match_data->type =3D=3D DMA_TYPE_BCDMA && - uc->config.dir =3D=3D DMA_DEV_TO_MEM && - uc->cyclic) { - /* - * For cyclic mode with BCDMA we have to set EOP in each TR to - * prevent short packet errors seen on channel teardown. So the - * PDMA must close the packet after every TR transfer by setting - * burst count equal to the number of bytes transferred. - */ - struct cppi5_tr_type1_t *tr_req =3D d->hwdesc[0].tr_req_base; - - d->static_tr.bstcnt =3D - (tr_req->icnt0 * tr_req->icnt1) / dev_width; - } else { - d->static_tr.bstcnt =3D 0; - } - - if (uc->config.dir =3D=3D DMA_DEV_TO_MEM && - d->static_tr.bstcnt > uc->ud->match_data->statictr_z_mask) - return -EINVAL; - - return 0; -} - -static struct udma_desc * -udma_prep_slave_sg_pkt(struct udma_chan *uc, struct scatterlist *sgl, - unsigned int sglen, enum dma_transfer_direction dir, - unsigned long tx_flags, void *context) -{ - struct scatterlist *sgent; - struct cppi5_host_desc_t *h_desc =3D NULL; - struct udma_desc *d; - u32 ring_id; - unsigned int i; - u64 asel; - - d =3D kzalloc(struct_size(d, hwdesc, sglen), GFP_NOWAIT); - if (!d) - return NULL; - - d->sglen =3D sglen; - d->hwdesc_count =3D sglen; - - if (dir =3D=3D DMA_DEV_TO_MEM) - ring_id =3D k3_ringacc_get_ring_id(uc->rflow->r_ring); - else - ring_id =3D k3_ringacc_get_ring_id(uc->tchan->tc_ring); - - if (uc->ud->match_data->type =3D=3D DMA_TYPE_UDMA) - asel =3D 0; - else - asel =3D (u64)uc->config.asel << K3_ADDRESS_ASEL_SHIFT; - - for_each_sg(sgl, sgent, sglen, i) { - struct udma_hwdesc *hwdesc =3D &d->hwdesc[i]; - dma_addr_t sg_addr =3D sg_dma_address(sgent); - struct cppi5_host_desc_t *desc; - size_t sg_len =3D sg_dma_len(sgent); - - hwdesc->cppi5_desc_vaddr =3D dma_pool_zalloc(uc->hdesc_pool, - GFP_NOWAIT, - &hwdesc->cppi5_desc_paddr); - if (!hwdesc->cppi5_desc_vaddr) { - dev_err(uc->ud->dev, - "descriptor%d allocation failed\n", i); - - udma_free_hwdesc(uc, d); - kfree(d); - return NULL; - } - - d->residue +=3D sg_len; - hwdesc->cppi5_desc_size =3D uc->config.hdesc_size; - desc =3D hwdesc->cppi5_desc_vaddr; - - if (i =3D=3D 0) { - cppi5_hdesc_init(desc, 0, 0); - /* Flow and Packed ID */ - cppi5_desc_set_pktids(&desc->hdr, uc->id, - CPPI5_INFO1_DESC_FLOWID_DEFAULT); - cppi5_desc_set_retpolicy(&desc->hdr, 0, ring_id); - } else { - cppi5_hdesc_reset_hbdesc(desc); - cppi5_desc_set_retpolicy(&desc->hdr, 0, 0xffff); - } - - /* attach the sg buffer to the descriptor */ - sg_addr |=3D asel; - cppi5_hdesc_attach_buf(desc, sg_addr, sg_len, sg_addr, sg_len); - - /* Attach link as host buffer descriptor */ - if (h_desc) - cppi5_hdesc_link_hbdesc(h_desc, - hwdesc->cppi5_desc_paddr | asel); - - if (uc->ud->match_data->type =3D=3D DMA_TYPE_PKTDMA || - dir =3D=3D DMA_MEM_TO_DEV) - h_desc =3D desc; - } - - if (d->residue >=3D SZ_4M) { - dev_err(uc->ud->dev, - "%s: Transfer size %u is over the supported 4M range\n", - __func__, d->residue); - udma_free_hwdesc(uc, d); - kfree(d); - return NULL; - } - - h_desc =3D d->hwdesc[0].cppi5_desc_vaddr; - cppi5_hdesc_set_pktlen(h_desc, d->residue); - - return d; -} - -static int udma_attach_metadata(struct dma_async_tx_descriptor *desc, - void *data, size_t len) -{ - struct udma_desc *d =3D to_udma_desc(desc); - struct udma_chan *uc =3D to_udma_chan(desc->chan); - struct cppi5_host_desc_t *h_desc; - u32 psd_size =3D len; - u32 flags =3D 0; - - if (!uc->config.pkt_mode || !uc->config.metadata_size) - return -ENOTSUPP; - - if (!data || len > uc->config.metadata_size) - return -EINVAL; - - if (uc->config.needs_epib && len < CPPI5_INFO0_HDESC_EPIB_SIZE) - return -EINVAL; - - h_desc =3D d->hwdesc[0].cppi5_desc_vaddr; - if (d->dir =3D=3D DMA_MEM_TO_DEV) - memcpy(h_desc->epib, data, len); - - if (uc->config.needs_epib) - psd_size -=3D CPPI5_INFO0_HDESC_EPIB_SIZE; - - d->metadata =3D data; - d->metadata_size =3D len; - if (uc->config.needs_epib) - flags |=3D CPPI5_INFO0_HDESC_EPIB_PRESENT; - - cppi5_hdesc_update_flags(h_desc, flags); - cppi5_hdesc_update_psdata_size(h_desc, psd_size); - - return 0; -} - -static void *udma_get_metadata_ptr(struct dma_async_tx_descriptor *desc, - size_t *payload_len, size_t *max_len) -{ - struct udma_desc *d =3D to_udma_desc(desc); - struct udma_chan *uc =3D to_udma_chan(desc->chan); - struct cppi5_host_desc_t *h_desc; - - if (!uc->config.pkt_mode || !uc->config.metadata_size) - return ERR_PTR(-ENOTSUPP); - - h_desc =3D d->hwdesc[0].cppi5_desc_vaddr; - - *max_len =3D uc->config.metadata_size; - - *payload_len =3D cppi5_hdesc_epib_present(&h_desc->hdr) ? - CPPI5_INFO0_HDESC_EPIB_SIZE : 0; - *payload_len +=3D cppi5_hdesc_get_psdata_size(h_desc); - - return h_desc->epib; -} - -static int udma_set_metadata_len(struct dma_async_tx_descriptor *desc, - size_t payload_len) -{ - struct udma_desc *d =3D to_udma_desc(desc); - struct udma_chan *uc =3D to_udma_chan(desc->chan); - struct cppi5_host_desc_t *h_desc; - u32 psd_size =3D payload_len; - u32 flags =3D 0; - - if (!uc->config.pkt_mode || !uc->config.metadata_size) - return -ENOTSUPP; - - if (payload_len > uc->config.metadata_size) - return -EINVAL; - - if (uc->config.needs_epib && payload_len < CPPI5_INFO0_HDESC_EPIB_SIZE) - return -EINVAL; - - h_desc =3D d->hwdesc[0].cppi5_desc_vaddr; - - if (uc->config.needs_epib) { - psd_size -=3D CPPI5_INFO0_HDESC_EPIB_SIZE; - flags |=3D CPPI5_INFO0_HDESC_EPIB_PRESENT; - } - - cppi5_hdesc_update_flags(h_desc, flags); - cppi5_hdesc_update_psdata_size(h_desc, psd_size); - - return 0; -} - -static struct dma_descriptor_metadata_ops metadata_ops =3D { - .attach =3D udma_attach_metadata, - .get_ptr =3D udma_get_metadata_ptr, - .set_len =3D udma_set_metadata_len, -}; - -static struct dma_async_tx_descriptor * -udma_prep_slave_sg(struct dma_chan *chan, struct scatterlist *sgl, - unsigned int sglen, enum dma_transfer_direction dir, - unsigned long tx_flags, void *context) -{ - struct udma_chan *uc =3D to_udma_chan(chan); - enum dma_slave_buswidth dev_width; - struct udma_desc *d; - u32 burst; - - if (dir !=3D uc->config.dir && - (uc->config.dir =3D=3D DMA_MEM_TO_MEM && !uc->config.tr_trigger_type)= ) { - dev_err(chan->device->dev, - "%s: chan%d is for %s, not supporting %s\n", - __func__, uc->id, - dmaengine_get_direction_text(uc->config.dir), - dmaengine_get_direction_text(dir)); - return NULL; - } - - if (dir =3D=3D DMA_DEV_TO_MEM) { - dev_width =3D uc->cfg.src_addr_width; - burst =3D uc->cfg.src_maxburst; - } else if (dir =3D=3D DMA_MEM_TO_DEV) { - dev_width =3D uc->cfg.dst_addr_width; - burst =3D uc->cfg.dst_maxburst; - } else { - dev_err(chan->device->dev, "%s: bad direction?\n", __func__); - return NULL; - } - - if (!burst) - burst =3D 1; - - uc->config.tx_flags =3D tx_flags; - - if (uc->config.pkt_mode) - d =3D udma_prep_slave_sg_pkt(uc, sgl, sglen, dir, tx_flags, - context); - else if (is_slave_direction(uc->config.dir)) - d =3D udma_prep_slave_sg_tr(uc, sgl, sglen, dir, tx_flags, - context); - else - d =3D udma_prep_slave_sg_triggered_tr(uc, sgl, sglen, dir, - tx_flags, context); - - if (!d) - return NULL; - - d->dir =3D dir; - d->desc_idx =3D 0; - d->tr_idx =3D 0; - - /* static TR for remote PDMA */ - if (udma_configure_statictr(uc, d, dev_width, burst)) { - dev_err(uc->ud->dev, - "%s: StaticTR Z is limited to maximum %u (%u)\n", - __func__, uc->ud->match_data->statictr_z_mask, - d->static_tr.bstcnt); - - udma_free_hwdesc(uc, d); - kfree(d); - return NULL; - } - - if (uc->config.metadata_size) - d->vd.tx.metadata_ops =3D &metadata_ops; - - return vchan_tx_prep(&uc->vc, &d->vd, tx_flags); -} - -static struct udma_desc * -udma_prep_dma_cyclic_tr(struct udma_chan *uc, dma_addr_t buf_addr, - size_t buf_len, size_t period_len, - enum dma_transfer_direction dir, unsigned long flags) -{ - struct udma_desc *d; - size_t tr_size, period_addr; - struct cppi5_tr_type1_t *tr_req; - unsigned int periods =3D buf_len / period_len; - u16 tr0_cnt0, tr0_cnt1, tr1_cnt0; - unsigned int i; - int num_tr; - u32 period_csf =3D 0; - - num_tr =3D udma_get_tr_counters(period_len, __ffs(buf_addr), &tr0_cnt0, - &tr0_cnt1, &tr1_cnt0); - if (num_tr < 0) { - dev_err(uc->ud->dev, "size %zu is not supported\n", - period_len); - return NULL; - } - - /* Now allocate and setup the descriptor. */ - tr_size =3D sizeof(struct cppi5_tr_type1_t); - d =3D udma_alloc_tr_desc(uc, tr_size, periods * num_tr, dir); - if (!d) - return NULL; - - tr_req =3D d->hwdesc[0].tr_req_base; - if (uc->ud->match_data->type =3D=3D DMA_TYPE_UDMA) - period_addr =3D buf_addr; - else - period_addr =3D buf_addr | - ((u64)uc->config.asel << K3_ADDRESS_ASEL_SHIFT); - - /* - * For BCDMA <-> PDMA transfers, the EOP flag needs to be set on the - * last TR of a descriptor, to mark the packet as complete. - * This is required for getting the teardown completion message in case - * of TX, and to avoid short-packet error in case of RX. - * - * As we are in cyclic mode, we do not know which period might be the - * last one, so set the flag for each period. - */ - if (uc->config.ep_type =3D=3D PSIL_EP_PDMA_XY && - uc->ud->match_data->type =3D=3D DMA_TYPE_BCDMA) { - period_csf =3D CPPI5_TR_CSF_EOP; - } - - for (i =3D 0; i < periods; i++) { - int tr_idx =3D i * num_tr; - - cppi5_tr_init(&tr_req[tr_idx].flags, CPPI5_TR_TYPE1, false, - false, CPPI5_TR_EVENT_SIZE_COMPLETION, 0); - - tr_req[tr_idx].addr =3D period_addr; - tr_req[tr_idx].icnt0 =3D tr0_cnt0; - tr_req[tr_idx].icnt1 =3D tr0_cnt1; - tr_req[tr_idx].dim1 =3D tr0_cnt0; - - if (num_tr =3D=3D 2) { - cppi5_tr_csf_set(&tr_req[tr_idx].flags, - CPPI5_TR_CSF_SUPR_EVT); - tr_idx++; - - cppi5_tr_init(&tr_req[tr_idx].flags, CPPI5_TR_TYPE1, - false, false, - CPPI5_TR_EVENT_SIZE_COMPLETION, 0); - - tr_req[tr_idx].addr =3D period_addr + tr0_cnt1 * tr0_cnt0; - tr_req[tr_idx].icnt0 =3D tr1_cnt0; - tr_req[tr_idx].icnt1 =3D 1; - tr_req[tr_idx].dim1 =3D tr1_cnt0; - } - - if (!(flags & DMA_PREP_INTERRUPT)) - period_csf |=3D CPPI5_TR_CSF_SUPR_EVT; - - if (period_csf) - cppi5_tr_csf_set(&tr_req[tr_idx].flags, period_csf); - - period_addr +=3D period_len; - } - - return d; -} - -static struct udma_desc * -udma_prep_dma_cyclic_pkt(struct udma_chan *uc, dma_addr_t buf_addr, - size_t buf_len, size_t period_len, - enum dma_transfer_direction dir, unsigned long flags) -{ - struct udma_desc *d; - u32 ring_id; - int i; - int periods =3D buf_len / period_len; - - if (periods > (K3_UDMA_DEFAULT_RING_SIZE - 1)) - return NULL; - - if (period_len >=3D SZ_4M) - return NULL; - - d =3D kzalloc(struct_size(d, hwdesc, periods), GFP_NOWAIT); - if (!d) - return NULL; - - d->hwdesc_count =3D periods; - - /* TODO: re-check this... */ - if (dir =3D=3D DMA_DEV_TO_MEM) - ring_id =3D k3_ringacc_get_ring_id(uc->rflow->r_ring); - else - ring_id =3D k3_ringacc_get_ring_id(uc->tchan->tc_ring); - - if (uc->ud->match_data->type !=3D DMA_TYPE_UDMA) - buf_addr |=3D (u64)uc->config.asel << K3_ADDRESS_ASEL_SHIFT; + udma_free_tx_resources(uc); + udma_free_rx_resources(uc); =20 - for (i =3D 0; i < periods; i++) { - struct udma_hwdesc *hwdesc =3D &d->hwdesc[i]; - dma_addr_t period_addr =3D buf_addr + (period_len * i); - struct cppi5_host_desc_t *h_desc; + udma_reset_uchan(uc); =20 - hwdesc->cppi5_desc_vaddr =3D dma_pool_zalloc(uc->hdesc_pool, - GFP_NOWAIT, - &hwdesc->cppi5_desc_paddr); - if (!hwdesc->cppi5_desc_vaddr) { - dev_err(uc->ud->dev, - "descriptor%d allocation failed\n", i); + if (uc->use_dma_pool) { + dma_pool_destroy(uc->hdesc_pool); + uc->use_dma_pool =3D false; + } =20 - udma_free_hwdesc(uc, d); - kfree(d); - return NULL; - } + return ret; +} =20 - hwdesc->cppi5_desc_size =3D uc->config.hdesc_size; - h_desc =3D hwdesc->cppi5_desc_vaddr; +static int bcdma_router_config(struct dma_chan *chan) +{ + struct k3_event_route_data *router_data =3D chan->route_data; + struct udma_chan *uc =3D to_udma_chan(chan); + u32 trigger_event; =20 - cppi5_hdesc_init(h_desc, 0, 0); - cppi5_hdesc_set_pktlen(h_desc, period_len); + if (!uc->bchan) + return -EINVAL; =20 - /* Flow and Packed ID */ - cppi5_desc_set_pktids(&h_desc->hdr, uc->id, - CPPI5_INFO1_DESC_FLOWID_DEFAULT); - cppi5_desc_set_retpolicy(&h_desc->hdr, 0, ring_id); + if (uc->config.tr_trigger_type !=3D 1 && uc->config.tr_trigger_type !=3D = 2) + return -EINVAL; =20 - /* attach each period to a new descriptor */ - cppi5_hdesc_attach_buf(h_desc, - period_addr, period_len, - period_addr, period_len); - } + trigger_event =3D uc->ud->soc_data->bcdma_trigger_event_offset; + trigger_event +=3D (uc->bchan->id * 2) + uc->config.tr_trigger_type - 1; =20 - return d; + return router_data->set_event(router_data->priv, trigger_event); } =20 -static struct dma_async_tx_descriptor * -udma_prep_dma_cyclic(struct dma_chan *chan, dma_addr_t buf_addr, size_t bu= f_len, - size_t period_len, enum dma_transfer_direction dir, - unsigned long flags) +static int pktdma_alloc_chan_resources(struct dma_chan *chan) { struct udma_chan *uc =3D to_udma_chan(chan); - enum dma_slave_buswidth dev_width; - struct udma_desc *d; - u32 burst; - - if (dir !=3D uc->config.dir) { - dev_err(chan->device->dev, - "%s: chan%d is for %s, not supporting %s\n", - __func__, uc->id, - dmaengine_get_direction_text(uc->config.dir), - dmaengine_get_direction_text(dir)); - return NULL; - } + struct udma_dev *ud =3D to_udma_dev(chan->device); + const struct udma_oes_offsets *oes =3D &ud->soc_data->oes; + u32 irq_ring_idx; + int ret; + + /* + * Make sure that the completion is in a known state: + * No teardown, the channel is idle + */ + reinit_completion(&uc->teardown_completed); + complete_all(&uc->teardown_completed); + uc->state =3D UDMA_CHAN_IS_IDLE; =20 - uc->cyclic =3D true; + switch (uc->config.dir) { + case DMA_MEM_TO_DEV: + /* Slave transfer synchronized - mem to dev (TX) trasnfer */ + dev_dbg(uc->ud->dev, "%s: chan%d as MEM-to-DEV\n", __func__, + uc->id); =20 - if (dir =3D=3D DMA_DEV_TO_MEM) { - dev_width =3D uc->cfg.src_addr_width; - burst =3D uc->cfg.src_maxburst; - } else if (dir =3D=3D DMA_MEM_TO_DEV) { - dev_width =3D uc->cfg.dst_addr_width; - burst =3D uc->cfg.dst_maxburst; - } else { - dev_err(uc->ud->dev, "%s: bad direction?\n", __func__); - return NULL; - } + ret =3D udma_alloc_tx_resources(uc); + if (ret) { + uc->config.remote_thread_id =3D -1; + return ret; + } =20 - if (!burst) - burst =3D 1; + uc->config.src_thread =3D ud->psil_base + uc->tchan->id; + uc->config.dst_thread =3D uc->config.remote_thread_id; + uc->config.dst_thread |=3D K3_PSIL_DST_THREAD_ID_OFFSET; =20 - if (uc->config.pkt_mode) - d =3D udma_prep_dma_cyclic_pkt(uc, buf_addr, buf_len, period_len, - dir, flags); - else - d =3D udma_prep_dma_cyclic_tr(uc, buf_addr, buf_len, period_len, - dir, flags); + irq_ring_idx =3D uc->tchan->tflow_id + oes->pktdma_tchan_flow; =20 - if (!d) - return NULL; + ret =3D pktdma_tisci_tx_channel_config(uc); + break; + case DMA_DEV_TO_MEM: + /* Slave transfer synchronized - dev to mem (RX) trasnfer */ + dev_dbg(uc->ud->dev, "%s: chan%d as DEV-to-MEM\n", __func__, + uc->id); =20 - d->sglen =3D buf_len / period_len; + ret =3D udma_alloc_rx_resources(uc); + if (ret) { + uc->config.remote_thread_id =3D -1; + return ret; + } =20 - d->dir =3D dir; - d->residue =3D buf_len; + uc->config.src_thread =3D uc->config.remote_thread_id; + uc->config.dst_thread =3D (ud->psil_base + uc->rchan->id) | + K3_PSIL_DST_THREAD_ID_OFFSET; =20 - /* static TR for remote PDMA */ - if (udma_configure_statictr(uc, d, dev_width, burst)) { - dev_err(uc->ud->dev, - "%s: StaticTR Z is limited to maximum %u (%u)\n", - __func__, uc->ud->match_data->statictr_z_mask, - d->static_tr.bstcnt); + irq_ring_idx =3D uc->rflow->id + oes->pktdma_rchan_flow; =20 - udma_free_hwdesc(uc, d); - kfree(d); - return NULL; + ret =3D pktdma_tisci_rx_channel_config(uc); + break; + default: + /* Can not happen */ + dev_err(uc->ud->dev, "%s: chan%d invalid direction (%u)\n", + __func__, uc->id, uc->config.dir); + return -EINVAL; } =20 - if (uc->config.metadata_size) - d->vd.tx.metadata_ops =3D &metadata_ops; - - return vchan_tx_prep(&uc->vc, &d->vd, flags); -} + /* check if the channel configuration was successful */ + if (ret) + goto err_res_free; =20 -static struct dma_async_tx_descriptor * -udma_prep_dma_memcpy(struct dma_chan *chan, dma_addr_t dest, dma_addr_t sr= c, - size_t len, unsigned long tx_flags) -{ - struct udma_chan *uc =3D to_udma_chan(chan); - struct udma_desc *d; - struct cppi5_tr_type15_t *tr_req; - int num_tr; - size_t tr_size =3D sizeof(struct cppi5_tr_type15_t); - u16 tr0_cnt0, tr0_cnt1, tr1_cnt0; - u32 csf =3D CPPI5_TR_CSF_SUPR_EVT; - - if (uc->config.dir !=3D DMA_MEM_TO_MEM) { - dev_err(chan->device->dev, - "%s: chan%d is for %s, not supporting %s\n", - __func__, uc->id, - dmaengine_get_direction_text(uc->config.dir), - dmaengine_get_direction_text(DMA_MEM_TO_MEM)); - return NULL; + if (udma_is_chan_running(uc)) { + dev_warn(ud->dev, "chan%d: is running!\n", uc->id); + udma_reset_chan(uc, false); + if (udma_is_chan_running(uc)) { + dev_err(ud->dev, "chan%d: won't stop!\n", uc->id); + ret =3D -EBUSY; + goto err_res_free; + } } =20 - num_tr =3D udma_get_tr_counters(len, __ffs(src | dest), &tr0_cnt0, - &tr0_cnt1, &tr1_cnt0); - if (num_tr < 0) { - dev_err(uc->ud->dev, "size %zu is not supported\n", - len); - return NULL; + uc->dma_dev =3D dmaengine_get_dma_device(chan); + uc->hdesc_pool =3D dma_pool_create(uc->name, uc->dma_dev, + uc->config.hdesc_size, ud->desc_align, + 0); + if (!uc->hdesc_pool) { + dev_err(ud->ddev.dev, + "Descriptor pool allocation failed\n"); + uc->use_dma_pool =3D false; + ret =3D -ENOMEM; + goto err_res_free; } =20 - d =3D udma_alloc_tr_desc(uc, tr_size, num_tr, DMA_MEM_TO_MEM); - if (!d) - return NULL; + uc->use_dma_pool =3D true; + + /* PSI-L pairing */ + ret =3D navss_psil_pair(ud, uc->config.src_thread, uc->config.dst_thread); + if (ret) { + dev_err(ud->dev, "PSI-L pairing failed: 0x%04x -> 0x%04x\n", + uc->config.src_thread, uc->config.dst_thread); + goto err_res_free; + } =20 - d->dir =3D DMA_MEM_TO_MEM; - d->desc_idx =3D 0; - d->tr_idx =3D 0; - d->residue =3D len; + uc->psil_paired =3D true; =20 - if (uc->ud->match_data->type !=3D DMA_TYPE_UDMA) { - src |=3D (u64)uc->ud->asel << K3_ADDRESS_ASEL_SHIFT; - dest |=3D (u64)uc->ud->asel << K3_ADDRESS_ASEL_SHIFT; - } else { - csf |=3D CPPI5_TR_CSF_EOL_ICNT0; + uc->irq_num_ring =3D msi_get_virq(ud->dev, irq_ring_idx); + if (uc->irq_num_ring <=3D 0) { + dev_err(ud->dev, "Failed to get ring irq (index: %u)\n", + irq_ring_idx); + ret =3D -EINVAL; + goto err_psi_free; } =20 - tr_req =3D d->hwdesc[0].tr_req_base; - - cppi5_tr_init(&tr_req[0].flags, CPPI5_TR_TYPE15, false, true, - CPPI5_TR_EVENT_SIZE_COMPLETION, 0); - cppi5_tr_csf_set(&tr_req[0].flags, csf); - - tr_req[0].addr =3D src; - tr_req[0].icnt0 =3D tr0_cnt0; - tr_req[0].icnt1 =3D tr0_cnt1; - tr_req[0].icnt2 =3D 1; - tr_req[0].icnt3 =3D 1; - tr_req[0].dim1 =3D tr0_cnt0; - - tr_req[0].daddr =3D dest; - tr_req[0].dicnt0 =3D tr0_cnt0; - tr_req[0].dicnt1 =3D tr0_cnt1; - tr_req[0].dicnt2 =3D 1; - tr_req[0].dicnt3 =3D 1; - tr_req[0].ddim1 =3D tr0_cnt0; - - if (num_tr =3D=3D 2) { - cppi5_tr_init(&tr_req[1].flags, CPPI5_TR_TYPE15, false, true, - CPPI5_TR_EVENT_SIZE_COMPLETION, 0); - cppi5_tr_csf_set(&tr_req[1].flags, csf); - - tr_req[1].addr =3D src + tr0_cnt1 * tr0_cnt0; - tr_req[1].icnt0 =3D tr1_cnt0; - tr_req[1].icnt1 =3D 1; - tr_req[1].icnt2 =3D 1; - tr_req[1].icnt3 =3D 1; - - tr_req[1].daddr =3D dest + tr0_cnt1 * tr0_cnt0; - tr_req[1].dicnt0 =3D tr1_cnt0; - tr_req[1].dicnt1 =3D 1; - tr_req[1].dicnt2 =3D 1; - tr_req[1].dicnt3 =3D 1; + ret =3D request_irq(uc->irq_num_ring, udma_ring_irq_handler, + IRQF_TRIGGER_HIGH, uc->name, uc); + if (ret) { + dev_err(ud->dev, "chan%d: ring irq request failed\n", uc->id); + goto err_irq_free; } =20 - cppi5_tr_csf_set(&tr_req[num_tr - 1].flags, csf | CPPI5_TR_CSF_EOP); + uc->irq_num_udma =3D 0; + + udma_reset_rings(uc); =20 - if (uc->config.metadata_size) - d->vd.tx.metadata_ops =3D &metadata_ops; + INIT_DELAYED_WORK_ONSTACK(&uc->tx_drain.work, + udma_check_tx_completion); =20 - return vchan_tx_prep(&uc->vc, &d->vd, tx_flags); -} + if (uc->tchan) + dev_dbg(ud->dev, + "chan%d: tchan%d, tflow%d, Remote thread: 0x%04x\n", + uc->id, uc->tchan->id, uc->tchan->tflow_id, + uc->config.remote_thread_id); + else if (uc->rchan) + dev_dbg(ud->dev, + "chan%d: rchan%d, rflow%d, Remote thread: 0x%04x\n", + uc->id, uc->rchan->id, uc->rflow->id, + uc->config.remote_thread_id); + return 0; =20 -static void udma_issue_pending(struct dma_chan *chan) -{ - struct udma_chan *uc =3D to_udma_chan(chan); - unsigned long flags; +err_irq_free: + uc->irq_num_ring =3D 0; +err_psi_free: + navss_psil_unpair(ud, uc->config.src_thread, uc->config.dst_thread); + uc->psil_paired =3D false; +err_res_free: + udma_free_tx_resources(uc); + udma_free_rx_resources(uc); =20 - spin_lock_irqsave(&uc->vc.lock, flags); + udma_reset_uchan(uc); =20 - /* If we have something pending and no active descriptor, then */ - if (vchan_issue_pending(&uc->vc) && !uc->desc) { - /* - * start a descriptor if the channel is NOT [marked as - * terminating _and_ it is still running (teardown has not - * completed yet)]. - */ - if (!(uc->state =3D=3D UDMA_CHAN_IS_TERMINATING && - udma_is_chan_running(uc))) - udma_start(uc); - } + dma_pool_destroy(uc->hdesc_pool); + uc->use_dma_pool =3D false; =20 - spin_unlock_irqrestore(&uc->vc.lock, flags); + return ret; } =20 static enum dma_status udma_tx_status(struct dma_chan *chan, @@ -3928,207 +1692,23 @@ static int udma_resume(struct dma_chan *chan) break; case DMA_MEM_TO_DEV: udma_tchanrt_update_bits(uc, UDMA_CHAN_RT_PEER_RT_EN_REG, - UDMA_PEER_RT_EN_PAUSE, 0); - break; - case DMA_MEM_TO_MEM: - udma_tchanrt_update_bits(uc, UDMA_CHAN_RT_CTL_REG, - UDMA_CHAN_RT_CTL_PAUSE, 0); - break; - default: - return -EINVAL; - } - - return 0; -} - -static int udma_terminate_all(struct dma_chan *chan) -{ - struct udma_chan *uc =3D to_udma_chan(chan); - unsigned long flags; - LIST_HEAD(head); - - spin_lock_irqsave(&uc->vc.lock, flags); - - if (udma_is_chan_running(uc)) - udma_stop(uc); - - if (uc->desc) { - uc->terminated_desc =3D uc->desc; - uc->desc =3D NULL; - uc->terminated_desc->terminated =3D true; - cancel_delayed_work(&uc->tx_drain.work); - } - - uc->paused =3D false; - - vchan_get_all_descriptors(&uc->vc, &head); - spin_unlock_irqrestore(&uc->vc.lock, flags); - vchan_dma_desc_free_list(&uc->vc, &head); - - return 0; -} - -static void udma_synchronize(struct dma_chan *chan) -{ - struct udma_chan *uc =3D to_udma_chan(chan); - unsigned long timeout =3D msecs_to_jiffies(1000); - - vchan_synchronize(&uc->vc); - - if (uc->state =3D=3D UDMA_CHAN_IS_TERMINATING) { - timeout =3D wait_for_completion_timeout(&uc->teardown_completed, - timeout); - if (!timeout) { - dev_warn(uc->ud->dev, "chan%d teardown timeout!\n", - uc->id); - udma_dump_chan_stdata(uc); - udma_reset_chan(uc, true); - } - } - - udma_reset_chan(uc, false); - if (udma_is_chan_running(uc)) - dev_warn(uc->ud->dev, "chan%d refused to stop!\n", uc->id); - - cancel_delayed_work_sync(&uc->tx_drain.work); - udma_reset_rings(uc); -} - -static void udma_desc_pre_callback(struct virt_dma_chan *vc, - struct virt_dma_desc *vd, - struct dmaengine_result *result) -{ - struct udma_chan *uc =3D to_udma_chan(&vc->chan); - struct udma_desc *d; - u8 status; - - if (!vd) - return; - - d =3D to_udma_desc(&vd->tx); - - if (d->metadata_size) - udma_fetch_epib(uc, d); - - if (result) { - void *desc_vaddr =3D udma_curr_cppi5_desc_vaddr(d, d->desc_idx); - - if (cppi5_desc_get_type(desc_vaddr) =3D=3D - CPPI5_INFO0_DESC_TYPE_VAL_HOST) { - /* Provide residue information for the client */ - result->residue =3D d->residue - - cppi5_hdesc_get_pktlen(desc_vaddr); - if (result->residue) - result->result =3D DMA_TRANS_ABORTED; - else - result->result =3D DMA_TRANS_NOERROR; - } else { - result->residue =3D 0; - /* Propagate TR Response errors to the client */ - status =3D d->hwdesc[0].tr_resp_base->status; - if (status) - result->result =3D DMA_TRANS_ABORTED; - else - result->result =3D DMA_TRANS_NOERROR; - } - } -} - -/* - * This tasklet handles the completion of a DMA descriptor by - * calling its callback and freeing it. - */ -static void udma_vchan_complete(struct tasklet_struct *t) -{ - struct virt_dma_chan *vc =3D from_tasklet(vc, t, task); - struct virt_dma_desc *vd, *_vd; - struct dmaengine_desc_callback cb; - LIST_HEAD(head); - - spin_lock_irq(&vc->lock); - list_splice_tail_init(&vc->desc_completed, &head); - vd =3D vc->cyclic; - if (vd) { - vc->cyclic =3D NULL; - dmaengine_desc_get_callback(&vd->tx, &cb); - } else { - memset(&cb, 0, sizeof(cb)); - } - spin_unlock_irq(&vc->lock); - - udma_desc_pre_callback(vc, vd, NULL); - dmaengine_desc_callback_invoke(&cb, NULL); - - list_for_each_entry_safe(vd, _vd, &head, node) { - struct dmaengine_result result; - - dmaengine_desc_get_callback(&vd->tx, &cb); - - list_del(&vd->node); - - udma_desc_pre_callback(vc, vd, &result); - dmaengine_desc_callback_invoke(&cb, &result); - - vchan_vdesc_fini(vd); - } -} - -static void udma_free_chan_resources(struct dma_chan *chan) -{ - struct udma_chan *uc =3D to_udma_chan(chan); - struct udma_dev *ud =3D to_udma_dev(chan->device); - - udma_terminate_all(chan); - if (uc->terminated_desc) { - udma_reset_chan(uc, false); - udma_reset_rings(uc); - } - - cancel_delayed_work_sync(&uc->tx_drain.work); - - if (uc->irq_num_ring > 0) { - free_irq(uc->irq_num_ring, uc); - - uc->irq_num_ring =3D 0; - } - if (uc->irq_num_udma > 0) { - free_irq(uc->irq_num_udma, uc); - - uc->irq_num_udma =3D 0; - } - - /* Release PSI-L pairing */ - if (uc->psil_paired) { - navss_psil_unpair(ud, uc->config.src_thread, - uc->config.dst_thread); - uc->psil_paired =3D false; - } - - vchan_free_chan_resources(&uc->vc); - tasklet_kill(&uc->vc.task); - - bcdma_free_bchan_resources(uc); - udma_free_tx_resources(uc); - udma_free_rx_resources(uc); - udma_reset_uchan(uc); - - if (uc->use_dma_pool) { - dma_pool_destroy(uc->hdesc_pool); - uc->use_dma_pool =3D false; + UDMA_PEER_RT_EN_PAUSE, 0); + break; + case DMA_MEM_TO_MEM: + udma_tchanrt_update_bits(uc, UDMA_CHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_PAUSE, 0); + break; + default: + return -EINVAL; } + + return 0; } =20 static struct platform_driver udma_driver; static struct platform_driver bcdma_driver; static struct platform_driver pktdma_driver; =20 -struct udma_filter_param { - int remote_thread_id; - u32 atype; - u32 asel; - u32 tr_trigger_type; -}; - static bool udma_dma_filter_fn(struct dma_chan *chan, void *param) { struct udma_chan_config *ucc; @@ -4555,849 +2135,6 @@ static int udma_get_mmrs(struct platform_device *pd= ev, struct udma_dev *ud) return 0; } =20 -static void udma_mark_resource_ranges(struct udma_dev *ud, unsigned long *= map, - struct ti_sci_resource_desc *rm_desc, - char *name) -{ - bitmap_clear(map, rm_desc->start, rm_desc->num); - bitmap_clear(map, rm_desc->start_sec, rm_desc->num_sec); - dev_dbg(ud->dev, "ti_sci resource range for %s: %d:%d | %d:%d\n", name, - rm_desc->start, rm_desc->num, rm_desc->start_sec, - rm_desc->num_sec); -} - -static const char * const range_names[] =3D { - [RM_RANGE_BCHAN] =3D "ti,sci-rm-range-bchan", - [RM_RANGE_TCHAN] =3D "ti,sci-rm-range-tchan", - [RM_RANGE_RCHAN] =3D "ti,sci-rm-range-rchan", - [RM_RANGE_RFLOW] =3D "ti,sci-rm-range-rflow", - [RM_RANGE_TFLOW] =3D "ti,sci-rm-range-tflow", -}; - -static int udma_setup_resources(struct udma_dev *ud) -{ - int ret, i, j; - struct device *dev =3D ud->dev; - struct ti_sci_resource *rm_res, irq_res; - struct udma_tisci_rm *tisci_rm =3D &ud->tisci_rm; - u32 cap3; - - /* Set up the throughput level start indexes */ - cap3 =3D udma_read(ud->mmrs[MMR_GCFG], 0x2c); - if (of_device_is_compatible(dev->of_node, - "ti,am654-navss-main-udmap")) { - ud->tchan_tpl.levels =3D 2; - ud->tchan_tpl.start_idx[0] =3D 8; - } else if (of_device_is_compatible(dev->of_node, - "ti,am654-navss-mcu-udmap")) { - ud->tchan_tpl.levels =3D 2; - ud->tchan_tpl.start_idx[0] =3D 2; - } else if (UDMA_CAP3_UCHAN_CNT(cap3)) { - ud->tchan_tpl.levels =3D 3; - ud->tchan_tpl.start_idx[1] =3D UDMA_CAP3_UCHAN_CNT(cap3); - ud->tchan_tpl.start_idx[0] =3D UDMA_CAP3_HCHAN_CNT(cap3); - } else if (UDMA_CAP3_HCHAN_CNT(cap3)) { - ud->tchan_tpl.levels =3D 2; - ud->tchan_tpl.start_idx[0] =3D UDMA_CAP3_HCHAN_CNT(cap3); - } else { - ud->tchan_tpl.levels =3D 1; - } - - ud->rchan_tpl.levels =3D ud->tchan_tpl.levels; - ud->rchan_tpl.start_idx[0] =3D ud->tchan_tpl.start_idx[0]; - ud->rchan_tpl.start_idx[1] =3D ud->tchan_tpl.start_idx[1]; - - ud->tchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->tchan_cnt), - sizeof(unsigned long), GFP_KERNEL); - ud->tchans =3D devm_kcalloc(dev, ud->tchan_cnt, sizeof(*ud->tchans), - GFP_KERNEL); - ud->rchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->rchan_cnt), - sizeof(unsigned long), GFP_KERNEL); - ud->rchans =3D devm_kcalloc(dev, ud->rchan_cnt, sizeof(*ud->rchans), - GFP_KERNEL); - ud->rflow_gp_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->rflow_cnt), - sizeof(unsigned long), - GFP_KERNEL); - ud->rflow_gp_map_allocated =3D devm_kcalloc(dev, - BITS_TO_LONGS(ud->rflow_cnt), - sizeof(unsigned long), - GFP_KERNEL); - ud->rflow_in_use =3D devm_kcalloc(dev, BITS_TO_LONGS(ud->rflow_cnt), - sizeof(unsigned long), - GFP_KERNEL); - ud->rflows =3D devm_kcalloc(dev, ud->rflow_cnt, sizeof(*ud->rflows), - GFP_KERNEL); - - if (!ud->tchan_map || !ud->rchan_map || !ud->rflow_gp_map || - !ud->rflow_gp_map_allocated || !ud->tchans || !ud->rchans || - !ud->rflows || !ud->rflow_in_use) - return -ENOMEM; - - /* - * RX flows with the same Ids as RX channels are reserved to be used - * as default flows if remote HW can't generate flow_ids. Those - * RX flows can be requested only explicitly by id. - */ - bitmap_set(ud->rflow_gp_map_allocated, 0, ud->rchan_cnt); - - /* by default no GP rflows are assigned to Linux */ - bitmap_set(ud->rflow_gp_map, 0, ud->rflow_cnt); - - /* Get resource ranges from tisci */ - for (i =3D 0; i < RM_RANGE_LAST; i++) { - if (i =3D=3D RM_RANGE_BCHAN || i =3D=3D RM_RANGE_TFLOW) - continue; - - tisci_rm->rm_ranges[i] =3D - devm_ti_sci_get_of_resource(tisci_rm->tisci, dev, - tisci_rm->tisci_dev_id, - (char *)range_names[i]); - } - - /* tchan ranges */ - rm_res =3D tisci_rm->rm_ranges[RM_RANGE_TCHAN]; - if (IS_ERR(rm_res)) { - bitmap_zero(ud->tchan_map, ud->tchan_cnt); - irq_res.sets =3D 1; - } else { - bitmap_fill(ud->tchan_map, ud->tchan_cnt); - for (i =3D 0; i < rm_res->sets; i++) - udma_mark_resource_ranges(ud, ud->tchan_map, - &rm_res->desc[i], "tchan"); - irq_res.sets =3D rm_res->sets; - } - - /* rchan and matching default flow ranges */ - rm_res =3D tisci_rm->rm_ranges[RM_RANGE_RCHAN]; - if (IS_ERR(rm_res)) { - bitmap_zero(ud->rchan_map, ud->rchan_cnt); - irq_res.sets++; - } else { - bitmap_fill(ud->rchan_map, ud->rchan_cnt); - for (i =3D 0; i < rm_res->sets; i++) - udma_mark_resource_ranges(ud, ud->rchan_map, - &rm_res->desc[i], "rchan"); - irq_res.sets +=3D rm_res->sets; - } - - irq_res.desc =3D kcalloc(irq_res.sets, sizeof(*irq_res.desc), GFP_KERNEL); - if (!irq_res.desc) - return -ENOMEM; - rm_res =3D tisci_rm->rm_ranges[RM_RANGE_TCHAN]; - if (IS_ERR(rm_res)) { - irq_res.desc[0].start =3D 0; - irq_res.desc[0].num =3D ud->tchan_cnt; - i =3D 1; - } else { - for (i =3D 0; i < rm_res->sets; i++) { - irq_res.desc[i].start =3D rm_res->desc[i].start; - irq_res.desc[i].num =3D rm_res->desc[i].num; - irq_res.desc[i].start_sec =3D rm_res->desc[i].start_sec; - irq_res.desc[i].num_sec =3D rm_res->desc[i].num_sec; - } - } - rm_res =3D tisci_rm->rm_ranges[RM_RANGE_RCHAN]; - if (IS_ERR(rm_res)) { - irq_res.desc[i].start =3D 0; - irq_res.desc[i].num =3D ud->rchan_cnt; - } else { - for (j =3D 0; j < rm_res->sets; j++, i++) { - if (rm_res->desc[j].num) { - irq_res.desc[i].start =3D rm_res->desc[j].start + - ud->soc_data->oes.udma_rchan; - irq_res.desc[i].num =3D rm_res->desc[j].num; - } - if (rm_res->desc[j].num_sec) { - irq_res.desc[i].start_sec =3D rm_res->desc[j].start_sec + - ud->soc_data->oes.udma_rchan; - irq_res.desc[i].num_sec =3D rm_res->desc[j].num_sec; - } - } - } - ret =3D ti_sci_inta_msi_domain_alloc_irqs(ud->dev, &irq_res); - kfree(irq_res.desc); - if (ret) { - dev_err(ud->dev, "Failed to allocate MSI interrupts\n"); - return ret; - } - - /* GP rflow ranges */ - rm_res =3D tisci_rm->rm_ranges[RM_RANGE_RFLOW]; - if (IS_ERR(rm_res)) { - /* all gp flows are assigned exclusively to Linux */ - bitmap_clear(ud->rflow_gp_map, ud->rchan_cnt, - ud->rflow_cnt - ud->rchan_cnt); - } else { - for (i =3D 0; i < rm_res->sets; i++) - udma_mark_resource_ranges(ud, ud->rflow_gp_map, - &rm_res->desc[i], "gp-rflow"); - } - - return 0; -} - -static int bcdma_setup_resources(struct udma_dev *ud) -{ - int ret, i, j; - struct device *dev =3D ud->dev; - struct ti_sci_resource *rm_res, irq_res; - struct udma_tisci_rm *tisci_rm =3D &ud->tisci_rm; - const struct udma_oes_offsets *oes =3D &ud->soc_data->oes; - u32 cap; - - /* Set up the throughput level start indexes */ - cap =3D udma_read(ud->mmrs[MMR_GCFG], 0x2c); - if (BCDMA_CAP3_UBCHAN_CNT(cap)) { - ud->bchan_tpl.levels =3D 3; - ud->bchan_tpl.start_idx[1] =3D BCDMA_CAP3_UBCHAN_CNT(cap); - ud->bchan_tpl.start_idx[0] =3D BCDMA_CAP3_HBCHAN_CNT(cap); - } else if (BCDMA_CAP3_HBCHAN_CNT(cap)) { - ud->bchan_tpl.levels =3D 2; - ud->bchan_tpl.start_idx[0] =3D BCDMA_CAP3_HBCHAN_CNT(cap); - } else { - ud->bchan_tpl.levels =3D 1; - } - - cap =3D udma_read(ud->mmrs[MMR_GCFG], 0x30); - if (BCDMA_CAP4_URCHAN_CNT(cap)) { - ud->rchan_tpl.levels =3D 3; - ud->rchan_tpl.start_idx[1] =3D BCDMA_CAP4_URCHAN_CNT(cap); - ud->rchan_tpl.start_idx[0] =3D BCDMA_CAP4_HRCHAN_CNT(cap); - } else if (BCDMA_CAP4_HRCHAN_CNT(cap)) { - ud->rchan_tpl.levels =3D 2; - ud->rchan_tpl.start_idx[0] =3D BCDMA_CAP4_HRCHAN_CNT(cap); - } else { - ud->rchan_tpl.levels =3D 1; - } - - if (BCDMA_CAP4_UTCHAN_CNT(cap)) { - ud->tchan_tpl.levels =3D 3; - ud->tchan_tpl.start_idx[1] =3D BCDMA_CAP4_UTCHAN_CNT(cap); - ud->tchan_tpl.start_idx[0] =3D BCDMA_CAP4_HTCHAN_CNT(cap); - } else if (BCDMA_CAP4_HTCHAN_CNT(cap)) { - ud->tchan_tpl.levels =3D 2; - ud->tchan_tpl.start_idx[0] =3D BCDMA_CAP4_HTCHAN_CNT(cap); - } else { - ud->tchan_tpl.levels =3D 1; - } - - ud->bchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->bchan_cnt), - sizeof(unsigned long), GFP_KERNEL); - ud->bchans =3D devm_kcalloc(dev, ud->bchan_cnt, sizeof(*ud->bchans), - GFP_KERNEL); - ud->tchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->tchan_cnt), - sizeof(unsigned long), GFP_KERNEL); - ud->tchans =3D devm_kcalloc(dev, ud->tchan_cnt, sizeof(*ud->tchans), - GFP_KERNEL); - ud->rchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->rchan_cnt), - sizeof(unsigned long), GFP_KERNEL); - ud->rchans =3D devm_kcalloc(dev, ud->rchan_cnt, sizeof(*ud->rchans), - GFP_KERNEL); - /* BCDMA do not really have flows, but the driver expect it */ - ud->rflow_in_use =3D devm_kcalloc(dev, BITS_TO_LONGS(ud->rchan_cnt), - sizeof(unsigned long), - GFP_KERNEL); - ud->rflows =3D devm_kcalloc(dev, ud->rchan_cnt, sizeof(*ud->rflows), - GFP_KERNEL); - - if (!ud->bchan_map || !ud->tchan_map || !ud->rchan_map || - !ud->rflow_in_use || !ud->bchans || !ud->tchans || !ud->rchans || - !ud->rflows) - return -ENOMEM; - - /* Get resource ranges from tisci */ - for (i =3D 0; i < RM_RANGE_LAST; i++) { - if (i =3D=3D RM_RANGE_RFLOW || i =3D=3D RM_RANGE_TFLOW) - continue; - if (i =3D=3D RM_RANGE_BCHAN && ud->bchan_cnt =3D=3D 0) - continue; - if (i =3D=3D RM_RANGE_TCHAN && ud->tchan_cnt =3D=3D 0) - continue; - if (i =3D=3D RM_RANGE_RCHAN && ud->rchan_cnt =3D=3D 0) - continue; - - tisci_rm->rm_ranges[i] =3D - devm_ti_sci_get_of_resource(tisci_rm->tisci, dev, - tisci_rm->tisci_dev_id, - (char *)range_names[i]); - } - - irq_res.sets =3D 0; - - /* bchan ranges */ - if (ud->bchan_cnt) { - rm_res =3D tisci_rm->rm_ranges[RM_RANGE_BCHAN]; - if (IS_ERR(rm_res)) { - bitmap_zero(ud->bchan_map, ud->bchan_cnt); - irq_res.sets++; - } else { - bitmap_fill(ud->bchan_map, ud->bchan_cnt); - for (i =3D 0; i < rm_res->sets; i++) - udma_mark_resource_ranges(ud, ud->bchan_map, - &rm_res->desc[i], - "bchan"); - irq_res.sets +=3D rm_res->sets; - } - } - - /* tchan ranges */ - if (ud->tchan_cnt) { - rm_res =3D tisci_rm->rm_ranges[RM_RANGE_TCHAN]; - if (IS_ERR(rm_res)) { - bitmap_zero(ud->tchan_map, ud->tchan_cnt); - irq_res.sets +=3D 2; - } else { - bitmap_fill(ud->tchan_map, ud->tchan_cnt); - for (i =3D 0; i < rm_res->sets; i++) - udma_mark_resource_ranges(ud, ud->tchan_map, - &rm_res->desc[i], - "tchan"); - irq_res.sets +=3D rm_res->sets * 2; - } - } - - /* rchan ranges */ - if (ud->rchan_cnt) { - rm_res =3D tisci_rm->rm_ranges[RM_RANGE_RCHAN]; - if (IS_ERR(rm_res)) { - bitmap_zero(ud->rchan_map, ud->rchan_cnt); - irq_res.sets +=3D 2; - } else { - bitmap_fill(ud->rchan_map, ud->rchan_cnt); - for (i =3D 0; i < rm_res->sets; i++) - udma_mark_resource_ranges(ud, ud->rchan_map, - &rm_res->desc[i], - "rchan"); - irq_res.sets +=3D rm_res->sets * 2; - } - } - - irq_res.desc =3D kcalloc(irq_res.sets, sizeof(*irq_res.desc), GFP_KERNEL); - if (!irq_res.desc) - return -ENOMEM; - if (ud->bchan_cnt) { - rm_res =3D tisci_rm->rm_ranges[RM_RANGE_BCHAN]; - if (IS_ERR(rm_res)) { - irq_res.desc[0].start =3D oes->bcdma_bchan_ring; - irq_res.desc[0].num =3D ud->bchan_cnt; - i =3D 1; - } else { - for (i =3D 0; i < rm_res->sets; i++) { - irq_res.desc[i].start =3D rm_res->desc[i].start + - oes->bcdma_bchan_ring; - irq_res.desc[i].num =3D rm_res->desc[i].num; - - if (rm_res->desc[i].num_sec) { - irq_res.desc[i].start_sec =3D rm_res->desc[i].start_sec + - oes->bcdma_bchan_ring; - irq_res.desc[i].num_sec =3D rm_res->desc[i].num_sec; - } - } - } - } else { - i =3D 0; - } - - if (ud->tchan_cnt) { - rm_res =3D tisci_rm->rm_ranges[RM_RANGE_TCHAN]; - if (IS_ERR(rm_res)) { - irq_res.desc[i].start =3D oes->bcdma_tchan_data; - irq_res.desc[i].num =3D ud->tchan_cnt; - irq_res.desc[i + 1].start =3D oes->bcdma_tchan_ring; - irq_res.desc[i + 1].num =3D ud->tchan_cnt; - i +=3D 2; - } else { - for (j =3D 0; j < rm_res->sets; j++, i +=3D 2) { - irq_res.desc[i].start =3D rm_res->desc[j].start + - oes->bcdma_tchan_data; - irq_res.desc[i].num =3D rm_res->desc[j].num; - - irq_res.desc[i + 1].start =3D rm_res->desc[j].start + - oes->bcdma_tchan_ring; - irq_res.desc[i + 1].num =3D rm_res->desc[j].num; - - if (rm_res->desc[j].num_sec) { - irq_res.desc[i].start_sec =3D rm_res->desc[j].start_sec + - oes->bcdma_tchan_data; - irq_res.desc[i].num_sec =3D rm_res->desc[j].num_sec; - irq_res.desc[i + 1].start_sec =3D rm_res->desc[j].start_sec + - oes->bcdma_tchan_ring; - irq_res.desc[i + 1].num_sec =3D rm_res->desc[j].num_sec; - } - } - } - } - if (ud->rchan_cnt) { - rm_res =3D tisci_rm->rm_ranges[RM_RANGE_RCHAN]; - if (IS_ERR(rm_res)) { - irq_res.desc[i].start =3D oes->bcdma_rchan_data; - irq_res.desc[i].num =3D ud->rchan_cnt; - irq_res.desc[i + 1].start =3D oes->bcdma_rchan_ring; - irq_res.desc[i + 1].num =3D ud->rchan_cnt; - i +=3D 2; - } else { - for (j =3D 0; j < rm_res->sets; j++, i +=3D 2) { - irq_res.desc[i].start =3D rm_res->desc[j].start + - oes->bcdma_rchan_data; - irq_res.desc[i].num =3D rm_res->desc[j].num; - - irq_res.desc[i + 1].start =3D rm_res->desc[j].start + - oes->bcdma_rchan_ring; - irq_res.desc[i + 1].num =3D rm_res->desc[j].num; - - if (rm_res->desc[j].num_sec) { - irq_res.desc[i].start_sec =3D rm_res->desc[j].start_sec + - oes->bcdma_rchan_data; - irq_res.desc[i].num_sec =3D rm_res->desc[j].num_sec; - irq_res.desc[i + 1].start_sec =3D rm_res->desc[j].start_sec + - oes->bcdma_rchan_ring; - irq_res.desc[i + 1].num_sec =3D rm_res->desc[j].num_sec; - } - } - } - } - - ret =3D ti_sci_inta_msi_domain_alloc_irqs(ud->dev, &irq_res); - kfree(irq_res.desc); - if (ret) { - dev_err(ud->dev, "Failed to allocate MSI interrupts\n"); - return ret; - } - - return 0; -} - -static int pktdma_setup_resources(struct udma_dev *ud) -{ - int ret, i, j; - struct device *dev =3D ud->dev; - struct ti_sci_resource *rm_res, irq_res; - struct udma_tisci_rm *tisci_rm =3D &ud->tisci_rm; - const struct udma_oes_offsets *oes =3D &ud->soc_data->oes; - u32 cap3; - - /* Set up the throughput level start indexes */ - cap3 =3D udma_read(ud->mmrs[MMR_GCFG], 0x2c); - if (UDMA_CAP3_UCHAN_CNT(cap3)) { - ud->tchan_tpl.levels =3D 3; - ud->tchan_tpl.start_idx[1] =3D UDMA_CAP3_UCHAN_CNT(cap3); - ud->tchan_tpl.start_idx[0] =3D UDMA_CAP3_HCHAN_CNT(cap3); - } else if (UDMA_CAP3_HCHAN_CNT(cap3)) { - ud->tchan_tpl.levels =3D 2; - ud->tchan_tpl.start_idx[0] =3D UDMA_CAP3_HCHAN_CNT(cap3); - } else { - ud->tchan_tpl.levels =3D 1; - } - - ud->rchan_tpl.levels =3D ud->tchan_tpl.levels; - ud->rchan_tpl.start_idx[0] =3D ud->tchan_tpl.start_idx[0]; - ud->rchan_tpl.start_idx[1] =3D ud->tchan_tpl.start_idx[1]; - - ud->tchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->tchan_cnt), - sizeof(unsigned long), GFP_KERNEL); - ud->tchans =3D devm_kcalloc(dev, ud->tchan_cnt, sizeof(*ud->tchans), - GFP_KERNEL); - ud->rchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->rchan_cnt), - sizeof(unsigned long), GFP_KERNEL); - ud->rchans =3D devm_kcalloc(dev, ud->rchan_cnt, sizeof(*ud->rchans), - GFP_KERNEL); - ud->rflow_in_use =3D devm_kcalloc(dev, BITS_TO_LONGS(ud->rflow_cnt), - sizeof(unsigned long), - GFP_KERNEL); - ud->rflows =3D devm_kcalloc(dev, ud->rflow_cnt, sizeof(*ud->rflows), - GFP_KERNEL); - ud->tflow_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->tflow_cnt), - sizeof(unsigned long), GFP_KERNEL); - - if (!ud->tchan_map || !ud->rchan_map || !ud->tflow_map || !ud->tchans || - !ud->rchans || !ud->rflows || !ud->rflow_in_use) - return -ENOMEM; - - /* Get resource ranges from tisci */ - for (i =3D 0; i < RM_RANGE_LAST; i++) { - if (i =3D=3D RM_RANGE_BCHAN) - continue; - - tisci_rm->rm_ranges[i] =3D - devm_ti_sci_get_of_resource(tisci_rm->tisci, dev, - tisci_rm->tisci_dev_id, - (char *)range_names[i]); - } - - /* tchan ranges */ - rm_res =3D tisci_rm->rm_ranges[RM_RANGE_TCHAN]; - if (IS_ERR(rm_res)) { - bitmap_zero(ud->tchan_map, ud->tchan_cnt); - } else { - bitmap_fill(ud->tchan_map, ud->tchan_cnt); - for (i =3D 0; i < rm_res->sets; i++) - udma_mark_resource_ranges(ud, ud->tchan_map, - &rm_res->desc[i], "tchan"); - } - - /* rchan ranges */ - rm_res =3D tisci_rm->rm_ranges[RM_RANGE_RCHAN]; - if (IS_ERR(rm_res)) { - bitmap_zero(ud->rchan_map, ud->rchan_cnt); - } else { - bitmap_fill(ud->rchan_map, ud->rchan_cnt); - for (i =3D 0; i < rm_res->sets; i++) - udma_mark_resource_ranges(ud, ud->rchan_map, - &rm_res->desc[i], "rchan"); - } - - /* rflow ranges */ - rm_res =3D tisci_rm->rm_ranges[RM_RANGE_RFLOW]; - if (IS_ERR(rm_res)) { - /* all rflows are assigned exclusively to Linux */ - bitmap_zero(ud->rflow_in_use, ud->rflow_cnt); - irq_res.sets =3D 1; - } else { - bitmap_fill(ud->rflow_in_use, ud->rflow_cnt); - for (i =3D 0; i < rm_res->sets; i++) - udma_mark_resource_ranges(ud, ud->rflow_in_use, - &rm_res->desc[i], "rflow"); - irq_res.sets =3D rm_res->sets; - } - - /* tflow ranges */ - rm_res =3D tisci_rm->rm_ranges[RM_RANGE_TFLOW]; - if (IS_ERR(rm_res)) { - /* all tflows are assigned exclusively to Linux */ - bitmap_zero(ud->tflow_map, ud->tflow_cnt); - irq_res.sets++; - } else { - bitmap_fill(ud->tflow_map, ud->tflow_cnt); - for (i =3D 0; i < rm_res->sets; i++) - udma_mark_resource_ranges(ud, ud->tflow_map, - &rm_res->desc[i], "tflow"); - irq_res.sets +=3D rm_res->sets; - } - - irq_res.desc =3D kcalloc(irq_res.sets, sizeof(*irq_res.desc), GFP_KERNEL); - if (!irq_res.desc) - return -ENOMEM; - rm_res =3D tisci_rm->rm_ranges[RM_RANGE_TFLOW]; - if (IS_ERR(rm_res)) { - irq_res.desc[0].start =3D oes->pktdma_tchan_flow; - irq_res.desc[0].num =3D ud->tflow_cnt; - i =3D 1; - } else { - for (i =3D 0; i < rm_res->sets; i++) { - irq_res.desc[i].start =3D rm_res->desc[i].start + - oes->pktdma_tchan_flow; - irq_res.desc[i].num =3D rm_res->desc[i].num; - - if (rm_res->desc[i].num_sec) { - irq_res.desc[i].start_sec =3D rm_res->desc[i].start_sec + - oes->pktdma_tchan_flow; - irq_res.desc[i].num_sec =3D rm_res->desc[i].num_sec; - } - } - } - rm_res =3D tisci_rm->rm_ranges[RM_RANGE_RFLOW]; - if (IS_ERR(rm_res)) { - irq_res.desc[i].start =3D oes->pktdma_rchan_flow; - irq_res.desc[i].num =3D ud->rflow_cnt; - } else { - for (j =3D 0; j < rm_res->sets; j++, i++) { - irq_res.desc[i].start =3D rm_res->desc[j].start + - oes->pktdma_rchan_flow; - irq_res.desc[i].num =3D rm_res->desc[j].num; - - if (rm_res->desc[j].num_sec) { - irq_res.desc[i].start_sec =3D rm_res->desc[j].start_sec + - oes->pktdma_rchan_flow; - irq_res.desc[i].num_sec =3D rm_res->desc[j].num_sec; - } - } - } - ret =3D ti_sci_inta_msi_domain_alloc_irqs(ud->dev, &irq_res); - kfree(irq_res.desc); - if (ret) { - dev_err(ud->dev, "Failed to allocate MSI interrupts\n"); - return ret; - } - - return 0; -} - -static int setup_resources(struct udma_dev *ud) -{ - struct device *dev =3D ud->dev; - int ch_count, ret; - - switch (ud->match_data->type) { - case DMA_TYPE_UDMA: - ret =3D udma_setup_resources(ud); - break; - case DMA_TYPE_BCDMA: - ret =3D bcdma_setup_resources(ud); - break; - case DMA_TYPE_PKTDMA: - ret =3D pktdma_setup_resources(ud); - break; - default: - return -EINVAL; - } - - if (ret) - return ret; - - ch_count =3D ud->bchan_cnt + ud->tchan_cnt + ud->rchan_cnt; - if (ud->bchan_cnt) - ch_count -=3D bitmap_weight(ud->bchan_map, ud->bchan_cnt); - ch_count -=3D bitmap_weight(ud->tchan_map, ud->tchan_cnt); - ch_count -=3D bitmap_weight(ud->rchan_map, ud->rchan_cnt); - if (!ch_count) - return -ENODEV; - - ud->channels =3D devm_kcalloc(dev, ch_count, sizeof(*ud->channels), - GFP_KERNEL); - if (!ud->channels) - return -ENOMEM; - - switch (ud->match_data->type) { - case DMA_TYPE_UDMA: - dev_info(dev, - "Channels: %d (tchan: %u, rchan: %u, gp-rflow: %u)\n", - ch_count, - ud->tchan_cnt - bitmap_weight(ud->tchan_map, - ud->tchan_cnt), - ud->rchan_cnt - bitmap_weight(ud->rchan_map, - ud->rchan_cnt), - ud->rflow_cnt - bitmap_weight(ud->rflow_gp_map, - ud->rflow_cnt)); - break; - case DMA_TYPE_BCDMA: - dev_info(dev, - "Channels: %d (bchan: %u, tchan: %u, rchan: %u)\n", - ch_count, - ud->bchan_cnt - bitmap_weight(ud->bchan_map, - ud->bchan_cnt), - ud->tchan_cnt - bitmap_weight(ud->tchan_map, - ud->tchan_cnt), - ud->rchan_cnt - bitmap_weight(ud->rchan_map, - ud->rchan_cnt)); - break; - case DMA_TYPE_PKTDMA: - dev_info(dev, - "Channels: %d (tchan: %u, rchan: %u)\n", - ch_count, - ud->tchan_cnt - bitmap_weight(ud->tchan_map, - ud->tchan_cnt), - ud->rchan_cnt - bitmap_weight(ud->rchan_map, - ud->rchan_cnt)); - break; - default: - break; - } - - return ch_count; -} - -static int udma_setup_rx_flush(struct udma_dev *ud) -{ - struct udma_rx_flush *rx_flush =3D &ud->rx_flush; - struct cppi5_desc_hdr_t *tr_desc; - struct cppi5_tr_type1_t *tr_req; - struct cppi5_host_desc_t *desc; - struct device *dev =3D ud->dev; - struct udma_hwdesc *hwdesc; - size_t tr_size; - - /* Allocate 1K buffer for discarded data on RX channel teardown */ - rx_flush->buffer_size =3D SZ_1K; - rx_flush->buffer_vaddr =3D devm_kzalloc(dev, rx_flush->buffer_size, - GFP_KERNEL); - if (!rx_flush->buffer_vaddr) - return -ENOMEM; - - rx_flush->buffer_paddr =3D dma_map_single(dev, rx_flush->buffer_vaddr, - rx_flush->buffer_size, - DMA_TO_DEVICE); - if (dma_mapping_error(dev, rx_flush->buffer_paddr)) - return -ENOMEM; - - /* Set up descriptor to be used for TR mode */ - hwdesc =3D &rx_flush->hwdescs[0]; - tr_size =3D sizeof(struct cppi5_tr_type1_t); - hwdesc->cppi5_desc_size =3D cppi5_trdesc_calc_size(tr_size, 1); - hwdesc->cppi5_desc_size =3D ALIGN(hwdesc->cppi5_desc_size, - ud->desc_align); - - hwdesc->cppi5_desc_vaddr =3D devm_kzalloc(dev, hwdesc->cppi5_desc_size, - GFP_KERNEL); - if (!hwdesc->cppi5_desc_vaddr) - return -ENOMEM; - - hwdesc->cppi5_desc_paddr =3D dma_map_single(dev, hwdesc->cppi5_desc_vaddr, - hwdesc->cppi5_desc_size, - DMA_TO_DEVICE); - if (dma_mapping_error(dev, hwdesc->cppi5_desc_paddr)) - return -ENOMEM; - - /* Start of the TR req records */ - hwdesc->tr_req_base =3D hwdesc->cppi5_desc_vaddr + tr_size; - /* Start address of the TR response array */ - hwdesc->tr_resp_base =3D hwdesc->tr_req_base + tr_size; - - tr_desc =3D hwdesc->cppi5_desc_vaddr; - cppi5_trdesc_init(tr_desc, 1, tr_size, 0, 0); - cppi5_desc_set_pktids(tr_desc, 0, CPPI5_INFO1_DESC_FLOWID_DEFAULT); - cppi5_desc_set_retpolicy(tr_desc, 0, 0); - - tr_req =3D hwdesc->tr_req_base; - cppi5_tr_init(&tr_req->flags, CPPI5_TR_TYPE1, false, false, - CPPI5_TR_EVENT_SIZE_COMPLETION, 0); - cppi5_tr_csf_set(&tr_req->flags, CPPI5_TR_CSF_SUPR_EVT); - - tr_req->addr =3D rx_flush->buffer_paddr; - tr_req->icnt0 =3D rx_flush->buffer_size; - tr_req->icnt1 =3D 1; - - dma_sync_single_for_device(dev, hwdesc->cppi5_desc_paddr, - hwdesc->cppi5_desc_size, DMA_TO_DEVICE); - - /* Set up descriptor to be used for packet mode */ - hwdesc =3D &rx_flush->hwdescs[1]; - hwdesc->cppi5_desc_size =3D ALIGN(sizeof(struct cppi5_host_desc_t) + - CPPI5_INFO0_HDESC_EPIB_SIZE + - CPPI5_INFO0_HDESC_PSDATA_MAX_SIZE, - ud->desc_align); - - hwdesc->cppi5_desc_vaddr =3D devm_kzalloc(dev, hwdesc->cppi5_desc_size, - GFP_KERNEL); - if (!hwdesc->cppi5_desc_vaddr) - return -ENOMEM; - - hwdesc->cppi5_desc_paddr =3D dma_map_single(dev, hwdesc->cppi5_desc_vaddr, - hwdesc->cppi5_desc_size, - DMA_TO_DEVICE); - if (dma_mapping_error(dev, hwdesc->cppi5_desc_paddr)) - return -ENOMEM; - - desc =3D hwdesc->cppi5_desc_vaddr; - cppi5_hdesc_init(desc, 0, 0); - cppi5_desc_set_pktids(&desc->hdr, 0, CPPI5_INFO1_DESC_FLOWID_DEFAULT); - cppi5_desc_set_retpolicy(&desc->hdr, 0, 0); - - cppi5_hdesc_attach_buf(desc, - rx_flush->buffer_paddr, rx_flush->buffer_size, - rx_flush->buffer_paddr, rx_flush->buffer_size); - - dma_sync_single_for_device(dev, hwdesc->cppi5_desc_paddr, - hwdesc->cppi5_desc_size, DMA_TO_DEVICE); - return 0; -} - -#ifdef CONFIG_DEBUG_FS -static void udma_dbg_summary_show_chan(struct seq_file *s, - struct dma_chan *chan) -{ - struct udma_chan *uc =3D to_udma_chan(chan); - struct udma_chan_config *ucc =3D &uc->config; - - seq_printf(s, " %-13s| %s", dma_chan_name(chan), - chan->dbg_client_name ?: "in-use"); - if (ucc->tr_trigger_type) - seq_puts(s, " (triggered, "); - else - seq_printf(s, " (%s, ", - dmaengine_get_direction_text(uc->config.dir)); - - switch (uc->config.dir) { - case DMA_MEM_TO_MEM: - if (uc->ud->match_data->type =3D=3D DMA_TYPE_BCDMA) { - seq_printf(s, "bchan%d)\n", uc->bchan->id); - return; - } - - seq_printf(s, "chan%d pair [0x%04x -> 0x%04x], ", uc->tchan->id, - ucc->src_thread, ucc->dst_thread); - break; - case DMA_DEV_TO_MEM: - seq_printf(s, "rchan%d [0x%04x -> 0x%04x], ", uc->rchan->id, - ucc->src_thread, ucc->dst_thread); - if (uc->ud->match_data->type =3D=3D DMA_TYPE_PKTDMA) - seq_printf(s, "rflow%d, ", uc->rflow->id); - break; - case DMA_MEM_TO_DEV: - seq_printf(s, "tchan%d [0x%04x -> 0x%04x], ", uc->tchan->id, - ucc->src_thread, ucc->dst_thread); - if (uc->ud->match_data->type =3D=3D DMA_TYPE_PKTDMA) - seq_printf(s, "tflow%d, ", uc->tchan->tflow_id); - break; - default: - seq_printf(s, ")\n"); - return; - } - - if (ucc->ep_type =3D=3D PSIL_EP_NATIVE) { - seq_printf(s, "PSI-L Native"); - if (ucc->metadata_size) { - seq_printf(s, "[%s", ucc->needs_epib ? " EPIB" : ""); - if (ucc->psd_size) - seq_printf(s, " PSDsize:%u", ucc->psd_size); - seq_printf(s, " ]"); - } - } else { - seq_printf(s, "PDMA"); - if (ucc->enable_acc32 || ucc->enable_burst) - seq_printf(s, "[%s%s ]", - ucc->enable_acc32 ? " ACC32" : "", - ucc->enable_burst ? " BURST" : ""); - } - - seq_printf(s, ", %s)\n", ucc->pkt_mode ? "Packet mode" : "TR mode"); -} - -static void udma_dbg_summary_show(struct seq_file *s, - struct dma_device *dma_dev) -{ - struct dma_chan *chan; - - list_for_each_entry(chan, &dma_dev->channels, device_node) { - if (chan->client_count) - udma_dbg_summary_show_chan(s, chan); - } -} -#endif /* CONFIG_DEBUG_FS */ - -static enum dmaengine_alignment udma_get_copy_align(struct udma_dev *ud) -{ - const struct udma_match_data *match_data =3D ud->match_data; - u8 tpl; - - if (!match_data->enable_memcpy_support) - return DMAENGINE_ALIGN_8_BYTES; - - /* Get the highest TPL level the device supports for memcpy */ - if (ud->bchan_cnt) - tpl =3D udma_get_chan_tpl_index(&ud->bchan_tpl, 0); - else if (ud->tchan_cnt) - tpl =3D udma_get_chan_tpl_index(&ud->tchan_tpl, 0); - else - return DMAENGINE_ALIGN_8_BYTES; - - switch (match_data->burst_size[tpl]) { - case TI_SCI_RM_UDMAP_CHAN_BURST_SIZE_256_BYTES: - return DMAENGINE_ALIGN_256_BYTES; - case TI_SCI_RM_UDMAP_CHAN_BURST_SIZE_128_BYTES: - return DMAENGINE_ALIGN_128_BYTES; - case TI_SCI_RM_UDMAP_CHAN_BURST_SIZE_64_BYTES: - fallthrough; - default: - return DMAENGINE_ALIGN_64_BYTES; - } -} - -#define TI_UDMAC_BUSWIDTHS (BIT(DMA_SLAVE_BUSWIDTH_1_BYTE) | \ - BIT(DMA_SLAVE_BUSWIDTH_2_BYTES) | \ - BIT(DMA_SLAVE_BUSWIDTH_3_BYTES) | \ - BIT(DMA_SLAVE_BUSWIDTH_4_BYTES) | \ - BIT(DMA_SLAVE_BUSWIDTH_8_BYTES)) - static int udma_probe(struct platform_device *pdev) { struct device_node *navss_node =3D pdev->dev.parent->of_node; @@ -5433,6 +2170,13 @@ static int udma_probe(struct platform_device *pdev) ud->soc_data =3D soc->data; } =20 + // Setup function pointers + ud->udma_start =3D udma_start; + ud->udma_stop =3D udma_stop; + ud->udma_reset_chan =3D udma_reset_chan; + ud->udma_is_desc_really_done =3D udma_is_desc_really_done; + ud->udma_decrement_byte_counters =3D udma_decrement_byte_counters; + ret =3D udma_get_mmrs(pdev, ud); if (ret) return ret; @@ -5710,6 +2454,3 @@ static struct platform_driver udma_driver =3D { module_platform_driver(udma_driver); MODULE_DESCRIPTION("Texas Instruments UDMA support"); MODULE_LICENSE("GPL v2"); - -/* Private interfaces to UDMA */ -#include "k3-udma-private.c" diff --git a/drivers/dma/ti/k3-udma.h b/drivers/dma/ti/k3-udma.h index 9062a237cd167..4de6f38089ce7 100644 --- a/drivers/dma/ti/k3-udma.h +++ b/drivers/dma/ti/k3-udma.h @@ -6,7 +6,33 @@ #ifndef K3_UDMA_H_ #define K3_UDMA_H_ =20 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include #include +#include +#include +#include + +#include "../virt-dma.h" +#include "k3-psil-priv.h" =20 /* Global registers */ #define UDMA_REV_REG 0x0 @@ -97,10 +123,50 @@ /* Address Space Select */ #define K3_ADDRESS_ASEL_SHIFT 48 =20 -struct udma_dev; -struct udma_tchan; -struct udma_rchan; -struct udma_rflow; +#define K3_UDMA_MAX_RFLOWS 1024 +#define K3_UDMA_DEFAULT_RING_SIZE 16 + +/* How SRC/DST tag should be updated by UDMA in the descriptor's Word 3 */ +#define UDMA_RFLOW_SRCTAG_NONE 0 +#define UDMA_RFLOW_SRCTAG_CFG_TAG 1 +#define UDMA_RFLOW_SRCTAG_FLOW_ID 2 +#define UDMA_RFLOW_SRCTAG_SRC_TAG 4 + +#define UDMA_RFLOW_DSTTAG_NONE 0 +#define UDMA_RFLOW_DSTTAG_CFG_TAG 1 +#define UDMA_RFLOW_DSTTAG_FLOW_ID 2 +#define UDMA_RFLOW_DSTTAG_DST_TAG_LO 4 +#define UDMA_RFLOW_DSTTAG_DST_TAG_HI 5 + +/* Device capability flags */ +#define UDMA_FLAG_PDMA_ACC32 BIT(0) +#define UDMA_FLAG_PDMA_BURST BIT(1) +#define UDMA_FLAG_TDTYPE BIT(2) +#define UDMA_FLAG_BURST_SIZE BIT(3) +#define UDMA_FLAGS_J7_CLASS (UDMA_FLAG_PDMA_ACC32 | \ + UDMA_FLAG_PDMA_BURST | \ + UDMA_FLAG_TDTYPE | \ + UDMA_FLAG_BURST_SIZE) + +#define TI_UDMAC_BUSWIDTHS (BIT(DMA_SLAVE_BUSWIDTH_1_BYTE) | \ + BIT(DMA_SLAVE_BUSWIDTH_2_BYTES) | \ + BIT(DMA_SLAVE_BUSWIDTH_3_BYTES) | \ + BIT(DMA_SLAVE_BUSWIDTH_4_BYTES) | \ + BIT(DMA_SLAVE_BUSWIDTH_8_BYTES)) + +enum k3_dma_type { + DMA_TYPE_UDMA =3D 0, + DMA_TYPE_BCDMA, + DMA_TYPE_PKTDMA, +}; + +enum udma_mmr { + MMR_GCFG =3D 0, + MMR_BCHANRT, + MMR_RCHANRT, + MMR_TCHANRT, + MMR_LAST, +}; =20 enum udma_rm_range { RM_RANGE_BCHAN =3D 0, @@ -111,6 +177,65 @@ enum udma_rm_range { RM_RANGE_LAST, }; =20 +enum udma_chan_state { + UDMA_CHAN_IS_IDLE =3D 0, /* not active, no teardown is in progress */ + UDMA_CHAN_IS_ACTIVE, /* Normal operation */ + UDMA_CHAN_IS_TERMINATING, /* channel is being terminated */ +}; + +struct udma_filter_param { + int remote_thread_id; + u32 atype; + u32 asel; + u32 tr_trigger_type; +}; + +struct udma_static_tr { + u8 elsize; /* RPSTR0 */ + u16 elcnt; /* RPSTR0 */ + u16 bstcnt; /* RPSTR1 */ +}; + +struct udma_tchan { + void __iomem *reg_rt; + + int id; + struct k3_ring *t_ring; /* Transmit ring */ + struct k3_ring *tc_ring; /* Transmit Completion ring */ + int tflow_id; /* applicable only for PKTDMA */ +}; + +#define udma_bchan udma_tchan + +struct udma_rflow { + int id; + struct k3_ring *fd_ring; /* Free Descriptor ring */ + struct k3_ring *r_ring; /* Receive ring */ +}; + +struct udma_rchan { + void __iomem *reg_rt; + + int id; +}; + +struct udma_oes_offsets { + /* K3 UDMA Output Event Offset */ + u32 udma_rchan; + + /* BCDMA Output Event Offsets */ + u32 bcdma_bchan_data; + u32 bcdma_bchan_ring; + u32 bcdma_tchan_data; + u32 bcdma_tchan_ring; + u32 bcdma_rchan_data; + u32 bcdma_rchan_ring; + + /* PKTDMA Output Event Offsets */ + u32 pktdma_tchan_flow; + u32 pktdma_rchan_flow; +}; + struct udma_tisci_rm { const struct ti_sci_handle *tisci; const struct ti_sci_rm_udmap_ops *tisci_udmap_ops; @@ -123,6 +248,421 @@ struct udma_tisci_rm { struct ti_sci_resource *rm_ranges[RM_RANGE_LAST]; }; =20 +struct udma_match_data { + enum k3_dma_type type; + u32 psil_base; + bool enable_memcpy_support; + u32 flags; + u32 statictr_z_mask; + u8 burst_size[3]; + struct udma_soc_data *soc_data; +}; + +struct udma_soc_data { + struct udma_oes_offsets oes; + u32 bcdma_trigger_event_offset; +}; + +struct udma_hwdesc { + size_t cppi5_desc_size; + void *cppi5_desc_vaddr; + dma_addr_t cppi5_desc_paddr; + + /* TR descriptor internal pointers */ + void *tr_req_base; + struct cppi5_tr_resp_t *tr_resp_base; +}; + +struct udma_rx_flush { + struct udma_hwdesc hwdescs[2]; + + size_t buffer_size; + void *buffer_vaddr; + dma_addr_t buffer_paddr; +}; + +struct udma_tpl { + u8 levels; + u32 start_idx[3]; +}; + +struct udma_desc { + struct virt_dma_desc vd; + + bool terminated; + + enum dma_transfer_direction dir; + + struct udma_static_tr static_tr; + u32 residue; + + unsigned int sglen; + unsigned int desc_idx; /* Only used for cyclic in packet mode */ + unsigned int tr_idx; + + u32 metadata_size; + void *metadata; /* pointer to provided metadata buffer (EPIP, PSdata) */ + + unsigned int hwdesc_count; + struct udma_hwdesc hwdesc[]; +}; + +struct udma_tx_drain { + struct delayed_work work; + ktime_t tstamp; + u32 residue; +}; + +struct udma_chan_config { + bool pkt_mode; /* TR or packet */ + bool needs_epib; /* EPIB is needed for the communication or not */ + u32 psd_size; /* size of Protocol Specific Data */ + u32 metadata_size; /* (needs_epib ? 16:0) + psd_size */ + u32 hdesc_size; /* Size of a packet descriptor in packet mode */ + bool notdpkt; /* Suppress sending TDC packet */ + int remote_thread_id; + u32 atype; + u32 asel; + u32 src_thread; + u32 dst_thread; + enum psil_endpoint_type ep_type; + bool enable_acc32; + bool enable_burst; + enum udma_tp_level channel_tpl; /* Channel Throughput Level */ + + u32 tr_trigger_type; + unsigned long tx_flags; + + /* PKDMA mapped channel */ + int mapped_channel_id; + /* PKTDMA default tflow or rflow for mapped channel */ + int default_flow_id; + + enum dma_transfer_direction dir; +}; + +struct udma_dev { + struct dma_device ddev; + struct device *dev; + void __iomem *mmrs[MMR_LAST]; + const struct udma_match_data *match_data; + const struct udma_soc_data *soc_data; + + struct udma_tpl bchan_tpl; + struct udma_tpl tchan_tpl; + struct udma_tpl rchan_tpl; + + size_t desc_align; /* alignment to use for descriptors */ + + struct udma_tisci_rm tisci_rm; + + struct k3_ringacc *ringacc; + + struct work_struct purge_work; + struct list_head desc_to_purge; + spinlock_t lock; + + struct udma_rx_flush rx_flush; + + int bchan_cnt; + int tchan_cnt; + int echan_cnt; + int rchan_cnt; + int rflow_cnt; + int tflow_cnt; + unsigned long *bchan_map; + unsigned long *tchan_map; + unsigned long *rchan_map; + unsigned long *rflow_gp_map; + unsigned long *rflow_gp_map_allocated; + unsigned long *rflow_in_use; + unsigned long *tflow_map; + + struct udma_bchan *bchans; + struct udma_tchan *tchans; + struct udma_rchan *rchans; + struct udma_rflow *rflows; + + struct udma_chan *channels; + u32 psil_base; + u32 atype; + u32 asel; + + int (*udma_start)(struct udma_chan *uc); + int (*udma_stop)(struct udma_chan *uc); + int (*udma_reset_chan)(struct udma_chan *uc, bool hard); + bool (*udma_is_desc_really_done)(struct udma_chan *uc, struct udma_desc *= d); + void (*udma_decrement_byte_counters)(struct udma_chan *uc, u32 val); +}; + + +struct udma_chan { + struct virt_dma_chan vc; + struct dma_slave_config cfg; + struct udma_dev *ud; + struct device *dma_dev; + struct udma_desc *desc; + struct udma_desc *terminated_desc; + struct udma_static_tr static_tr; + char *name; + + struct udma_bchan *bchan; + struct udma_tchan *tchan; + struct udma_rchan *rchan; + struct udma_rflow *rflow; + + bool psil_paired; + + int irq_num_ring; + int irq_num_udma; + + bool cyclic; + bool paused; + + enum udma_chan_state state; + struct completion teardown_completed; + + struct udma_tx_drain tx_drain; + + /* Channel configuration parameters */ + struct udma_chan_config config; + /* Channel configuration parameters (backup) */ + struct udma_chan_config backup_config; + + /* dmapool for packet mode descriptors */ + bool use_dma_pool; + struct dma_pool *hdesc_pool; + + u32 id; +}; + +/* K3 UDMA helper functions */ +static inline struct udma_dev *to_udma_dev(struct dma_device *d) +{ + return container_of(d, struct udma_dev, ddev); +} + +static inline struct udma_chan *to_udma_chan(struct dma_chan *c) +{ + return container_of(c, struct udma_chan, vc.chan); +} + +static inline struct udma_desc *to_udma_desc(struct dma_async_tx_descripto= r *t) +{ + return container_of(t, struct udma_desc, vd.tx); +} + +/* Generic register access functions */ +static inline u32 udma_read(void __iomem *base, int reg) +{ + return readl(base + reg); +} + +static inline void udma_write(void __iomem *base, int reg, u32 val) +{ + writel(val, base + reg); +} + +static inline void udma_update_bits(void __iomem *base, int reg, + u32 mask, u32 val) +{ + u32 tmp, orig; + + orig =3D readl(base + reg); + tmp =3D orig & ~mask; + tmp |=3D (val & mask); + + if (tmp !=3D orig) + writel(tmp, base + reg); +} + +/* TCHANRT */ +static inline u32 udma_tchanrt_read(struct udma_chan *uc, int reg) +{ + if (!uc->tchan) + return 0; + return udma_read(uc->tchan->reg_rt, reg); +} + +static inline void udma_tchanrt_write(struct udma_chan *uc, int reg, u32 v= al) +{ + if (!uc->tchan) + return; + udma_write(uc->tchan->reg_rt, reg, val); +} + +static inline void udma_tchanrt_update_bits(struct udma_chan *uc, int reg, + u32 mask, u32 val) +{ + if (!uc->tchan) + return; + udma_update_bits(uc->tchan->reg_rt, reg, mask, val); +} + +/* RCHANRT */ +static inline u32 udma_rchanrt_read(struct udma_chan *uc, int reg) +{ + if (!uc->rchan) + return 0; + return udma_read(uc->rchan->reg_rt, reg); +} + +static inline void udma_rchanrt_write(struct udma_chan *uc, int reg, u32 v= al) +{ + if (!uc->rchan) + return; + udma_write(uc->rchan->reg_rt, reg, val); +} + +static inline void udma_rchanrt_update_bits(struct udma_chan *uc, int reg, + u32 mask, u32 val) +{ + if (!uc->rchan) + return; + udma_update_bits(uc->rchan->reg_rt, reg, mask, val); +} + +static inline dma_addr_t udma_curr_cppi5_desc_paddr(struct udma_desc *d, + int idx) +{ + return d->hwdesc[idx].cppi5_desc_paddr; +} + +static inline void *udma_curr_cppi5_desc_vaddr(struct udma_desc *d, int id= x) +{ + return d->hwdesc[idx].cppi5_desc_vaddr; +} + +static inline dma_addr_t udma_get_rx_flush_hwdesc_paddr(struct udma_chan *= uc) +{ + return uc->ud->rx_flush.hwdescs[uc->config.pkt_mode].cppi5_desc_paddr; +} + +static inline void udma_fetch_epib(struct udma_chan *uc, struct udma_desc = *d) +{ + struct cppi5_host_desc_t *h_desc =3D d->hwdesc[0].cppi5_desc_vaddr; + + memcpy(d->metadata, h_desc->epib, d->metadata_size); +} + +void udma_start_desc(struct udma_chan *uc); +bool udma_chan_needs_reconfiguration(struct udma_chan *uc); +void udma_cyclic_packet_elapsed(struct udma_chan *uc); +void udma_check_tx_completion(struct work_struct *work); +void udma_issue_pending(struct dma_chan *chan); +void udma_free_chan_resources(struct dma_chan *chan); +int setup_resources(struct udma_dev *ud); +void udma_mark_resource_ranges(struct udma_dev *ud, unsigned long *map, + struct ti_sci_resource_desc *rm_desc, + char *name); +int udma_setup_resources(struct udma_dev *ud); +int bcdma_setup_resources(struct udma_dev *ud); +int pktdma_setup_resources(struct udma_dev *ud); + +void k3_configure_chan_coherency(struct dma_chan *chan, u32 asel); +u8 udma_get_chan_tpl_index(struct udma_tpl *tpl_map, int chan_id); +void udma_reset_uchan(struct udma_chan *uc); +void udma_dump_chan_stdata(struct udma_chan *uc); +struct udma_desc *udma_udma_desc_from_paddr(struct udma_chan *uc, + dma_addr_t paddr); +void udma_free_hwdesc(struct udma_chan *uc, struct udma_desc *d); +void udma_purge_desc_work(struct work_struct *work); +void udma_desc_free(struct virt_dma_desc *vd); +bool udma_is_chan_running(struct udma_chan *uc); +void udma_reset_rings(struct udma_chan *uc); +int udma_push_to_ring(struct udma_chan *uc, int idx); +bool udma_desc_is_rx_flush(struct udma_chan *uc, dma_addr_t addr); +int udma_pop_from_ring(struct udma_chan *uc, dma_addr_t *addr); + +int __udma_alloc_gp_rflow_range(struct udma_dev *ud, int from, int cnt); +int __udma_free_gp_rflow_range(struct udma_dev *ud, int from, int cnt); +struct udma_rflow *__udma_get_rflow(struct udma_dev *ud, int id); +void __udma_put_rflow(struct udma_dev *ud, struct udma_rflow *rflow); + +struct udma_bchan *__udma_reserve_bchan(struct udma_dev *ud, enum udma_tp_= level tpl, int id); +struct udma_tchan *__udma_reserve_tchan(struct udma_dev *ud, enum udma_tp_= level tpl, int id); +struct udma_rchan *__udma_reserve_rchan(struct udma_dev *ud, enum udma_tp_= level tpl, int id); + +int udma_get_tchan(struct udma_chan *uc); +int udma_get_rchan(struct udma_chan *uc); +int udma_get_chan_pair(struct udma_chan *uc); +int udma_get_rflow(struct udma_chan *uc, int flow_id); +void bcdma_put_bchan(struct udma_chan *uc); +void udma_put_rchan(struct udma_chan *uc); +void udma_put_tchan(struct udma_chan *uc); +void udma_put_rflow(struct udma_chan *uc); +void bcdma_free_bchan_resources(struct udma_chan *uc); +void udma_free_tx_resources(struct udma_chan *uc); +void udma_free_rx_resources(struct udma_chan *uc); +int udma_slave_config(struct dma_chan *chan, + struct dma_slave_config *cfg); +struct udma_desc *udma_alloc_tr_desc(struct udma_chan *uc, + size_t tr_size, int tr_count, + enum dma_transfer_direction dir); +int udma_get_tr_counters(size_t len, unsigned long align_to, + u16 *tr0_cnt0, u16 *tr0_cnt1, u16 *tr1_cnt0); +struct udma_desc * +udma_prep_slave_sg_tr(struct udma_chan *uc, struct scatterlist *sgl, + unsigned int sglen, enum dma_transfer_direction dir, + unsigned long tx_flags, void *context); +struct udma_desc * +udma_prep_slave_sg_triggered_tr(struct udma_chan *uc, struct scatterlist *= sgl, + unsigned int sglen, + enum dma_transfer_direction dir, + unsigned long tx_flags, void *context); +int udma_configure_statictr(struct udma_chan *uc, struct udma_desc *d, + enum dma_slave_buswidth dev_width, + u16 elcnt); +struct udma_desc * +udma_prep_slave_sg_pkt(struct udma_chan *uc, struct scatterlist *sgl, + unsigned int sglen, enum dma_transfer_direction dir, + unsigned long tx_flags, void *context); +int udma_attach_metadata(struct dma_async_tx_descriptor *desc, + void *data, size_t len); +void *udma_get_metadata_ptr(struct dma_async_tx_descriptor *desc, + size_t *payload_len, size_t *max_len); +int udma_set_metadata_len(struct dma_async_tx_descriptor *desc, + size_t payload_len); +struct dma_async_tx_descriptor * +udma_prep_slave_sg(struct dma_chan *chan, struct scatterlist *sgl, + unsigned int sglen, enum dma_transfer_direction dir, + unsigned long tx_flags, void *context); +struct udma_desc * +udma_prep_dma_cyclic_tr(struct udma_chan *uc, dma_addr_t buf_addr, + size_t buf_len, size_t period_len, + enum dma_transfer_direction dir, unsigned long flags); +struct udma_desc * +udma_prep_dma_cyclic_pkt(struct udma_chan *uc, dma_addr_t buf_addr, + size_t buf_len, size_t period_len, + enum dma_transfer_direction dir, unsigned long flags); +struct dma_async_tx_descriptor * +udma_prep_dma_cyclic(struct dma_chan *chan, dma_addr_t buf_addr, size_t bu= f_len, + size_t period_len, enum dma_transfer_direction dir, + unsigned long flags); +struct dma_async_tx_descriptor * +udma_prep_dma_memcpy(struct dma_chan *chan, dma_addr_t dest, dma_addr_t sr= c, + size_t len, unsigned long tx_flags); +int udma_terminate_all(struct dma_chan *chan); +void udma_synchronize(struct dma_chan *chan); +void udma_desc_pre_callback(struct virt_dma_chan *vc, + struct virt_dma_desc *vd, + struct dmaengine_result *result); + +void udma_vchan_complete(struct tasklet_struct *t); +int udma_setup_rx_flush(struct udma_dev *ud); + +#ifdef CONFIG_DEBUG_FS +void udma_dbg_summary_show_chan(struct seq_file *s, + struct dma_chan *chan); +void udma_dbg_summary_show(struct seq_file *s, + struct dma_device *dma_dev); +#endif /* CONFIG_DEBUG_FS */ + +enum dmaengine_alignment udma_get_copy_align(struct udma_dev *ud); +int navss_psil_pair(struct udma_dev *ud, u32 src_thread, u32 dst_thread); +int navss_psil_unpair(struct udma_dev *ud, u32 src_thread, + u32 dst_thread); + /* Direct access to UDMA low lever resources for the glue layer */ int xudma_navss_psil_pair(struct udma_dev *ud, u32 src_thread, u32 dst_thr= ead); int xudma_navss_psil_unpair(struct udma_dev *ud, u32 src_thread, --=20 2.34.1 From nobody Sun Feb 8 20:13:33 2026 Received: from lelvem-ot01.ext.ti.com (lelvem-ot01.ext.ti.com [198.47.23.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 71A861D5AC6; Mon, 28 Apr 2025 07:21:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.47.23.234 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745824889; cv=none; b=aneRc0k2xnqRq74AEv5aE5mW6TnEe3jioOM3ff9CEKHrp5cPJ55vfQNj0FaVvMqecokIS6vb9ilx2mpjI3E2hIpoBytqE05/bCoDqszcCgFRrLLc1lVanwCtoEaATmWUmtNfu7ZAEdSCil2dRgr9uSf1BtPsZbZQ8ukxAh7RafM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745824889; c=relaxed/simple; bh=b4gr6GJDTCdFqz+GIIsR8/t+wBTMzYnL6Bpxe2rVhmw=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=tYH0otynE/2be+lPplf+gqJw0AS9ZXpkZWQeIaVZjxdvCdpLGZ6U/B9HqZnjvxev5uzbm2ejbg6IJFoaaPrK+6Cv8kG0iLrbJ9Q+qYp3MIMeS1gOa+s1wXTM+4bbSMUDxK763C8xZAbyf66W0WdqpN/Bxo/Iu63ZghakgNYQvgc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ti.com; spf=pass smtp.mailfrom=ti.com; dkim=pass (1024-bit key) header.d=ti.com header.i=@ti.com header.b=r0heJAOv; arc=none smtp.client-ip=198.47.23.234 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ti.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ti.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ti.com header.i=@ti.com header.b="r0heJAOv" Received: from lelv0265.itg.ti.com ([10.180.67.224]) by lelvem-ot01.ext.ti.com (8.15.2/8.15.2) with ESMTPS id 53S7LFp02713999 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 28 Apr 2025 02:21:15 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1745824875; bh=FazsueVaj+YhhF+9bYhnapWdoUDSbmNSp2aLiF/uiLo=; h=From:To:Subject:Date:In-Reply-To:References; b=r0heJAOviv+TG126zn1S1REfyWVEpYO/OF4Vh8lzX+hB8Q4N2zVIpJHk0k8RKtBPk IDBNL2FAZ1G+s5lZoUejr9VysNLc+VTuRw3Q/4hVqIDV7KyOuWdv3SOXIIvJXljWOD wispgbMOLoVIYNHMebXgp16ti6GdMnABHwojy5vA= Received: from DFLE102.ent.ti.com (dfle102.ent.ti.com [10.64.6.23]) by lelv0265.itg.ti.com (8.15.2/8.15.2) with ESMTPS id 53S7LEbs003563 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 28 Apr 2025 02:21:14 -0500 Received: from DFLE107.ent.ti.com (10.64.6.28) by DFLE102.ent.ti.com (10.64.6.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.23; Mon, 28 Apr 2025 02:21:14 -0500 Received: from lelvsmtp6.itg.ti.com (10.180.75.249) by DFLE107.ent.ti.com (10.64.6.28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.23 via Frontend Transport; Mon, 28 Apr 2025 02:21:14 -0500 Received: from uda0498651.dhcp.ti.com (uda0498651.dhcp.ti.com [172.24.227.7]) by lelvsmtp6.itg.ti.com (8.15.2/8.15.2) with ESMTP id 53S7KdMd068873; Mon, 28 Apr 2025 02:21:10 -0500 From: Sai Sree Kartheek Adivi To: , , , , , , , , , , , , , , , Subject: [PATCH 4/8] dmaengine: ti: k3-psil-am62l: Add AM62Lx PSIL and PDMA data Date: Mon, 28 Apr 2025 12:50:28 +0530 Message-ID: <20250428072032.946008-5-s-adivi@ti.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250428072032.946008-1-s-adivi@ti.com> References: <20250428072032.946008-1-s-adivi@ti.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-C2ProcessedOrg: 333ef613-75bf-4e12-a4b1-8e3623f5dcea Content-Type: text/plain; charset="utf-8" Add PSIL and PDMA data for AM62Lx SoC. Signed-off-by: Sai Sree Kartheek Adivi --- drivers/dma/ti/Makefile | 1 + drivers/dma/ti/k3-psil-am62l.c | 132 +++++++++++++++++++++++++++++++++ drivers/dma/ti/k3-psil-priv.h | 1 + drivers/dma/ti/k3-psil.c | 1 + 4 files changed, 135 insertions(+) create mode 100644 drivers/dma/ti/k3-psil-am62l.c diff --git a/drivers/dma/ti/Makefile b/drivers/dma/ti/Makefile index 257e8141d7fe0..b03235a78d6cc 100644 --- a/drivers/dma/ti/Makefile +++ b/drivers/dma/ti/Makefile @@ -12,6 +12,7 @@ k3-psil-lib-objs :=3D k3-psil.o \ k3-psil-j721s2.o \ k3-psil-am62.o \ k3-psil-am62a.o \ + k3-psil-am62l.o \ k3-psil-j784s4.o \ k3-psil-am62p.o obj-$(CONFIG_TI_K3_PSIL) +=3D k3-psil-lib.o diff --git a/drivers/dma/ti/k3-psil-am62l.c b/drivers/dma/ti/k3-psil-am62l.c new file mode 100644 index 0000000000000..45f5aac32f6a0 --- /dev/null +++ b/drivers/dma/ti/k3-psil-am62l.c @@ -0,0 +1,132 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2024-2025 Texas Instruments Incorporated - https://www.t= i.com + */ + +#include + +#include "k3-psil-priv.h" + +#define PSIL_PDMA_XY_TR(x, ch) \ + { \ + .thread_id =3D x, \ + .ep_config =3D { \ + .ep_type =3D PSIL_EP_PDMA_XY, \ + .mapped_channel_id =3D ch, \ + .default_flow_id =3D -1, \ + }, \ + } + +#define PSIL_PDMA_XY_PKT(x, ch) \ + { \ + .thread_id =3D x, \ + .ep_config =3D { \ + .ep_type =3D PSIL_EP_PDMA_XY, \ + .mapped_channel_id =3D ch, \ + .pkt_mode =3D 1, \ + .default_flow_id =3D -1 \ + }, \ + } + +#define PSIL_ETHERNET(x, ch, flow_base, flow_cnt) \ + { \ + .thread_id =3D x, \ + .ep_config =3D { \ + .ep_type =3D PSIL_EP_NATIVE, \ + .pkt_mode =3D 1, \ + .needs_epib =3D 1, \ + .psd_size =3D 16, \ + .mapped_channel_id =3D ch, \ + .flow_start =3D flow_base, \ + .flow_num =3D flow_cnt, \ + .default_flow_id =3D flow_base, \ + }, \ + } + +#define PSIL_PDMA_MCASP(x, ch) \ + { \ + .thread_id =3D x, \ + .ep_config =3D { \ + .ep_type =3D PSIL_EP_PDMA_XY, \ + .pdma_acc32 =3D 1, \ + .pdma_burst =3D 1, \ + .mapped_channel_id =3D ch, \ + }, \ + } + +/* PSI-L source thread IDs, used for RX (DMA_DEV_TO_MEM) */ +static struct psil_ep am62l_src_ep_map[] =3D { + /* PDMA_MAIN1 - UART0-6 */ + PSIL_PDMA_XY_PKT(0x4400, 0), + PSIL_PDMA_XY_PKT(0x4401, 2), + PSIL_PDMA_XY_PKT(0x4402, 4), + PSIL_PDMA_XY_PKT(0x4403, 6), + PSIL_PDMA_XY_PKT(0x4404, 8), + PSIL_PDMA_XY_PKT(0x4405, 10), + PSIL_PDMA_XY_PKT(0x4406, 12), + /* PDMA_MAIN0 - SPI0 - CH0-3 */ + PSIL_PDMA_XY_TR(0x4300, 16), + /* PDMA_MAIN0 - SPI1 - CH0-3 */ + PSIL_PDMA_XY_TR(0x4301, 24), + /* PDMA_MAIN0 - SPI2 - CH0-3 */ + PSIL_PDMA_XY_TR(0x4302, 32), + /* PDMA_MAIN0 - SPI3 - CH0-3 */ + PSIL_PDMA_XY_TR(0x4303, 40), + /* PDMA_MAIN2 - MCASP0-2 */ + PSIL_PDMA_MCASP(0x4500, 48), + PSIL_PDMA_MCASP(0x4501, 50), + PSIL_PDMA_MCASP(0x4502, 52), + /* PDMA_MAIN0 - AES */ + PSIL_PDMA_XY_TR(0x4700, 65), + /* PDMA_MAIN0 - ADC */ + PSIL_PDMA_XY_TR(0x4503, 80), + PSIL_PDMA_XY_TR(0x4504, 81), + PSIL_ETHERNET(0x4600, 96, 96, 16), +}; + +/* PSI-L destination thread IDs, used for TX (DMA_MEM_TO_DEV) */ +static struct psil_ep am62l_dst_ep_map[] =3D { + /* PDMA_MAIN1 - UART0-6 */ + PSIL_PDMA_XY_PKT(0xC400, 1), + PSIL_PDMA_XY_PKT(0xC401, 3), + PSIL_PDMA_XY_PKT(0xC402, 5), + PSIL_PDMA_XY_PKT(0xC403, 7), + PSIL_PDMA_XY_PKT(0xC404, 9), + PSIL_PDMA_XY_PKT(0xC405, 11), + PSIL_PDMA_XY_PKT(0xC406, 13), + /* PDMA_MAIN0 - SPI0 - CH0-3 */ + PSIL_PDMA_XY_TR(0xC300, 17), + /* PDMA_MAIN0 - SPI1 - CH0-3 */ + PSIL_PDMA_XY_TR(0xC301, 25), + /* PDMA_MAIN0 - SPI2 - CH0-3 */ + PSIL_PDMA_XY_TR(0xC302, 33), + /* PDMA_MAIN0 - SPI3 - CH0-3 */ + PSIL_PDMA_XY_TR(0xC303, 41), + /* PDMA_MAIN2 - MCASP0-2 */ + PSIL_PDMA_MCASP(0xC500, 49), + PSIL_PDMA_MCASP(0xC501, 51), + PSIL_PDMA_MCASP(0xC502, 53), + /* PDMA_MAIN0 - SHA */ + PSIL_PDMA_XY_TR(0xC700, 64), + /* PDMA_MAIN0 - AES */ + PSIL_PDMA_XY_TR(0xC701, 66), + /* PDMA_MAIN0 - CRC32 - CH0-1 */ + PSIL_PDMA_XY_TR(0xC702, 67), + /* CPSW3G */ + PSIL_ETHERNET(0xc600, 64, 64, 2), + PSIL_ETHERNET(0xc601, 66, 66, 2), + PSIL_ETHERNET(0xc602, 68, 68, 2), + PSIL_ETHERNET(0xc603, 70, 70, 2), + PSIL_ETHERNET(0xc604, 72, 72, 2), + PSIL_ETHERNET(0xc605, 74, 74, 2), + PSIL_ETHERNET(0xc606, 76, 76, 2), + PSIL_ETHERNET(0xc607, 78, 78, 2), +}; + +struct psil_ep_map am62l_ep_map =3D { + .name =3D "am62l", + .src =3D am62l_src_ep_map, + .src_count =3D ARRAY_SIZE(am62l_src_ep_map), + .dst =3D am62l_dst_ep_map, + .dst_count =3D ARRAY_SIZE(am62l_dst_ep_map), +}; diff --git a/drivers/dma/ti/k3-psil-priv.h b/drivers/dma/ti/k3-psil-priv.h index a577be97e3447..b2eb6d3b63e1b 100644 --- a/drivers/dma/ti/k3-psil-priv.h +++ b/drivers/dma/ti/k3-psil-priv.h @@ -44,6 +44,7 @@ extern struct psil_ep_map am64_ep_map; extern struct psil_ep_map j721s2_ep_map; extern struct psil_ep_map am62_ep_map; extern struct psil_ep_map am62a_ep_map; +extern struct psil_ep_map am62l_ep_map; extern struct psil_ep_map j784s4_ep_map; extern struct psil_ep_map am62p_ep_map; =20 diff --git a/drivers/dma/ti/k3-psil.c b/drivers/dma/ti/k3-psil.c index c4b6f0df46861..27c1bf347b248 100644 --- a/drivers/dma/ti/k3-psil.c +++ b/drivers/dma/ti/k3-psil.c @@ -25,6 +25,7 @@ static const struct soc_device_attribute k3_soc_devices[]= =3D { { .family =3D "J721S2", .data =3D &j721s2_ep_map }, { .family =3D "AM62X", .data =3D &am62_ep_map }, { .family =3D "AM62AX", .data =3D &am62a_ep_map }, + { .family =3D "AM62LX", .data =3D &am62l_ep_map }, { .family =3D "J784S4", .data =3D &j784s4_ep_map }, { .family =3D "AM62PX", .data =3D &am62p_ep_map }, { .family =3D "J722S", .data =3D &am62p_ep_map }, --=20 2.34.1 From nobody Sun Feb 8 20:13:33 2026 Received: from lelvem-ot01.ext.ti.com (lelvem-ot01.ext.ti.com [198.47.23.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB6D41DF99C; Mon, 28 Apr 2025 07:21:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.47.23.234 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745824894; cv=none; b=HXrzTNuHWoR73McfdS1a37AClOmcIzDPxEHX8fPXzBGwOwZaLM6e0Ah4CvN/AdKv1zyPxkXkdaEvKqeLMAr5b0ufbpN5DGyr3Vjj+7VAdCjTYfGs2CmIGen1EeF0XWYwm2Cfs2fXb7jYd63USKefxKi7UaHhli3G6+jgspBijxs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745824894; c=relaxed/simple; bh=3te5LPmJkpHjXH2Vc5I9HPtlBqUQBVrsokKzIQqVe3g=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=HxFDU1zb/vwcbd5WnVg95wUTNa5rsY5e2HOIPvGBC1ypbCUAG1boNAmpBi7vHgZY9M3z/NnBv5e2WmWvErZnTFs5VyO6yay7YNgyhiN97Z0rauK9k4bMIuJ8dLvIm6UI6Y0MxUZhxDa8gtMN5b+srXKJry+fSBG1mYavCFSnC/E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ti.com; spf=pass smtp.mailfrom=ti.com; dkim=pass (1024-bit key) header.d=ti.com header.i=@ti.com header.b=fzAUs6Rd; arc=none smtp.client-ip=198.47.23.234 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ti.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ti.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ti.com header.i=@ti.com header.b="fzAUs6Rd" Received: from fllv0035.itg.ti.com ([10.64.41.0]) by lelvem-ot01.ext.ti.com (8.15.2/8.15.2) with ESMTPS id 53S7LK8q2714007 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 28 Apr 2025 02:21:21 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1745824881; bh=JsyrQTZY+NVH5nil6RuG7SvmccuQbuMeBRKoaahJSPk=; h=From:To:Subject:Date:In-Reply-To:References; b=fzAUs6RdtB25icZdW+8rVWmiNJGsRC+dzNX27ABqyRKjmiF5iQ8dcSkPp4lO9Aio2 XpxqMD4RhiuNs1FlcJipWzrP7u9beTzlIT0CFtQ630OCEDLGp5bneY1diVgJwfmN0o 1DyT2ClM9WpaE3CThV6np5EXpA3RcnV5zCO+eJew= Received: from DLEE101.ent.ti.com (dlee101.ent.ti.com [157.170.170.31]) by fllv0035.itg.ti.com (8.15.2/8.15.2) with ESMTPS id 53S7LKEF016562 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 28 Apr 2025 02:21:20 -0500 Received: from DLEE110.ent.ti.com (157.170.170.21) by DLEE101.ent.ti.com (157.170.170.31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.23; Mon, 28 Apr 2025 02:21:20 -0500 Received: from lelvsmtp6.itg.ti.com (10.180.75.249) by DLEE110.ent.ti.com (157.170.170.21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.23 via Frontend Transport; Mon, 28 Apr 2025 02:21:20 -0500 Received: from uda0498651.dhcp.ti.com (uda0498651.dhcp.ti.com [172.24.227.7]) by lelvsmtp6.itg.ti.com (8.15.2/8.15.2) with ESMTP id 53S7KdMe068873; Mon, 28 Apr 2025 02:21:16 -0500 From: Sai Sree Kartheek Adivi To: , , , , , , , , , , , , , , , Subject: [PATCH 5/8] drivers: soc: ti: k3-ringacc: handle absence of tisci Date: Mon, 28 Apr 2025 12:50:29 +0530 Message-ID: <20250428072032.946008-6-s-adivi@ti.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250428072032.946008-1-s-adivi@ti.com> References: <20250428072032.946008-1-s-adivi@ti.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-C2ProcessedOrg: 333ef613-75bf-4e12-a4b1-8e3623f5dcea Content-Type: text/plain; charset="utf-8" Handle absence of tisci with direct register writes. This will support platforms that do not have tisci firmware like AM62L. Signed-off-by: Sai Sree Kartheek Adivi --- drivers/soc/ti/k3-ringacc.c | 162 +++++++++++++++++++++++++----- include/linux/soc/ti/k3-ringacc.h | 4 + 2 files changed, 142 insertions(+), 24 deletions(-) diff --git a/drivers/soc/ti/k3-ringacc.c b/drivers/soc/ti/k3-ringacc.c index 82a15cad1c6c4..49e0483676a14 100644 --- a/drivers/soc/ti/k3-ringacc.c +++ b/drivers/soc/ti/k3-ringacc.c @@ -45,6 +45,38 @@ struct k3_ring_rt_regs { u32 hwindx; }; =20 +#define K3_RINGACC_RT_CFG_REGS_OFS 0x40 +#define K3_DMARING_CFG_ADDR_HI_MASK GENMASK(3, 0) +#define K3_DMARING_CFG_ASEL_SHIFT 16 +#define K3_DMARING_CFG_SIZE_MASK GENMASK(15, 0) + +/** + * struct k3_ring_cfg_regs - The RA Configuration Registers region + * + * @ba_lo: Ring Base Address Low Register + * @ba_hi: Ring Base Address High Register + * @size: Ring Size Register + */ +struct k3_ring_cfg_regs { + u32 ba_lo; + u32 ba_hi; + u32 size; +}; + +#define K3_RINGACC_RT_INT_REGS_OFS 0x140 +#define K3_RINGACC_RT_INT_ENABLE_SET_COMPLETE BIT(0) +#define K3_RINGACC_RT_INT_ENABLE_SET_TR BIT(2) + +struct k3_ring_intr_regs { + u32 enable_set; + u32 resv_4; + u32 clr; + u32 resv_16; + u32 status_set; + u32 resv_8; + u32 status; +}; + #define K3_RINGACC_RT_REGS_STEP 0x1000 #define K3_DMARING_RT_REGS_STEP 0x2000 #define K3_DMARING_RT_REGS_REVERSE_OFS 0x1000 @@ -157,6 +189,8 @@ struct k3_ring_state { */ struct k3_ring { struct k3_ring_rt_regs __iomem *rt; + struct k3_ring_cfg_regs __iomem *cfg; + struct k3_ring_intr_regs __iomem *intr; struct k3_ring_fifo_regs __iomem *fifos; struct k3_ringacc_proxy_target_regs __iomem *proxy; dma_addr_t ring_mem_dma; @@ -465,16 +499,30 @@ static void k3_ringacc_ring_reset_sci(struct k3_ring = *ring) struct ti_sci_msg_rm_ring_cfg ring_cfg =3D { 0 }; struct k3_ringacc *ringacc =3D ring->parent; int ret; + u32 reg; =20 - ring_cfg.nav_id =3D ringacc->tisci_dev_id; - ring_cfg.index =3D ring->ring_id; - ring_cfg.valid_params =3D TI_SCI_MSG_VALUE_RM_RING_COUNT_VALID; - ring_cfg.count =3D ring->size; + if (!ringacc->tisci) { + if (ring->cfg =3D=3D NULL) + return; + reg =3D readl(&ring->cfg->size); + reg &=3D ~K3_DMARING_CFG_SIZE_MASK; =20 - ret =3D ringacc->tisci_ring_ops->set_cfg(ringacc->tisci, &ring_cfg); - if (ret) - dev_err(ringacc->dev, "TISCI reset ring fail (%d) ring_idx %d\n", - ret, ring->ring_id); + writel(reg, &ring->cfg->size); + wmb(); + reg |=3D ring->size; + + writel(reg, &ring->cfg->size); + } else { + ring_cfg.nav_id =3D ringacc->tisci_dev_id; + ring_cfg.index =3D ring->ring_id; + ring_cfg.valid_params =3D TI_SCI_MSG_VALUE_RM_RING_COUNT_VALID; + ring_cfg.count =3D ring->size; + + ret =3D ringacc->tisci_ring_ops->set_cfg(ringacc->tisci, &ring_cfg); + if (ret) + dev_err(ringacc->dev, "TISCI reset ring fail (%d) ring_idx %d\n", + ret, ring->ring_id); + } } =20 void k3_ringacc_ring_reset(struct k3_ring *ring) @@ -494,16 +542,30 @@ static void k3_ringacc_ring_reconfig_qmode_sci(struct= k3_ring *ring, struct ti_sci_msg_rm_ring_cfg ring_cfg =3D { 0 }; struct k3_ringacc *ringacc =3D ring->parent; int ret; + u32 reg; =20 ring_cfg.nav_id =3D ringacc->tisci_dev_id; ring_cfg.index =3D ring->ring_id; ring_cfg.valid_params =3D TI_SCI_MSG_VALUE_RM_RING_MODE_VALID; ring_cfg.mode =3D mode; =20 - ret =3D ringacc->tisci_ring_ops->set_cfg(ringacc->tisci, &ring_cfg); - if (ret) - dev_err(ringacc->dev, "TISCI reconf qmode fail (%d) ring_idx %d\n", - ret, ring->ring_id); + if (!ringacc->tisci) { + writel(ring_cfg.addr_lo, &ring->cfg->ba_lo); + writel((ring_cfg.addr_hi & K3_DMARING_CFG_ADDR_HI_MASK) + + (ring_cfg.asel << K3_DMARING_CFG_ASEL_SHIFT), + &ring->cfg->ba_hi); + + reg =3D readl(&ring->cfg->size); + reg &=3D ~K3_DMARING_CFG_SIZE_MASK; + reg |=3D ring_cfg.count & K3_DMARING_CFG_SIZE_MASK; + + writel(reg, &ring->cfg->size); + } else { + ret =3D ringacc->tisci_ring_ops->set_cfg(ringacc->tisci, &ring_cfg); + if (ret) + dev_err(ringacc->dev, "TISCI reconf qmode fail (%d) ring_idx %d\n", + ret, ring->ring_id); + } } =20 void k3_ringacc_ring_reset_dma(struct k3_ring *ring, u32 occ) @@ -570,15 +632,29 @@ static void k3_ringacc_ring_free_sci(struct k3_ring *= ring) struct ti_sci_msg_rm_ring_cfg ring_cfg =3D { 0 }; struct k3_ringacc *ringacc =3D ring->parent; int ret; + u32 reg; =20 ring_cfg.nav_id =3D ringacc->tisci_dev_id; ring_cfg.index =3D ring->ring_id; ring_cfg.valid_params =3D TI_SCI_MSG_VALUE_RM_ALL_NO_ORDER; =20 - ret =3D ringacc->tisci_ring_ops->set_cfg(ringacc->tisci, &ring_cfg); - if (ret) - dev_err(ringacc->dev, "TISCI ring free fail (%d) ring_idx %d\n", - ret, ring->ring_id); + if (!ringacc->tisci) { + writel(ring_cfg.addr_lo, &ring->cfg->ba_lo); + writel((ring_cfg.addr_hi & K3_DMARING_CFG_ADDR_HI_MASK) + + (ring_cfg.asel << K3_DMARING_CFG_ASEL_SHIFT), + &ring->cfg->ba_hi); + + reg =3D readl(&ring->cfg->size); + reg &=3D ~K3_DMARING_CFG_SIZE_MASK; + reg |=3D ring_cfg.count & K3_DMARING_CFG_SIZE_MASK; + + writel(reg, &ring->cfg->size); + } else { + ret =3D ringacc->tisci_ring_ops->set_cfg(ringacc->tisci, &ring_cfg); + if (ret) + dev_err(ringacc->dev, "TISCI ring free fail (%d) ring_idx %d\n", + ret, ring->ring_id); + } } =20 int k3_ringacc_ring_free(struct k3_ring *ring) @@ -669,15 +745,31 @@ int k3_ringacc_get_ring_irq_num(struct k3_ring *ring) } EXPORT_SYMBOL_GPL(k3_ringacc_get_ring_irq_num); =20 +u32 k3_ringacc_ring_get_irq_status(struct k3_ring *ring) +{ + struct k3_ringacc *ringacc =3D ring->parent; + struct k3_ring *ring2 =3D &ringacc->rings[ring->ring_id]; + + return readl(&ring2->intr->status); +} +EXPORT_SYMBOL_GPL(k3_ringacc_ring_get_irq_status); + +void k3_ringacc_ring_clear_irq(struct k3_ring *ring) +{ + struct k3_ringacc *ringacc =3D ring->parent; + struct k3_ring *ring2 =3D &ringacc->rings[ring->ring_id]; + + writel(0xFF, &ring2->intr->status); +} +EXPORT_SYMBOL_GPL(k3_ringacc_ring_clear_irq); + static int k3_ringacc_ring_cfg_sci(struct k3_ring *ring) { struct ti_sci_msg_rm_ring_cfg ring_cfg =3D { 0 }; struct k3_ringacc *ringacc =3D ring->parent; + u32 reg; int ret; =20 - if (!ringacc->tisci) - return -EINVAL; - ring_cfg.nav_id =3D ringacc->tisci_dev_id; ring_cfg.index =3D ring->ring_id; ring_cfg.valid_params =3D TI_SCI_MSG_VALUE_RM_ALL_NO_ORDER; @@ -688,11 +780,26 @@ static int k3_ringacc_ring_cfg_sci(struct k3_ring *ri= ng) ring_cfg.size =3D ring->elm_size; ring_cfg.asel =3D ring->asel; =20 + if (!ringacc->tisci) { + writel(ring_cfg.addr_lo, &ring->cfg->ba_lo); + writel((ring_cfg.addr_hi & K3_DMARING_CFG_ADDR_HI_MASK) + + (ring_cfg.asel << K3_DMARING_CFG_ASEL_SHIFT), + &ring->cfg->ba_hi); + + reg =3D readl(&ring->cfg->size); + reg &=3D ~K3_DMARING_CFG_SIZE_MASK; + reg |=3D ring_cfg.count & K3_DMARING_CFG_SIZE_MASK; + + writel(reg, &ring->cfg->size); + writel(K3_RINGACC_RT_INT_ENABLE_SET_COMPLETE | K3_RINGACC_RT_INT_ENABLE_= SET_TR, + &ring->intr->enable_set); + return 0; + } + ret =3D ringacc->tisci_ring_ops->set_cfg(ringacc->tisci, &ring_cfg); if (ret) dev_err(ringacc->dev, "TISCI config ring fail (%d) ring_idx %d\n", - ret, ring->ring_id); - + ret, ring->ring_id); return ret; } =20 @@ -1480,9 +1587,12 @@ struct k3_ringacc *k3_ringacc_dmarings_init(struct p= latform_device *pdev, =20 mutex_init(&ringacc->req_lock); =20 - base_rt =3D devm_platform_ioremap_resource_byname(pdev, "ringrt"); - if (IS_ERR(base_rt)) - return ERR_CAST(base_rt); + base_rt =3D data->base_rt; + if (!base_rt) { + base_rt =3D devm_platform_ioremap_resource_byname(pdev, "ringrt"); + if (IS_ERR(base_rt)) + return ERR_CAST(base_rt); + } =20 ringacc->rings =3D devm_kzalloc(dev, sizeof(*ringacc->rings) * @@ -1498,6 +1608,10 @@ struct k3_ringacc *k3_ringacc_dmarings_init(struct p= latform_device *pdev, struct k3_ring *ring =3D &ringacc->rings[i]; =20 ring->rt =3D base_rt + K3_DMARING_RT_REGS_STEP * i; + ring->cfg =3D base_rt + K3_RINGACC_RT_CFG_REGS_OFS + + K3_DMARING_RT_REGS_STEP * i; + ring->intr =3D base_rt + K3_RINGACC_RT_INT_REGS_OFS + + K3_DMARING_RT_REGS_STEP * i; ring->parent =3D ringacc; ring->ring_id =3D i; ring->proxy_id =3D K3_RINGACC_PROXY_NOT_USED; diff --git a/include/linux/soc/ti/k3-ringacc.h b/include/linux/soc/ti/k3-ri= ngacc.h index 39b022b925986..fcf6fbd4a8594 100644 --- a/include/linux/soc/ti/k3-ringacc.h +++ b/include/linux/soc/ti/k3-ringacc.h @@ -158,6 +158,9 @@ u32 k3_ringacc_get_ring_id(struct k3_ring *ring); */ int k3_ringacc_get_ring_irq_num(struct k3_ring *ring); =20 +u32 k3_ringacc_ring_get_irq_status(struct k3_ring *ring); +void k3_ringacc_ring_clear_irq(struct k3_ring *ring); + /** * k3_ringacc_ring_cfg - ring configure * @ring: pointer on ring @@ -262,6 +265,7 @@ struct k3_ringacc_init_data { const struct ti_sci_handle *tisci; u32 tisci_dev_id; u32 num_rings; + void __iomem *base_rt; }; =20 struct k3_ringacc *k3_ringacc_dmarings_init(struct platform_device *pdev, --=20 2.34.1 From nobody Sun Feb 8 20:13:33 2026 Received: from lelvem-ot02.ext.ti.com (lelvem-ot02.ext.ti.com [198.47.23.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E41FD1D8E07; Mon, 28 Apr 2025 07:21:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.47.23.235 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745824905; cv=none; b=jVzhfDsPAcO+PXFoNCT2qVODtacL+OFyfEmhYTgcQsHGaEPiIWet/UBNNd4jbLUgcljL+/is9QVAwU/ki3QO9lHrjVnyg+1cOUSUSJVilKOqZiKL16efR5ejm4PpdEs8Mq0h0DjvwbcYaCC6UdbNmNl3imoZDeXFcKJaacTd74A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745824905; c=relaxed/simple; bh=nlSW2adgbr1e62nvNsDM0yD02xXHGbRNQ3/a7JJcSJk=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=OcByezzixM1AnGeZdG4VlWi0SXglFZMlBnghIX1wGqf45cgC/m5SCJp2ME0Jbcrn3v6jOe0kA87lxS6x1TARID//J7gq0rdpCLJYUj/7bEYIXslz3gRXIJLiO8Eb6vj5sW80kBLStqCrG4/oE+sidEDtw5wXGjQXKzrRP3tSU5w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ti.com; spf=pass smtp.mailfrom=ti.com; dkim=pass (1024-bit key) header.d=ti.com header.i=@ti.com header.b=F2rWBxA6; arc=none smtp.client-ip=198.47.23.235 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ti.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ti.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ti.com header.i=@ti.com header.b="F2rWBxA6" Received: from lelv0265.itg.ti.com ([10.180.67.224]) by lelvem-ot02.ext.ti.com (8.15.2/8.15.2) with ESMTPS id 53S7LQGZ3475787 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 28 Apr 2025 02:21:26 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1745824886; bh=kChFHX/gFAfOC/4OJnMr8CD0BeUuRE+BT8Md6wvLP0I=; h=From:To:Subject:Date:In-Reply-To:References; b=F2rWBxA6X1qir4idgkCRnv9mBpS/VCG9K3zA78+ZCK4JJxvHM6bxf+qGapbgMetNq 1kstyRSQp96pWLeK10Rob3dFiD6MEq2nLk5fTNBf9OFSSap/JbL/5bLm6YDdfwrOym k8qpjOpws8gk+b84qGvMU+8pTna6zTiwfclhAB/s= Received: from DFLE102.ent.ti.com (dfle102.ent.ti.com [10.64.6.23]) by lelv0265.itg.ti.com (8.15.2/8.15.2) with ESMTPS id 53S7LQFg003660 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 28 Apr 2025 02:21:26 -0500 Received: from DFLE107.ent.ti.com (10.64.6.28) by DFLE102.ent.ti.com (10.64.6.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.23; Mon, 28 Apr 2025 02:21:26 -0500 Received: from lelvsmtp6.itg.ti.com (10.180.75.249) by DFLE107.ent.ti.com (10.64.6.28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.23 via Frontend Transport; Mon, 28 Apr 2025 02:21:25 -0500 Received: from uda0498651.dhcp.ti.com (uda0498651.dhcp.ti.com [172.24.227.7]) by lelvsmtp6.itg.ti.com (8.15.2/8.15.2) with ESMTP id 53S7KdMf068873; Mon, 28 Apr 2025 02:21:21 -0500 From: Sai Sree Kartheek Adivi To: , , , , , , , , , , , , , , , Subject: [PATCH 6/8] dmaengine: ti: New driver for K3 BCDMA_V2 Date: Mon, 28 Apr 2025 12:50:30 +0530 Message-ID: <20250428072032.946008-7-s-adivi@ti.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250428072032.946008-1-s-adivi@ti.com> References: <20250428072032.946008-1-s-adivi@ti.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-C2ProcessedOrg: 333ef613-75bf-4e12-a4b1-8e3623f5dcea Content-Type: text/plain; charset="utf-8" Add support for BCDMA_V2. The BCDMA_V2 is different than the existing BCDMA supported by the k3-udma driver. The changes in BCDMA_V2 are: - Autopair: There is no longer a need for PSIL pair and AUTOPAIR bit needs to set in the RT_CTL register. - Static channel mapping: Each channel is mapped to a single peripheral. - Direct IRQs: There is no INT-A and interrupt lines from DMA are directly connected to GIC. - Remote side configuration handled by DMA. So no need to write to PEER registers to START / STOP / PAUSE / TEARDOWN. Signed-off-by: Sai Sree Kartheek Adivi --- drivers/dma/ti/Kconfig | 14 +- drivers/dma/ti/Makefile | 1 + drivers/dma/ti/k3-udma-common.c | 76 +- drivers/dma/ti/k3-udma-v2.c | 1322 +++++++++++++++++++++++++++++ drivers/dma/ti/k3-udma.h | 118 +-- include/linux/soc/ti/k3-ringacc.h | 3 + 6 files changed, 1467 insertions(+), 67 deletions(-) create mode 100644 drivers/dma/ti/k3-udma-v2.c diff --git a/drivers/dma/ti/Kconfig b/drivers/dma/ti/Kconfig index 2adc2cca10e92..bb37015074834 100644 --- a/drivers/dma/ti/Kconfig +++ b/drivers/dma/ti/Kconfig @@ -47,17 +47,27 @@ config TI_K3_UDMA Enable support for the TI UDMA (Unified DMA) controller. This DMA engine is used in AM65x and j721e. =20 +config TI_K3_UDMA_V2 + tristate "Texas Instruments AM62L UDMA v2 support" + depends on ARCH_K3 + select DMA_ENGINE + select DMA_VIRTUAL_CHANNELS + select TI_K3_RINGACC + select TI_K3_PSIL + help + Enable support for the TI UDMA (Unified DMA) v2 controller. + config TI_K3_UDMA_GLUE_LAYER tristate "Texas Instruments UDMA Glue layer for non DMAengine users" depends on ARCH_K3 - depends on TI_K3_UDMA + depends on TI_K3_UDMA || TI_K3_UDMA_AM62L help Say y here to support the K3 NAVSS DMA glue interface If unsure, say N. =20 config TI_K3_PSIL tristate - default TI_K3_UDMA + default TI_K3_UDMA || TI_K3_UDMA_AM62L =20 config TI_DMA_CROSSBAR bool diff --git a/drivers/dma/ti/Makefile b/drivers/dma/ti/Makefile index b03235a78d6cc..f4e06412ab53f 100644 --- a/drivers/dma/ti/Makefile +++ b/drivers/dma/ti/Makefile @@ -3,6 +3,7 @@ obj-$(CONFIG_TI_CPPI41) +=3D cppi41.o obj-$(CONFIG_TI_EDMA) +=3D edma.o obj-$(CONFIG_DMA_OMAP) +=3D omap-dma.o obj-$(CONFIG_TI_K3_UDMA) +=3D k3-udma.o k3-udma-common.o +obj-$(CONFIG_TI_K3_UDMA_V2) +=3D k3-udma-v2.o k3-udma-common.o obj-$(CONFIG_TI_K3_UDMA_GLUE_LAYER) +=3D k3-udma-glue.o k3-psil-lib-objs :=3D k3-psil.o \ k3-psil-am654.o \ diff --git a/drivers/dma/ti/k3-udma-common.c b/drivers/dma/ti/k3-udma-commo= n.c index 078b018b22830..462ad17c74604 100644 --- a/drivers/dma/ti/k3-udma-common.c +++ b/drivers/dma/ti/k3-udma-common.c @@ -597,6 +597,8 @@ int udma_get_tchan(struct udma_chan *uc) uc->tchan =3D NULL; return ret; } + if (ud->match_data->type =3D=3D DMA_TYPE_BCDMA_V2) + uc->chan =3D uc->tchan; =20 if (ud->tflow_cnt) { int tflow_id; @@ -646,6 +648,8 @@ int udma_get_rchan(struct udma_chan *uc) uc->rchan =3D NULL; return ret; } + if (ud->match_data->type =3D=3D DMA_TYPE_BCDMA_V2) + uc->chan =3D uc->rchan; =20 return 0; } @@ -967,6 +971,7 @@ udma_prep_slave_sg_tr(struct udma_chan *uc, struct scat= terlist *sgl, size_t tr_size; int num_tr =3D 0; int tr_idx =3D 0; + u32 extra_flags =3D 0; u64 asel; =20 /* estimate the number of TRs we will need */ @@ -990,6 +995,11 @@ udma_prep_slave_sg_tr(struct udma_chan *uc, struct sca= tterlist *sgl, else asel =3D (u64)uc->config.asel << K3_ADDRESS_ASEL_SHIFT; =20 + + if (dir =3D=3D DMA_MEM_TO_DEV && uc->ud->match_data->type =3D=3D DMA_TYPE= _BCDMA_V2) + extra_flags =3D CPPI5_TR_CSF_EOP; + + tr_req =3D d->hwdesc[0].tr_req_base; for_each_sg(sgl, sgent, sglen, i) { dma_addr_t sg_addr =3D sg_dma_address(sgent); @@ -1006,7 +1016,7 @@ udma_prep_slave_sg_tr(struct udma_chan *uc, struct sc= atterlist *sgl, =20 cppi5_tr_init(&tr_req[tr_idx].flags, CPPI5_TR_TYPE1, false, false, CPPI5_TR_EVENT_SIZE_COMPLETION, 0); - cppi5_tr_csf_set(&tr_req[tr_idx].flags, CPPI5_TR_CSF_SUPR_EVT); + cppi5_tr_csf_set(&tr_req[tr_idx].flags, CPPI5_TR_CSF_SUPR_EVT | extra_fl= ags); =20 sg_addr |=3D asel; tr_req[tr_idx].addr =3D sg_addr; @@ -1020,7 +1030,7 @@ udma_prep_slave_sg_tr(struct udma_chan *uc, struct sc= atterlist *sgl, false, false, CPPI5_TR_EVENT_SIZE_COMPLETION, 0); cppi5_tr_csf_set(&tr_req[tr_idx].flags, - CPPI5_TR_CSF_SUPR_EVT); + CPPI5_TR_CSF_SUPR_EVT | extra_flags); =20 tr_req[tr_idx].addr =3D sg_addr + tr0_cnt1 * tr0_cnt0; tr_req[tr_idx].icnt0 =3D tr1_cnt0; @@ -1280,7 +1290,8 @@ int udma_configure_statictr(struct udma_chan *uc, str= uct udma_desc *d, d->static_tr.bstcnt =3D d->residue / d->sglen / div; else d->static_tr.bstcnt =3D d->residue / div; - } else if (uc->ud->match_data->type =3D=3D DMA_TYPE_BCDMA && + } else if ((uc->ud->match_data->type =3D=3D DMA_TYPE_BCDMA || + uc->ud->match_data->type =3D=3D DMA_TYPE_BCDMA_V2) && uc->config.dir =3D=3D DMA_DEV_TO_MEM && uc->cyclic) { /* @@ -1605,7 +1616,8 @@ udma_prep_dma_cyclic_tr(struct udma_chan *uc, dma_add= r_t buf_addr, * last one, so set the flag for each period. */ if (uc->config.ep_type =3D=3D PSIL_EP_PDMA_XY && - uc->ud->match_data->type =3D=3D DMA_TYPE_BCDMA) { + (uc->ud->match_data->type =3D=3D DMA_TYPE_BCDMA || + uc->ud->match_data->type =3D=3D DMA_TYPE_BCDMA_V2)) { period_csf =3D CPPI5_TR_CSF_EOP; } =20 @@ -2057,8 +2069,9 @@ void udma_free_chan_resources(struct dma_chan *chan) =20 /* Release PSI-L pairing */ if (uc->psil_paired) { - navss_psil_unpair(ud, uc->config.src_thread, - uc->config.dst_thread); + if (ud->match_data->type < DMA_TYPE_BCDMA_V2 && IS_ENABLED(CONFIG_TI_K3_= UDMA)) + navss_psil_unpair(ud, uc->config.src_thread, + uc->config.dst_thread); uc->psil_paired =3D false; } =20 @@ -2086,6 +2099,7 @@ int setup_resources(struct udma_dev *ud) ret =3D udma_setup_resources(ud); break; case DMA_TYPE_BCDMA: + case DMA_TYPE_BCDMA_V2: ret =3D bcdma_setup_resources(ud); break; case DMA_TYPE_PKTDMA: @@ -2098,11 +2112,18 @@ int setup_resources(struct udma_dev *ud) if (ret) return ret; =20 - ch_count =3D ud->bchan_cnt + ud->tchan_cnt + ud->rchan_cnt; - if (ud->bchan_cnt) - ch_count -=3D bitmap_weight(ud->bchan_map, ud->bchan_cnt); - ch_count -=3D bitmap_weight(ud->tchan_map, ud->tchan_cnt); - ch_count -=3D bitmap_weight(ud->rchan_map, ud->rchan_cnt); + if (ud->match_data->type =3D=3D DMA_TYPE_BCDMA_V2) { + ch_count =3D ud->bchan_cnt + ud->tchan_cnt; + if (ud->bchan_cnt) + ch_count -=3D bitmap_weight(ud->bchan_map, ud->bchan_cnt); + ch_count -=3D bitmap_weight(ud->tchan_map, ud->tchan_cnt); + } else { + ch_count =3D ud->bchan_cnt + ud->tchan_cnt + ud->rchan_cnt; + if (ud->bchan_cnt) + ch_count -=3D bitmap_weight(ud->bchan_map, ud->bchan_cnt); + ch_count -=3D bitmap_weight(ud->tchan_map, ud->tchan_cnt); + ch_count -=3D bitmap_weight(ud->rchan_map, ud->rchan_cnt); + } if (!ch_count) return -ENODEV; =20 @@ -2134,6 +2155,15 @@ int setup_resources(struct udma_dev *ud) ud->rchan_cnt - bitmap_weight(ud->rchan_map, ud->rchan_cnt)); break; + case DMA_TYPE_BCDMA_V2: + dev_info(dev, + "Channels: %d (bchan: %u, chan: %u)\n", + ch_count, + ud->bchan_cnt - bitmap_weight(ud->bchan_map, + ud->bchan_cnt), + ud->chan_cnt - bitmap_weight(ud->chan_map, + ud->chan_cnt)); + break; case DMA_TYPE_PKTDMA: dev_info(dev, "Channels: %d (tchan: %u, rchan: %u)\n", @@ -2370,16 +2400,26 @@ int bcdma_setup_resources(struct udma_dev *ud) =20 ud->bchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->bchan_cnt), sizeof(unsigned long), GFP_KERNEL); + bitmap_zero(ud->bchan_map, ud->bchan_cnt); ud->bchans =3D devm_kcalloc(dev, ud->bchan_cnt, sizeof(*ud->bchans), GFP_KERNEL); ud->tchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->tchan_cnt), sizeof(unsigned long), GFP_KERNEL); + bitmap_zero(ud->tchan_map, ud->tchan_cnt); ud->tchans =3D devm_kcalloc(dev, ud->tchan_cnt, sizeof(*ud->tchans), GFP_KERNEL); - ud->rchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->rchan_cnt), - sizeof(unsigned long), GFP_KERNEL); - ud->rchans =3D devm_kcalloc(dev, ud->rchan_cnt, sizeof(*ud->rchans), - GFP_KERNEL); + if (ud->match_data->type =3D=3D DMA_TYPE_BCDMA_V2) { + ud->rchan_map =3D ud->tchan_map; + ud->rchans =3D ud->tchans; + ud->chan_map =3D ud->tchan_map; + ud->chans =3D ud->tchans; + } else { + ud->rchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->rchan_cnt), + sizeof(unsigned long), GFP_KERNEL); + bitmap_zero(ud->rchan_map, ud->rchan_cnt); + ud->rchans =3D devm_kcalloc(dev, ud->rchan_cnt, sizeof(*ud->rchans), + GFP_KERNEL); + } /* BCDMA do not really have flows, but the driver expect it */ ud->rflow_in_use =3D devm_kcalloc(dev, BITS_TO_LONGS(ud->rchan_cnt), sizeof(unsigned long), @@ -2392,6 +2432,9 @@ int bcdma_setup_resources(struct udma_dev *ud) !ud->rflows) return -ENOMEM; =20 + if (ud->match_data->type =3D=3D DMA_TYPE_BCDMA_V2) + return 0; + /* Get resource ranges from tisci */ for (i =3D 0; i < RM_RANGE_LAST; i++) { if (i =3D=3D RM_RANGE_RFLOW || i =3D=3D RM_RANGE_TFLOW) @@ -2821,7 +2864,8 @@ void udma_dbg_summary_show_chan(struct seq_file *s, =20 switch (uc->config.dir) { case DMA_MEM_TO_MEM: - if (uc->ud->match_data->type =3D=3D DMA_TYPE_BCDMA) { + if (uc->ud->match_data->type =3D=3D DMA_TYPE_BCDMA || + uc->ud->match_data->type =3D=3D DMA_TYPE_BCDMA_V2) { seq_printf(s, "bchan%d)\n", uc->bchan->id); return; } diff --git a/drivers/dma/ti/k3-udma-v2.c b/drivers/dma/ti/k3-udma-v2.c new file mode 100644 index 0000000000000..90b5ac5e00ead --- /dev/null +++ b/drivers/dma/ti/k3-udma-v2.c @@ -0,0 +1,1322 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Derived from K3 UDMA driver (k3-udma.c) + * Copyright (C) 2024-2025 Texas Instruments Incorporated - http://www.ti= .com + * Author: Peter Ujfalusi + * Author: Sai Sree Kartheek Adivi + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../virt-dma.h" +#include "k3-udma.h" +#include "k3-psil-priv.h" + +#define UDMA_CHAN_RT_STATIC_TR_XY_REG 0x800 +#define UDMA_CHAN_RT_STATIC_TR_Z_REG 0x804 +#define UDMA_CHAN_RT_PERIPH_BCNT_REG 0x810 + +static const char * const v2_mmr_names[] =3D { + [V2_MMR_GCFG] =3D "gcfg", + [V2_MMR_BCHANRT] =3D "bchanrt", + [V2_MMR_CHANRT] =3D "chanrt", +}; + +static int udma_v2_check_chan_autopair_completion(struct udma_chan *uc) +{ + u32 val; + + val =3D udma_chanrt_read(uc, UDMA_CHAN_RT_CTL_REG); + if (val & UDMA_CHAN_RT_CTL_PAIR_TIMEOUT) + return -ETIMEDOUT; + else if (val & UDMA_CHAN_RT_CTL_PAIR_COMPLETE) + return 1; + + /* timeout didn't occur and also pairing didn't happen yet. */ + return 0; +} + +static bool udma_v2_is_chan_paused(struct udma_chan *uc) +{ + u32 val, pause_mask; + + if (uc->config.dir =3D=3D DMA_MEM_TO_MEM) { + val =3D udma_chanrt_read(uc, UDMA_CHAN_RT_CTL_REG); + pause_mask =3D UDMA_CHAN_RT_CTL_PAUSE; + } else { + val =3D udma_chanrt_read(uc, UDMA_CHAN_RT_PDMA_STATE_REG); + pause_mask =3D UDMA_CHAN_RT_PDMA_STATE_PAUSE; + } + + if (val & pause_mask) + return true; + + return false; +} + +static void udma_v2_decrement_byte_counters(struct udma_chan *uc, u32 val) +{ + udma_chanrt_write(uc, UDMA_CHAN_RT_BCNT_REG, val); + udma_chanrt_write(uc, UDMA_CHAN_RT_SBCNT_REG, val); + if (uc->config.ep_type !=3D PSIL_EP_NATIVE) + udma_chanrt_write(uc, UDMA_CHAN_RT_PERIPH_BCNT_REG, val); +} + +static void udma_v2_reset_counters(struct udma_chan *uc) +{ + u32 val; + + val =3D udma_chanrt_read(uc, UDMA_CHAN_RT_BCNT_REG); + udma_chanrt_write(uc, UDMA_CHAN_RT_BCNT_REG, val); + + val =3D udma_chanrt_read(uc, UDMA_CHAN_RT_SBCNT_REG); + udma_chanrt_write(uc, UDMA_CHAN_RT_SBCNT_REG, val); + + val =3D udma_chanrt_read(uc, UDMA_CHAN_RT_PCNT_REG); + udma_chanrt_write(uc, UDMA_CHAN_RT_PCNT_REG, val); + + if (!uc->bchan) { + val =3D udma_chanrt_read(uc, UDMA_CHAN_RT_PERIPH_BCNT_REG); + udma_chanrt_write(uc, UDMA_CHAN_RT_PERIPH_BCNT_REG, val); + } +} + +static int udma_v2_reset_chan(struct udma_chan *uc, bool hard) +{ + udma_chanrt_write(uc, UDMA_CHAN_RT_CTL_REG, 0); + + /* Reset all counters */ + udma_v2_reset_counters(uc); + + /* Hard reset: re-initialize the channel to reset */ + if (hard) { + struct udma_chan_config ucc_backup; + int ret; + + memcpy(&ucc_backup, &uc->config, sizeof(uc->config)); + uc->ud->ddev.device_free_chan_resources(&uc->vc.chan); + + /* restore the channel configuration */ + memcpy(&uc->config, &ucc_backup, sizeof(uc->config)); + ret =3D uc->ud->ddev.device_alloc_chan_resources(&uc->vc.chan); + if (ret) + return ret; + + /* + * Setting forced teardown after forced reset helps recovering + * the rchan. + */ + if (uc->config.dir =3D=3D DMA_DEV_TO_MEM) + udma_chanrt_update_bits(uc, UDMA_CHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_EN | UDMA_CHAN_RT_CTL_TDOWN | + UDMA_CHAN_RT_CTL_FTDOWN, + UDMA_CHAN_RT_CTL_EN | UDMA_CHAN_RT_CTL_TDOWN | + UDMA_CHAN_RT_CTL_FTDOWN); + } + uc->state =3D UDMA_CHAN_IS_IDLE; + + return 0; +} + +static int udma_v2_start(struct udma_chan *uc) +{ + struct virt_dma_desc *vd =3D vchan_next_desc(&uc->vc); + struct udma_dev *ud =3D uc->ud; + int status, ret; + + if (!vd) { + uc->desc =3D NULL; + return -ENOENT; + } + + list_del(&vd->node); + + uc->desc =3D to_udma_desc(&vd->tx); + + /* Channel is already running and does not need reconfiguration */ + if (udma_is_chan_running(uc) && !udma_chan_needs_reconfiguration(uc)) { + udma_start_desc(uc); + goto out; + } + + /* Make sure that we clear the teardown bit, if it is set */ + ud->udma_reset_chan(uc, false); + + /* Push descriptors before we start the channel */ + udma_start_desc(uc); + + switch (uc->desc->dir) { + case DMA_DEV_TO_MEM: + /* Config remote TR */ + if (uc->config.ep_type =3D=3D PSIL_EP_PDMA_XY) { + u32 val =3D PDMA_STATIC_TR_Y(uc->desc->static_tr.elcnt) | + PDMA_STATIC_TR_X(uc->desc->static_tr.elsize); + const struct udma_match_data *match_data =3D + uc->ud->match_data; + + if (uc->config.enable_acc32) + val |=3D PDMA_STATIC_TR_XY_ACC32; + if (uc->config.enable_burst) + val |=3D PDMA_STATIC_TR_XY_BURST; + + udma_chanrt_write(uc, + UDMA_CHAN_RT_STATIC_TR_XY_REG, + val); + + udma_chanrt_write(uc, + UDMA_CHAN_RT_STATIC_TR_Z_REG, + PDMA_STATIC_TR_Z(uc->desc->static_tr.bstcnt, + match_data->statictr_z_mask)); + + /* save the current staticTR configuration */ + memcpy(&uc->static_tr, &uc->desc->static_tr, + sizeof(uc->static_tr)); + } + + udma_chanrt_write(uc, UDMA_CHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_EN | UDMA_CHAN_RT_CTL_AUTOPAIR); + + /* Poll for autopair completion */ + ret =3D read_poll_timeout_atomic(udma_v2_check_chan_autopair_completion, + status, status !=3D 0, 100, 500, false, uc); + + if (status <=3D 0) + return -ETIMEDOUT; + + break; + case DMA_MEM_TO_DEV: + /* Config remote TR */ + if (uc->config.ep_type =3D=3D PSIL_EP_PDMA_XY) { + u32 val =3D PDMA_STATIC_TR_Y(uc->desc->static_tr.elcnt) | + PDMA_STATIC_TR_X(uc->desc->static_tr.elsize); + + if (uc->config.enable_acc32) + val |=3D PDMA_STATIC_TR_XY_ACC32; + if (uc->config.enable_burst) + val |=3D PDMA_STATIC_TR_XY_BURST; + + udma_chanrt_write(uc, + UDMA_CHAN_RT_STATIC_TR_XY_REG, + val); + + /* save the current staticTR configuration */ + memcpy(&uc->static_tr, &uc->desc->static_tr, + sizeof(uc->static_tr)); + } + + udma_chanrt_write(uc, UDMA_CHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_EN | UDMA_CHAN_RT_CTL_AUTOPAIR); + + /* Poll for autopair completion */ + ret =3D read_poll_timeout_atomic(udma_v2_check_chan_autopair_completion, + status, status !=3D 0, 100, 500, false, uc); + + if (status <=3D 0) + return -ETIMEDOUT; + + break; + case DMA_MEM_TO_MEM: + udma_bchanrt_write(uc, UDMA_CHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_EN); + udma_bchanrt_write(uc, UDMA_CHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_EN); + + break; + default: + return -EINVAL; + } + + uc->state =3D UDMA_CHAN_IS_ACTIVE; +out: + + return 0; +} + +static int udma_v2_stop(struct udma_chan *uc) +{ + uc->state =3D UDMA_CHAN_IS_TERMINATING; + reinit_completion(&uc->teardown_completed); + + if (uc->config.dir =3D=3D DMA_DEV_TO_MEM) { + if (!uc->cyclic && !uc->desc) + udma_push_to_ring(uc, -1); + } + + udma_chanrt_write(uc, UDMA_CHAN_RT_PEER_REG(8), UDMA_CHAN_RT_PEER_REG8_FL= USH); + udma_chanrt_update_bits(uc, UDMA_CHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_EN | UDMA_CHAN_RT_CTL_TDOWN, + UDMA_CHAN_RT_CTL_EN | UDMA_CHAN_RT_CTL_TDOWN); + + return 0; +} + +static bool udma_v2_is_desc_really_done(struct udma_chan *uc, struct udma_= desc *d) +{ + u32 peer_bcnt, bcnt; + + /* + * Only TX towards PDMA is affected. + * If DMA_PREP_INTERRUPT is not set by consumer then skip the transfer + * completion calculation, consumer must ensure that there is no stale + * data in DMA fabric in this case. + */ + if (uc->config.ep_type =3D=3D PSIL_EP_NATIVE || + uc->config.dir !=3D DMA_MEM_TO_DEV || !(uc->config.tx_flags & DMA_PRE= P_INTERRUPT)) + return true; + + peer_bcnt =3D udma_chanrt_read(uc, UDMA_CHAN_RT_PERIPH_BCNT_REG); + bcnt =3D udma_chanrt_read(uc, UDMA_CHAN_RT_BCNT_REG); + + /* Transfer is incomplete, store current residue and time stamp */ + if (peer_bcnt < bcnt) { + uc->tx_drain.residue =3D bcnt - peer_bcnt; + uc->tx_drain.tstamp =3D ktime_get(); + return false; + } + + return true; +} + +static irqreturn_t udma_v2_udma_irq_handler(int irq, void *data) +{ + struct udma_chan *uc =3D data; + struct udma_dev *ud =3D uc->ud; + struct udma_desc *d; + + switch (uc->config.dir) { + case DMA_DEV_TO_MEM: + k3_ringacc_ring_clear_irq(uc->rflow->r_ring); + break; + case DMA_MEM_TO_DEV: + case DMA_MEM_TO_MEM: + k3_ringacc_ring_clear_irq(uc->tchan->tc_ring); + break; + default: + return -ENOENT; + } + + spin_lock(&uc->vc.lock); + d =3D uc->desc; + if (d) { + d->tr_idx =3D (d->tr_idx + 1) % d->sglen; + + if (uc->cyclic) { + vchan_cyclic_callback(&d->vd); + } else { + /* TODO: figure out the real amount of data */ + ud->udma_decrement_byte_counters(uc, d->residue); + ud->udma_start(uc); + vchan_cookie_complete(&d->vd); + } + } + + spin_unlock(&uc->vc.lock); + + return IRQ_HANDLED; +} + +static irqreturn_t udma_v2_ring_irq_handler(int irq, void *data) +{ + struct udma_chan *uc =3D data; + struct udma_dev *ud =3D uc->ud; + struct udma_desc *d; + dma_addr_t paddr =3D 0; + u32 intr_status, reg; + + switch (uc->config.dir) { + case DMA_DEV_TO_MEM: + intr_status =3D k3_ringacc_ring_get_irq_status(uc->rflow->r_ring); + break; + case DMA_MEM_TO_DEV: + case DMA_MEM_TO_MEM: + intr_status =3D k3_ringacc_ring_get_irq_status(uc->tchan->tc_ring); + break; + default: + return -ENOENT; + } + + reg =3D udma_chanrt_read(uc, UDMA_CHAN_RT_CTL_REG); + + if (intr_status & K3_RINGACC_RT_INT_STATUS_TR) { + /* check teardown status */ + if ((reg & UDMA_CHAN_RT_CTL_TDOWN) && !(reg & UDMA_CHAN_RT_CTL_EN)) + complete_all(&uc->teardown_completed); + return udma_v2_udma_irq_handler(irq, data); + } + + if (udma_pop_from_ring(uc, &paddr) || !paddr) + return IRQ_HANDLED; + + spin_lock(&uc->vc.lock); + + /* Teardown completion message */ + if (cppi5_desc_is_tdcm(paddr)) { + complete_all(&uc->teardown_completed); + + if (uc->terminated_desc) { + udma_desc_free(&uc->terminated_desc->vd); + uc->terminated_desc =3D NULL; + } + + if (!uc->desc) + ud->udma_start(uc); + + goto out; + } + + d =3D udma_udma_desc_from_paddr(uc, paddr); + + if (d) { + dma_addr_t desc_paddr =3D udma_curr_cppi5_desc_paddr(d, + d->desc_idx); + if (desc_paddr !=3D paddr) { + dev_err(uc->ud->dev, "not matching descriptors!\n"); + goto out; + } + + if (d =3D=3D uc->desc) { + /* active descriptor */ + if (uc->cyclic) { + udma_cyclic_packet_elapsed(uc); + vchan_cyclic_callback(&d->vd); + } else { + if (ud->udma_is_desc_really_done(uc, d)) { + ud->udma_decrement_byte_counters(uc, d->residue); + ud->udma_start(uc); + vchan_cookie_complete(&d->vd); + } else { + schedule_delayed_work(&uc->tx_drain.work, + 0); + } + } + } else { + /* + * terminated descriptor, mark the descriptor as + * completed to update the channel's cookie marker + */ + dma_cookie_complete(&d->vd.tx); + } + } +out: + spin_unlock(&uc->vc.lock); + + return IRQ_HANDLED; +} + +static int bcdma_v2_get_bchan(struct udma_chan *uc) +{ + struct udma_dev *ud =3D uc->ud; + enum udma_tp_level tpl; + int ret; + + if (uc->bchan) { + dev_dbg(ud->dev, "chan%d: already have bchan%d allocated\n", + uc->id, uc->bchan->id); + return 0; + } + + /* + * Use normal channels for peripherals, and highest TPL channel for + * mem2mem + */ + if (uc->config.tr_trigger_type) + tpl =3D 0; + else + tpl =3D ud->bchan_tpl.levels - 1; + + uc->bchan =3D __udma_reserve_bchan(ud, tpl, uc->id); + if (IS_ERR(uc->bchan)) { + ret =3D PTR_ERR(uc->bchan); + uc->bchan =3D NULL; + return ret; + } + uc->chan =3D uc->bchan; + uc->tchan =3D uc->bchan; + + return 0; +} + +static int bcdma_v2_alloc_bchan_resources(struct udma_chan *uc) +{ + struct k3_ring_cfg ring_cfg; + struct udma_dev *ud =3D uc->ud; + int ret; + + ret =3D bcdma_v2_get_bchan(uc); + if (ret) + return ret; + + ret =3D k3_ringacc_request_rings_pair(ud->ringacc, ud->match_data->chan_c= nt + uc->id, -1, + &uc->bchan->t_ring, + &uc->bchan->tc_ring); + if (ret) { + ret =3D -EBUSY; + goto err_ring; + } + + memset(&ring_cfg, 0, sizeof(ring_cfg)); + ring_cfg.size =3D K3_UDMA_DEFAULT_RING_SIZE; + ring_cfg.elm_size =3D K3_RINGACC_RING_ELSIZE_8; + ring_cfg.mode =3D K3_RINGACC_RING_MODE_RING; + + k3_configure_chan_coherency(&uc->vc.chan, ud->asel); + ring_cfg.asel =3D ud->asel; + ring_cfg.dma_dev =3D dmaengine_get_dma_device(&uc->vc.chan); + + ret =3D k3_ringacc_ring_cfg(uc->bchan->t_ring, &ring_cfg); + if (ret) + goto err_ringcfg; + + return 0; + +err_ringcfg: + k3_ringacc_ring_free(uc->bchan->tc_ring); + uc->bchan->tc_ring =3D NULL; + k3_ringacc_ring_free(uc->bchan->t_ring); + uc->bchan->t_ring =3D NULL; + k3_configure_chan_coherency(&uc->vc.chan, 0); +err_ring: + bcdma_put_bchan(uc); + + return ret; +} + +static int udma_v2_alloc_tx_resources(struct udma_chan *uc) +{ + struct k3_ring_cfg ring_cfg; + struct udma_dev *ud =3D uc->ud; + struct udma_tchan *tchan; + int ring_idx, ret; + + ret =3D udma_get_tchan(uc); + if (ret) + return ret; + + tchan =3D uc->tchan; + if (tchan->tflow_id >=3D 0) + ring_idx =3D tchan->tflow_id; + else + ring_idx =3D tchan->id; + + ret =3D k3_ringacc_request_rings_pair(ud->ringacc, ring_idx, -1, + &tchan->t_ring, + &tchan->tc_ring); + if (ret) { + ret =3D -EBUSY; + goto err_ring; + } + + memset(&ring_cfg, 0, sizeof(ring_cfg)); + ring_cfg.size =3D K3_UDMA_DEFAULT_RING_SIZE; + ring_cfg.elm_size =3D K3_RINGACC_RING_ELSIZE_8; + ring_cfg.mode =3D K3_RINGACC_RING_MODE_RING; + + k3_configure_chan_coherency(&uc->vc.chan, uc->config.asel); + ring_cfg.asel =3D uc->config.asel; + ring_cfg.dma_dev =3D dmaengine_get_dma_device(&uc->vc.chan); + + ret =3D k3_ringacc_ring_cfg(tchan->t_ring, &ring_cfg); + ret |=3D k3_ringacc_ring_cfg(tchan->tc_ring, &ring_cfg); + + if (ret) + goto err_ringcfg; + + return 0; + +err_ringcfg: + k3_ringacc_ring_free(uc->tchan->tc_ring); + uc->tchan->tc_ring =3D NULL; + k3_ringacc_ring_free(uc->tchan->t_ring); + uc->tchan->t_ring =3D NULL; +err_ring: + udma_put_tchan(uc); + + return ret; +} + +static int udma_v2_alloc_rx_resources(struct udma_chan *uc) +{ + struct udma_dev *ud =3D uc->ud; + struct k3_ring_cfg ring_cfg; + struct udma_rflow *rflow; + int fd_ring_id; + int ret; + + ret =3D udma_get_rchan(uc); + if (ret) + return ret; + + /* For MEM_TO_MEM we don't need rflow or rings */ + if (uc->config.dir =3D=3D DMA_MEM_TO_MEM) + return 0; + + if (uc->config.default_flow_id >=3D 0) + ret =3D udma_get_rflow(uc, uc->config.default_flow_id); + else + ret =3D udma_get_rflow(uc, uc->rchan->id); + + if (ret) { + ret =3D -EBUSY; + goto err_rflow; + } + + rflow =3D uc->rflow; + if (ud->tflow_cnt) + fd_ring_id =3D ud->tflow_cnt + rflow->id; + else + fd_ring_id =3D uc->rchan->id; + ret =3D k3_ringacc_request_rings_pair(ud->ringacc, fd_ring_id, -1, + &rflow->fd_ring, &rflow->r_ring); + if (ret) { + ret =3D -EBUSY; + goto err_ring; + } + + memset(&ring_cfg, 0, sizeof(ring_cfg)); + + ring_cfg.elm_size =3D K3_RINGACC_RING_ELSIZE_8; + ring_cfg.size =3D K3_UDMA_DEFAULT_RING_SIZE; + ring_cfg.mode =3D K3_RINGACC_RING_MODE_RING; + + k3_configure_chan_coherency(&uc->vc.chan, uc->config.asel); + ring_cfg.asel =3D uc->config.asel; + ring_cfg.dma_dev =3D dmaengine_get_dma_device(&uc->vc.chan); + + ret =3D k3_ringacc_ring_cfg(rflow->fd_ring, &ring_cfg); + + ring_cfg.size =3D K3_UDMA_DEFAULT_RING_SIZE; + ret |=3D k3_ringacc_ring_cfg(rflow->r_ring, &ring_cfg); + + if (ret) + goto err_ringcfg; + + return 0; + +err_ringcfg: + k3_ringacc_ring_free(rflow->r_ring); + rflow->r_ring =3D NULL; + k3_ringacc_ring_free(rflow->fd_ring); + rflow->fd_ring =3D NULL; +err_ring: + udma_put_rflow(uc); +err_rflow: + udma_put_rchan(uc); + + return ret; +} + +static int bcdma_v2_alloc_chan_resources(struct dma_chan *chan) +{ + struct udma_chan *uc =3D to_udma_chan(chan); + struct udma_dev *ud =3D to_udma_dev(chan->device); + u32 irq_ring_idx; + __be32 addr[2] =3D {0, 0}; + struct of_phandle_args out_irq; + int ret; + + /* Only TR mode is supported */ + uc->config.pkt_mode =3D false; + + /* + * Make sure that the completion is in a known state: + * No teardown, the channel is idle + */ + reinit_completion(&uc->teardown_completed); + complete_all(&uc->teardown_completed); + uc->state =3D UDMA_CHAN_IS_IDLE; + + switch (uc->config.dir) { + case DMA_MEM_TO_MEM: + /* Non synchronized - mem to mem type of transfer */ + dev_dbg(uc->ud->dev, "%s: chan%d as MEM-to-MEM\n", __func__, + uc->id); + + ret =3D bcdma_v2_alloc_bchan_resources(uc); + if (ret) + return ret; + + irq_ring_idx =3D ud->match_data->chan_cnt + uc->id; + break; + case DMA_MEM_TO_DEV: + /* Slave transfer synchronized - mem to dev (TX) trasnfer */ + dev_dbg(uc->ud->dev, "%s: chan%d as MEM-to-DEV\n", __func__, + uc->id); + + ret =3D udma_v2_alloc_tx_resources(uc); + if (ret) { + uc->config.remote_thread_id =3D -1; + return ret; + } + + uc->config.src_thread =3D ud->psil_base + uc->tchan->id; + uc->config.dst_thread =3D uc->config.remote_thread_id; + uc->config.dst_thread |=3D K3_PSIL_DST_THREAD_ID_OFFSET; + + irq_ring_idx =3D uc->tchan->id; + + break; + case DMA_DEV_TO_MEM: + /* Slave transfer synchronized - dev to mem (RX) trasnfer */ + dev_dbg(uc->ud->dev, "%s: chan%d as DEV-to-MEM\n", __func__, + uc->id); + + ret =3D udma_v2_alloc_rx_resources(uc); + if (ret) { + uc->config.remote_thread_id =3D -1; + return ret; + } + + uc->config.src_thread =3D uc->config.remote_thread_id; + uc->config.dst_thread =3D (ud->psil_base + uc->rchan->id) | + K3_PSIL_DST_THREAD_ID_OFFSET; + + irq_ring_idx =3D uc->rchan->id; + + break; + default: + /* Can not happen */ + dev_err(uc->ud->dev, "%s: chan%d invalid direction (%u)\n", + __func__, uc->id, uc->config.dir); + return -EINVAL; + } + + /* check if the channel configuration was successful */ + if (ret) + goto err_res_free; + + if (udma_is_chan_running(uc)) { + dev_warn(ud->dev, "chan%d: is running!\n", uc->id); + ud->udma_reset_chan(uc, false); + if (udma_is_chan_running(uc)) { + dev_err(ud->dev, "chan%d: won't stop!\n", uc->id); + ret =3D -EBUSY; + goto err_res_free; + } + } + + uc->dma_dev =3D dmaengine_get_dma_device(chan); + if (uc->config.dir =3D=3D DMA_MEM_TO_MEM && !uc->config.tr_trigger_type)= { + uc->config.hdesc_size =3D cppi5_trdesc_calc_size( + sizeof(struct cppi5_tr_type15_t), 2); + + uc->hdesc_pool =3D dma_pool_create(uc->name, ud->ddev.dev, + uc->config.hdesc_size, + ud->desc_align, + 0); + if (!uc->hdesc_pool) { + dev_err(ud->ddev.dev, + "Descriptor pool allocation failed\n"); + uc->use_dma_pool =3D false; + ret =3D -ENOMEM; + goto err_res_free; + } + + uc->use_dma_pool =3D true; + } else if (uc->config.dir !=3D DMA_MEM_TO_MEM) { + uc->psil_paired =3D true; + } + + out_irq.np =3D dev_of_node(ud->dev); + out_irq.args_count =3D 1; + out_irq.args[0] =3D irq_ring_idx; + ret =3D of_irq_parse_raw(addr, &out_irq); + if (ret) + return ret; + + uc->irq_num_ring =3D irq_create_of_mapping(&out_irq); + + ret =3D devm_request_irq(ud->dev, uc->irq_num_ring, udma_v2_ring_irq_hand= ler, + IRQF_TRIGGER_HIGH, uc->name, uc); + if (ret) { + dev_err(ud->dev, "chan%d: ring irq request failed\n", uc->id); + goto err_irq_free; + } + + udma_reset_rings(uc); + + INIT_DELAYED_WORK_ONSTACK(&uc->tx_drain.work, + udma_check_tx_completion); + return 0; + +err_irq_free: + uc->irq_num_ring =3D 0; + uc->irq_num_udma =3D 0; +err_res_free: + bcdma_free_bchan_resources(uc); + udma_free_tx_resources(uc); + udma_free_rx_resources(uc); + + udma_reset_uchan(uc); + + if (uc->use_dma_pool) { + dma_pool_destroy(uc->hdesc_pool); + uc->use_dma_pool =3D false; + } + + return ret; +} + +static enum dma_status udma_v2_tx_status(struct dma_chan *chan, + dma_cookie_t cookie, + struct dma_tx_state *txstate) +{ + struct udma_chan *uc =3D to_udma_chan(chan); + enum dma_status ret; + unsigned long flags; + + spin_lock_irqsave(&uc->vc.lock, flags); + + ret =3D dma_cookie_status(chan, cookie, txstate); + + if (!udma_is_chan_running(uc)) + ret =3D DMA_COMPLETE; + + if (ret =3D=3D DMA_IN_PROGRESS && udma_v2_is_chan_paused(uc)) + ret =3D DMA_PAUSED; + + if (ret =3D=3D DMA_COMPLETE || !txstate) + goto out; + + if (uc->desc && uc->desc->vd.tx.cookie =3D=3D cookie) { + u32 peer_bcnt =3D 0; + u32 bcnt =3D 0; + u32 residue =3D uc->desc->residue; + u32 delay =3D 0; + + if (uc->desc->dir =3D=3D DMA_MEM_TO_DEV) { + bcnt =3D udma_chanrt_read(uc, UDMA_CHAN_RT_SBCNT_REG); + + if (uc->config.ep_type !=3D PSIL_EP_NATIVE) { + peer_bcnt =3D udma_chanrt_read(uc, 0x810); + + if (bcnt > peer_bcnt) + delay =3D bcnt - peer_bcnt; + } + } else if (uc->desc->dir =3D=3D DMA_DEV_TO_MEM) { + bcnt =3D udma_chanrt_read(uc, UDMA_CHAN_RT_BCNT_REG); + + if (uc->config.ep_type !=3D PSIL_EP_NATIVE) { + peer_bcnt =3D udma_chanrt_read(uc, 0x810); + + if (peer_bcnt > bcnt) + delay =3D peer_bcnt - bcnt; + } + } else { + bcnt =3D udma_chanrt_read(uc, UDMA_CHAN_RT_BCNT_REG); + } + + if (bcnt && !(bcnt % uc->desc->residue)) + residue =3D 0; + else + residue -=3D bcnt % uc->desc->residue; + + if (!residue && (uc->config.dir =3D=3D DMA_DEV_TO_MEM || !delay)) { + ret =3D DMA_COMPLETE; + delay =3D 0; + } + + dma_set_residue(txstate, residue); + dma_set_in_flight_bytes(txstate, delay); + + } else { + ret =3D DMA_COMPLETE; + } + +out: + spin_unlock_irqrestore(&uc->vc.lock, flags); + return ret; +} + +static int udma_v2_pause(struct dma_chan *chan) +{ + struct udma_chan *uc =3D to_udma_chan(chan); + + /* pause the channel */ + udma_chanrt_update_bits(uc, UDMA_CHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_PAUSE, UDMA_CHAN_RT_CTL_PAUSE); + + return 0; +} + +static int udma_v2_resume(struct dma_chan *chan) +{ + struct udma_chan *uc =3D to_udma_chan(chan); + + /* resume the channel */ + udma_chanrt_update_bits(uc, UDMA_CHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_PAUSE, 0); + + return 0; +} + +static struct platform_driver bcdma_v2_driver; + +static bool udma_v2_dma_filter_fn(struct dma_chan *chan, void *param) +{ + struct udma_chan_config *ucc; + struct psil_endpoint_config *ep_config; + struct udma_v2_filter_param *filter_param; + struct udma_chan *uc; + struct udma_dev *ud; + + if (chan->device->dev->driver !=3D &bcdma_v2_driver.driver) + return false; + + uc =3D to_udma_chan(chan); + ucc =3D &uc->config; + ud =3D uc->ud; + filter_param =3D param; + + if (filter_param->asel > 15) { + dev_err(ud->dev, "Invalid channel asel: %u\n", + filter_param->asel); + return false; + } + + ucc->remote_thread_id =3D filter_param->remote_thread_id; + ucc->asel =3D filter_param->asel; + ucc->tr_trigger_type =3D filter_param->tr_trigger_type; + + if (ucc->tr_trigger_type) { + ucc->dir =3D DMA_MEM_TO_MEM; + goto triggered_bchan; + } else if (ucc->remote_thread_id & K3_PSIL_DST_THREAD_ID_OFFSET) { + ucc->dir =3D DMA_MEM_TO_DEV; + } else { + ucc->dir =3D DMA_DEV_TO_MEM; + } + + ep_config =3D psil_get_ep_config(ucc->remote_thread_id); + if (IS_ERR(ep_config)) { + dev_err(ud->dev, "No configuration for psi-l thread 0x%04x\n", + ucc->remote_thread_id); + ucc->dir =3D DMA_MEM_TO_MEM; + ucc->remote_thread_id =3D -1; + ucc->atype =3D 0; + ucc->asel =3D 0; + return false; + } + + ucc->pkt_mode =3D ep_config->pkt_mode; + ucc->channel_tpl =3D ep_config->channel_tpl; + ucc->notdpkt =3D ep_config->notdpkt; + ucc->ep_type =3D ep_config->ep_type; + + if ((ud->match_data->type =3D=3D DMA_TYPE_BCDMA_V2) && + ep_config->mapped_channel_id >=3D 0) { + ucc->mapped_channel_id =3D ep_config->mapped_channel_id; + ucc->default_flow_id =3D ep_config->default_flow_id; + } else { + ucc->mapped_channel_id =3D -1; + ucc->default_flow_id =3D -1; + } + + ucc->needs_epib =3D ep_config->needs_epib; + ucc->psd_size =3D ep_config->psd_size; + ucc->metadata_size =3D + (ucc->needs_epib ? CPPI5_INFO0_HDESC_EPIB_SIZE : 0) + + ucc->psd_size; + + if (ucc->ep_type !=3D PSIL_EP_NATIVE) { + const struct udma_match_data *match_data =3D ud->match_data; + + if ((match_data->flags & UDMA_FLAG_PDMA_ACC32) && (ep_config->pdma_acc32= )) + ucc->enable_acc32 =3D true; + else + ucc->enable_acc32 =3D false; + + if ((match_data->flags & UDMA_FLAG_PDMA_BURST) && (ep_config->pdma_burst= )) + ucc->enable_burst =3D true; + else + ucc->enable_burst =3D false; + } + if (ucc->pkt_mode) + ucc->hdesc_size =3D ALIGN(sizeof(struct cppi5_host_desc_t) + + ucc->metadata_size, ud->desc_align); + + dev_dbg(ud->dev, "chan%d: Remote thread: 0x%04x (%s)\n", uc->id, + ucc->remote_thread_id, dmaengine_get_direction_text(ucc->dir)); + + return true; + +triggered_bchan: + dev_dbg(ud->dev, "chan%d: triggered channel (type: %u)\n", uc->id, + ucc->tr_trigger_type); + + return true; +} + +static struct dma_chan *udma_v2_of_xlate(struct of_phandle_args *dma_spec, + struct of_dma *ofdma) +{ + struct udma_dev *ud =3D ofdma->of_dma_data; + dma_cap_mask_t mask =3D ud->ddev.cap_mask; + struct udma_v2_filter_param filter_param; + struct dma_chan *chan; + + if (ud->match_data->type =3D=3D DMA_TYPE_BCDMA_V2) { + if (dma_spec->args_count !=3D 4) + return NULL; + + filter_param.tr_trigger_type =3D dma_spec->args[0]; + filter_param.trigger_param =3D dma_spec->args[1]; + filter_param.remote_thread_id =3D dma_spec->args[2]; + filter_param.asel =3D dma_spec->args[3]; + } else { + if (dma_spec->args_count !=3D 1 && dma_spec->args_count !=3D 2) + return NULL; + + filter_param.remote_thread_id =3D dma_spec->args[0]; + filter_param.tr_trigger_type =3D 0; + if (dma_spec->args_count =3D=3D 2) + filter_param.asel =3D dma_spec->args[1]; + else + filter_param.asel =3D 0; + } + + chan =3D __dma_request_channel(&mask, udma_v2_dma_filter_fn, &filter_para= m, + ofdma->of_node); + if (!chan) { + dev_err(ud->dev, "get channel fail in %s.\n", __func__); + return ERR_PTR(-EINVAL); + } + + return chan; +} + +static struct udma_match_data bcdma_v2_data =3D { + .type =3D DMA_TYPE_BCDMA_V2, + .psil_base =3D 0x2000, /* for tchan and rchan, not applicable to bchan */ + .enable_memcpy_support =3D true, /* Supported via bchan */ + .flags =3D UDMA_FLAGS_J7_CLASS, + .statictr_z_mask =3D GENMASK(23, 0), + .burst_size =3D { + TI_SCI_RM_UDMAP_CHAN_BURST_SIZE_64_BYTES, /* Normal Channels */ + 0, /* No H Channels */ + 0, /* No UH Channels */ + }, + .bchan_cnt =3D 16, + .chan_cnt =3D 128, + .tchan_cnt =3D 128, + .rchan_cnt =3D 128, +}; + +static const struct of_device_id udma_of_match[] =3D { + { + .compatible =3D "ti,dmss-bcdma-v2", + .data =3D &bcdma_v2_data, + }, + { /* Sentinel */ }, +}; + +static const struct soc_device_attribute k3_soc_devices[] =3D { + { .family =3D "AM62LX", }, + { /* sentinel */ } +}; +MODULE_DEVICE_TABLE(of, udma_of_match); + +static int udma_v2_get_mmrs(struct platform_device *pdev, struct udma_dev = *ud) +{ + u32 cap2, cap3; + int i; + + ud->mmrs[V2_MMR_GCFG] =3D devm_platform_ioremap_resource_byname(pdev, + v2_mmr_names[V2_MMR_GCFG]); + if (IS_ERR(ud->mmrs[V2_MMR_GCFG])) + return PTR_ERR(ud->mmrs[V2_MMR_GCFG]); + + cap2 =3D udma_read(ud->mmrs[V2_MMR_GCFG], 0x28); + cap3 =3D udma_read(ud->mmrs[V2_MMR_GCFG], 0x2c); + + ud->bchan_cnt =3D ud->match_data->bchan_cnt; + /* There are no tchan and rchan in BCDMA_V2. + * Duplicate chan as tchan and rchan to keep the common code + * in k3-udma-common.c functional for BCDMA_V2. + */ + ud->chan_cnt =3D ud->match_data->chan_cnt; + ud->tchan_cnt =3D ud->match_data->chan_cnt; + ud->rchan_cnt =3D ud->match_data->chan_cnt; + ud->rflow_cnt =3D ud->chan_cnt; + + for (i =3D 1; i < V2_MMR_LAST; i++) { + if (i =3D=3D V2_MMR_BCHANRT && ud->bchan_cnt =3D=3D 0) + continue; + if (i =3D=3D V2_MMR_CHANRT && ud->chan_cnt =3D=3D 0) + continue; + + ud->mmrs[i] =3D devm_platform_ioremap_resource_byname(pdev, v2_mmr_names= [i]); + if (IS_ERR(ud->mmrs[i])) + return PTR_ERR(ud->mmrs[i]); + } + + return 0; +} + +static int udma_v2_probe(struct platform_device *pdev) +{ + const struct soc_device_attribute *soc; + struct device *dev =3D &pdev->dev; + struct udma_dev *ud; + const struct of_device_id *match; + int i, ret; + int ch_count; + + ret =3D dma_coerce_mask_and_coherent(dev, DMA_BIT_MASK(48)); + if (ret) + dev_err(dev, "failed to set dma mask stuff\n"); + + ud =3D devm_kzalloc(dev, sizeof(*ud), GFP_KERNEL); + if (!ud) + return -ENOMEM; + + match =3D of_match_node(udma_of_match, dev->of_node); + if (!match) { + dev_err(dev, "No compatible match found\n"); + return -ENODEV; + } + ud->match_data =3D match->data; + + ud->soc_data =3D ud->match_data->soc_data; + if (!ud->soc_data) { + soc =3D soc_device_match(k3_soc_devices); + if (!soc) { + dev_err(dev, "No compatible SoC found\n"); + return -ENODEV; + } + ud->soc_data =3D soc->data; + } + // Setup function pointers + ud->udma_start =3D udma_v2_start; + ud->udma_stop =3D udma_v2_stop; + ud->udma_reset_chan =3D udma_v2_reset_chan; + ud->udma_is_desc_really_done =3D udma_v2_is_desc_really_done; + ud->udma_decrement_byte_counters =3D udma_v2_decrement_byte_counters; + + ret =3D udma_v2_get_mmrs(pdev, ud); + if (ret) + return ret; + + struct k3_ringacc_init_data ring_init_data =3D {0}; + + ring_init_data.num_rings =3D ud->bchan_cnt + ud->chan_cnt; + + ud->ringacc =3D k3_ringacc_dmarings_init(pdev, &ring_init_data); + + if (IS_ERR(ud->ringacc)) + return PTR_ERR(ud->ringacc); + + dma_cap_set(DMA_SLAVE, ud->ddev.cap_mask); + + dma_cap_set(DMA_CYCLIC, ud->ddev.cap_mask); + ud->ddev.device_prep_dma_cyclic =3D udma_prep_dma_cyclic; + + ud->ddev.device_config =3D udma_slave_config; + ud->ddev.device_prep_slave_sg =3D udma_prep_slave_sg; + ud->ddev.device_issue_pending =3D udma_issue_pending; + ud->ddev.device_tx_status =3D udma_v2_tx_status; + ud->ddev.device_pause =3D udma_v2_pause; + ud->ddev.device_resume =3D udma_v2_resume; + ud->ddev.device_terminate_all =3D udma_terminate_all; + ud->ddev.device_synchronize =3D udma_synchronize; +#ifdef CONFIG_DEBUG_FS + ud->ddev.dbg_summary_show =3D udma_dbg_summary_show; +#endif + + ud->ddev.device_alloc_chan_resources =3D + bcdma_v2_alloc_chan_resources; + + ud->ddev.device_free_chan_resources =3D udma_free_chan_resources; + + ud->ddev.src_addr_widths =3D TI_UDMAC_BUSWIDTHS; + ud->ddev.dst_addr_widths =3D TI_UDMAC_BUSWIDTHS; + ud->ddev.directions =3D BIT(DMA_DEV_TO_MEM) | BIT(DMA_MEM_TO_DEV); + ud->ddev.residue_granularity =3D DMA_RESIDUE_GRANULARITY_BURST; + ud->ddev.desc_metadata_modes =3D DESC_METADATA_CLIENT | + DESC_METADATA_ENGINE; + if (ud->match_data->enable_memcpy_support && + !(ud->match_data->type =3D=3D DMA_TYPE_BCDMA && ud->bchan_cnt =3D=3D = 0)) { + dma_cap_set(DMA_MEMCPY, ud->ddev.cap_mask); + ud->ddev.device_prep_dma_memcpy =3D udma_prep_dma_memcpy; + ud->ddev.directions |=3D BIT(DMA_MEM_TO_MEM); + } + + ud->ddev.dev =3D dev; + ud->dev =3D dev; + ud->psil_base =3D ud->match_data->psil_base; + + INIT_LIST_HEAD(&ud->ddev.channels); + INIT_LIST_HEAD(&ud->desc_to_purge); + + ch_count =3D setup_resources(ud); + if (ch_count <=3D 0) + return ch_count; + + spin_lock_init(&ud->lock); + INIT_WORK(&ud->purge_work, udma_purge_desc_work); + + ud->desc_align =3D 64; + if (ud->desc_align < dma_get_cache_alignment()) + ud->desc_align =3D dma_get_cache_alignment(); + + ret =3D udma_setup_rx_flush(ud); + if (ret) + return ret; + + for (i =3D 0; i < ud->bchan_cnt; i++) { + struct udma_bchan *bchan =3D &ud->bchans[i]; + + bchan->id =3D i; + bchan->reg_rt =3D ud->mmrs[V2_MMR_BCHANRT] + i * 0x1000; + } + + for (i =3D 0; i < ud->tchan_cnt; i++) { + struct udma_tchan *tchan =3D &ud->tchans[i]; + + tchan->id =3D i; + tchan->reg_rt =3D ud->mmrs[V2_MMR_CHANRT] + i * 0x1000; + } + + for (i =3D 0; i < ud->rchan_cnt; i++) { + struct udma_rchan *rchan =3D &ud->rchans[i]; + + rchan->id =3D i; + rchan->reg_rt =3D ud->mmrs[V2_MMR_CHANRT] + i * 0x1000; + } + + for (i =3D 0; i < ud->rflow_cnt; i++) { + struct udma_rflow *rflow =3D &ud->rflows[i]; + + rflow->id =3D i; + rflow->reg_rt =3D ud->rflow_rt + i * 0x2000; + } + + for (i =3D 0; i < ch_count; i++) { + struct udma_chan *uc =3D &ud->channels[i]; + + uc->ud =3D ud; + uc->vc.desc_free =3D udma_desc_free; + uc->id =3D i; + uc->bchan =3D NULL; + uc->tchan =3D NULL; + uc->rchan =3D NULL; + uc->config.remote_thread_id =3D -1; + uc->config.mapped_channel_id =3D -1; + uc->config.default_flow_id =3D -1; + uc->config.dir =3D DMA_MEM_TO_MEM; + uc->name =3D devm_kasprintf(dev, GFP_KERNEL, "%s chan%d", + dev_name(dev), i); + + vchan_init(&uc->vc, &ud->ddev); + /* Use custom vchan completion handling */ + tasklet_setup(&uc->vc.task, udma_vchan_complete); + init_completion(&uc->teardown_completed); + INIT_DELAYED_WORK(&uc->tx_drain.work, udma_check_tx_completion); + } + + /* Configure the copy_align to the maximum burst size the device supports= */ + ud->ddev.copy_align =3D udma_get_copy_align(ud); + + ret =3D dma_async_device_register(&ud->ddev); + if (ret) { + dev_err(dev, "failed to register slave DMA engine: %d\n", ret); + return ret; + } + + platform_set_drvdata(pdev, ud); + + ret =3D of_dma_controller_register(dev->of_node, udma_v2_of_xlate, ud); + if (ret) { + dev_err(dev, "failed to register of_dma controller\n"); + dma_async_device_unregister(&ud->ddev); + } + + return ret; +} + +static int __maybe_unused udma_v2_pm_suspend(struct device *dev) +{ + struct udma_dev *ud =3D dev_get_drvdata(dev); + struct dma_device *dma_dev =3D &ud->ddev; + struct dma_chan *chan; + struct udma_chan *uc; + + list_for_each_entry(chan, &dma_dev->channels, device_node) { + if (chan->client_count) { + uc =3D to_udma_chan(chan); + /* backup the channel configuration */ + memcpy(&uc->backup_config, &uc->config, + sizeof(struct udma_chan_config)); + dev_dbg(dev, "Suspending channel %s\n", + dma_chan_name(chan)); + ud->ddev.device_free_chan_resources(chan); + } + } + + return 0; +} + +static int __maybe_unused udma_v2_pm_resume(struct device *dev) +{ + struct udma_dev *ud =3D dev_get_drvdata(dev); + struct dma_device *dma_dev =3D &ud->ddev; + struct dma_chan *chan; + struct udma_chan *uc; + int ret; + + list_for_each_entry(chan, &dma_dev->channels, device_node) { + if (chan->client_count) { + uc =3D to_udma_chan(chan); + /* restore the channel configuration */ + memcpy(&uc->config, &uc->backup_config, + sizeof(struct udma_chan_config)); + dev_dbg(dev, "Resuming channel %s\n", + dma_chan_name(chan)); + ret =3D ud->ddev.device_alloc_chan_resources(chan); + if (ret) + return ret; + } + } + + return 0; +} + +static const struct dev_pm_ops udma_pm_ops =3D { + SET_LATE_SYSTEM_SLEEP_PM_OPS(udma_v2_pm_suspend, udma_v2_pm_resume) +}; + +static struct platform_driver bcdma_v2_driver =3D { + .driver =3D { + .name =3D "ti-udma-v2", + .of_match_table =3D udma_of_match, + .suppress_bind_attrs =3D true, + .pm =3D &udma_pm_ops, + }, + .probe =3D udma_v2_probe, +}; + +module_platform_driver(bcdma_v2_driver); +MODULE_DESCRIPTION("Texas Instruments K3 UDMA v2 support"); +MODULE_LICENSE("GPL"); + diff --git a/drivers/dma/ti/k3-udma.h b/drivers/dma/ti/k3-udma.h index 4de6f38089ce7..c054113640bf4 100644 --- a/drivers/dma/ti/k3-udma.h +++ b/drivers/dma/ti/k3-udma.h @@ -49,6 +49,8 @@ #define UDMA_CHAN_RT_SWTRIG_REG 0x8 #define UDMA_CHAN_RT_STDATA_REG 0x80 =20 +#define UDMA_CHAN_RT_PDMA_STATE_REG 0x80c + #define UDMA_CHAN_RT_PEER_REG(i) (0x200 + ((i) * 0x4)) #define UDMA_CHAN_RT_PEER_STATIC_TR_XY_REG \ UDMA_CHAN_RT_PEER_REG(0) /* PSI-L: 0x400 */ @@ -88,8 +90,16 @@ #define UDMA_CHAN_RT_CTL_TDOWN BIT(30) #define UDMA_CHAN_RT_CTL_PAUSE BIT(29) #define UDMA_CHAN_RT_CTL_FTDOWN BIT(28) +#define UDMA_CHAN_RT_CTL_AUTOPAIR BIT(23) +#define UDMA_CHAN_RT_CTL_PAIR_TIMEOUT BIT(17) +#define UDMA_CHAN_RT_CTL_PAIR_COMPLETE BIT(16) #define UDMA_CHAN_RT_CTL_ERROR BIT(0) =20 +/* UDMA_CHAN_RT_PDMA_STATE_REG */ +#define UDMA_CHAN_RT_PDMA_STATE_IN_EVT BIT(31) +#define UDMA_CHAN_RT_PDMA_STATE_TDOWN BIT(30) +#define UDMA_CHAN_RT_PDMA_STATE_PAUSE BIT(29) + /* UDMA_CHAN_RT_PEER_RT_EN_REG */ #define UDMA_PEER_RT_EN_ENABLE BIT(31) #define UDMA_PEER_RT_EN_TEARDOWN BIT(30) @@ -120,6 +130,9 @@ */ #define PDMA_STATIC_TR_Z(x, mask) ((x) & (mask)) =20 +/* UDMA_CHAN_RT_PEER_REG(8) */ +#define UDMA_CHAN_RT_PEER_REG8_FLUSH 0x09000000 + /* Address Space Select */ #define K3_ADDRESS_ASEL_SHIFT 48 =20 @@ -154,10 +167,15 @@ BIT(DMA_SLAVE_BUSWIDTH_4_BYTES) | \ BIT(DMA_SLAVE_BUSWIDTH_8_BYTES)) =20 +struct udma_chan; +struct udma_dev; + enum k3_dma_type { DMA_TYPE_UDMA =3D 0, DMA_TYPE_BCDMA, DMA_TYPE_PKTDMA, + DMA_TYPE_BCDMA_V2, + DMA_TYPE_PKTDMA_V2, }; =20 enum udma_mmr { @@ -168,6 +186,13 @@ enum udma_mmr { MMR_LAST, }; =20 +enum udma_v2_mmr { + V2_MMR_GCFG =3D 0, + V2_MMR_BCHANRT, + V2_MMR_CHANRT, + V2_MMR_LAST, +}; + enum udma_rm_range { RM_RANGE_BCHAN =3D 0, RM_RANGE_TCHAN, @@ -190,6 +215,13 @@ struct udma_filter_param { u32 tr_trigger_type; }; =20 +struct udma_v2_filter_param { + u32 tr_trigger_type; + u32 trigger_param; + int remote_thread_id; + u32 asel; +}; + struct udma_static_tr { u8 elsize; /* RPSTR0 */ u16 elcnt; /* RPSTR0 */ @@ -206,17 +238,13 @@ struct udma_tchan { }; =20 #define udma_bchan udma_tchan +#define udma_rchan udma_tchan =20 struct udma_rflow { int id; struct k3_ring *fd_ring; /* Free Descriptor ring */ struct k3_ring *r_ring; /* Receive ring */ -}; - -struct udma_rchan { void __iomem *reg_rt; - - int id; }; =20 struct udma_oes_offsets { @@ -256,6 +284,12 @@ struct udma_match_data { u32 statictr_z_mask; u8 burst_size[3]; struct udma_soc_data *soc_data; + u32 bchan_cnt; + u32 chan_cnt; + u32 tchan_cnt; + u32 rchan_cnt; + u32 tflow_cnt; + u32 rflow_cnt; }; =20 struct udma_soc_data { @@ -345,6 +379,7 @@ struct udma_dev { struct dma_device ddev; struct device *dev; void __iomem *mmrs[MMR_LAST]; + void __iomem *rflow_rt; const struct udma_match_data *match_data; const struct udma_soc_data *soc_data; =20 @@ -365,12 +400,14 @@ struct udma_dev { struct udma_rx_flush rx_flush; =20 int bchan_cnt; + int chan_cnt; int tchan_cnt; int echan_cnt; int rchan_cnt; int rflow_cnt; int tflow_cnt; unsigned long *bchan_map; + unsigned long *chan_map; unsigned long *tchan_map; unsigned long *rchan_map; unsigned long *rflow_gp_map; @@ -379,6 +416,7 @@ struct udma_dev { unsigned long *tflow_map; =20 struct udma_bchan *bchans; + struct udma_tchan *chans; struct udma_tchan *tchans; struct udma_rchan *rchans; struct udma_rflow *rflows; @@ -395,7 +433,6 @@ struct udma_dev { void (*udma_decrement_byte_counters)(struct udma_chan *uc, u32 val); }; =20 - struct udma_chan { struct virt_dma_chan vc; struct dma_slave_config cfg; @@ -407,6 +444,7 @@ struct udma_chan { char *name; =20 struct udma_bchan *bchan; + struct udma_tchan *chan; struct udma_tchan *tchan; struct udma_rchan *rchan; struct udma_rflow *rflow; @@ -476,51 +514,33 @@ static inline void udma_update_bits(void __iomem *bas= e, int reg, writel(tmp, base + reg); } =20 -/* TCHANRT */ -static inline u32 udma_tchanrt_read(struct udma_chan *uc, int reg) -{ - if (!uc->tchan) - return 0; - return udma_read(uc->tchan->reg_rt, reg); -} - -static inline void udma_tchanrt_write(struct udma_chan *uc, int reg, u32 v= al) -{ - if (!uc->tchan) - return; - udma_write(uc->tchan->reg_rt, reg, val); +#define _UDMA_REG_ACCESS(channel) \ +static inline u32 udma_##channel##rt_read(struct udma_chan *uc, int reg) \ +{ \ + if (!uc->channel) \ + return 0; \ + return udma_read(uc->channel->reg_rt, reg); \ +} \ +\ +static inline void udma_##channel##rt_write(struct udma_chan *uc, int reg,= u32 val) \ +{ \ + if (!uc->channel) \ + return; \ + udma_write(uc->channel->reg_rt, reg, val); \ +} \ +\ +static inline void udma_##channel##rt_update_bits(struct udma_chan *uc, in= t reg, \ + u32 mask, u32 val) \ +{ \ + if (!uc->channel) \ + return; \ + udma_update_bits(uc->channel->reg_rt, reg, mask, val); \ } =20 -static inline void udma_tchanrt_update_bits(struct udma_chan *uc, int reg, - u32 mask, u32 val) -{ - if (!uc->tchan) - return; - udma_update_bits(uc->tchan->reg_rt, reg, mask, val); -} - -/* RCHANRT */ -static inline u32 udma_rchanrt_read(struct udma_chan *uc, int reg) -{ - if (!uc->rchan) - return 0; - return udma_read(uc->rchan->reg_rt, reg); -} - -static inline void udma_rchanrt_write(struct udma_chan *uc, int reg, u32 v= al) -{ - if (!uc->rchan) - return; - udma_write(uc->rchan->reg_rt, reg, val); -} - -static inline void udma_rchanrt_update_bits(struct udma_chan *uc, int reg, - u32 mask, u32 val) -{ - if (!uc->rchan) - return; - udma_update_bits(uc->rchan->reg_rt, reg, mask, val); -} +_UDMA_REG_ACCESS(chan); +_UDMA_REG_ACCESS(bchan); +_UDMA_REG_ACCESS(tchan); +_UDMA_REG_ACCESS(rchan); =20 static inline dma_addr_t udma_curr_cppi5_desc_paddr(struct udma_desc *d, int idx) diff --git a/include/linux/soc/ti/k3-ringacc.h b/include/linux/soc/ti/k3-ri= ngacc.h index fcf6fbd4a8594..2711f4119b0ee 100644 --- a/include/linux/soc/ti/k3-ringacc.h +++ b/include/linux/soc/ti/k3-ringacc.h @@ -10,6 +10,9 @@ =20 #include =20 +#define K3_RINGACC_RT_INT_STATUS_COMPLETE BIT(0) +#define K3_RINGACC_RT_INT_STATUS_TR BIT(2) + struct device_node; =20 /** --=20 2.34.1 From nobody Sun Feb 8 20:13:33 2026 Received: from lelvem-ot01.ext.ti.com (lelvem-ot01.ext.ti.com [198.47.23.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 261F01D7E5C; Mon, 28 Apr 2025 07:21:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.47.23.234 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745824900; cv=none; b=QLwnknsorrRCSXVCaGyeYYHAs+H0O4u9gu29mVxxEZUmoxaUIZpBtzyj1g0KUtcyfi6XZPuUho8SNjIlI01et6Tt+GBkQJXbbNcpVb4S0b4H8a6iHkVQx8yDQbGXT3pflmvjLUeQAujrwZWVsUquIDfAP7xokhN3FKARWX6gRbQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745824900; c=relaxed/simple; bh=wg0myzF40CZqwLO7OXWp3SxjOKGy2Yyp2f95aUUXSYI=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=nbiaJmk4k3Ru51CmZRV7GNH3k14Ml0wCglVbrT1MxvmW7cthF+HDjabF4uUuSesCtzCx8oJln6foXSwUX0LeEINPaKKJlmw27TsPdVzZ2wbBYWMyVtcBU2INL7dtIpXnvQaGIUAOuqNXaP56jna5CHIXe3skVDF4qHzUjpx6GfY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ti.com; spf=pass smtp.mailfrom=ti.com; dkim=pass (1024-bit key) header.d=ti.com header.i=@ti.com header.b=lo4dTSEg; arc=none smtp.client-ip=198.47.23.234 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ti.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ti.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ti.com header.i=@ti.com header.b="lo4dTSEg" Received: from fllv0034.itg.ti.com ([10.64.40.246]) by lelvem-ot01.ext.ti.com (8.15.2/8.15.2) with ESMTPS id 53S7LWEa2714021 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 28 Apr 2025 02:21:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1745824892; bh=9ZdEjYKfU9dlzcOxSg5PF1U/m2NbiauMiKDgkc2Umqk=; h=From:To:Subject:Date:In-Reply-To:References; b=lo4dTSEgOBFODM7yoJXBkzDKrRcx/aOz1/rdC9+BYVmlEujr+fUJxzBuGc+R5iHXa rPAxZlz2Kva5NVl81deRtOZMWdScSk6d6yjNpALyzYMhqHFCAytgadJ3lX/GUbS20v 2uPBt/M5V8I+k5BqgBo4i5YaazlqiPN6vek02E/4= Received: from DFLE108.ent.ti.com (dfle108.ent.ti.com [10.64.6.29]) by fllv0034.itg.ti.com (8.15.2/8.15.2) with ESMTPS id 53S7LWIq073799 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 28 Apr 2025 02:21:32 -0500 Received: from DFLE104.ent.ti.com (10.64.6.25) by DFLE108.ent.ti.com (10.64.6.29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.23; Mon, 28 Apr 2025 02:21:31 -0500 Received: from lelvsmtp6.itg.ti.com (10.180.75.249) by DFLE104.ent.ti.com (10.64.6.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.23 via Frontend Transport; Mon, 28 Apr 2025 02:21:31 -0500 Received: from uda0498651.dhcp.ti.com (uda0498651.dhcp.ti.com [172.24.227.7]) by lelvsmtp6.itg.ti.com (8.15.2/8.15.2) with ESMTP id 53S7KdMg068873; Mon, 28 Apr 2025 02:21:27 -0500 From: Sai Sree Kartheek Adivi To: , , , , , , , , , , , , , , , Subject: [PATCH 7/8] dmaengine: ti: k3-udma-v2: Add support for PKTDMA V2 Date: Mon, 28 Apr 2025 12:50:31 +0530 Message-ID: <20250428072032.946008-8-s-adivi@ti.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250428072032.946008-1-s-adivi@ti.com> References: <20250428072032.946008-1-s-adivi@ti.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-C2ProcessedOrg: 333ef613-75bf-4e12-a4b1-8e3623f5dcea Content-Type: text/plain; charset="utf-8" The PKTDMA V2 is different than the existing PKTDMA supported by the k3-udma driver. The changes in PKTDMA V2 are: - Autopair: There is no longer a need for PSIL pair and AUTOPAIR bit needs to set in the RT_CTL register. - Static channel mapping: Each channel is mapped to a single peripheral. - Direct IRQs: There is no INT-A and interrupt lines from DMA are directly connected to GIC. - Remote side configuration handled by DMA. So no need to write to PEER registers to START / STOP / PAUSE / TEARDOWN. Signed-off-by: Sai Sree Kartheek Adivi --- drivers/dma/ti/k3-udma-common.c | 33 ++++- drivers/dma/ti/k3-udma-v2.c | 219 ++++++++++++++++++++++++++++++-- drivers/dma/ti/k3-udma.h | 3 + 3 files changed, 235 insertions(+), 20 deletions(-) diff --git a/drivers/dma/ti/k3-udma-common.c b/drivers/dma/ti/k3-udma-commo= n.c index 462ad17c74604..b3de76893149a 100644 --- a/drivers/dma/ti/k3-udma-common.c +++ b/drivers/dma/ti/k3-udma-common.c @@ -2103,6 +2103,7 @@ int setup_resources(struct udma_dev *ud) ret =3D bcdma_setup_resources(ud); break; case DMA_TYPE_PKTDMA: + case DMA_TYPE_PKTDMA_V2: ret =3D pktdma_setup_resources(ud); break; default: @@ -2112,7 +2113,7 @@ int setup_resources(struct udma_dev *ud) if (ret) return ret; =20 - if (ud->match_data->type =3D=3D DMA_TYPE_BCDMA_V2) { + if (ud->match_data->type >=3D DMA_TYPE_BCDMA_V2) { ch_count =3D ud->bchan_cnt + ud->tchan_cnt; if (ud->bchan_cnt) ch_count -=3D bitmap_weight(ud->bchan_map, ud->bchan_cnt); @@ -2157,7 +2158,7 @@ int setup_resources(struct udma_dev *ud) break; case DMA_TYPE_BCDMA_V2: dev_info(dev, - "Channels: %d (bchan: %u, chan: %u)\n", + "Channels: %d (bchan: %u, tchan + rchan: %u)\n", ch_count, ud->bchan_cnt - bitmap_weight(ud->bchan_map, ud->bchan_cnt), @@ -2173,6 +2174,13 @@ int setup_resources(struct udma_dev *ud) ud->rchan_cnt - bitmap_weight(ud->rchan_map, ud->rchan_cnt)); break; + case DMA_TYPE_PKTDMA_V2: + dev_info(dev, + "Channels: %d (tchan + rchan: %u)\n", + ch_count, + ud->chan_cnt - bitmap_weight(ud->chan_map, + ud->chan_cnt)); + break; default: break; } @@ -2625,12 +2633,21 @@ int pktdma_setup_resources(struct udma_dev *ud) =20 ud->tchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->tchan_cnt), sizeof(unsigned long), GFP_KERNEL); + bitmap_zero(ud->tchan_map, ud->tchan_cnt); ud->tchans =3D devm_kcalloc(dev, ud->tchan_cnt, sizeof(*ud->tchans), GFP_KERNEL); - ud->rchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->rchan_cnt), - sizeof(unsigned long), GFP_KERNEL); - ud->rchans =3D devm_kcalloc(dev, ud->rchan_cnt, sizeof(*ud->rchans), - GFP_KERNEL); + if (ud->match_data->type =3D=3D DMA_TYPE_PKTDMA_V2) { + ud->rchan_map =3D ud->tchan_map; + ud->rchans =3D ud->tchans; + ud->chan_map =3D ud->tchan_map; + ud->chans =3D ud->tchans; + } else { + ud->rchan_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->rchan_cnt), + sizeof(unsigned long), GFP_KERNEL); + bitmap_zero(ud->rchan_map, ud->rchan_cnt); + ud->rchans =3D devm_kcalloc(dev, ud->rchan_cnt, sizeof(*ud->rchans), + GFP_KERNEL); + } ud->rflow_in_use =3D devm_kcalloc(dev, BITS_TO_LONGS(ud->rflow_cnt), sizeof(unsigned long), GFP_KERNEL); @@ -2638,11 +2655,15 @@ int pktdma_setup_resources(struct udma_dev *ud) GFP_KERNEL); ud->tflow_map =3D devm_kmalloc_array(dev, BITS_TO_LONGS(ud->tflow_cnt), sizeof(unsigned long), GFP_KERNEL); + bitmap_zero(ud->tflow_map, ud->tflow_cnt); =20 if (!ud->tchan_map || !ud->rchan_map || !ud->tflow_map || !ud->tchans || !ud->rchans || !ud->rflows || !ud->rflow_in_use) return -ENOMEM; =20 + if (ud->match_data->type =3D=3D DMA_TYPE_PKTDMA_V2) + return 0; + /* Get resource ranges from tisci */ for (i =3D 0; i < RM_RANGE_LAST; i++) { if (i =3D=3D RM_RANGE_BCHAN) diff --git a/drivers/dma/ti/k3-udma-v2.c b/drivers/dma/ti/k3-udma-v2.c index 90b5ac5e00ead..1e7fc39a4600e 100644 --- a/drivers/dma/ti/k3-udma-v2.c +++ b/drivers/dma/ti/k3-udma-v2.c @@ -778,6 +778,147 @@ static int bcdma_v2_alloc_chan_resources(struct dma_c= han *chan) return ret; } =20 +static int pktdma_v2_alloc_chan_resources(struct dma_chan *chan) +{ + struct udma_chan *uc =3D to_udma_chan(chan); + struct udma_dev *ud =3D to_udma_dev(chan->device); + u32 irq_ring_idx; + __be32 addr[2] =3D {0, 0}; + struct of_phandle_args out_irq; + int ret; + + /* + * Make sure that the completion is in a known state: + * No teardown, the channel is idle + */ + reinit_completion(&uc->teardown_completed); + complete_all(&uc->teardown_completed); + uc->state =3D UDMA_CHAN_IS_IDLE; + + switch (uc->config.dir) { + case DMA_MEM_TO_DEV: + /* Slave transfer synchronized - mem to dev (TX) trasnfer */ + dev_dbg(uc->ud->dev, "%s: chan%d as MEM-to-DEV\n", __func__, + uc->id); + + ret =3D udma_v2_alloc_tx_resources(uc); + if (ret) { + uc->config.remote_thread_id =3D -1; + return ret; + } + + uc->config.src_thread =3D ud->psil_base + uc->tchan->id; + uc->config.dst_thread =3D uc->config.remote_thread_id; + uc->config.dst_thread |=3D K3_PSIL_DST_THREAD_ID_OFFSET; + + + irq_ring_idx =3D uc->config.mapped_channel_id; + break; + case DMA_DEV_TO_MEM: + /* Slave transfer synchronized - dev to mem (RX) trasnfer */ + dev_dbg(uc->ud->dev, "%s: chan%d as DEV-to-MEM\n", __func__, + uc->id); + + ret =3D udma_v2_alloc_rx_resources(uc); + if (ret) { + uc->config.remote_thread_id =3D -1; + return ret; + } + + uc->config.src_thread =3D uc->config.remote_thread_id; + uc->config.dst_thread =3D (ud->psil_base + uc->rchan->id) | + K3_PSIL_DST_THREAD_ID_OFFSET; + + irq_ring_idx =3D uc->config.mapped_channel_id; + udma_write(uc->rflow->reg_rt, UDMA_RX_FLOWRT_RFA, BIT(28)); + break; + default: + /* Can not happen */ + dev_err(uc->ud->dev, "%s: chan%d invalid direction (%u)\n", + __func__, uc->id, uc->config.dir); + return -EINVAL; + } + + /* check if the channel configuration was successful */ + if (ret) + goto err_res_free; + + if (udma_is_chan_running(uc)) { + dev_warn(ud->dev, "chan%d: is running!\n", uc->id); + ud->udma_reset_chan(uc, false); + if (udma_is_chan_running(uc)) { + dev_err(ud->dev, "chan%d: won't stop!\n", uc->id); + ret =3D -EBUSY; + goto err_res_free; + } + } + + uc->dma_dev =3D dmaengine_get_dma_device(chan); + uc->hdesc_pool =3D dma_pool_create(uc->name, uc->dma_dev, + uc->config.hdesc_size, ud->desc_align, + 0); + if (!uc->hdesc_pool) { + dev_err(ud->ddev.dev, + "Descriptor pool allocation failed\n"); + uc->use_dma_pool =3D false; + ret =3D -ENOMEM; + goto err_res_free; + } + + uc->use_dma_pool =3D true; + + uc->psil_paired =3D true; + + out_irq.np =3D dev_of_node(ud->dev); + out_irq.args_count =3D 1; + out_irq.args[0] =3D irq_ring_idx; + ret =3D of_irq_parse_raw(addr, &out_irq); + if (ret) + return ret; + + uc->irq_num_ring =3D irq_create_of_mapping(&out_irq); + + ret =3D devm_request_irq(ud->dev, uc->irq_num_ring, udma_v2_ring_irq_hand= ler, + IRQF_TRIGGER_HIGH, uc->name, uc); + + if (ret) { + dev_err(ud->dev, "chan%d: ring irq request failed\n", uc->id); + goto err_irq_free; + } + + uc->irq_num_udma =3D 0; + + udma_reset_rings(uc); + + INIT_DELAYED_WORK_ONSTACK(&uc->tx_drain.work, + udma_check_tx_completion); + + if (uc->tchan) + dev_dbg(ud->dev, + "chan%d: tchan%d, tflow%d, Remote thread: 0x%04x\n", + uc->id, uc->tchan->id, uc->tchan->tflow_id, + uc->config.remote_thread_id); + else if (uc->rchan) + dev_dbg(ud->dev, + "chan%d: rchan%d, rflow%d, Remote thread: 0x%04x\n", + uc->id, uc->rchan->id, uc->rflow->id, + uc->config.remote_thread_id); + return 0; + +err_irq_free: + uc->irq_num_ring =3D 0; +err_res_free: + udma_free_tx_resources(uc); + udma_free_rx_resources(uc); + + udma_reset_uchan(uc); + + dma_pool_destroy(uc->hdesc_pool); + uc->use_dma_pool =3D false; + + return ret; +} + static enum dma_status udma_v2_tx_status(struct dma_chan *chan, dma_cookie_t cookie, struct dma_tx_state *txstate) @@ -872,6 +1013,7 @@ static int udma_v2_resume(struct dma_chan *chan) } =20 static struct platform_driver bcdma_v2_driver; +static struct platform_driver pktdma_v2_driver; =20 static bool udma_v2_dma_filter_fn(struct dma_chan *chan, void *param) { @@ -881,7 +1023,8 @@ static bool udma_v2_dma_filter_fn(struct dma_chan *cha= n, void *param) struct udma_chan *uc; struct udma_dev *ud; =20 - if (chan->device->dev->driver !=3D &bcdma_v2_driver.driver) + if (chan->device->dev->driver !=3D &bcdma_v2_driver.driver && + chan->device->dev->driver !=3D &pktdma_v2_driver.driver) return false; =20 uc =3D to_udma_chan(chan); @@ -924,7 +1067,7 @@ static bool udma_v2_dma_filter_fn(struct dma_chan *cha= n, void *param) ucc->notdpkt =3D ep_config->notdpkt; ucc->ep_type =3D ep_config->ep_type; =20 - if ((ud->match_data->type =3D=3D DMA_TYPE_BCDMA_V2) && + if ((ud->match_data->type >=3D DMA_TYPE_BCDMA_V2) && ep_config->mapped_channel_id >=3D 0) { ucc->mapped_channel_id =3D ep_config->mapped_channel_id; ucc->default_flow_id =3D ep_config->default_flow_id; @@ -1023,11 +1166,33 @@ static struct udma_match_data bcdma_v2_data =3D { .rchan_cnt =3D 128, }; =20 +static struct udma_match_data pktdma_v2_data =3D { + .type =3D DMA_TYPE_PKTDMA_V2, + .psil_base =3D 0x1000, + .enable_memcpy_support =3D false, /* PKTDMA does not support MEM_TO_MEM */ + .flags =3D UDMA_FLAGS_J7_CLASS, + .statictr_z_mask =3D GENMASK(23, 0), + .burst_size =3D { + TI_SCI_RM_UDMAP_CHAN_BURST_SIZE_64_BYTES, /* Normal Channels */ + 0, /* No H Channels */ + 0, /* No UH Channels */ + }, + .tchan_cnt =3D 97, + .rchan_cnt =3D 97, + .chan_cnt =3D 97, + .tflow_cnt =3D 112, + .rflow_cnt =3D 112, +}; + static const struct of_device_id udma_of_match[] =3D { { .compatible =3D "ti,dmss-bcdma-v2", .data =3D &bcdma_v2_data, }, + { + .compatible =3D "ti,dmss-pktdma-v2", + .data =3D &pktdma_v2_data, + }, { /* Sentinel */ }, }; =20 @@ -1050,15 +1215,22 @@ static int udma_v2_get_mmrs(struct platform_device = *pdev, struct udma_dev *ud) cap2 =3D udma_read(ud->mmrs[V2_MMR_GCFG], 0x28); cap3 =3D udma_read(ud->mmrs[V2_MMR_GCFG], 0x2c); =20 - ud->bchan_cnt =3D ud->match_data->bchan_cnt; - /* There are no tchan and rchan in BCDMA_V2. + /* There are no tchan and rchan in BCDMA_V2 and PKTDMA_V2. * Duplicate chan as tchan and rchan to keep the common code - * in k3-udma-common.c functional for BCDMA_V2. + * in k3-udma-common.c functional. */ - ud->chan_cnt =3D ud->match_data->chan_cnt; - ud->tchan_cnt =3D ud->match_data->chan_cnt; - ud->rchan_cnt =3D ud->match_data->chan_cnt; - ud->rflow_cnt =3D ud->chan_cnt; + if (ud->match_data->type =3D=3D DMA_TYPE_BCDMA_V2) { + ud->bchan_cnt =3D ud->match_data->bchan_cnt; + ud->chan_cnt =3D ud->match_data->chan_cnt; + ud->tchan_cnt =3D ud->match_data->chan_cnt; + ud->rchan_cnt =3D ud->match_data->chan_cnt; + ud->rflow_cnt =3D ud->chan_cnt; + } else if (ud->match_data->type =3D=3D DMA_TYPE_PKTDMA_V2) { + ud->chan_cnt =3D ud->match_data->chan_cnt; + ud->tchan_cnt =3D ud->match_data->tchan_cnt; + ud->rchan_cnt =3D ud->match_data->rchan_cnt; + ud->rflow_cnt =3D ud->match_data->rflow_cnt; + } =20 for (i =3D 1; i < V2_MMR_LAST; i++) { if (i =3D=3D V2_MMR_BCHANRT && ud->bchan_cnt =3D=3D 0) @@ -1120,7 +1292,14 @@ static int udma_v2_probe(struct platform_device *pde= v) =20 struct k3_ringacc_init_data ring_init_data =3D {0}; =20 - ring_init_data.num_rings =3D ud->bchan_cnt + ud->chan_cnt; + if (ud->match_data->type =3D=3D DMA_TYPE_BCDMA_V2) { + ring_init_data.num_rings =3D ud->bchan_cnt + ud->chan_cnt; + } else if (ud->match_data->type =3D=3D DMA_TYPE_PKTDMA_V2) { + ring_init_data.num_rings =3D ud->rflow_cnt; + + ud->rflow_rt =3D devm_platform_ioremap_resource_byname(pdev, "ringrt"); + ring_init_data.base_rt =3D ud->rflow_rt; + } =20 ud->ringacc =3D k3_ringacc_dmarings_init(pdev, &ring_init_data); =20 @@ -1129,8 +1308,10 @@ static int udma_v2_probe(struct platform_device *pde= v) =20 dma_cap_set(DMA_SLAVE, ud->ddev.cap_mask); =20 - dma_cap_set(DMA_CYCLIC, ud->ddev.cap_mask); - ud->ddev.device_prep_dma_cyclic =3D udma_prep_dma_cyclic; + if (ud->match_data->type !=3D DMA_TYPE_PKTDMA_V2) { + dma_cap_set(DMA_CYCLIC, ud->ddev.cap_mask); + ud->ddev.device_prep_dma_cyclic =3D udma_prep_dma_cyclic; + } =20 ud->ddev.device_config =3D udma_slave_config; ud->ddev.device_prep_slave_sg =3D udma_prep_slave_sg; @@ -1144,8 +1325,18 @@ static int udma_v2_probe(struct platform_device *pde= v) ud->ddev.dbg_summary_show =3D udma_dbg_summary_show; #endif =20 - ud->ddev.device_alloc_chan_resources =3D - bcdma_v2_alloc_chan_resources; + switch (ud->match_data->type) { + case DMA_TYPE_BCDMA_V2: + ud->ddev.device_alloc_chan_resources =3D + bcdma_v2_alloc_chan_resources; + break; + case DMA_TYPE_PKTDMA_V2: + ud->ddev.device_alloc_chan_resources =3D + pktdma_v2_alloc_chan_resources; + break; + default: + return -EINVAL; + } =20 ud->ddev.device_free_chan_resources =3D udma_free_chan_resources; =20 diff --git a/drivers/dma/ti/k3-udma.h b/drivers/dma/ti/k3-udma.h index c054113640bf4..a112ce4186ca9 100644 --- a/drivers/dma/ti/k3-udma.h +++ b/drivers/dma/ti/k3-udma.h @@ -44,8 +44,11 @@ #define UDMA_RX_FLOW_ID_FW_OES_REG 0x80 #define UDMA_RX_FLOW_ID_FW_STATUS_REG 0x88 =20 +#define UDMA_RX_FLOWRT_RFA 0x8 + /* BCHANRT/TCHANRT/RCHANRT registers */ #define UDMA_CHAN_RT_CTL_REG 0x0 +#define UDMA_CHAN_RT_CFG_REG 0x4 #define UDMA_CHAN_RT_SWTRIG_REG 0x8 #define UDMA_CHAN_RT_STDATA_REG 0x80 =20 --=20 2.34.1 From nobody Sun Feb 8 20:13:33 2026 Received: from lelvem-ot01.ext.ti.com (lelvem-ot01.ext.ti.com [198.47.23.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8BDF81E9B04; Mon, 28 Apr 2025 07:21:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.47.23.234 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745824908; cv=none; b=l0NhsB5Gpzo/+bs9Rz/F5/+KK5wZDy5yxoW5cVizBtjwxUwVAS8Tyc5hoEB1//yKt8AbTx0nL6vEH9FxxIZPafXzRk69QUJPPjzeY8Cy4z3W5QNrjRHP52rwmYmk7qvteidBqqGLc8qUgtzh2bZeYt608giPo0a33Djak9xJmsw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745824908; c=relaxed/simple; bh=9QIsBgJpumkpCS/3IBMZvkRjF8Ope94HemVtN3JBdmw=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=OAma/nqm5hu6h5wJaTZMmrpbeKalJet7AnFVXagVh1YfXOViB+rk9+WhOOJSGjk5U5opGjnRzqb4VSMcBKbR34V9153/0gTD/bOHYrcpP48twnmMHBfyqJjkeQy6xAt/V2XoFHIBVwPNuxA9yo52rnV5RMXiP43KiW1cM7+XE0M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ti.com; spf=pass smtp.mailfrom=ti.com; dkim=pass (1024-bit key) header.d=ti.com header.i=@ti.com header.b=cBx64ppD; arc=none smtp.client-ip=198.47.23.234 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ti.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ti.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ti.com header.i=@ti.com header.b="cBx64ppD" Received: from lelv0266.itg.ti.com ([10.180.67.225]) by lelvem-ot01.ext.ti.com (8.15.2/8.15.2) with ESMTPS id 53S7Ld9L2714025 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 28 Apr 2025 02:21:39 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1745824899; bh=WARFXUGJtQhHRkobO2Jt9PhPdz2OjBnXVVYXvYthk2U=; h=From:To:Subject:Date:In-Reply-To:References; b=cBx64ppDg50BmUdjKDgbTwgqAwnllsq2Sh8xyvhNwLdjjWWGKWnO4l65LekGp8aZY 7cM9WIUwZeEEAa6G4polt4AJxt/Ft9ZjHpQlP9H2wVR3KBT///dmpxCypRnEu9Ehz9 46hYKHpm0h6khB8Bxx0JOjz+y1ZRcby2Ysy5B2k0= Received: from DLEE110.ent.ti.com (dlee110.ent.ti.com [157.170.170.21]) by lelv0266.itg.ti.com (8.15.2/8.15.2) with ESMTPS id 53S7LdUb121754 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 28 Apr 2025 02:21:39 -0500 Received: from DLEE111.ent.ti.com (157.170.170.22) by DLEE110.ent.ti.com (157.170.170.21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.23; Mon, 28 Apr 2025 02:21:39 -0500 Received: from lelvsmtp6.itg.ti.com (10.180.75.249) by DLEE111.ent.ti.com (157.170.170.22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2507.23 via Frontend Transport; Mon, 28 Apr 2025 02:21:39 -0500 Received: from uda0498651.dhcp.ti.com (uda0498651.dhcp.ti.com [172.24.227.7]) by lelvsmtp6.itg.ti.com (8.15.2/8.15.2) with ESMTP id 53S7KdMh068873; Mon, 28 Apr 2025 02:21:34 -0500 From: Sai Sree Kartheek Adivi To: , , , , , , , , , , , , , , , Subject: [PATCH 8/8] dmaengine: ti: k3-udma-v2: Update glue layer to support PKTDMA V2 Date: Mon, 28 Apr 2025 12:50:32 +0530 Message-ID: <20250428072032.946008-9-s-adivi@ti.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250428072032.946008-1-s-adivi@ti.com> References: <20250428072032.946008-1-s-adivi@ti.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-C2ProcessedOrg: 333ef613-75bf-4e12-a4b1-8e3623f5dcea Content-Type: text/plain; charset="utf-8" Update glue layer to support PKTDMA V2 for non DMAengine users. The updates include - Handling absence of TISCI - Direct IRQs - Autopair: Lack of PSIL pair. Signed-off-by: Sai Sree Kartheek Adivi --- drivers/dma/ti/k3-udma-glue.c | 91 ++++++++++++++++++++++---------- drivers/dma/ti/k3-udma-private.c | 48 +++++++++++++++-- drivers/dma/ti/k3-udma.h | 2 + 3 files changed, 110 insertions(+), 31 deletions(-) diff --git a/drivers/dma/ti/k3-udma-glue.c b/drivers/dma/ti/k3-udma-glue.c index f87d244cc2d67..886d57dadacae 100644 --- a/drivers/dma/ti/k3-udma-glue.c +++ b/drivers/dma/ti/k3-udma-glue.c @@ -244,6 +244,9 @@ static int k3_udma_glue_cfg_tx_chn(struct k3_udma_glue_= tx_channel *tx_chn) const struct udma_tisci_rm *tisci_rm =3D tx_chn->common.tisci_rm; struct ti_sci_msg_rm_udmap_tx_ch_cfg req; =20 + if (!tisci_rm->tisci) + return 0; + memset(&req, 0, sizeof(req)); =20 req.valid_params =3D TI_SCI_MSG_VALUE_RM_UDMAP_CH_PAUSE_ON_ERR_VALID | @@ -502,21 +505,26 @@ int k3_udma_glue_enable_tx_chn(struct k3_udma_glue_tx= _channel *tx_chn) { int ret; =20 - ret =3D xudma_navss_psil_pair(tx_chn->common.udmax, - tx_chn->common.src_thread, - tx_chn->common.dst_thread); - if (ret) { - dev_err(tx_chn->common.dev, "PSI-L request err %d\n", ret); - return ret; - } + if (tx_chn->common.udmax->match_data->type =3D=3D DMA_TYPE_PKTDMA_V2) { + xudma_tchanrt_write(tx_chn->udma_tchanx, UDMA_CHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_AUTOPAIR | UDMA_CHAN_RT_CTL_EN); + } else { + ret =3D xudma_navss_psil_pair(tx_chn->common.udmax, + tx_chn->common.src_thread, + tx_chn->common.dst_thread); + if (ret) { + dev_err(tx_chn->common.dev, "PSI-L request err %d\n", ret); + return ret; + } =20 - tx_chn->psil_paired =3D true; + tx_chn->psil_paired =3D true; =20 - xudma_tchanrt_write(tx_chn->udma_tchanx, UDMA_CHAN_RT_PEER_RT_EN_REG, - UDMA_PEER_RT_EN_ENABLE); + xudma_tchanrt_write(tx_chn->udma_tchanx, UDMA_CHAN_RT_PEER_RT_EN_REG, + UDMA_PEER_RT_EN_ENABLE); =20 - xudma_tchanrt_write(tx_chn->udma_tchanx, UDMA_CHAN_RT_CTL_REG, - UDMA_CHAN_RT_CTL_EN); + xudma_tchanrt_write(tx_chn->udma_tchanx, UDMA_CHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_EN); + } =20 k3_udma_glue_dump_tx_rt_chn(tx_chn, "txchn en"); return 0; @@ -682,7 +690,6 @@ static int k3_udma_glue_cfg_rx_chn(struct k3_udma_glue_= rx_channel *rx_chn) TI_SCI_MSG_VALUE_RM_UDMAP_CH_CHAN_TYPE_VALID | TI_SCI_MSG_VALUE_RM_UDMAP_CH_ATYPE_VALID; =20 - req.nav_id =3D tisci_rm->tisci_dev_id; req.index =3D rx_chn->udma_rchan_id; req.rx_fetch_size =3D rx_chn->common.hdesc_size >> 2; /* @@ -702,11 +709,18 @@ static int k3_udma_glue_cfg_rx_chn(struct k3_udma_glu= e_rx_channel *rx_chn) req.rx_chan_type =3D TI_SCI_RM_UDMAP_CHAN_TYPE_PKT_PBRR; req.rx_atype =3D rx_chn->common.atype_asel; =20 + if (!tisci_rm->tisci) { + // TODO: look at the chan settings + xudma_rchanrt_write(rx_chn->udma_rchanx, UDMA_CHAN_RT_CFG_REG, + UDMA_CHAN_RT_CTL_TDOWN | UDMA_CHAN_RT_CTL_PAUSE); + return 0; + } + + req.nav_id =3D tisci_rm->tisci_dev_id; ret =3D tisci_rm->tisci_udmap_ops->rx_ch_cfg(tisci_rm->tisci, &req); if (ret) dev_err(rx_chn->common.dev, "rchan%d cfg failed %d\n", - rx_chn->udma_rchan_id, ret); - + rx_chn->udma_rchan_id, ret); return ret; } =20 @@ -755,8 +769,11 @@ static int k3_udma_glue_cfg_rx_flow(struct k3_udma_glu= e_rx_channel *rx_chn, } =20 if (xudma_is_pktdma(rx_chn->common.udmax)) { - rx_ringfdq_id =3D flow->udma_rflow_id + + if (tisci_rm->tisci) + rx_ringfdq_id =3D flow->udma_rflow_id + xudma_get_rflow_ring_offset(rx_chn->common.udmax); + else + rx_ringfdq_id =3D flow->udma_rflow_id; rx_ring_id =3D 0; } else { rx_ring_id =3D flow_cfg->ring_rxq_id; @@ -803,6 +820,13 @@ static int k3_udma_glue_cfg_rx_flow(struct k3_udma_glu= e_rx_channel *rx_chn, rx_ringfdq_id =3D k3_ringacc_get_ring_id(flow->ringrxfdq); } =20 + if (!tisci_rm->tisci) { + xudma_rflowrt_write(flow->udma_rflow, UDMA_RX_FLOWRT_RFA, + UDMA_CHAN_RT_CTL_TDOWN | UDMA_CHAN_RT_CTL_PAUSE); + rx_chn->flows_ready++; + return 0; + } + memset(&req, 0, sizeof(req)); =20 req.valid_params =3D @@ -1307,6 +1331,9 @@ int k3_udma_glue_rx_flow_enable(struct k3_udma_glue_r= x_channel *rx_chn, if (!rx_chn->remote) return -EINVAL; =20 + if (!tisci_rm->tisci) + return 0; + rx_ring_id =3D k3_ringacc_get_ring_id(flow->ringrx); rx_ringfdq_id =3D k3_ringacc_get_ring_id(flow->ringrxfdq); =20 @@ -1348,6 +1375,9 @@ int k3_udma_glue_rx_flow_disable(struct k3_udma_glue_= rx_channel *rx_chn, if (!rx_chn->remote) return -EINVAL; =20 + if (!tisci_rm->tisci) + return 0; + memset(&req, 0, sizeof(req)); req.valid_params =3D TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_DEST_QNUM_VALID | @@ -1383,21 +1413,26 @@ int k3_udma_glue_enable_rx_chn(struct k3_udma_glue_= rx_channel *rx_chn) if (rx_chn->flows_ready < rx_chn->flow_num) return -EINVAL; =20 - ret =3D xudma_navss_psil_pair(rx_chn->common.udmax, - rx_chn->common.src_thread, - rx_chn->common.dst_thread); - if (ret) { - dev_err(rx_chn->common.dev, "PSI-L request err %d\n", ret); - return ret; - } + if (rx_chn->common.udmax->match_data->type =3D=3D DMA_TYPE_PKTDMA_V2) { + xudma_rchanrt_write(rx_chn->udma_rchanx, UDMA_CHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_AUTOPAIR | UDMA_CHAN_RT_CTL_EN); + } else { + ret =3D xudma_navss_psil_pair(rx_chn->common.udmax, + rx_chn->common.src_thread, + rx_chn->common.dst_thread); + if (ret) { + dev_err(rx_chn->common.dev, "PSI-L request err %d\n", ret); + return ret; + } =20 - rx_chn->psil_paired =3D true; + rx_chn->psil_paired =3D true; =20 - xudma_rchanrt_write(rx_chn->udma_rchanx, UDMA_CHAN_RT_CTL_REG, - UDMA_CHAN_RT_CTL_EN); + xudma_rchanrt_write(rx_chn->udma_rchanx, UDMA_CHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_EN); =20 - xudma_rchanrt_write(rx_chn->udma_rchanx, UDMA_CHAN_RT_PEER_RT_EN_REG, - UDMA_PEER_RT_EN_ENABLE); + xudma_rchanrt_write(rx_chn->udma_rchanx, UDMA_CHAN_RT_PEER_RT_EN_REG, + UDMA_PEER_RT_EN_ENABLE); + } =20 k3_udma_glue_dump_rx_rt_chn(rx_chn, "rxrt en"); return 0; diff --git a/drivers/dma/ti/k3-udma-private.c b/drivers/dma/ti/k3-udma-priv= ate.c index 05228bf000333..5fccb8d18c898 100644 --- a/drivers/dma/ti/k3-udma-private.c +++ b/drivers/dma/ti/k3-udma-private.c @@ -3,18 +3,28 @@ * Copyright (C) 2019 Texas Instruments Incorporated - http://www.ti.com * Author: Peter Ujfalusi */ +#include +#include +#include +#include #include #include =20 int xudma_navss_psil_pair(struct udma_dev *ud, u32 src_thread, u32 dst_thr= ead) { - return navss_psil_pair(ud, src_thread, dst_thread); + if (IS_ENABLED(CONFIG_TI_K3_UDMA)) + return navss_psil_pair(ud, src_thread, dst_thread); + + return 0; } EXPORT_SYMBOL(xudma_navss_psil_pair); =20 int xudma_navss_psil_unpair(struct udma_dev *ud, u32 src_thread, u32 dst_t= hread) { - return navss_psil_unpair(ud, src_thread, dst_thread); + if (IS_ENABLED(CONFIG_TI_K3_UDMA)) + return navss_psil_unpair(ud, src_thread, dst_thread); + + return 0; } EXPORT_SYMBOL(xudma_navss_psil_unpair); =20 @@ -159,15 +169,32 @@ void xudma_##res##rt_write(struct udma_##res *p, int = reg, u32 val) \ EXPORT_SYMBOL(xudma_##res##rt_write) XUDMA_RT_IO_FUNCTIONS(tchan); XUDMA_RT_IO_FUNCTIONS(rchan); +XUDMA_RT_IO_FUNCTIONS(rflow); =20 int xudma_is_pktdma(struct udma_dev *ud) { - return ud->match_data->type =3D=3D DMA_TYPE_PKTDMA; + return (ud->match_data->type =3D=3D DMA_TYPE_PKTDMA || + ud->match_data->type =3D=3D DMA_TYPE_PKTDMA_V2); } EXPORT_SYMBOL(xudma_is_pktdma); =20 int xudma_pktdma_tflow_get_irq(struct udma_dev *ud, int udma_tflow_id) { + if (ud->match_data->type =3D=3D DMA_TYPE_PKTDMA_V2) { + __be32 addr[2] =3D {0, 0}; + struct of_phandle_args out_irq; + int ret; + + out_irq.np =3D dev_of_node(ud->dev); + out_irq.args_count =3D 1; + out_irq.args[0] =3D udma_tflow_id; + ret =3D of_irq_parse_raw(addr, &out_irq); + if (ret) + return ret; + + return irq_create_of_mapping(&out_irq); + } + const struct udma_oes_offsets *oes =3D &ud->soc_data->oes; =20 return msi_get_virq(ud->dev, udma_tflow_id + oes->pktdma_tchan_flow); @@ -176,6 +203,21 @@ EXPORT_SYMBOL(xudma_pktdma_tflow_get_irq); =20 int xudma_pktdma_rflow_get_irq(struct udma_dev *ud, int udma_rflow_id) { + if (ud->match_data->type =3D=3D DMA_TYPE_PKTDMA_V2) { + __be32 addr[2] =3D {0, 0}; + struct of_phandle_args out_irq; + int ret; + + out_irq.np =3D dev_of_node(ud->dev); + out_irq.args_count =3D 1; + out_irq.args[0] =3D udma_rflow_id; + ret =3D of_irq_parse_raw(addr, &out_irq); + if (ret) + return ret; + + return irq_create_of_mapping(&out_irq); + } + const struct udma_oes_offsets *oes =3D &ud->soc_data->oes; =20 return msi_get_virq(ud->dev, udma_rflow_id + oes->pktdma_rchan_flow); diff --git a/drivers/dma/ti/k3-udma.h b/drivers/dma/ti/k3-udma.h index a112ce4186ca9..b09a7339a6442 100644 --- a/drivers/dma/ti/k3-udma.h +++ b/drivers/dma/ti/k3-udma.h @@ -718,6 +718,8 @@ u32 xudma_rchanrt_read(struct udma_rchan *rchan, int re= g); void xudma_rchanrt_write(struct udma_rchan *rchan, int reg, u32 val); bool xudma_rflow_is_gp(struct udma_dev *ud, int id); int xudma_get_rflow_ring_offset(struct udma_dev *ud); +u32 xudma_rflowrt_read(struct udma_rflow *rflow, int reg); +void xudma_rflowrt_write(struct udma_rflow *rflow, int reg, u32 val); =20 int xudma_is_pktdma(struct udma_dev *ud); =20 --=20 2.34.1