D1 is a RISC-V SoC from Allwinner's sunxi family. This series adds IOMMU
binding and driver support.

One piece is still missing to use the IOMMU for DMA allocations: a call
to iommu_setup_dma_ops(). On ARM64 this is handled by the architecture's
code. RISC-V does not currently select ARCH_HAS_SETUP_DMA_OPS, but it
will once Zicbom support[1] is merged.

[1]: https://lore.kernel.org/lkml/20220307224620.1933061-2-heiko@sntech.de/

So I cannot follow virtio-iommu.c and call iommu_setup_dma_ops() when
ARCH_HAS_SETUP_DMA_OPS=n. However, if I apply the following patch on top
of Heiko's non-coherent DMA series, the display engine successfully uses
the IOMMU to allocate its framebuffer:

--- a/arch/riscv/mm/dma-noncoherent.c
+++ b/arch/riscv/mm/dma-noncoherent.c
@@ -6,6 +6,7 @@
  */

 #include <linux/dma-direct.h>
+#include <linux/dma-iommu.h>
 #include <linux/dma-map-ops.h>
 #include <linux/mm.h>

@@ -53,4 +54,7 @@
 {
 	/* If a specific device is dma-coherent, set it here */
 	dev->dma_coherent = coherent;
+
+	if (iommu)
+		iommu_setup_dma_ops(dev, dma_base, dma_base + size - 1);
 }


Samuel Holland (5):
  dt-bindings: iommu: sun50i: Add compatible for Allwinner D1
  iommu/sun50i: Support variants without an external reset
  iommu/sun50i: Ensure bypass is disabled
  iommu/sun50i: Add support for the D1 variant
  iommu/sun50i: Ensure the IOMMU can be used for DMA

 .../iommu/allwinner,sun50i-h6-iommu.yaml      | 16 +++++++++++--
 drivers/iommu/Kconfig                         |  1 +
 drivers/iommu/sun50i-iommu.c                  | 24 +++++++++++++++++--
 3 files changed, 37 insertions(+), 4 deletions(-)

-- 
2.35.1

Hi Samuel!

Dne četrtek, 28. april 2022 ob 03:03:55 CEST je Samuel Holland napisal(a):
> D1 is a RISC-V SoC from Allwinner's sunxi family. This series adds IOMMU
> binding and driver support.
> 
> One piece is still missing to use the IOMMU for DMA allocations: a call
> to iommu_setup_dma_ops(). On ARM64 this is handled by the architecture's
> code. RISC-V does not currently select ARCH_HAS_SETUP_DMA_OPS, but it
> will once Zicbom support[1] is merged.
> 
> [1]: https://lore.kernel.org/lkml/20220307224620.1933061-2-heiko@sntech.de/
> 
> So I cannot follow virtio-iommu.c and call iommu_setup_dma_ops() when
> ARCH_HAS_SETUP_DMA_OPS=n. However, if I apply the following patch on top
> of Heiko's non-coherent DMA series, the display engine successfully uses
> the IOMMU to allocate its framebuffer:

Did you test this on any other device than display pipeline? It should be 
supported by Cedrus too, right? I think there are still some corner cases to 
fix on Cedrus before IOMMU fully works.

Best regards,
Jernej

> 
> --- a/arch/riscv/mm/dma-noncoherent.c
> +++ b/arch/riscv/mm/dma-noncoherent.c
> @@ -6,6 +6,7 @@
>   */
> 
>  #include <linux/dma-direct.h>
> +#include <linux/dma-iommu.h>
>  #include <linux/dma-map-ops.h>
>  #include <linux/mm.h>
> 
> @@ -53,4 +54,7 @@
>  {
>  	/* If a specific device is dma-coherent, set it here */
>  	dev->dma_coherent = coherent;
> +
> +	if (iommu)
> +		iommu_setup_dma_ops(dev, dma_base, dma_base + size - 1);
>  }
> 
> 
> Samuel Holland (5):
>   dt-bindings: iommu: sun50i: Add compatible for Allwinner D1
>   iommu/sun50i: Support variants without an external reset
>   iommu/sun50i: Ensure bypass is disabled
>   iommu/sun50i: Add support for the D1 variant
>   iommu/sun50i: Ensure the IOMMU can be used for DMA
> 
>  .../iommu/allwinner,sun50i-h6-iommu.yaml      | 16 +++++++++++--
>  drivers/iommu/Kconfig                         |  1 +
>  drivers/iommu/sun50i-iommu.c                  | 24 +++++++++++++++++--
>  3 files changed, 37 insertions(+), 4 deletions(-)

D1 contains an IOMMU similar to the one in the H6 SoC, but the D1
variant has no external reset signal.

Signed-off-by: Samuel Holland <samuel@sholland.org>
---

 .../iommu/allwinner,sun50i-h6-iommu.yaml         | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/Documentation/devicetree/bindings/iommu/allwinner,sun50i-h6-iommu.yaml b/Documentation/devicetree/bindings/iommu/allwinner,sun50i-h6-iommu.yaml
index 5e125cf2a88b..18d3451d4dd5 100644
--- a/Documentation/devicetree/bindings/iommu/allwinner,sun50i-h6-iommu.yaml
+++ b/Documentation/devicetree/bindings/iommu/allwinner,sun50i-h6-iommu.yaml
@@ -17,7 +17,9 @@ properties:
       The content of the cell is the master ID.
 
   compatible:
-    const: allwinner,sun50i-h6-iommu
+    enum:
+      - allwinner,sun20i-d1-iommu
+      - allwinner,sun50i-h6-iommu
 
   reg:
     maxItems: 1
@@ -37,7 +39,17 @@ required:
   - reg
   - interrupts
   - clocks
-  - resets
+
+if:
+  properties:
+    compatible:
+      contains:
+        enum:
+          - allwinner,sun50i-h6-iommu
+
+then:
+  required:
+    - resets
 
 additionalProperties: false
 
-- 
2.35.1

On 28/04/2022 03:03, Samuel Holland wrote:

Thank you for your patch. There is something to discuss/improve.

> +
> +if:
> +  properties:
> +    compatible:
> +      contains:
> +        enum:
> +          - allwinner,sun50i-h6-iommu
> +
> +then:
> +  required:
> +    - resets

else:
  properties:
    resets: false

>  


Best regards,
Krzysztof

The IOMMU in the Allwinner D1 SoC does not have an external reset line.

Only attempt to get the reset on hardware variants which should have one
according to the binding. And switch from the deprecated function to the
explicit "exclusive" variant.

Signed-off-by: Samuel Holland <samuel@sholland.org>
---

 drivers/iommu/sun50i-iommu.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/sun50i-iommu.c b/drivers/iommu/sun50i-iommu.c
index c54ab477b8fd..ec07b60016d3 100644
--- a/drivers/iommu/sun50i-iommu.c
+++ b/drivers/iommu/sun50i-iommu.c
@@ -92,6 +92,10 @@
 #define NUM_PT_ENTRIES			256
 #define PT_SIZE				(NUM_PT_ENTRIES * PT_ENTRY_SIZE)
 
+struct sun50i_iommu_variant {
+	bool has_reset;
+};
+
 struct sun50i_iommu {
 	struct iommu_device iommu;
 
@@ -905,9 +909,14 @@ static irqreturn_t sun50i_iommu_irq(int irq, void *dev_id)
 
 static int sun50i_iommu_probe(struct platform_device *pdev)
 {
+	const struct sun50i_iommu_variant *variant;
 	struct sun50i_iommu *iommu;
 	int ret, irq;
 
+	variant = of_device_get_match_data(&pdev->dev);
+	if (!variant)
+		return -EINVAL;
+
 	iommu = devm_kzalloc(&pdev->dev, sizeof(*iommu), GFP_KERNEL);
 	if (!iommu)
 		return -ENOMEM;
@@ -947,7 +956,8 @@ static int sun50i_iommu_probe(struct platform_device *pdev)
 		goto err_free_group;
 	}
 
-	iommu->reset = devm_reset_control_get(&pdev->dev, NULL);
+	if (variant->has_reset)
+		iommu->reset = devm_reset_control_get_exclusive(&pdev->dev, NULL);
 	if (IS_ERR(iommu->reset)) {
 		dev_err(&pdev->dev, "Couldn't get our reset line.\n");
 		ret = PTR_ERR(iommu->reset);
@@ -987,8 +997,12 @@ static int sun50i_iommu_probe(struct platform_device *pdev)
 	return ret;
 }
 
+static const struct sun50i_iommu_variant sun50i_h6_iommu = {
+	.has_reset = true,
+};
+
 static const struct of_device_id sun50i_iommu_dt[] = {
-	{ .compatible = "allwinner,sun50i-h6-iommu", },
+	{ .compatible = "allwinner,sun50i-h6-iommu", .data = &sun50i_h6_iommu },
 	{ /* sentinel */ },
 };
 MODULE_DEVICE_TABLE(of, sun50i_iommu_dt);
-- 
2.35.1

Hi Samuel,

On Mi, 2022-04-27 at 20:03 -0500, Samuel Holland wrote:
> The IOMMU in the Allwinner D1 SoC does not have an external reset line.
> 
> Only attempt to get the reset on hardware variants which should have one
> according to the binding. And switch from the deprecated function to the
> explicit "exclusive" variant.
> 
> Signed-off-by: Samuel Holland <samuel@sholland.org>

Reviewed-by: Philipp Zabel <p.zabel@pengutronix.de>

regards
Philipp

Dne četrtek, 28. april 2022 ob 03:03:57 CEST je Samuel Holland napisal(a):
> The IOMMU in the Allwinner D1 SoC does not have an external reset line.
> 
> Only attempt to get the reset on hardware variants which should have one
> according to the binding. And switch from the deprecated function to the
> explicit "exclusive" variant.
> 
> Signed-off-by: Samuel Holland <samuel@sholland.org>

Reviewed-by: Jernej Skrabec <jernej.skrabec@gmail.com>

Best regards,
Jernej

> ---
> 
>  drivers/iommu/sun50i-iommu.c | 18 ++++++++++++++++--
>  1 file changed, 16 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/iommu/sun50i-iommu.c b/drivers/iommu/sun50i-iommu.c
> index c54ab477b8fd..ec07b60016d3 100644
> --- a/drivers/iommu/sun50i-iommu.c
> +++ b/drivers/iommu/sun50i-iommu.c
> @@ -92,6 +92,10 @@
>  #define NUM_PT_ENTRIES			256
>  #define PT_SIZE				(NUM_PT_ENTRIES * 
PT_ENTRY_SIZE)
> 
> +struct sun50i_iommu_variant {
> +	bool has_reset;
> +};
> +
>  struct sun50i_iommu {
>  	struct iommu_device iommu;
> 
> @@ -905,9 +909,14 @@ static irqreturn_t sun50i_iommu_irq(int irq, void
> *dev_id)
> 
>  static int sun50i_iommu_probe(struct platform_device *pdev)
>  {
> +	const struct sun50i_iommu_variant *variant;
>  	struct sun50i_iommu *iommu;
>  	int ret, irq;
> 
> +	variant = of_device_get_match_data(&pdev->dev);
> +	if (!variant)
> +		return -EINVAL;
> +
>  	iommu = devm_kzalloc(&pdev->dev, sizeof(*iommu), GFP_KERNEL);
>  	if (!iommu)
>  		return -ENOMEM;
> @@ -947,7 +956,8 @@ static int sun50i_iommu_probe(struct platform_device
> *pdev) goto err_free_group;
>  	}
> 
> -	iommu->reset = devm_reset_control_get(&pdev->dev, NULL);
> +	if (variant->has_reset)
> +		iommu->reset = devm_reset_control_get_exclusive(&pdev-
>dev, NULL);
>  	if (IS_ERR(iommu->reset)) {
>  		dev_err(&pdev->dev, "Couldn't get our reset line.\n");
>  		ret = PTR_ERR(iommu->reset);
> @@ -987,8 +997,12 @@ static int sun50i_iommu_probe(struct platform_device
> *pdev) return ret;
>  }
> 
> +static const struct sun50i_iommu_variant sun50i_h6_iommu = {
> +	.has_reset = true,
> +};
> +
>  static const struct of_device_id sun50i_iommu_dt[] = {
> -	{ .compatible = "allwinner,sun50i-h6-iommu", },
> +	{ .compatible = "allwinner,sun50i-h6-iommu", .data = 
&sun50i_h6_iommu },
>  	{ /* sentinel */ },
>  };
>  MODULE_DEVICE_TABLE(of, sun50i_iommu_dt);

The H6 variant of the hardware disables bypass by default. The D1
variant of the hardware enables bypass for all masters by default.

Since the driver expects bypass to be disabled, ensure that is the case.

Signed-off-by: Samuel Holland <samuel@sholland.org>
---

 drivers/iommu/sun50i-iommu.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/iommu/sun50i-iommu.c b/drivers/iommu/sun50i-iommu.c
index ec07b60016d3..b9e644b93637 100644
--- a/drivers/iommu/sun50i-iommu.c
+++ b/drivers/iommu/sun50i-iommu.c
@@ -374,6 +374,8 @@ static int sun50i_iommu_enable(struct sun50i_iommu *iommu)
 
 	spin_lock_irqsave(&iommu->iommu_lock, flags);
 
+	iommu_write(iommu, IOMMU_BYPASS_REG, 0);
+
 	iommu_write(iommu, IOMMU_TTB_REG, sun50i_domain->dt_dma);
 	iommu_write(iommu, IOMMU_TLB_PREFETCH_REG,
 		    IOMMU_TLB_PREFETCH_MASTER_ENABLE(0) |
-- 
2.35.1

Dne četrtek, 28. april 2022 ob 03:03:58 CEST je Samuel Holland napisal(a):
> The H6 variant of the hardware disables bypass by default. The D1
> variant of the hardware enables bypass for all masters by default.
> 
> Since the driver expects bypass to be disabled, ensure that is the case.
> 
> Signed-off-by: Samuel Holland <samuel@sholland.org>

Actually, it would be better to set bypass to 0xff and in 
sun50i_iommu_attach_device() clear bypass bit for that particular device. As 
you might notice, index in phandle is currently not used. This would also help 
expose bugs, like missing second iommu channel for Cedrus on H6, but that's 
easy to fix.

Best regards,
Jernej

> ---
> 
>  drivers/iommu/sun50i-iommu.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/iommu/sun50i-iommu.c b/drivers/iommu/sun50i-iommu.c
> index ec07b60016d3..b9e644b93637 100644
> --- a/drivers/iommu/sun50i-iommu.c
> +++ b/drivers/iommu/sun50i-iommu.c
> @@ -374,6 +374,8 @@ static int sun50i_iommu_enable(struct sun50i_iommu
> *iommu)
> 
>  	spin_lock_irqsave(&iommu->iommu_lock, flags);
> 
> +	iommu_write(iommu, IOMMU_BYPASS_REG, 0);
> +
>  	iommu_write(iommu, IOMMU_TTB_REG, sun50i_domain->dt_dma);
>  	iommu_write(iommu, IOMMU_TLB_PREFETCH_REG,
>  		    IOMMU_TLB_PREFETCH_MASTER_ENABLE(0) |

D1 contains an IOMMU similar to the one in the H6 SoC, but the D1
variant has no external reset signal. It also has some register
definition changes, but none that affect the current driver.

Signed-off-by: Samuel Holland <samuel@sholland.org>
---

 drivers/iommu/sun50i-iommu.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/iommu/sun50i-iommu.c b/drivers/iommu/sun50i-iommu.c
index b9e644b93637..1fb707e37fb3 100644
--- a/drivers/iommu/sun50i-iommu.c
+++ b/drivers/iommu/sun50i-iommu.c
@@ -999,11 +999,15 @@ static int sun50i_iommu_probe(struct platform_device *pdev)
 	return ret;
 }
 
+static const struct sun50i_iommu_variant sun20i_d1_iommu = {
+};
+
 static const struct sun50i_iommu_variant sun50i_h6_iommu = {
 	.has_reset = true,
 };
 
 static const struct of_device_id sun50i_iommu_dt[] = {
+	{ .compatible = "allwinner,sun20i-d1-iommu", .data = &sun20i_d1_iommu },
 	{ .compatible = "allwinner,sun50i-h6-iommu", .data = &sun50i_h6_iommu },
 	{ /* sentinel */ },
 };
-- 
2.35.1

Dne četrtek, 28. april 2022 ob 03:03:59 CEST je Samuel Holland napisal(a):
> D1 contains an IOMMU similar to the one in the H6 SoC, but the D1
> variant has no external reset signal. It also has some register
> definition changes, but none that affect the current driver.
> 
> Signed-off-by: Samuel Holland <samuel@sholland.org>

Reviewed-by: Jernej Skrabec <jernej.skrabec@gmail.com>

Best regards,
Jernej

> ---
> 
>  drivers/iommu/sun50i-iommu.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/drivers/iommu/sun50i-iommu.c b/drivers/iommu/sun50i-iommu.c
> index b9e644b93637..1fb707e37fb3 100644
> --- a/drivers/iommu/sun50i-iommu.c
> +++ b/drivers/iommu/sun50i-iommu.c
> @@ -999,11 +999,15 @@ static int sun50i_iommu_probe(struct platform_device
> *pdev) return ret;
>  }
> 
> +static const struct sun50i_iommu_variant sun20i_d1_iommu = {
> +};
> +
>  static const struct sun50i_iommu_variant sun50i_h6_iommu = {
>  	.has_reset = true,
>  };
> 
>  static const struct of_device_id sun50i_iommu_dt[] = {
> +	{ .compatible = "allwinner,sun20i-d1-iommu", .data = 
&sun20i_d1_iommu },
>  	{ .compatible = "allwinner,sun50i-h6-iommu", .data = 
&sun50i_h6_iommu },
>  	{ /* sentinel */ },
>  };

So far, the driver has relied on arch/arm64/Kconfig to select IOMMU_DMA.
Unsurprisingly, this does not work on RISC-V, so the driver must select
IOMMU_DMA itself.

Signed-off-by: Samuel Holland <samuel@sholland.org>
---

 drivers/iommu/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index c79a0df090c0..70a0bfa6d907 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -223,6 +223,7 @@ config SUN50I_IOMMU
 	depends on ARCH_SUNXI || COMPILE_TEST
 	select ARM_DMA_USE_IOMMU
 	select IOMMU_API
+	select IOMMU_DMA
 	help
 	  Support for the IOMMU introduced in the Allwinner H6 SoCs.
 
-- 
2.35.1

On 2022-04-28 02:04, Samuel Holland wrote:
> So far, the driver has relied on arch/arm64/Kconfig to select IOMMU_DMA.
> Unsurprisingly, this does not work on RISC-V, so the driver must select
> IOMMU_DMA itself.

No, IOMMU_DMA should only be selected by the architecture code that's 
also responsible for calling iommu_setup_dma_ops(). Without that, this 
select will do nothing other than add some unused code to the kernel image.

I appreciate that the current state of the x86 IOMMU drivers being 
tightly-coupled to the x86 arch code might be confusing (which reminds 
me I'd totally forgotten about [1]). I'm about to start reworking the 
whole area anyway, but for now please just follow the existing intent.

Thanks,
Robin.

[1] 
https://lore.kernel.org/linux-iommu/9ba6f2e8568a3ff6a94fade66668d99705433c44.1631536879.git.robin.murphy@arm.com/

> Signed-off-by: Samuel Holland <samuel@sholland.org>
> ---
> 
>   drivers/iommu/Kconfig | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
> index c79a0df090c0..70a0bfa6d907 100644
> --- a/drivers/iommu/Kconfig
> +++ b/drivers/iommu/Kconfig
> @@ -223,6 +223,7 @@ config SUN50I_IOMMU
>   	depends on ARCH_SUNXI || COMPILE_TEST
>   	select ARM_DMA_USE_IOMMU
>   	select IOMMU_API
> +	select IOMMU_DMA
>   	help
>   	  Support for the IOMMU introduced in the Allwinner H6 SoCs.
>

Dne četrtek, 28. april 2022 ob 03:04:00 CEST je Samuel Holland napisal(a):
> So far, the driver has relied on arch/arm64/Kconfig to select IOMMU_DMA.
> Unsurprisingly, this does not work on RISC-V, so the driver must select
> IOMMU_DMA itself.
> 
> Signed-off-by: Samuel Holland <samuel@sholland.org>

Reviewed-by: Jernej Skrabec <jernej.skrabec@gmail.com>

Best regards,
Jernej

> ---
> 
>  drivers/iommu/Kconfig | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
> index c79a0df090c0..70a0bfa6d907 100644
> --- a/drivers/iommu/Kconfig
> +++ b/drivers/iommu/Kconfig
> @@ -223,6 +223,7 @@ config SUN50I_IOMMU
>  	depends on ARCH_SUNXI || COMPILE_TEST
>  	select ARM_DMA_USE_IOMMU
>  	select IOMMU_API
> +	select IOMMU_DMA
>  	help
>  	  Support for the IOMMU introduced in the Allwinner H6 SoCs.