1. Patchew
  2. Xen
  3. [PATCH V2 00/10] Add support for Renesas R-Car S4 IPMMU and other misc changes
  4. View patch

[PATCH V2 09/10] iommu/ipmmu-vmsa: Use refcount for the micro-TLBs

Oleksandr Tyshchenko posted 10 patches 4 years, 1 month ago
[PATCH V2 09/10] iommu/ipmmu-vmsa: Use refcount for the micro-TLBs
Posted by Oleksandr Tyshchenko 4 years, 1 month ago 
From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

Reference-count the micro-TLBs as several bus masters can be
connected to the same micro-TLB (and drop TODO comment).
This wasn't an issue so far, since the platform devices
(this driver deals with) get assigned/deassigned together during
domain creation/destruction. But, in order to support PCI devices
(which are hot-pluggable) in the near future we will need to
take care of.

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Reviewed-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
Changes V1 -> V2:
   - add R-b
   - add ASSERT() in ipmmu_utlb_disable()
---
 xen/drivers/passthrough/arm/ipmmu-vmsa.c | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
index 649b9f6..1224ea4 100644
--- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
+++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
@@ -127,6 +127,7 @@ struct ipmmu_vmsa_device {
     spinlock_t lock;    /* Protects ctx and domains[] */
     DECLARE_BITMAP(ctx, IPMMU_CTX_MAX);
     struct ipmmu_vmsa_domain *domains[IPMMU_CTX_MAX];
+    unsigned int utlb_refcount[IPMMU_UTLB_MAX];
     const struct ipmmu_features *features;
 };
 
@@ -467,13 +468,12 @@ static int ipmmu_utlb_enable(struct ipmmu_vmsa_domain *domain,
         }
     }
 
-    /*
-     * TODO: Reference-count the micro-TLB as several bus masters can be
-     * connected to the same micro-TLB.
-     */
-    ipmmu_imuasid_write(mmu, utlb, 0);
-    ipmmu_imuctr_write(mmu, utlb, imuctr |
-                       IMUCTR_TTSEL_MMU(domain->context_id) | IMUCTR_MMUEN);
+    if ( mmu->utlb_refcount[utlb]++ == 0 )
+    {
+        ipmmu_imuasid_write(mmu, utlb, 0);
+        ipmmu_imuctr_write(mmu, utlb, imuctr |
+                           IMUCTR_TTSEL_MMU(domain->context_id) | IMUCTR_MMUEN);
+    }
 
     return 0;
 }
@@ -484,7 +484,10 @@ static void ipmmu_utlb_disable(struct ipmmu_vmsa_domain *domain,
 {
     struct ipmmu_vmsa_device *mmu = domain->mmu;
 
-    ipmmu_imuctr_write(mmu, utlb, 0);
+    ASSERT(mmu->utlb_refcount[utlb] > 0);
+
+    if ( --mmu->utlb_refcount[utlb] == 0 )
+        ipmmu_imuctr_write(mmu, utlb, 0);
 }
 
 /* Domain/Context Management */
-- 
2.7.4
Re: [PATCH V2 09/10] iommu/ipmmu-vmsa: Use refcount for the micro-TLBs
Posted by Julien Grall 4 years ago 
Hi Oleksandr,

On 20/12/2021 21:15, Oleksandr Tyshchenko wrote:
> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> 
> Reference-count the micro-TLBs as several bus masters can be
> connected to the same micro-TLB (and drop TODO comment).
> This wasn't an issue so far, since the platform devices
> (this driver deals with) get assigned/deassigned together during
> domain creation/destruction. But, in order to support PCI devices
> (which are hot-pluggable) in the near future we will need to
> take care of.

Looking at the code, it is not possible to share the micro-TLB between 
domains (or even Xen). So technically, we will still want to {, 
un}hotplug the devices using the same u-TLB together.

Therefore, I would clarify that this is necessary because even if we 
have to remove all the devices together, the IOMMU driver will be 
de-assigning them one-by-one.

I would add a similar comment in the code as well.

> 
> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> Reviewed-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> ---
> Changes V1 -> V2:
>     - add R-b
>     - add ASSERT() in ipmmu_utlb_disable()
> ---
>   xen/drivers/passthrough/arm/ipmmu-vmsa.c | 19 +++++++++++--------
>   1 file changed, 11 insertions(+), 8 deletions(-)
> 
> diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
> index 649b9f6..1224ea4 100644
> --- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
> +++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
> @@ -127,6 +127,7 @@ struct ipmmu_vmsa_device {
>       spinlock_t lock;    /* Protects ctx and domains[] */
>       DECLARE_BITMAP(ctx, IPMMU_CTX_MAX);
>       struct ipmmu_vmsa_domain *domains[IPMMU_CTX_MAX];
> +    unsigned int utlb_refcount[IPMMU_UTLB_MAX];
>       const struct ipmmu_features *features;
>   };
>   
> @@ -467,13 +468,12 @@ static int ipmmu_utlb_enable(struct ipmmu_vmsa_domain *domain,
>           }
>       }
>   
> -    /*
> -     * TODO: Reference-count the micro-TLB as several bus masters can be
> -     * connected to the same micro-TLB.
> -     */
> -    ipmmu_imuasid_write(mmu, utlb, 0);
> -    ipmmu_imuctr_write(mmu, utlb, imuctr |
> -                       IMUCTR_TTSEL_MMU(domain->context_id) | IMUCTR_MMUEN);
> +    if ( mmu->utlb_refcount[utlb]++ == 0 )
> +    {
> +        ipmmu_imuasid_write(mmu, utlb, 0);
> +        ipmmu_imuctr_write(mmu, utlb, imuctr |
> +                           IMUCTR_TTSEL_MMU(domain->context_id) | IMUCTR_MMUEN);
> +    }
>   
>       return 0;
>   }
> @@ -484,7 +484,10 @@ static void ipmmu_utlb_disable(struct ipmmu_vmsa_domain *domain,
>   {
>       struct ipmmu_vmsa_device *mmu = domain->mmu;
>   
> -    ipmmu_imuctr_write(mmu, utlb, 0);
> +    ASSERT(mmu->utlb_refcount[utlb] > 0);
> +
> +    if ( --mmu->utlb_refcount[utlb] == 0 )
> +        ipmmu_imuctr_write(mmu, utlb, 0);
>   }
>   
>   /* Domain/Context Management */

Cheers,

-- 
Julien Grall
Re: [PATCH V2 09/10] iommu/ipmmu-vmsa: Use refcount for the micro-TLBs
Posted by Oleksandr 4 years ago 
On 27.01.22 13:48, Julien Grall wrote:
> Hi Oleksandr,

Hi Julien


>
> On 20/12/2021 21:15, Oleksandr Tyshchenko wrote:
>> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
>>
>> Reference-count the micro-TLBs as several bus masters can be
>> connected to the same micro-TLB (and drop TODO comment).
>> This wasn't an issue so far, since the platform devices
>> (this driver deals with) get assigned/deassigned together during
>> domain creation/destruction. But, in order to support PCI devices
>> (which are hot-pluggable) in the near future we will need to
>> take care of.
>
> Looking at the code, it is not possible to share the micro-TLB between 
> domains (or even Xen). So technically, we will still want to {, 
> un}hotplug the devices using the same u-TLB together.
>
> Therefore, I would clarify that this is necessary because even if we 
> have to remove all the devices together, the IOMMU driver will be 
> de-assigning them one-by-one.
>
> I would add a similar comment in the code as well.

ok, will add.


>
>
>>
>> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
>> Reviewed-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
>> ---
>> Changes V1 -> V2:
>>     - add R-b
>>     - add ASSERT() in ipmmu_utlb_disable()
>> ---
>>   xen/drivers/passthrough/arm/ipmmu-vmsa.c | 19 +++++++++++--------
>>   1 file changed, 11 insertions(+), 8 deletions(-)
>>
>> diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c 
>> b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
>> index 649b9f6..1224ea4 100644
>> --- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
>> +++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
>> @@ -127,6 +127,7 @@ struct ipmmu_vmsa_device {
>>       spinlock_t lock;    /* Protects ctx and domains[] */
>>       DECLARE_BITMAP(ctx, IPMMU_CTX_MAX);
>>       struct ipmmu_vmsa_domain *domains[IPMMU_CTX_MAX];
>> +    unsigned int utlb_refcount[IPMMU_UTLB_MAX];
>>       const struct ipmmu_features *features;
>>   };
>>   @@ -467,13 +468,12 @@ static int ipmmu_utlb_enable(struct 
>> ipmmu_vmsa_domain *domain,
>>           }
>>       }
>>   -    /*
>> -     * TODO: Reference-count the micro-TLB as several bus masters 
>> can be
>> -     * connected to the same micro-TLB.
>> -     */
>> -    ipmmu_imuasid_write(mmu, utlb, 0);
>> -    ipmmu_imuctr_write(mmu, utlb, imuctr |
>> -                       IMUCTR_TTSEL_MMU(domain->context_id) | 
>> IMUCTR_MMUEN);
>> +    if ( mmu->utlb_refcount[utlb]++ == 0 )
>> +    {
>> +        ipmmu_imuasid_write(mmu, utlb, 0);
>> +        ipmmu_imuctr_write(mmu, utlb, imuctr |
>> + IMUCTR_TTSEL_MMU(domain->context_id) | IMUCTR_MMUEN);
>> +    }
>>         return 0;
>>   }
>> @@ -484,7 +484,10 @@ static void ipmmu_utlb_disable(struct 
>> ipmmu_vmsa_domain *domain,
>>   {
>>       struct ipmmu_vmsa_device *mmu = domain->mmu;
>>   -    ipmmu_imuctr_write(mmu, utlb, 0);
>> +    ASSERT(mmu->utlb_refcount[utlb] > 0);
>> +
>> +    if ( --mmu->utlb_refcount[utlb] == 0 )
>> +        ipmmu_imuctr_write(mmu, utlb, 0);
>>   }
>>     /* Domain/Context Management */
>
> Cheers,
>
-- 
Regards,

Oleksandr Tyshchenko

Patchew 2.1-devel-b67e4c2

© 2016 - 2026 Red Hat, Inc.