iommu/vt-d: Add ESRTPS & ESIRTPS check

[PATCH 1/2] iommu/vt-d: Avoid unnecessary global IRTE cache invalidation

Posted by Lu Baolu 3 years, 6 months ago

Some VT-d hardware implementations invalidate all interrupt remapping
hardware translation caches as part of SIRTP flow. The VT-d spec adds
a ESIRTPS (Enhanced Set Interrupt Remap Table Pointer Support, section
11.4.2 in VT-d spec) capability bit to indicate this. With this bit set,
software has no need to issue the global invalidation request.

Signed-off-by: Jacob Pan <jacob.jun.pan@linux.intel.com>
Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
---
 drivers/iommu/intel/iommu.h         | 1 +
 drivers/iommu/intel/irq_remapping.c | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h
index eef1a166b855..5407d82df4d1 100644
--- a/drivers/iommu/intel/iommu.h
+++ b/drivers/iommu/intel/iommu.h
@@ -146,6 +146,7 @@
 /*
  * Decoding Capability Register
  */
+#define cap_esirtps(c)		(((c) >> 62) & 1)
 #define cap_fl5lp_support(c)	(((c) >> 60) & 1)
 #define cap_pi_support(c)	(((c) >> 59) & 1)
 #define cap_fl1gp_support(c)	(((c) >> 56) & 1)
diff --git a/drivers/iommu/intel/irq_remapping.c b/drivers/iommu/intel/irq_remapping.c
index 2e9683e970f8..b4a91fbd1c2f 100644
--- a/drivers/iommu/intel/irq_remapping.c
+++ b/drivers/iommu/intel/irq_remapping.c
@@ -494,7 +494,8 @@ static void iommu_set_irq_remapping(struct intel_iommu *iommu, int mode)
 	 * Global invalidation of interrupt entry cache to make sure the
 	 * hardware uses the new irq remapping table.
 	 */
-	qi_global_iec(iommu);
+	if (!cap_esirtps(iommu->cap))
+		qi_global_iec(iommu);
 }
 
 static void iommu_enable_irq_remapping(struct intel_iommu *iommu)
-- 
2.34.1

RE: [PATCH 1/2] iommu/vt-d: Avoid unnecessary global IRTE cache invalidation

Posted by Tian, Kevin 3 years, 6 months ago

> From: Lu Baolu <baolu.lu@linux.intel.com>
> Sent: Monday, September 19, 2022 2:25 PM
> 
> Some VT-d hardware implementations invalidate all interrupt remapping
> hardware translation caches as part of SIRTP flow. The VT-d spec adds
> a ESIRTPS (Enhanced Set Interrupt Remap Table Pointer Support, section
> 11.4.2 in VT-d spec) capability bit to indicate this. With this bit set,
> software has no need to issue the global invalidation request.
> 
> Signed-off-by: Jacob Pan <jacob.jun.pan@linux.intel.com>
> Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
> ---
>  drivers/iommu/intel/iommu.h         | 1 +
>  drivers/iommu/intel/irq_remapping.c | 3 ++-
>  2 files changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h
> index eef1a166b855..5407d82df4d1 100644
> --- a/drivers/iommu/intel/iommu.h
> +++ b/drivers/iommu/intel/iommu.h
> @@ -146,6 +146,7 @@
>  /*
>   * Decoding Capability Register
>   */
> +#define cap_esirtps(c)		(((c) >> 62) & 1)
>  #define cap_fl5lp_support(c)	(((c) >> 60) & 1)
>  #define cap_pi_support(c)	(((c) >> 59) & 1)
>  #define cap_fl1gp_support(c)	(((c) >> 56) & 1)
> diff --git a/drivers/iommu/intel/irq_remapping.c
> b/drivers/iommu/intel/irq_remapping.c
> index 2e9683e970f8..b4a91fbd1c2f 100644
> --- a/drivers/iommu/intel/irq_remapping.c
> +++ b/drivers/iommu/intel/irq_remapping.c
> @@ -494,7 +494,8 @@ static void iommu_set_irq_remapping(struct
> intel_iommu *iommu, int mode)
>  	 * Global invalidation of interrupt entry cache to make sure the
>  	 * hardware uses the new irq remapping table.
>  	 */
> -	qi_global_iec(iommu);
> +	if (!cap_esirtps(iommu->cap))
> +		qi_global_iec(iommu);
>  }
> 

Same check is also required in iommu_disable_irq_remapping().

The spec says that:

--
For implementations reporting the Enhanced Set Interrupt Remap Table
Pointer Support (ESIRTPS) field as Set, hardware performs global
invalidation on all Interrupt remapping caches as part of Interrupt
Remapping Disable operation.
--

While looking at that code I think it's problematic to invalidate interrupt
cache before disabling interrupt remapping. Presumably invalidations
should be done only after a configuration is changed, no matter it's 'enable'
or 'disable'.

Thanks
Kevin

Re: [PATCH 1/2] iommu/vt-d: Avoid unnecessary global IRTE cache invalidation

Posted by Baolu Lu 3 years, 6 months ago

On 2022/9/20 15:51, Tian, Kevin wrote:
>> From: Lu Baolu <baolu.lu@linux.intel.com>
>> Sent: Monday, September 19, 2022 2:25 PM
>>
>> Some VT-d hardware implementations invalidate all interrupt remapping
>> hardware translation caches as part of SIRTP flow. The VT-d spec adds
>> a ESIRTPS (Enhanced Set Interrupt Remap Table Pointer Support, section
>> 11.4.2 in VT-d spec) capability bit to indicate this. With this bit set,
>> software has no need to issue the global invalidation request.
>>
>> Signed-off-by: Jacob Pan <jacob.jun.pan@linux.intel.com>
>> Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
>> ---
>>   drivers/iommu/intel/iommu.h         | 1 +
>>   drivers/iommu/intel/irq_remapping.c | 3 ++-
>>   2 files changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h
>> index eef1a166b855..5407d82df4d1 100644
>> --- a/drivers/iommu/intel/iommu.h
>> +++ b/drivers/iommu/intel/iommu.h
>> @@ -146,6 +146,7 @@
>>   /*
>>    * Decoding Capability Register
>>    */
>> +#define cap_esirtps(c)		(((c) >> 62) & 1)
>>   #define cap_fl5lp_support(c)	(((c) >> 60) & 1)
>>   #define cap_pi_support(c)	(((c) >> 59) & 1)
>>   #define cap_fl1gp_support(c)	(((c) >> 56) & 1)
>> diff --git a/drivers/iommu/intel/irq_remapping.c
>> b/drivers/iommu/intel/irq_remapping.c
>> index 2e9683e970f8..b4a91fbd1c2f 100644
>> --- a/drivers/iommu/intel/irq_remapping.c
>> +++ b/drivers/iommu/intel/irq_remapping.c
>> @@ -494,7 +494,8 @@ static void iommu_set_irq_remapping(struct
>> intel_iommu *iommu, int mode)
>>   	 * Global invalidation of interrupt entry cache to make sure the
>>   	 * hardware uses the new irq remapping table.
>>   	 */
>> -	qi_global_iec(iommu);
>> +	if (!cap_esirtps(iommu->cap))
>> +		qi_global_iec(iommu);
>>   }
>>
> 
> Same check is also required in iommu_disable_irq_remapping().
> 
> The spec says that:
> 
> --
> For implementations reporting the Enhanced Set Interrupt Remap Table
> Pointer Support (ESIRTPS) field as Set, hardware performs global
> invalidation on all Interrupt remapping caches as part of Interrupt
> Remapping Disable operation.
> --

Yes. I will add the same check in iommu_disable_irq_remapping(). Thanks
for the reminding.

> While looking at that code I think it's problematic to invalidate interrupt
> cache before disabling interrupt remapping. Presumably invalidations
> should be done only after a configuration is changed, no matter it's 'enable'
> or 'disable'.

I understand your concern. The spec doesn't state the sequence in the
interrupt remapping disabling flow. Need to check it with the spec
developers before any change here.

Best regards,
baolu

Re: [PATCH 1/2] iommu/vt-d: Avoid unnecessary global IRTE cache invalidation

Posted by Jerry Snitselaar 3 years, 6 months ago

On Mon, Sep 19, 2022 at 02:25:22PM +0800, Lu Baolu wrote:
> Some VT-d hardware implementations invalidate all interrupt remapping
> hardware translation caches as part of SIRTP flow. The VT-d spec adds
> a ESIRTPS (Enhanced Set Interrupt Remap Table Pointer Support, section
> 11.4.2 in VT-d spec) capability bit to indicate this. With this bit set,
> software has no need to issue the global invalidation request.
> 
> Signed-off-by: Jacob Pan <jacob.jun.pan@linux.intel.com>
> Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>

Reviewed-by: Jerry Snitselaar <jsnitsel@redhat.com

> ---
>  drivers/iommu/intel/iommu.h         | 1 +
>  drivers/iommu/intel/irq_remapping.c | 3 ++-
>  2 files changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h
> index eef1a166b855..5407d82df4d1 100644
> --- a/drivers/iommu/intel/iommu.h
> +++ b/drivers/iommu/intel/iommu.h
> @@ -146,6 +146,7 @@
>  /*
>   * Decoding Capability Register
>   */
> +#define cap_esirtps(c)		(((c) >> 62) & 1)
>  #define cap_fl5lp_support(c)	(((c) >> 60) & 1)
>  #define cap_pi_support(c)	(((c) >> 59) & 1)
>  #define cap_fl1gp_support(c)	(((c) >> 56) & 1)
> diff --git a/drivers/iommu/intel/irq_remapping.c b/drivers/iommu/intel/irq_remapping.c
> index 2e9683e970f8..b4a91fbd1c2f 100644
> --- a/drivers/iommu/intel/irq_remapping.c
> +++ b/drivers/iommu/intel/irq_remapping.c
> @@ -494,7 +494,8 @@ static void iommu_set_irq_remapping(struct intel_iommu *iommu, int mode)
>  	 * Global invalidation of interrupt entry cache to make sure the
>  	 * hardware uses the new irq remapping table.
>  	 */
> -	qi_global_iec(iommu);
> +	if (!cap_esirtps(iommu->cap))
> +		qi_global_iec(iommu);
>  }
>  
>  static void iommu_enable_irq_remapping(struct intel_iommu *iommu)
> -- 
> 2.34.1
>