[PATCH for-5.2 v4 00/11] SMMUv3.2 Range-based TLB Invalidation Support

Eric Auger posted 11 patches 2 weeks ago
Test docker-quick@centos7 failed
Test docker-mingw@fedora failed
Test checkpatch failed
Test FreeBSD failed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20200728150815.11446-1-eric.auger@redhat.com
Maintainers: Peter Maydell <peter.maydell@linaro.org>, Eric Auger <eric.auger@redhat.com>
hw/arm/smmu-internal.h       |   8 ++
hw/arm/smmuv3-internal.h     |  10 +-
include/hw/arm/smmu-common.h |  19 +++-
include/hw/arm/smmuv3.h      |   1 +
hw/arm/smmu-common.c         | 214 ++++++++++++++++++++++++-----------
hw/arm/smmuv3.c              | 142 +++++++++++------------
hw/arm/trace-events          |  12 +-
7 files changed, 258 insertions(+), 148 deletions(-)

[PATCH for-5.2 v4 00/11] SMMUv3.2 Range-based TLB Invalidation Support

Posted by Eric Auger 2 weeks ago
SMMU3.2 brings the support of range-based TLB invalidation and
level hint. When this feature is supported, the SMMUv3 driver
is allowed to send TLB invalidations for a range of IOVAs instead
of using page based invalidation.

Implementing this feature in the virtual SMMUv3 device is
mandated for DPDK on guest use case: DPDK uses hugepage
buffers and guest sends invalidations for blocks. Without
this feature, a guest invalidation of a block of 1GB for instance
translates into a storm of page invalidations. Each of them
is trapped by the VMM and cascaded downto the physical IOMMU.
This completely stalls the execution. This integration issue
was initially reported in [1].

Now SMMUv3.2 specifies additional parameters to NH_VA and NH_VAA
stage 1 invalidation commands so we can support those extensions.

Supporting block mappings in the IOTLB look sensible in terms of
TLB entry consumption. However looking at virtio/vhost device usage,
without block mapping and without range invalidation (< 5.7 kernels
it may be less performant. However for recent guest kernels
supporting range invalidations [2], the performance should be similar.

Best Regards

Eric

This series can be found at:
https://github.com/eauger/qemu.git
branch: v5.1.0-rc1-smmuv3-ril-v4

References:
[1] [RFC v2 4/4] iommu/arm-smmu-v3: add CMD_TLBI_NH_VA_AM command
for iova range invalidation
(https://lists.linuxfoundation.org/pipermail/iommu/2017-August/023679.html

[2] 5.7+ kernels featuring
6a481a95d4c1 iommu/arm-smmu-v3: Add SMMUv3.2 range invalidation support

History:

v3 -> v4:
- use tg in hash function and directly compare key fields in the
  equal function
- don't claim 3.2 as we don't have BBML support yet
- fixed a bunch of indent issues

v2 -> v3:
- restore the Jenkins hash function and keep the key as a struct
- simplify the logic in smmu_hash_remove_by_asid_iova
- added HAD support
- expose AIDR (advertise v3.2 support) and fix IIDR offset

v1 -> v2:
- added "hw/arm/smmu: Introduce smmu_get_iotlb_key()"
- removed "[PATCH 5/9] hw/arm/smmuv3: Store the starting level in
  SMMUTransTableInfo"
- Collected Peter's R-b
- In this version the key still features TG/LVL.
- More details in individual history logs


Eric Auger (11):
  hw/arm/smmu-common: Factorize some code in smmu_ptw_64()
  hw/arm/smmu-common: Add IOTLB helpers
  hw/arm/smmu: Introduce smmu_get_iotlb_key()
  hw/arm/smmu: Introduce SMMUTLBEntry for PTW and IOTLB value
  hw/arm/smmu-common: Manage IOTLB block entries
  hw/arm/smmuv3: Introduce smmuv3_s1_range_inval() helper
  hw/arm/smmuv3: Get prepared for range invalidation
  hw/arm/smmuv3: Fix IIDR offset
  hw/arm/smmuv3: Let AIDR advertise SMMUv3.0 support
  hw/arm/smmuv3: Support HAD and advertise SMMUv3.1 support
  hw/arm/smmuv3: Advertise SMMUv3.2 range invalidation

 hw/arm/smmu-internal.h       |   8 ++
 hw/arm/smmuv3-internal.h     |  10 +-
 include/hw/arm/smmu-common.h |  19 +++-
 include/hw/arm/smmuv3.h      |   1 +
 hw/arm/smmu-common.c         | 214 ++++++++++++++++++++++++-----------
 hw/arm/smmuv3.c              | 142 +++++++++++------------
 hw/arm/trace-events          |  12 +-
 7 files changed, 258 insertions(+), 148 deletions(-)

-- 
2.21.3


Re: [PATCH for-5.2 v4 00/11] SMMUv3.2 Range-based TLB Invalidation Support

Posted by Peter Maydell 1 week ago
On Tue, 28 Jul 2020 at 16:08, Eric Auger <eric.auger@redhat.com> wrote:
>
> SMMU3.2 brings the support of range-based TLB invalidation and
> level hint. When this feature is supported, the SMMUv3 driver
> is allowed to send TLB invalidations for a range of IOVAs instead
> of using page based invalidation.
>
> Implementing this feature in the virtual SMMUv3 device is
> mandated for DPDK on guest use case: DPDK uses hugepage
> buffers and guest sends invalidations for blocks. Without
> this feature, a guest invalidation of a block of 1GB for instance
> translates into a storm of page invalidations. Each of them
> is trapped by the VMM and cascaded downto the physical IOMMU.
> This completely stalls the execution. This integration issue
> was initially reported in [1].
>
> Now SMMUv3.2 specifies additional parameters to NH_VA and NH_VAA
> stage 1 invalidation commands so we can support those extensions.
>
> Supporting block mappings in the IOTLB look sensible in terms of
> TLB entry consumption. However looking at virtio/vhost device usage,
> without block mapping and without range invalidation (< 5.7 kernels
> it may be less performant. However for recent guest kernels
> supporting range invalidations [2], the performance should be similar.

I think this is all reviewed now; I've put it on my list
of series to apply to target-arm.next in once 5.1 is out
and we reopen the trunk for 5.2.

thanks
-- PMM

Re: [PATCH for-5.2 v4 00/11] SMMUv3.2 Range-based TLB Invalidation Support

Posted by Peter Maydell 5 days ago
On Thu, 30 Jul 2020 at 14:39, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Tue, 28 Jul 2020 at 16:08, Eric Auger <eric.auger@redhat.com> wrote:
> >
> > SMMU3.2 brings the support of range-based TLB invalidation and
> > level hint. When this feature is supported, the SMMUv3 driver
> > is allowed to send TLB invalidations for a range of IOVAs instead
> > of using page based invalidation.

> I think this is all reviewed now; I've put it on my list
> of series to apply to target-arm.next in once 5.1 is out
> and we reopen the trunk for 5.2.



Applied to target-arm.next for 5.2, thanks.

-- PMM