With a system having multiple SMMU physical instances, multiple vSMMUs can
be allocated for a VM that deivces behind different SMMUs are assigned to.
In such a use case, the IPA->PA mappings (i.e. the stage-2 I/O page table)
can be shared across the vSMMU instances.

With a shareable S2 parent domain, it is more natural to store a vmid per
vSMMU instance v.s. a shared S2 domain, since each physical SMMU instance
maintains its own vmid bitmap.

Have a pair of patches getting the functions ready for the vmid migration.
Decouple the vmid from S2 parent domains and move its allocation to vSMMU
instances. Note that a regular S2 domain (!nest_parent) has to retain the
s2_cfg and vmid for non-nesting use cases, if the SMMU HW doesn't support
stage 1. Then, an S2 invalidation has to be iterated for all the vmids in
the vSMMU list introduced in the S2 parent domain.

This is on Github:
https://github.com/nicolinc/iommufd/commits/smmuv3_vmid-v1

To test it with RMR:
https://github.com/nicolinc/iommufd/commits/smmuv3_vmid-v1-with-rmr
Pairing QEMU branch:
https://github.com/nicolinc/qemu/commits/wip/for_smmuv3_vmid-v1

Thanks
Nicolin

Nicolin Chen (4):
  iommu/arm-smmu-v3: Pass in vmid to arm_smmu_make_s2_domain_ste()
  iommu/arm-smmu-v3: Share arm_smmu_cmdq_issue_cmd_with_sync() with
    vsmmu
  iommu/arm-smmu-v3: Decouple vmid from S2 nest_parent domain
  iommu/arm-smmu-v3-iommufd: Allow a shared s2_parent to allocate vSMMU

 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   | 14 ++++-
 .../arm/arm-smmu-v3/arm-smmu-v3-iommufd.c     | 44 +++++++++++---
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c  |  3 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 57 ++++++++++++++-----
 4 files changed, 93 insertions(+), 25 deletions(-)


base-commit: 7eb172143d5508b4da468ed59ee857c6e5e01da6
-- 
2.43.0

On Tue, Mar 04, 2025 at 09:03:59PM -0800, Nicolin Chen wrote:

> Have a pair of patches getting the functions ready for the vmid migration.
> Decouple the vmid from S2 parent domains and move its allocation to vSMMU
> instances. Note that a regular S2 domain (!nest_parent) has to retain the
> s2_cfg and vmid for non-nesting use cases, if the SMMU HW doesn't support
> stage 1. Then, an S2 invalidation has to be iterated for all the vmids in
> the vSMMU list introduced in the S2 parent domain.

I was planning to also fix the S2 to be able to attach to multiple
IOMMU instances at the same time as getting VMID to the viommu.. It
doesn't quite make sense to me that viommu would allow multi-attach
but the normal cases wouldn't. Did you find a shortcut?

The main issue here was to do this without degrading the invalidation
workflow and harming SVA and DMA API performance..

Jason

On Wed, Mar 05, 2025 at 12:54:52PM -0400, Jason Gunthorpe wrote:
> On Tue, Mar 04, 2025 at 09:03:59PM -0800, Nicolin Chen wrote:
> 
> > Have a pair of patches getting the functions ready for the vmid migration.
> > Decouple the vmid from S2 parent domains and move its allocation to vSMMU
> > instances. Note that a regular S2 domain (!nest_parent) has to retain the
> > s2_cfg and vmid for non-nesting use cases, if the SMMU HW doesn't support
> > stage 1. Then, an S2 invalidation has to be iterated for all the vmids in
> > the vSMMU list introduced in the S2 parent domain.
> 
> I was planning to also fix the S2 to be able to attach to multiple
> IOMMU instances at the same time as getting VMID to the viommu.. It
> doesn't quite make sense to me that viommu would allow multi-attach
> but the normal cases wouldn't. Did you find a shortcut?

Hmm, not quite following the question. You mean vIOMMU attaching to
multiple S2 domains?

> The main issue here was to do this without degrading the invalidation
> workflow and harming SVA and DMA API performance..

I see.. The ATC_INV and TLBI_NH_ALL will be increased for the shared
S2, even if the S2 change is about a device behind one physical SMMU.

Thanks
Nicolin

On Wed, Mar 05, 2025 at 10:23:36AM -0800, Nicolin Chen wrote:
> On Wed, Mar 05, 2025 at 12:54:52PM -0400, Jason Gunthorpe wrote:
> > On Tue, Mar 04, 2025 at 09:03:59PM -0800, Nicolin Chen wrote:
> > 
> > > Have a pair of patches getting the functions ready for the vmid migration.
> > > Decouple the vmid from S2 parent domains and move its allocation to vSMMU
> > > instances. Note that a regular S2 domain (!nest_parent) has to retain the
> > > s2_cfg and vmid for non-nesting use cases, if the SMMU HW doesn't support
> > > stage 1. Then, an S2 invalidation has to be iterated for all the vmids in
> > > the vSMMU list introduced in the S2 parent domain.
> > 
> > I was planning to also fix the S2 to be able to attach to multiple
> > IOMMU instances at the same time as getting VMID to the viommu.. It
> > doesn't quite make sense to me that viommu would allow multi-attach
> > but the normal cases wouldn't. Did you find a shortcut?
> 
> Hmm, not quite following the question. You mean vIOMMU attaching to
> multiple S2 domains?

I mean a normal S2 domain attaching to multiple devices on multiple
instances.

Jason

On Wed, Mar 05, 2025 at 02:31:51PM -0400, Jason Gunthorpe wrote:
> On Wed, Mar 05, 2025 at 10:23:36AM -0800, Nicolin Chen wrote:
> > On Wed, Mar 05, 2025 at 12:54:52PM -0400, Jason Gunthorpe wrote:
> > > On Tue, Mar 04, 2025 at 09:03:59PM -0800, Nicolin Chen wrote:
> > > 
> > > > Have a pair of patches getting the functions ready for the vmid migration.
> > > > Decouple the vmid from S2 parent domains and move its allocation to vSMMU
> > > > instances. Note that a regular S2 domain (!nest_parent) has to retain the
> > > > s2_cfg and vmid for non-nesting use cases, if the SMMU HW doesn't support
> > > > stage 1. Then, an S2 invalidation has to be iterated for all the vmids in
> > > > the vSMMU list introduced in the S2 parent domain.
> > > 
> > > I was planning to also fix the S2 to be able to attach to multiple
> > > IOMMU instances at the same time as getting VMID to the viommu.. It
> > > doesn't quite make sense to me that viommu would allow multi-attach
> > > but the normal cases wouldn't. Did you find a shortcut?
> > 
> > Hmm, not quite following the question. You mean vIOMMU attaching to
> > multiple S2 domains?
> 
> I mean a normal S2 domain attaching to multiple devices on multiple
> instances.

Oh, I haven't thought about a !nest_parent S2 case.

A nest_parent case will not allow devices to attach the S2 but
always to a proxy nested S1 as we discussed previously. So, I
think the implementation could be very different?

Thanks
Nicolin

On Wed, Mar 05, 2025 at 10:51:38AM -0800, Nicolin Chen wrote:

> > I mean a normal S2 domain attaching to multiple devices on multiple
> > instances.
> 
> Oh, I haven't thought about a !nest_parent S2 case.
> 
> A nest_parent case will not allow devices to attach the S2 but
> always to a proxy nested S1 as we discussed previously. So, I
> think the implementation could be very different?

It could, and that is what you show here

But also, it could be the same implementation.

Jason

On Wed, Mar 05, 2025 at 03:29:50PM -0400, Jason Gunthorpe wrote:
> On Wed, Mar 05, 2025 at 10:51:38AM -0800, Nicolin Chen wrote:
> 
> > > I mean a normal S2 domain attaching to multiple devices on multiple
> > > instances.
> > 
> > Oh, I haven't thought about a !nest_parent S2 case.
> > 
> > A nest_parent case will not allow devices to attach the S2 but
> > always to a proxy nested S1 as we discussed previously. So, I
> > think the implementation could be very different?
> 
> It could, and that is what you show here
> 
> But also, it could be the same implementation.

OK.

Obviously I underestimated this part of work.

Implementation aside, this at lease needs some assessment of any
invalidation impact. I should do some extra tracing.

Thanks
Nicolin