The SMMUv3 supports two stages of translation. Each stage of translation can be
independently enabled. An incoming address is logically translated from VA to
IPA in stage 1, then the IPA is input to stage 2 which translates the IPA to
the output PA.

Stage 1 is intended to be used by a software entity to provide isolation or
translation to buffers within the entity, for example DMA isolation within an
OS. Stage 2 is intended to be available in systems supporting the
Virtualization Extensions and is intended to virtualize device DMA to guest VM
address spaces. When both stage 1 and stage 2 are enabled, the translation
configuration is called nested.

Stage 1 translation support is required to provide isolation between different
devices within OS. XEN already supports Stage 2 translation but there is no
support for Stage 1 translation. The goal of this work is to support Stage 1
translation for XEN guests. Stage 1 has to be configured within the guest to
provide isolation.

We cannot trust the guest OS to control the SMMUv3 hardware directly as
compromised guest OS can corrupt the SMMUv3 configuration and make the system
vulnerable. The guest gets the ownership of the stage 1 page tables and also
owns stage 1 configuration structures. The XEN handles the root configuration
structure (for security reasons), including the stage 2 configuration.

XEN will emulate the SMMUv3 hardware and exposes the virtual SMMUv3 to the
guest. Guest can use the native SMMUv3 driver to configure the stage 1
translation. When the guest configures the SMMUv3 for Stage 1, XEN will trap
the access and configure hardware.

SMMUv3 Driver(Guest OS) -> Configure the Stage-1 translation ->
XEN trap access -> XEN SMMUv3 driver configure the HW.

SMMUv3 driver has to be updated to support the Stage-1 translation support
based on work done by the KVM team to support Nested Stage translation:
https://github.com/eauger/linux/commits/v5.11-stallv12-2stage-v14
https://lwn.net/Articles/852299/

As the stage 1 translation is configured by XEN on behalf of the guest,
translation faults encountered during the translation process need to be
propagated up to the guest and re-injected into the guest. When the guest
invalidates stage 1 related caches, invalidations must be forwarded to the
SMMUv3 hardware.

This patch series is sent as RFC to get the initial feedback from the
community. This patch series consists of 21 patches which is a big number for
the reviewer to review the patches but to understand the feature end-to-end we
thought of sending this as a big series. Once we will get initial feedback, we
will divide the series into a small number of patches for review.

Jean-Philippe Brucker (1):
  xen/arm: smmuv3: Maintain a SID->device structure

Rahul Singh (20):
  xen/arm: smmuv3: Add support for stage-1 and nested stage translation
  xen/arm: smmuv3: Alloc io_domain for each device
  xen/arm: vIOMMU: add generic vIOMMU framework
  xen/arm: vsmmuv3: Add dummy support for virtual SMMUv3 for guests
  xen/domctl: Add XEN_DOMCTL_CONFIG_VIOMMU_* and viommu config param
  xen/arm: vIOMMU: Add cmdline boot option "viommu = <boolean>"
  xen/arm: vsmmuv3: Add support for registers emulation
  xen/arm: vsmmuv3: Add support for cmdqueue handling
  xen/arm: vsmmuv3: Add support for command CMD_CFGI_STE
  xen/arm: vsmmuv3: Attach Stage-1 configuration to SMMUv3 hardware
  xen/arm: vsmmuv3: Add support for event queue and global error
  xen/arm: vsmmuv3: Add "iommus" property node for dom0 devices
  xen/arm: vIOMMU: IOMMU device tree node for dom0
  xen/arm: vsmmuv3: Emulated SMMUv3 device tree node for dom0less
  arm/libxl: vsmmuv3: Emulated SMMUv3 device tree node in libxl
  xen/arm: vsmmuv3: Alloc virq for virtual SMMUv3
  xen/arm: iommu: skip the iommu-map property for PCI devices
  xen/arm: vsmmuv3: Add support to send stage-1 event to guest
  libxl/arm: vIOMMU: Modify the partial device tree for iommus
  xen/arm: vIOMMU: Modify the partial device tree for dom0less

 docs/man/xl.cfg.5.pod.in               |  11 +
 docs/misc/xen-command-line.pandoc      |   7 +
 tools/golang/xenlight/helpers.gen.go   |   2 +
 tools/golang/xenlight/types.gen.go     |   1 +
 tools/include/libxl.h                  |   5 +
 tools/libs/light/libxl_arm.c           | 121 +++-
 tools/libs/light/libxl_types.idl       |   6 +
 tools/xl/xl_parse.c                    |   9 +
 xen/arch/arm/domain.c                  |  18 +
 xen/arch/arm/domain_build.c            | 213 +++++-
 xen/arch/arm/include/asm/domain.h      |   4 +
 xen/arch/arm/include/asm/viommu.h      | 102 +++
 xen/drivers/passthrough/Kconfig        |  14 +
 xen/drivers/passthrough/arm/Makefile   |   2 +
 xen/drivers/passthrough/arm/smmu-v3.c  | 370 +++++++++--
 xen/drivers/passthrough/arm/smmu-v3.h  |  43 +-
 xen/drivers/passthrough/arm/viommu.c   |  87 +++
 xen/drivers/passthrough/arm/vsmmu-v3.c | 887 +++++++++++++++++++++++++
 xen/drivers/passthrough/arm/vsmmu-v3.h |  32 +
 xen/include/public/arch-arm.h          |  14 +-
 xen/include/public/device_tree_defs.h  |   1 +
 xen/include/xen/iommu.h                |  14 +
 22 files changed, 1913 insertions(+), 50 deletions(-)
 create mode 100644 xen/arch/arm/include/asm/viommu.h
 create mode 100644 xen/drivers/passthrough/arm/viommu.c
 create mode 100644 xen/drivers/passthrough/arm/vsmmu-v3.c
 create mode 100644 xen/drivers/passthrough/arm/vsmmu-v3.h

--
2.25.1

Hi Rahul,

On 01/12/2022 16:02, Rahul Singh wrote:
> This patch series is sent as RFC to get the initial feedback from the
> community. This patch series consists of 21 patches which is a big number for
> the reviewer to review the patches but to understand the feature end-to-end we
> thought of sending this as a big series. Once we will get initial feedback, we
> will divide the series into a small number of patches for review.

 From the cover letter, it is not clear to me what sort of input you are 
expecting for the RFC. Is this about the design itself?

If so, I think it would be more helpful to write an high level document 
on how you plan to emulate the vIOMMU in Xen. So there is one place to 
read/agree/verify rather than trying to collate all the information from 
the 20+ patches.

Briefly skimming through I think the main things that need to be 
addressed in order of priority:
   - How to secure the vIOMMU
   - 1 vs multiple vIOMMU

The questions are very similar to the vITS because the SMMUv3 is based 
on a queue. And given you are selling this feature as a security one, I 
don't think we can go forward with the review without any 
understanding/agreement on what needs to be implemented in order to have 
a safe/secure vIOMMU.

Cheers,

-- 
Julien Grall

On Sat, 3 Dec 2022, Julien Grall wrote:
> On 01/12/2022 16:02, Rahul Singh wrote:
> > This patch series is sent as RFC to get the initial feedback from the
> > community. This patch series consists of 21 patches which is a big number
> > for
> > the reviewer to review the patches but to understand the feature end-to-end
> > we
> > thought of sending this as a big series. Once we will get initial feedback,
> > we
> > will divide the series into a small number of patches for review.
> 
> From the cover letter, it is not clear to me what sort of input you are
> expecting for the RFC. Is this about the design itself?
> 
> If so, I think it would be more helpful to write an high level document on how
> you plan to emulate the vIOMMU in Xen. So there is one place to
> read/agree/verify rather than trying to collate all the information from the
> 20+ patches.
> 
> Briefly skimming through I think the main things that need to be addressed in
> order of priority:
>   - How to secure the vIOMMU
>   - 1 vs multiple vIOMMU
> 
> The questions are very similar to the vITS because the SMMUv3 is based on a
> queue. And given you are selling this feature as a security one, I don't think
> we can go forward with the review without any understanding/agreement on what
> needs to be implemented in order to have a safe/secure vIOMMU.

I think we are all aligned here, but let me try to clarify further.

As the vIOMMU is exposed to the guest, and exposing a queue-based
interface to the guest is not simple, it would be good to clarify in a
document the following points:

- how is the queue exposed to the guest
- how are guest-inputs sanitized
- how do the virtual queue resources map to the physical queue
  resources
- lifecycle of the resource mappings
- any memory allocations triggered by guest actions and their lifecycle

It is difficult to extrapole these details from 21 patches. Having these
key detailed written down in the 0/21 email would greatly help with the
review. It would make the review go a lot faster.

Hi Stefano,Julien

> On 5 Dec 2022, at 9:43 pm, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Sat, 3 Dec 2022, Julien Grall wrote:
>> On 01/12/2022 16:02, Rahul Singh wrote:
>>> This patch series is sent as RFC to get the initial feedback from the
>>> community. This patch series consists of 21 patches which is a big number
>>> for
>>> the reviewer to review the patches but to understand the feature end-to-end
>>> we
>>> thought of sending this as a big series. Once we will get initial feedback,
>>> we
>>> will divide the series into a small number of patches for review.
>> 
>> From the cover letter, it is not clear to me what sort of input you are
>> expecting for the RFC. Is this about the design itself?
>> 
>> If so, I think it would be more helpful to write an high level document on how
>> you plan to emulate the vIOMMU in Xen. So there is one place to
>> read/agree/verify rather than trying to collate all the information from the
>> 20+ patches.
>> 
>> Briefly skimming through I think the main things that need to be addressed in
>> order of priority:
>>  - How to secure the vIOMMU
>>  - 1 vs multiple vIOMMU
>> 
>> The questions are very similar to the vITS because the SMMUv3 is based on a
>> queue. And given you are selling this feature as a security one, I don't think
>> we can go forward with the review without any understanding/agreement on what
>> needs to be implemented in order to have a safe/secure vIOMMU.
> 
> I think we are all aligned here, but let me try to clarify further.
> 
> As the vIOMMU is exposed to the guest, and exposing a queue-based
> interface to the guest is not simple, it would be good to clarify in a
> document the following points:
> 
> - how is the queue exposed to the guest
> - how are guest-inputs sanitized
> - how do the virtual queue resources map to the physical queue
>  resources
> - lifecycle of the resource mappings
> - any memory allocations triggered by guest actions and their lifecycle
> 
> It is difficult to extrapole these details from 21 patches. Having these
> key detailed written down in the 0/21 email would greatly help with the
> review. It would make the review go a lot faster.

Ack. I will send the design docs by next week that will include all the requested
information.

Regards.
Rahul

Hi Rahul,

On 01/12/2022 17:02, Rahul Singh wrote:
> 
> 
> The SMMUv3 supports two stages of translation. Each stage of translation can be
> independently enabled. An incoming address is logically translated from VA to
> IPA in stage 1, then the IPA is input to stage 2 which translates the IPA to
> the output PA.
> 
> Stage 1 is intended to be used by a software entity to provide isolation or
> translation to buffers within the entity, for example DMA isolation within an
> OS. Stage 2 is intended to be available in systems supporting the
> Virtualization Extensions and is intended to virtualize device DMA to guest VM
> address spaces. When both stage 1 and stage 2 are enabled, the translation
> configuration is called nested.
> 
> Stage 1 translation support is required to provide isolation between different
> devices within OS. XEN already supports Stage 2 translation but there is no
> support for Stage 1 translation. The goal of this work is to support Stage 1
> translation for XEN guests. Stage 1 has to be configured within the guest to
> provide isolation.
> 
> We cannot trust the guest OS to control the SMMUv3 hardware directly as
> compromised guest OS can corrupt the SMMUv3 configuration and make the system
> vulnerable. The guest gets the ownership of the stage 1 page tables and also
> owns stage 1 configuration structures. The XEN handles the root configuration
> structure (for security reasons), including the stage 2 configuration.
> 
> XEN will emulate the SMMUv3 hardware and exposes the virtual SMMUv3 to the
> guest. Guest can use the native SMMUv3 driver to configure the stage 1
> translation. When the guest configures the SMMUv3 for Stage 1, XEN will trap
> the access and configure hardware.
> 
> SMMUv3 Driver(Guest OS) -> Configure the Stage-1 translation ->
> XEN trap access -> XEN SMMUv3 driver configure the HW.
> 
> SMMUv3 driver has to be updated to support the Stage-1 translation support
> based on work done by the KVM team to support Nested Stage translation:
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Feauger%2Flinux%2Fcommits%2Fv5.11-stallv12-2stage-v14&data=05%7C01%7Cmichal.orzel%40amd.com%7Cecb9075a29974c8f5ad608dad3b5916f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638055074068482160%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PdK4%2Bsps3%2FdXYJUDv3iCy%2Byaqbh1bOVb1AFzTtx1nts%3D&reserved=0
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F852299%2F&data=05%7C01%7Cmichal.orzel%40amd.com%7Cecb9075a29974c8f5ad608dad3b5916f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638055074068482160%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5Kp7023HiA4Qbfi28wcPL20JyC2xLwwiyEUZcxTSCOA%3D&reserved=0
> 
> As the stage 1 translation is configured by XEN on behalf of the guest,
> translation faults encountered during the translation process need to be
> propagated up to the guest and re-injected into the guest. When the guest
> invalidates stage 1 related caches, invalidations must be forwarded to the
> SMMUv3 hardware.
> 
> This patch series is sent as RFC to get the initial feedback from the
> community. This patch series consists of 21 patches which is a big number for
> the reviewer to review the patches but to understand the feature end-to-end we
> thought of sending this as a big series. Once we will get initial feedback, we
> will divide the series into a small number of patches for review.

Due to the very limited availability of the board we have, that is equipped with
DMA platform devices and SMMUv3 (I know that you tested PCI use case thoroughly),
I managed for now to do the testing on dom0 only.

By commenting out the code in Linux responsible for setting up Xen SWIOTLB DMA ops, I was able
to successfully verify the nested SMMU working properly for DMA platform devices on the
example of using ZDMA. Both the upstream dmatest client app as well as the VFIO user space driver
that I wrote for ZDMA passed the test!

I added some logs to verify the sync up between Linux and Xen during a VFIO test:

LINUX: SMMUv3: Setting the STE S1 Config 0x1405c000 for SID=0x210
XEN: vSMMUv3: guest config=ARM_SMMU_DOMAIN_NESTED
XEN: SMMUv3: Setting the STE S1 Config 0x1405c000 for SID=0x210

Before transfer example:
 src value: 0xdb71faf
 dst value: 0
Waiting for transfer completion...
After transfer example:
 src value: 0xdb71faf
 dst value: 0xdb71faf
TEST RESULT: PASS

LINUX: SMMUv3: Setting the STE S1 Config 0x12502000 for SID=0x210
XEN: vSMMUv3: guest config=ARM_SMMU_DOMAIN_NESTED
XEN: SMMUv3: Setting the STE S1 Config 0x12502000 for SID=0x210

~Michal

Hi Rahul,

On 02/12/2022 11:59, Michal Orzel wrote:
> Hi Rahul,
> 
> On 01/12/2022 17:02, Rahul Singh wrote:
>>
>>
>> The SMMUv3 supports two stages of translation. Each stage of translation can be
>> independently enabled. An incoming address is logically translated from VA to
>> IPA in stage 1, then the IPA is input to stage 2 which translates the IPA to
>> the output PA.
>>
>> Stage 1 is intended to be used by a software entity to provide isolation or
>> translation to buffers within the entity, for example DMA isolation within an
>> OS. Stage 2 is intended to be available in systems supporting the
>> Virtualization Extensions and is intended to virtualize device DMA to guest VM
>> address spaces. When both stage 1 and stage 2 are enabled, the translation
>> configuration is called nested.
>>
>> Stage 1 translation support is required to provide isolation between different
>> devices within OS. XEN already supports Stage 2 translation but there is no
>> support for Stage 1 translation. The goal of this work is to support Stage 1
>> translation for XEN guests. Stage 1 has to be configured within the guest to
>> provide isolation.
>>
>> We cannot trust the guest OS to control the SMMUv3 hardware directly as
>> compromised guest OS can corrupt the SMMUv3 configuration and make the system
>> vulnerable. The guest gets the ownership of the stage 1 page tables and also
>> owns stage 1 configuration structures. The XEN handles the root configuration
>> structure (for security reasons), including the stage 2 configuration.
>>
>> XEN will emulate the SMMUv3 hardware and exposes the virtual SMMUv3 to the
>> guest. Guest can use the native SMMUv3 driver to configure the stage 1
>> translation. When the guest configures the SMMUv3 for Stage 1, XEN will trap
>> the access and configure hardware.
>>
>> SMMUv3 Driver(Guest OS) -> Configure the Stage-1 translation ->
>> XEN trap access -> XEN SMMUv3 driver configure the HW.
>>
>> SMMUv3 driver has to be updated to support the Stage-1 translation support
>> based on work done by the KVM team to support Nested Stage translation:
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Feauger%2Flinux%2Fcommits%2Fv5.11-stallv12-2stage-v14&data=05%7C01%7Cmichal.orzel%40amd.com%7Cecb9075a29974c8f5ad608dad3b5916f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638055074068482160%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PdK4%2Bsps3%2FdXYJUDv3iCy%2Byaqbh1bOVb1AFzTtx1nts%3D&reserved=0
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F852299%2F&data=05%7C01%7Cmichal.orzel%40amd.com%7Cecb9075a29974c8f5ad608dad3b5916f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638055074068482160%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5Kp7023HiA4Qbfi28wcPL20JyC2xLwwiyEUZcxTSCOA%3D&reserved=0
>>
>> As the stage 1 translation is configured by XEN on behalf of the guest,
>> translation faults encountered during the translation process need to be
>> propagated up to the guest and re-injected into the guest. When the guest
>> invalidates stage 1 related caches, invalidations must be forwarded to the
>> SMMUv3 hardware.
>>
>> This patch series is sent as RFC to get the initial feedback from the
>> community. This patch series consists of 21 patches which is a big number for
>> the reviewer to review the patches but to understand the feature end-to-end we
>> thought of sending this as a big series. Once we will get initial feedback, we
>> will divide the series into a small number of patches for review.
> 
> Due to the very limited availability of the board we have, that is equipped with
> DMA platform devices and SMMUv3 (I know that you tested PCI use case thoroughly),
> I managed for now to do the testing on dom0 only.
> 
> By commenting out the code in Linux responsible for setting up Xen SWIOTLB DMA ops, I was able
> to successfully verify the nested SMMU working properly for DMA platform devices on the
> example of using ZDMA. Both the upstream dmatest client app as well as the VFIO user space driver
> that I wrote for ZDMA passed the test!
> 
> I added some logs to verify the sync up between Linux and Xen during a VFIO test:
> 
> LINUX: SMMUv3: Setting the STE S1 Config 0x1405c000 for SID=0x210
> XEN: vSMMUv3: guest config=ARM_SMMU_DOMAIN_NESTED
> XEN: SMMUv3: Setting the STE S1 Config 0x1405c000 for SID=0x210
> 
> Before transfer example:
>  src value: 0xdb71faf
>  dst value: 0
> Waiting for transfer completion...
> After transfer example:
>  src value: 0xdb71faf
>  dst value: 0xdb71faf
> TEST RESULT: PASS
> 
> LINUX: SMMUv3: Setting the STE S1 Config 0x12502000 for SID=0x210
> XEN: vSMMUv3: guest config=ARM_SMMU_DOMAIN_NESTED
> XEN: SMMUv3: Setting the STE S1 Config 0x12502000 for SID=0x210

I finished testing this series by also covering dom0less and xl domUs.
Tests passed so good job!
I do not have access to any board with more than one IOMMU so I could not validate
this behavior.

~Michal

Hi Michal,

> On 6 Dec 2022, at 9:33 am, Michal Orzel <michal.orzel@amd.com> wrote:
> 
> Hi Rahul,
> 
> On 02/12/2022 11:59, Michal Orzel wrote:
>> Hi Rahul,
>> 
>> On 01/12/2022 17:02, Rahul Singh wrote:
>>> 
>>> 
>>> The SMMUv3 supports two stages of translation. Each stage of translation can be
>>> independently enabled. An incoming address is logically translated from VA to
>>> IPA in stage 1, then the IPA is input to stage 2 which translates the IPA to
>>> the output PA.
>>> 
>>> Stage 1 is intended to be used by a software entity to provide isolation or
>>> translation to buffers within the entity, for example DMA isolation within an
>>> OS. Stage 2 is intended to be available in systems supporting the
>>> Virtualization Extensions and is intended to virtualize device DMA to guest VM
>>> address spaces. When both stage 1 and stage 2 are enabled, the translation
>>> configuration is called nested.
>>> 
>>> Stage 1 translation support is required to provide isolation between different
>>> devices within OS. XEN already supports Stage 2 translation but there is no
>>> support for Stage 1 translation. The goal of this work is to support Stage 1
>>> translation for XEN guests. Stage 1 has to be configured within the guest to
>>> provide isolation.
>>> 
>>> We cannot trust the guest OS to control the SMMUv3 hardware directly as
>>> compromised guest OS can corrupt the SMMUv3 configuration and make the system
>>> vulnerable. The guest gets the ownership of the stage 1 page tables and also
>>> owns stage 1 configuration structures. The XEN handles the root configuration
>>> structure (for security reasons), including the stage 2 configuration.
>>> 
>>> XEN will emulate the SMMUv3 hardware and exposes the virtual SMMUv3 to the
>>> guest. Guest can use the native SMMUv3 driver to configure the stage 1
>>> translation. When the guest configures the SMMUv3 for Stage 1, XEN will trap
>>> the access and configure hardware.
>>> 
>>> SMMUv3 Driver(Guest OS) -> Configure the Stage-1 translation ->
>>> XEN trap access -> XEN SMMUv3 driver configure the HW.
>>> 
>>> SMMUv3 driver has to be updated to support the Stage-1 translation support
>>> based on work done by the KVM team to support Nested Stage translation:
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Feauger%2Flinux%2Fcommits%2Fv5.11-stallv12-2stage-v14&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cecb9075a29974c8f5ad608dad3b5916f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638055074068482160%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=PdK4%2Bsps3%2FdXYJUDv3iCy%2Byaqbh1bOVb1AFzTtx1nts%3D&amp;reserved=0
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F852299%2F&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cecb9075a29974c8f5ad608dad3b5916f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638055074068482160%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=5Kp7023HiA4Qbfi28wcPL20JyC2xLwwiyEUZcxTSCOA%3D&amp;reserved=0
>>> 
>>> As the stage 1 translation is configured by XEN on behalf of the guest,
>>> translation faults encountered during the translation process need to be
>>> propagated up to the guest and re-injected into the guest. When the guest
>>> invalidates stage 1 related caches, invalidations must be forwarded to the
>>> SMMUv3 hardware.
>>> 
>>> This patch series is sent as RFC to get the initial feedback from the
>>> community. This patch series consists of 21 patches which is a big number for
>>> the reviewer to review the patches but to understand the feature end-to-end we
>>> thought of sending this as a big series. Once we will get initial feedback, we
>>> will divide the series into a small number of patches for review.
>> 
>> Due to the very limited availability of the board we have, that is equipped with
>> DMA platform devices and SMMUv3 (I know that you tested PCI use case thoroughly),
>> I managed for now to do the testing on dom0 only.
>> 
>> By commenting out the code in Linux responsible for setting up Xen SWIOTLB DMA ops, I was able
>> to successfully verify the nested SMMU working properly for DMA platform devices on the
>> example of using ZDMA. Both the upstream dmatest client app as well as the VFIO user space driver
>> that I wrote for ZDMA passed the test!
>> 
>> I added some logs to verify the sync up between Linux and Xen during a VFIO test:
>> 
>> LINUX: SMMUv3: Setting the STE S1 Config 0x1405c000 for SID=0x210
>> XEN: vSMMUv3: guest config=ARM_SMMU_DOMAIN_NESTED
>> XEN: SMMUv3: Setting the STE S1 Config 0x1405c000 for SID=0x210
>> 
>> Before transfer example:
>> src value: 0xdb71faf
>> dst value: 0
>> Waiting for transfer completion...
>> After transfer example:
>> src value: 0xdb71faf
>> dst value: 0xdb71faf
>> TEST RESULT: PASS
>> 
>> LINUX: SMMUv3: Setting the STE S1 Config 0x12502000 for SID=0x210
>> XEN: vSMMUv3: guest config=ARM_SMMU_DOMAIN_NESTED
>> XEN: SMMUv3: Setting the STE S1 Config 0x12502000 for SID=0x210
> 
> I finished testing this series by also covering dom0less and xl domUs.
> Tests passed so good job!
> I do not have access to any board with more than one IOMMU so I could not validate
> this behavior.
> 

Thank you much appreciated for testing the series as it helps a lot to confirm that this feature is
working as expected.

Regards,
Rahul