Documentation: x86: add documenation for AMD IOMMU

[PATCH] Documentation: x86: add documenation for AMD IOMMU

Posted by Alex Deucher 4 years, 3 months ago

Add preliminary documenation for AMD IOMMU.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 Documentation/x86/amd-iommu.rst   | 85 +++++++++++++++++++++++++++++++
 Documentation/x86/index.rst       |  1 +
 Documentation/x86/intel-iommu.rst |  2 +-
 3 files changed, 87 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/x86/amd-iommu.rst

diff --git a/Documentation/x86/amd-iommu.rst b/Documentation/x86/amd-iommu.rst
new file mode 100644
index 000000000000..89820140fefa
--- /dev/null
+++ b/Documentation/x86/amd-iommu.rst
@@ -0,0 +1,85 @@
+=================
+AMD IOMMU Support
+=================
+
+The architecture spec can be obtained from the below location.
+
+https://www.amd.com/system/files/TechDocs/48882_IOMMU.pdf
+
+This guide gives a quick cheat sheet for some basic understanding.
+
+Some Keywords
+
+- IVRS - I/O Virtualization Reporting Structure
+- IVDB - I/O Virtualization Definition Block
+- IVHD - I/O Virtualization Hardware Definition
+- IOVA - I/O Virtual Address.
+
+Basic stuff
+-----------
+
+ACPI enumerates and lists the different DMA engines in the platform, and
+device scope relationships between PCI devices and which DMA engine controls
+them.
+
+What is IVRS?
+-------------
+
+The architecture defines an ACPI-compatible data structure called an I/O
+Virtualization Reporting Structure (IVRS) that is used to convey information
+related to I/O virtualization to system software.  The IVRS describes the
+configuration and capabilities of the IOMMUs contained in the platform as
+well as information about the devices that each IOMMU virtualizes.
+
+The IVRS provides information about the following:
+- IOMMUs present in the platform including their capabilities and proper configuration
+- System I/O topology relevant to each IOMMU
+- Peripheral devices that cannot be otherwise enumerated
+- Memory regions used by SMI/SMM, platform firmware, and platform hardware. These are
+generally exclusion ranges to be configured by system software.
+
+How is IOVA generated?
+----------------------
+
+Well behaved drivers call pci_map_*() calls before sending command to device
+that needs to perform DMA. Once DMA is completed and mapping is no longer
+required, device performs a pci_unmap_*() calls to unmap the region.
+
+The AMD IOMMU driver allocates a virtual address per domain. Each PCIE
+device has its own domain (hence protection). Devices under p2p bridges
+share the virtual address with all devices under the p2p bridge due to
+transaction id aliasing for p2p bridges.
+
+IOVA generation is pretty generic. We used the same technique as vmalloc()
+but these are not global address spaces, but separate for each domain.
+Different DMA engines may support different number of domains.
+
+
+Fault reporting
+---------------
+When errors are reported, the DMA engine signals via an interrupt. The fault
+reason and device that caused it with fault reason is printed on console.
+
+See below for sample.
+
+
+Boot Message Sample
+-------------------
+
+Something like this gets printed indicating presence of the IOMMU.
+
+	iommu: Default domain type: Translated
+	iommu: DMA domain TLB invalidation policy: lazy mode
+
+
+PCI-DMA: Using AMD IOMMU
+------------------------
+
+Fault reporting
+^^^^^^^^^^^^^^^
+
+::
+
+	AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0007 address=0xffffc02000 flags=0x0000]
+	AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.0 domain=0x0007 address=0xffffc02000 flags=0x0000]
+
diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst
index f498f1d36cd3..15711134eb68 100644
--- a/Documentation/x86/index.rst
+++ b/Documentation/x86/index.rst
@@ -22,6 +22,7 @@ x86-specific Documentation
    mtrr
    pat
    intel-iommu
+   amd-iommu
    intel_txt
    amd-memory-encryption
    pti
diff --git a/Documentation/x86/intel-iommu.rst b/Documentation/x86/intel-iommu.rst
index 099f13d51d5f..4d3391c7bd3f 100644
--- a/Documentation/x86/intel-iommu.rst
+++ b/Documentation/x86/intel-iommu.rst
@@ -1,5 +1,5 @@
 ===================
-Linux IOMMU Support
+Intel IOMMU Support
 ===================
 
 The architecture spec can be obtained from the below location.
-- 
2.35.1

Re: [PATCH] Documentation: x86: add documenation for AMD IOMMU

Posted by Robin Murphy 4 years, 3 months ago

On 2022-03-08 19:04, Alex Deucher via iommu wrote:
> Add preliminary documenation for AMD IOMMU.
> 
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
>   Documentation/x86/amd-iommu.rst   | 85 +++++++++++++++++++++++++++++++
>   Documentation/x86/index.rst       |  1 +
>   Documentation/x86/intel-iommu.rst |  2 +-
>   3 files changed, 87 insertions(+), 1 deletion(-)
>   create mode 100644 Documentation/x86/amd-iommu.rst
> 
> diff --git a/Documentation/x86/amd-iommu.rst b/Documentation/x86/amd-iommu.rst
> new file mode 100644
> index 000000000000..89820140fefa
> --- /dev/null
> +++ b/Documentation/x86/amd-iommu.rst
> @@ -0,0 +1,85 @@
> +=================
> +AMD IOMMU Support
> +=================
> +
> +The architecture spec can be obtained from the below location.
> +
> +https://www.amd.com/system/files/TechDocs/48882_IOMMU.pdf
> +
> +This guide gives a quick cheat sheet for some basic understanding.
> +
> +Some Keywords
> +
> +- IVRS - I/O Virtualization Reporting Structure
> +- IVDB - I/O Virtualization Definition Block
> +- IVHD - I/O Virtualization Hardware Definition
> +- IOVA - I/O Virtual Address.
> +
> +Basic stuff
> +-----------
> +
> +ACPI enumerates and lists the different DMA engines in the platform, and
> +device scope relationships between PCI devices and which DMA engine controls
> +them.

"DMA engine" typically means a dedicated device for peripheral-to-memory 
or memory-to-memory transfers, or the responsible block within a general 
DMA-capable endpoint. In the context of the original Intel doc from 
whence I see this is copied, this probably should have said "DMAR unit" 
or similar; here I'd suggest picking your favourite vendor-appropriate 
term for "instance of IOMMU translation hardware". Let's not promote 
confusion more than necessary.

> +
> +What is IVRS?
> +-------------
> +
> +The architecture defines an ACPI-compatible data structure called an I/O
> +Virtualization Reporting Structure (IVRS) that is used to convey information
> +related to I/O virtualization to system software.  The IVRS describes the
> +configuration and capabilities of the IOMMUs contained in the platform as
> +well as information about the devices that each IOMMU virtualizes.
> +
> +The IVRS provides information about the following:
> +- IOMMUs present in the platform including their capabilities and proper configuration
> +- System I/O topology relevant to each IOMMU
> +- Peripheral devices that cannot be otherwise enumerated
> +- Memory regions used by SMI/SMM, platform firmware, and platform hardware. These are
> +generally exclusion ranges to be configured by system software.
> +
> +How is IOVA generated?
> +----------------------
> +
> +Well behaved drivers call pci_map_*() calls before sending command to device

Horribly out-of-date drivers call pci_map_*(). Modern well-behaved 
drivers call dma_map_*() ;)

> +that needs to perform DMA. Once DMA is completed and mapping is no longer
> +required, device performs a pci_unmap_*() calls to unmap the region.
> +
> +The AMD IOMMU driver allocates a virtual address per domain. Each PCIE
> +device has its own domain (hence protection). Devices under p2p bridges
> +share the virtual address with all devices under the p2p bridge due to
> +transaction id aliasing for p2p bridges.
> +
> +IOVA generation is pretty generic. We used the same technique as vmalloc()
> +but these are not global address spaces, but separate for each domain.
> +Different DMA engines may support different number of domains.

I'm not sure about this whole section, really - IOVA management was 
entirely decoupled from drivers some time ago. If there's value in 
having some overview documentation, then it's probably worth 
consolidating into a common "IOMMU API" doc that can be cross-referenced 
for a summary of domains, groups, and iommu_dma_ops.

> +
> +
> +Fault reporting
> +---------------
> +When errors are reported, the DMA engine signals via an interrupt. The fault

Again, here I instinctively read "DMA engine" as being the endpoint 
device *making* the DMA transaction that faulted, and indeed that might 
happen to raise its own error interrupt if it gets an unexpected abort 
back from the IOMMU, which is coincidental to a thoroughly misleading 
degree...

Thanks,
Robin.

> +reason and device that caused it with fault reason is printed on console.
> +
> +See below for sample.
> +
> +
> +Boot Message Sample
> +-------------------
> +
> +Something like this gets printed indicating presence of the IOMMU.
> +
> +	iommu: Default domain type: Translated
> +	iommu: DMA domain TLB invalidation policy: lazy mode
> +
> +
> +PCI-DMA: Using AMD IOMMU
> +------------------------
> +
> +Fault reporting
> +^^^^^^^^^^^^^^^
> +
> +::
> +
> +	AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0007 address=0xffffc02000 flags=0x0000]
> +	AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.0 domain=0x0007 address=0xffffc02000 flags=0x0000]
> +
> diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst
> index f498f1d36cd3..15711134eb68 100644
> --- a/Documentation/x86/index.rst
> +++ b/Documentation/x86/index.rst
> @@ -22,6 +22,7 @@ x86-specific Documentation
>      mtrr
>      pat
>      intel-iommu
> +   amd-iommu
>      intel_txt
>      amd-memory-encryption
>      pti
> diff --git a/Documentation/x86/intel-iommu.rst b/Documentation/x86/intel-iommu.rst
> index 099f13d51d5f..4d3391c7bd3f 100644
> --- a/Documentation/x86/intel-iommu.rst
> +++ b/Documentation/x86/intel-iommu.rst
> @@ -1,5 +1,5 @@
>   ===================
> -Linux IOMMU Support
> +Intel IOMMU Support
>   ===================
>   
>   The architecture spec can be obtained from the below location.

RE: [PATCH] Documentation: x86: add documenation for AMD IOMMU

Posted by Deucher, Alexander 4 years, 3 months ago

[Public]

> -----Original Message-----
> From: Robin Murphy <robin.murphy@arm.com>
> Sent: Tuesday, March 8, 2022 3:09 PM
> To: Deucher, Alexander <Alexander.Deucher@amd.com>; linux-
> doc@vger.kernel.org; linux-kernel@vger.kernel.org; corbet@lwn.net;
> hpa@zytor.com; x86@kernel.org; dave.hansen@linux.intel.com;
> bp@alien8.de; mingo@redhat.com; tglx@linutronix.de; joro@8bytes.org;
> Suthikulpanit, Suravee <Suravee.Suthikulpanit@amd.com>; will@kernel.org;
> iommu@lists.linux-foundation.org
> Subject: Re: [PATCH] Documentation: x86: add documenation for AMD
> IOMMU
> 
> On 2022-03-08 19:04, Alex Deucher via iommu wrote:
> > Add preliminary documenation for AMD IOMMU.
> >
> > Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> > ---
> >   Documentation/x86/amd-iommu.rst   | 85
> +++++++++++++++++++++++++++++++
> >   Documentation/x86/index.rst       |  1 +
> >   Documentation/x86/intel-iommu.rst |  2 +-
> >   3 files changed, 87 insertions(+), 1 deletion(-)
> >   create mode 100644 Documentation/x86/amd-iommu.rst
> >
> > diff --git a/Documentation/x86/amd-iommu.rst
> > b/Documentation/x86/amd-iommu.rst new file mode 100644 index
> > 000000000000..89820140fefa
> > --- /dev/null
> > +++ b/Documentation/x86/amd-iommu.rst
> > @@ -0,0 +1,85 @@
> > +=================
> > +AMD IOMMU Support
> > +=================
> > +
> > +The architecture spec can be obtained from the below location.
> > +
> >
> +https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw
> ww
> >
> +.amd.com%2Fsystem%2Ffiles%2FTechDocs%2F48882_IOMMU.pdf&amp;da
> ta=04%7C
> >
> +01%7Calexander.deucher%40amd.com%7C3adb51f8c3f1435e0deb08da013f
> 8172%7
> >
> +C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637823669974023501
> %7CUnkn
> >
> +own%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I
> k1haWw
> >
> +iLCJXVCI6Mn0%3D%7C3000&amp;sdata=9Wq07GM%2BdT9xt%2FCZ3xhue
> %2BrNIe6CnD
> > +cG32kwqosUEZ8%3D&amp;reserved=0
> > +
> > +This guide gives a quick cheat sheet for some basic understanding.
> > +
> > +Some Keywords
> > +
> > +- IVRS - I/O Virtualization Reporting Structure
> > +- IVDB - I/O Virtualization Definition Block
> > +- IVHD - I/O Virtualization Hardware Definition
> > +- IOVA - I/O Virtual Address.
> > +
> > +Basic stuff
> > +-----------
> > +
> > +ACPI enumerates and lists the different DMA engines in the platform,
> > +and device scope relationships between PCI devices and which DMA
> > +engine controls them.
> 
> "DMA engine" typically means a dedicated device for peripheral-to-memory
> or memory-to-memory transfers, or the responsible block within a general
> DMA-capable endpoint. In the context of the original Intel doc from whence I
> see this is copied, this probably should have said "DMAR unit"
> or similar; here I'd suggest picking your favourite vendor-appropriate term
> for "instance of IOMMU translation hardware". Let's not promote confusion
> more than necessary.
> 
> > +
> > +What is IVRS?
> > +-------------
> > +
> > +The architecture defines an ACPI-compatible data structure called an
> > +I/O Virtualization Reporting Structure (IVRS) that is used to convey
> > +information related to I/O virtualization to system software.  The
> > +IVRS describes the configuration and capabilities of the IOMMUs
> > +contained in the platform as well as information about the devices that
> each IOMMU virtualizes.
> > +
> > +The IVRS provides information about the following:
> > +- IOMMUs present in the platform including their capabilities and
> > +proper configuration
> > +- System I/O topology relevant to each IOMMU
> > +- Peripheral devices that cannot be otherwise enumerated
> > +- Memory regions used by SMI/SMM, platform firmware, and platform
> > +hardware. These are generally exclusion ranges to be configured by
> system software.
> > +
> > +How is IOVA generated?
> > +----------------------
> > +
> > +Well behaved drivers call pci_map_*() calls before sending command to
> > +device
> 
> Horribly out-of-date drivers call pci_map_*(). Modern well-behaved drivers
> call dma_map_*() ;)
> 
> > +that needs to perform DMA. Once DMA is completed and mapping is no
> > +longer required, device performs a pci_unmap_*() calls to unmap the
> region.
> > +
> > +The AMD IOMMU driver allocates a virtual address per domain. Each
> > +PCIE device has its own domain (hence protection). Devices under p2p
> > +bridges share the virtual address with all devices under the p2p
> > +bridge due to transaction id aliasing for p2p bridges.
> > +
> > +IOVA generation is pretty generic. We used the same technique as
> > +vmalloc() but these are not global address spaces, but separate for each
> domain.
> > +Different DMA engines may support different number of domains.
> 
> I'm not sure about this whole section, really - IOVA management was entirely
> decoupled from drivers some time ago. If there's value in having some
> overview documentation, then it's probably worth consolidating into a
> common "IOMMU API" doc that can be cross-referenced for a summary of
> domains, groups, and iommu_dma_ops.
> 
> > +
> > +
> > +Fault reporting
> > +---------------
> > +When errors are reported, the DMA engine signals via an interrupt. The
> fault
> 
> Again, here I instinctively read "DMA engine" as being the endpoint
> device *making* the DMA transaction that faulted, and indeed that might
> happen to raise its own error interrupt if it gets an unexpected abort
> back from the IOMMU, which is coincidental to a thoroughly misleading
> degree...

Thanks for the review Robin.  All good points.  I've fixed them up and also fixed up the Intel documentation.  Will send out v2 shortly.

Thanks,

Alex

> 
> Thanks,
> Robin.
> 
> > +reason and device that caused it with fault reason is printed on console.
> > +
> > +See below for sample.
> > +
> > +
> > +Boot Message Sample
> > +-------------------
> > +
> > +Something like this gets printed indicating presence of the IOMMU.
> > +
> > +	iommu: Default domain type: Translated
> > +	iommu: DMA domain TLB invalidation policy: lazy mode
> > +
> > +
> > +PCI-DMA: Using AMD IOMMU
> > +------------------------
> > +
> > +Fault reporting
> > +^^^^^^^^^^^^^^^
> > +
> > +::
> > +
> > +	AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0007
> address=0xffffc02000 flags=0x0000]
> > +	AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.0
> domain=0x0007 address=0xffffc02000 flags=0x0000]
> > +
> > diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst
> > index f498f1d36cd3..15711134eb68 100644
> > --- a/Documentation/x86/index.rst
> > +++ b/Documentation/x86/index.rst
> > @@ -22,6 +22,7 @@ x86-specific Documentation
> >      mtrr
> >      pat
> >      intel-iommu
> > +   amd-iommu
> >      intel_txt
> >      amd-memory-encryption
> >      pti
> > diff --git a/Documentation/x86/intel-iommu.rst
> b/Documentation/x86/intel-iommu.rst
> > index 099f13d51d5f..4d3391c7bd3f 100644
> > --- a/Documentation/x86/intel-iommu.rst
> > +++ b/Documentation/x86/intel-iommu.rst
> > @@ -1,5 +1,5 @@
> >   ===================
> > -Linux IOMMU Support
> > +Intel IOMMU Support
> >   ===================
> >
> >   The architecture spec can be obtained from the below location.