Series comparison

-[PATCH v4 0/7] SMMU handling for PCIe Passthrough on ARM
+[PATCH v7 0/8] SMMU handling for PCIe Passthrough on ARM
 This series introduces SMMU handling for PCIe passthrough on ARM. These patches
-are independent from (and don't depend on) the vPCI reference counting/locking
+should be able to be upstreamed independently from the vPCI series [1]. See [2]
-work in progress, and should be able to be upstreamed independently.
+for notes about test cases.
 [1] https://lists.xenproject.org/archives/html/xen-devel/2023-10/msg00660.html
 [2] https://lists.xenproject.org/archives/html/xen-devel/2023-06/msg01135.html
 v6->v7:
 * drop ("xen/arm: don't pass iommu properties to hwdom for iommu-map")
 v5->v6:
 * don't revert ("xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"")
 * add ("xen/arm: enable dom0 to use PCI devices with pci-passthrough=no")
 v4->v5:
 * drop ("xen/arm: Improve readability of check for registered devices")
 * drop ("xen/arm: Move is_protected flag to struct device")
 * add ("xen/arm: don't pass iommu properties to hwdom for iommu-map")
 * add ("xen/arm: Fix mapping for PCI bridge mmio region")
 * revert ("xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"")
 * add ("xen/arm: Map ITS doorbell register to IOMMU page tables.")
 * fix test case #1 with PCI device in dom0
 v3->v4:
 * split a change from ("xen/arm: Move is_protected flag to struct device") into
   a new separate patch
 * see individual patches for further details
 ...
 * drop "pci/arm: Use iommu_add_dt_pci_device()"
 * drop "RFC: pci/arm: don't do iommu call for phantom functions"
 * move invocation of sideband ID mapping function to add_device()
   platform_ops/iommu_ops hook
-v1->v2:
-* phantom device handling
-* shuffle around iommu_add_dt_pci_device()
 Oleksandr Andrushchenko (1):
   xen/arm: smmuv2: Add PCI devices support for SMMUv2
-Oleksandr Tyshchenko (4):
+Oleksandr Tyshchenko (2):
   xen/arm: Improve readability of check for registered devices
   xen/arm: Move is_protected flag to struct device
   iommu/arm: Add iommu_dt_xlate()
   iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API
-Rahul Singh (1):
+Rahul Singh (3):
   xen/arm: smmuv3: Add PCI devices support for SMMUv3
+  xen/arm: Fix mapping for PCI bridge mmio region
+  xen/arm: Map ITS doorbell register to IOMMU page tables
-Stewart Hildebrand (1):
+Stewart Hildebrand (2):
   iommu/arm: iommu_add_dt_pci_sideband_ids phantom handling
+  xen/arm: enable dom0 to use PCI devices with pci-passthrough=no
- xen/arch/arm/domain_build.c              |   4 +-
+ xen/arch/arm/device.c                 |   2 +-
- xen/arch/arm/include/asm/device.h        |  14 ++
+ xen/arch/arm/pci/pci.c                |   5 +-
- xen/common/device_tree.c                 |   2 +-
+ xen/arch/arm/vgic-v3-its.c            |  20 +++
- xen/drivers/passthrough/arm/ipmmu-vmsa.c |   4 +-
+ xen/arch/x86/include/asm/pci.h        |   6 -
- xen/drivers/passthrough/arm/smmu-v3.c    |  81 ++++++++-
+ xen/common/device-tree/device-tree.c  |  91 ++++++++++++
- xen/drivers/passthrough/arm/smmu.c       | 122 +++++++++++---
+ xen/drivers/passthrough/arm/smmu-v3.c | 117 ++++++++++++++--
- xen/drivers/passthrough/device_tree.c    | 205 ++++++++++++++++++++---
+ xen/drivers/passthrough/arm/smmu.c    | 190 ++++++++++++++++++++------
- xen/include/xen/device_tree.h            |  38 +++--
+ xen/drivers/passthrough/device_tree.c |  97 ++++++++++---
- xen/include/xen/iommu.h                  |  22 ++-
+ xen/drivers/pci/physdev.c             |   4 +-
-files changed, 423 insertions(+), 69 deletions(-)
+ xen/include/xen/device_tree.h         |  23 ++++
  xen/include/xen/iommu.h               |  31 ++++-
 files changed, 502 insertions(+), 84 deletions(-)
 --
-.40.1
+.34.1

-[PATCH v4 3/7] iommu/arm: Add iommu_dt_xlate()
+[PATCH v7 1/8] iommu/arm: Add iommu_dt_xlate()
 ...
 While at it introduce NO_IOMMU to avoid magic "1".
 Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
 Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
+Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
+Reviewed-by: Julien Grall <jgrall@amazon.com>
 ---
+v6->v7:
+* explained NO_IOMMU in comments
+v5->v6:
+* pass ops parameter to iommu_dt_xlate()
+* add Julien's R-b
+v4->v5:
+* rebase on top of "dynamic node programming using overlay dtbo" series
+* move #define NO_IOMMU 1 to header
+* s/these/this/ inside comment
 v3->v4:
 * make dt_phandle_args *iommu_spec const
 * move !ops->add_device check to helper
 v2->v3:
 ...
 (cherry picked from commit c26bab0415ca303df86aba1d06ef8edc713734d3 from
  the downstream branch poc/pci-passthrough from
  https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
 ---
- xen/drivers/passthrough/device_tree.c | 47 ++++++++++++++++++---------
+ xen/drivers/passthrough/device_tree.c | 48 +++++++++++++++++----------
-file changed, 31 insertions(+), 16 deletions(-)
+ xen/include/xen/iommu.h               |  3 ++
 files changed, 33 insertions(+), 18 deletions(-)
 diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
 index XXXXXXX..XXXXXXX 100644
 --- a/xen/drivers/passthrough/device_tree.c
 +++ b/xen/drivers/passthrough/device_tree.c
 @@ -XXX,XX +XXX,XX @@ int iommu_release_dt_devices(struct domain *d)
      return 0;
  }
-+/* This correlation must not be altered */
-+#define NO_IOMMU    1
-+
 +static int iommu_dt_xlate(struct device *dev,
-+                          const struct dt_phandle_args *iommu_spec)
++                          const struct dt_phandle_args *iommu_spec,
 +                          const struct iommu_ops *ops)
 +{
-+    const struct iommu_ops *ops = iommu_get_ops();
 +    int rc;
 +
 +    if ( !ops->dt_xlate )
 +        return -EINVAL;
 +
 ...
 +     * The driver is responsible to decide how to interpret them.
 +     */
 +    return ops->dt_xlate(dev, iommu_spec);
 +}
 +
- int iommu_add_dt_device(struct dt_device_node *np)
+ int iommu_remove_dt_device(struct dt_device_node *np)
  {
+     const struct iommu_ops *ops = iommu_get_ops();
+@@ -XXX,XX +XXX,XX @@ int iommu_remove_dt_device(struct dt_device_node *np)
+     ASSERT(rw_is_locked(&dt_host_lock));
+     if ( !iommu_enabled )
+-        return 1;
++        return NO_IOMMU;
+     if ( !ops )
+         return -EOPNOTSUPP;
+@@ -XXX,XX +XXX,XX @@ int iommu_add_dt_device(struct dt_device_node *np)
      const struct iommu_ops *ops = iommu_get_ops();
      struct dt_phandle_args iommu_spec;
      struct device *dev = dt_to_dev(np);
 -    int rc = 1, index = 0;
 +    int rc = NO_IOMMU, index = 0;
+     ASSERT(system_state < SYS_STATE_active || rw_is_locked(&dt_host_lock));
      if ( !iommu_enabled )
 -        return 1;
 +        return NO_IOMMU;
      if ( !ops )
          return -EINVAL;
 @@ -XXX,XX +XXX,XX @@ int iommu_add_dt_device(struct dt_device_node *np)
+     {
+         /*
           * The driver which supports generic IOMMU DT bindings must have
-          * these callback implemented.
+-         * these callback implemented.
 +         * this callback implemented.
           */
 -        if ( !ops->add_device || !ops->dt_xlate )
 +        if ( !ops->add_device )
-             return -EINVAL;
+         {
              rc = -EINVAL;
              goto fail;
          }
 -        if ( !dt_device_is_available(iommu_spec.np) )
 -            break;
 -
 -        rc = iommu_fwspec_init(dev, &iommu_spec.np->dev);
 ...
 -         * Provide DT IOMMU specifier which describes the IOMMU master
 -         * interfaces of that device (device IDs, etc) to the driver.
 -         * The driver is responsible to decide how to interpret them.
 -         */
 -        rc = ops->dt_xlate(dev, &iommu_spec);
-+        rc = iommu_dt_xlate(dev, &iommu_spec);
++        rc = iommu_dt_xlate(dev, &iommu_spec, ops);
          if ( rc )
              break;
+diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
+index XXXXXXX..XXXXXXX 100644
+--- a/xen/include/xen/iommu.h
++++ b/xen/include/xen/iommu.h
+@@ -XXX,XX +XXX,XX @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
+  */
+ int iommu_remove_dt_device(struct dt_device_node *np);
++/* Error code for reporting no IOMMU is present */
++#define NO_IOMMU    1
++
+ #endif /* HAS_DEVICE_TREE */
+ struct page_info;
 --
-.40.1
+.34.1

-[PATCH v4 4/7] iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API
+[PATCH v7 2/8] iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API
 ...
 This behaves similarly to the existing iommu_add_dt_device API, except it
 handles PCI devices, and it is to be invoked from the add_device hook in the
 SMMU driver.
-The function of_map_id to translate an ID through a downstream mapping
+The function dt_map_id to translate an ID through a downstream mapping
 (which is also suitable for mapping Requester ID) was borrowed from Linux
 (v5.10-rc6) and updated according to the Xen code base.
 [1] https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/pci-iommu.txt
 Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
 Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
+Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
 ---
+Regarding pci_for_each_dma_alias question: getting host bridge node
+directly seems like a simpler solution with the same result. AFAIU
+with pci_for_each_dma_alias in linux we would arrive to the host brige
+node anyway, but also try to call dt_map_id for each device along the
+way. I am not sure why exactly it is done this way in linux, as
+according to the pci-iommu.txt, iommu-map node can only be present in
+the PCI root.
+v6->v7:
+* put iommu_add_pci_sideband_ids under ifdef
+* remove ifdef CONFIG_APCI
+* style: add newline for symmetry
+v5->v6:
+* pass ops to iommu_dt_xlate()
+v4->v5:
+* style: add newlines after variable declarations and before return in iommu.h
+* drop device_is_protected() check in iommu_add_dt_pci_sideband_ids()
+* rebase on top of ("dynamic node programming using overlay dtbo") series
+* fix typo in commit message
+* remove #ifdef around dt_map_id() prototype
+* move dt_map_id() to xen/common/device_tree.c
+* add function name in error prints
+* use dprintk for debug prints
+* use GENMASK and #include <xen/bitops.h>
+* fix typo in comment
+* remove unnecessary (int) cast in loop condition
+* assign *id_out and return success in case of no translation in dt_map_id()
+* don't initialize local variable unnecessarily
+* return error in case of ACPI/no DT in iommu_add_{dt_}pci_sideband_ids()
 v3->v4:
 * wrap #include <asm/acpi.h> and if ( acpi_disabled ) in #ifdef CONFIG_ACPI
 * fix Michal's remarks about style, parenthesis, and print formats
 * remove !ops->dt_xlate check since it is already in iommu_dt_xlate helper
 * rename s/iommu_dt_pci_map_id/dt_map_id/ because it is generic, not specific
 ...
 (cherry picked from commit 734e3bf6ee77e7947667ab8fa96c25b349c2e1da from
  the downstream branch poc/pci-passthrough from
  https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
 ---
- xen/drivers/passthrough/device_tree.c | 134 ++++++++++++++++++++++++++
+ xen/common/device-tree/device-tree.c  | 91 +++++++++++++++++++++++++++
- xen/include/xen/device_tree.h         |  25 +++++
+ xen/drivers/passthrough/device_tree.c | 42 +++++++++++++
- xen/include/xen/iommu.h               |  22 ++++-
+ xen/include/xen/device_tree.h         | 23 +++++++
-files changed, 180 insertions(+), 1 deletion(-)
+ xen/include/xen/iommu.h               | 28 ++++++++-
+files changed, 183 insertions(+), 1 deletion(-)
 diff --git a/xen/common/device-tree/device-tree.c b/xen/common/device-tree/device-tree.c
 index XXXXXXX..XXXXXXX 100644
 --- a/xen/common/device-tree/device-tree.c
 +++ b/xen/common/device-tree/device-tree.c
@@ -XXX,XX +XXX,XX @@
   * published by the Free Software Foundation.
   */
 +#include <xen/bitops.h>
  #include <xen/types.h>
  #include <xen/init.h>
  #include <xen/guest_access.h>
@@ -XXX,XX +XXX,XX @@ int dt_get_pci_domain_nr(struct dt_device_node *node)
      return (u16)domain;
  }
 +int dt_map_id(const struct dt_device_node *np, uint32_t id,
 +              const char *map_name, const char *map_mask_name,
 +              struct dt_device_node **target, uint32_t *id_out)
 +{
 +    uint32_t map_mask, masked_id, map_len;
 +    const __be32 *map = NULL;
 +
 +    if ( !np || !map_name || (!target && !id_out) )
 +        return -EINVAL;
 +
 +    map = dt_get_property(np, map_name, &map_len);
 +    if ( !map )
 +    {
 +        if ( target )
 +            return -ENODEV;
 +
 +        /* Otherwise, no map implies no translation */
 +        *id_out = id;
 +        return 0;
 +    }
 +
 +    if ( !map_len || (map_len % (4 * sizeof(*map))) )
 +    {
 +        printk(XENLOG_ERR "%s(): %s: Error: Bad %s length: %u\n", __func__,
 +               np->full_name, map_name, map_len);
 +        return -EINVAL;
 +    }
 +
 +    /* The default is to select all bits. */
 +    map_mask = GENMASK(31, 0);
 +
 +    /*
 +     * Can be overridden by "{iommu,msi}-map-mask" property.
 +     * If dt_property_read_u32() fails, the default is used.
 +     */
 +    if ( map_mask_name )
 +        dt_property_read_u32(np, map_mask_name, &map_mask);
 +
 +    masked_id = map_mask & id;
 +    for ( ; map_len > 0; map_len -= 4 * sizeof(*map), map += 4 )
 +    {
 +        struct dt_device_node *phandle_node;
 +        uint32_t id_base = be32_to_cpup(map + 0);
 +        uint32_t phandle = be32_to_cpup(map + 1);
 +        uint32_t out_base = be32_to_cpup(map + 2);
 +        uint32_t id_len = be32_to_cpup(map + 3);
 +
 +        if ( id_base & ~map_mask )
 +        {
 +            printk(XENLOG_ERR "%s(): %s: Invalid %s translation - %s-mask (0x%"PRIx32") ignores id-base (0x%"PRIx32")\n",
 +                   __func__, np->full_name, map_name, map_name, map_mask,
 +                   id_base);
 +            return -EFAULT;
 +        }
 +
 +        if ( (masked_id < id_base) || (masked_id >= (id_base + id_len)) )
 +            continue;
 +
 +        phandle_node = dt_find_node_by_phandle(phandle);
 +        if ( !phandle_node )
 +            return -ENODEV;
 +
 +        if ( target )
 +        {
 +            if ( !*target )
 +                *target = phandle_node;
 +
 +            if ( *target != phandle_node )
 +                continue;
 +        }
 +
 +        if ( id_out )
 +            *id_out = masked_id - id_base + out_base;
 +
 +        dprintk(XENLOG_DEBUG, "%s: %s, using mask %08"PRIx32", id-base: %08"PRIx32", out-base: %08"PRIx32", length: %08"PRIx32", id: %08"PRIx32" -> %08"PRIx32"\n",
 +               np->full_name, map_name, map_mask, id_base, out_base, id_len, id,
 +               masked_id - id_base + out_base);
 +        return 0;
 +    }
 +
 +    dprintk(XENLOG_DEBUG, "%s: no %s translation for id 0x%"PRIx32" on %s\n",
 +           np->full_name, map_name, id,
 +           (target && *target) ? (*target)->full_name : NULL);
 +
 +    if ( id_out )
 +        *id_out = id;
 +
 +    return 0;
 +}
 +
  /*
   * Local variables:
   * mode: C
 diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
 index XXXXXXX..XXXXXXX 100644
 --- a/xen/drivers/passthrough/device_tree.c
 +++ b/xen/drivers/passthrough/device_tree.c
 @@ -XXX,XX +XXX,XX @@ static int iommu_dt_xlate(struct device *dev,
      return ops->dt_xlate(dev, iommu_spec);
  }
 +#ifdef CONFIG_HAS_PCI
-+int dt_map_id(const struct dt_device_node *np, uint32_t id,
-+              const char *map_name, const char *map_mask_name,
-+              struct dt_device_node **target, uint32_t *id_out)
-+{
-+    uint32_t map_mask, masked_id, map_len;
-+    const __be32 *map = NULL;
-+
-+    if ( !np || !map_name || (!target && !id_out) )
-+        return -EINVAL;
-+
-+    map = dt_get_property(np, map_name, &map_len);
-+    if ( !map )
-+    {
-+        if ( target )
-+            return -ENODEV;
-+
-+        /* Otherwise, no map implies no translation */
-+        *id_out = id;
-+        return 0;
-+    }
-+
-+    if ( !map_len || (map_len % (4 * sizeof(*map))) )
-+    {
-+        printk(XENLOG_ERR "%s: Error: Bad %s length: %u\n", np->full_name,
-+               map_name, map_len);
-+        return -EINVAL;
-+    }
-+
-+    /* The default is to select all bits. */
-+    map_mask = 0xffffffff;
-+
-+    /*
-+     * Can be overridden by "{iommu,msi}-map-mask" property.
-+     * If df_property_read_u32() fails, the default is used.
-+     */
-+    if ( map_mask_name )
-+        dt_property_read_u32(np, map_mask_name, &map_mask);
-+
-+    masked_id = map_mask & id;
-+    for ( ; (int)map_len > 0; map_len -= 4 * sizeof(*map), map += 4 )
-+    {
-+        struct dt_device_node *phandle_node;
-+        uint32_t id_base = be32_to_cpup(map + 0);
-+        uint32_t phandle = be32_to_cpup(map + 1);
-+        uint32_t out_base = be32_to_cpup(map + 2);
-+        uint32_t id_len = be32_to_cpup(map + 3);
-+
-+        if ( id_base & ~map_mask )
-+        {
-+            printk(XENLOG_ERR "%s: Invalid %s translation - %s-mask (0x%"PRIx32") ignores id-base (0x%"PRIx32")\n",
-+                   np->full_name, map_name, map_name, map_mask, id_base);
-+            return -EFAULT;
-+        }
-+
-+        if ( (masked_id < id_base) || (masked_id >= (id_base + id_len)) )
-+            continue;
-+
-+        phandle_node = dt_find_node_by_phandle(phandle);
-+        if ( !phandle_node )
-+            return -ENODEV;
-+
-+        if ( target )
-+        {
-+            if ( !*target )
-+                *target = phandle_node;
-+
-+            if ( *target != phandle_node )
-+                continue;
-+        }
-+
-+        if ( id_out )
-+            *id_out = masked_id - id_base + out_base;
-+
-+        printk(XENLOG_DEBUG "%s: %s, using mask %08"PRIx32", id-base: %08"PRIx32", out-base: %08"PRIx32", length: %08"PRIx32", id: %08"PRIx32" -> %08"PRIx32"\n",
-+               np->full_name, map_name, map_mask, id_base, out_base, id_len, id,
-+               masked_id - id_base + out_base);
-+        return 0;
-+    }
-+
-+    printk(XENLOG_ERR "%s: no %s translation for id 0x%"PRIx32" on %s\n",
-+           np->full_name, map_name, id, (target && *target) ? (*target)->full_name : NULL);
-+
-+    /*
-+     * NOTE: Linux bypasses translation without returning an error here,
-+     * but should we behave in the same way on Xen? Restrict for now.
-+     */
-+    return -EFAULT;
-+}
-+
 +int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)
 +{
 +    const struct iommu_ops *ops = iommu_get_ops();
 +    struct dt_phandle_args iommu_spec = { .args_count = 1 };
 +    struct device *dev = pci_to_dev(pdev);
 +    const struct dt_device_node *np;
-+    int rc = NO_IOMMU;
++    int rc;
 +
 +    if ( !iommu_enabled )
 +        return NO_IOMMU;
 +
 +    if ( !ops )
 +        return -EINVAL;
-+
-+    if ( device_is_protected(dev) )
-+        return 0;
 +
 +    if ( dev_iommu_fwspec_get(dev) )
 +        return -EEXIST;
 +
 +    np = pci_find_host_bridge_node(pdev);
 ...
 +    rc = dt_map_id(np, PCI_BDF(pdev->bus, pdev->devfn), "iommu-map",
 +                   "iommu-map-mask", &iommu_spec.np, iommu_spec.args);
 +    if ( rc )
 +        return (rc == -ENODEV) ? NO_IOMMU : rc;
 +
-+    rc = iommu_dt_xlate(dev, &iommu_spec);
++    rc = iommu_dt_xlate(dev, &iommu_spec, ops);
 +    if ( rc < 0 )
 +    {
 +        iommu_fwspec_free(dev);
 +        return -EINVAL;
 +    }
 +
 +    return rc;
 +}
 +#endif /* CONFIG_HAS_PCI */
 +
- int iommu_add_dt_device(struct dt_device_node *np)
+ int iommu_remove_dt_device(struct dt_device_node *np)
  {
      const struct iommu_ops *ops = iommu_get_ops();
 diff --git a/xen/include/xen/device_tree.h b/xen/include/xen/device_tree.h
 index XXXXXXX..XXXXXXX 100644
 --- a/xen/include/xen/device_tree.h
 +++ b/xen/include/xen/device_tree.h
 @@ -XXX,XX +XXX,XX @@ int dt_count_phandle_with_args(const struct dt_device_node *np,
   */
  int dt_get_pci_domain_nr(struct dt_device_node *node);
-+#ifdef CONFIG_HAS_PCI
 +/**
 + * dt_map_id - Translate an ID through a downstream mapping.
 + * @np: root complex device node.
 + * @id: device ID to map.
 + * @map_name: property name of the map to use.
 ...
 + * Return: 0 on success or a standard error code on failure.
 + */
 +int dt_map_id(const struct dt_device_node *np, uint32_t id,
 +              const char *map_name, const char *map_mask_name,
 +              struct dt_device_node **target, uint32_t *id_out);
-+#endif /* CONFIG_HAS_PCI */
 +
  struct dt_device_node *dt_find_node_by_phandle(dt_phandle handle);
  #ifdef CONFIG_DEVICE_TREE_DEBUG
 diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
 index XXXXXXX..XXXXXXX 100644
 --- a/xen/include/xen/iommu.h
 +++ b/xen/include/xen/iommu.h
 @@ -XXX,XX +XXX,XX @@
- #include <xen/spinlock.h>
+ #include <xen/errno.h>
  #include <public/domctl.h>
  #include <public/hvm/ioreq.h>
-+#ifdef CONFIG_ACPI
++#include <xen/acpi.h>
 +#include <asm/acpi.h>
 +#endif
  #include <asm/device.h>
  TYPE_SAFE(uint64_t, dfn);
 @@ -XXX,XX +XXX,XX @@ int iommu_dt_domain_init(struct domain *d);
  int iommu_release_dt_devices(struct domain *d);
 ...
   *      (IOMMU is not enabled/present or device is not connected to it).
   */
  int iommu_add_dt_device(struct dt_device_node *np);
 +int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev);
- int iommu_do_dt_domctl(struct xen_domctl *, struct domain *,
+ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
-                        XEN_GUEST_HANDLE_PARAM(xen_domctl_t));
+                        XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl);
@@ -XXX,XX +XXX,XX @@ int iommu_remove_dt_device(struct dt_device_node *np);
  /* Error code for reporting no IOMMU is present */
  #define NO_IOMMU    1
 +#else /* !HAS_DEVICE_TREE */
 +static inline int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)
 +{
-+    return 0;
++    return -ENOSYS;
 +}
 +
  #endif /* HAS_DEVICE_TREE */
++#ifdef CONFIG_HAS_PCI
 +static inline int iommu_add_pci_sideband_ids(struct pci_dev *pdev)
 +{
-+    int ret = 0;
++    int ret = -ENOSYS;
-+#ifdef CONFIG_ACPI
++
 +    if ( acpi_disabled )
++        ret = iommu_add_dt_pci_sideband_ids(pdev);
++
++    return ret;
++}
++#else /* !HAS_PCI */
++static inline int iommu_add_pci_sideband_ids(struct pci_dev *pdev)
++{
++    return -ENOSYS;
++}
 +#endif
-+        ret = iommu_add_dt_pci_sideband_ids(pdev);
-+    return ret;
-+}
 +
  struct page_info;
  /*
 --
-.40.1
+.34.1

-[PATCH v4 5/7] iommu/arm: iommu_add_dt_pci_sideband_ids phantom handling
+[PATCH v7 3/8] iommu/arm: iommu_add_dt_pci_sideband_ids phantom handling
+From: Stewart Hildebrand <stewart.hildebrand@amd.com>
 Handle phantom functions in iommu_add_dt_pci_sideband_ids(). Each phantom
 function will have a unique requestor ID (RID)/BDF. On ARM, we need to
 map/translate the RID/BDF to an AXI stream ID for each phantom function
 according to the pci-iommu device tree mapping [1]. The RID/BDF -> AXI stream ID
 mapping in DT could allow phantom devices (i.e. devices with phantom functions)
 to use different AXI stream IDs based on the (phantom) function.
 [1] https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/pci-iommu.txt
 Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
 ---
+v6->v7:
+* no change
+v5->v6:
+* no change
+v4->v5:
+* no change
 v3->v4:
 * s/iommu_dt_pci_map_id/dt_map_id/
 v2->v3:
 * new patch title (was: iommu/arm: iommu_add_dt_pci_device phantom handling)
 * rework loop to reduce duplication
 * s/iommu_fwspec_free(pci_to_dev(pdev))/iommu_fwspec_free(dev)/
 v1->v2:
 * new patch
+---
+TODO: investigate Jan's comment [2]
+[2] https://lore.kernel.org/xen-devel/806a2978-19fb-4d31-ab6a-35ea7317c8de@suse.com/
 ---
  xen/drivers/passthrough/device_tree.c | 33 ++++++++++++++++-----------
 file changed, 20 insertions(+), 13 deletions(-)
 diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
 index XXXXXXX..XXXXXXX 100644
 --- a/xen/drivers/passthrough/device_tree.c
 +++ b/xen/drivers/passthrough/device_tree.c
 @@ -XXX,XX +XXX,XX @@ int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)
      struct device *dev = pci_to_dev(pdev);
      const struct dt_device_node *np;
-     int rc = NO_IOMMU;
+     int rc;
 +    unsigned int devfn = pdev->devfn;
      if ( !iommu_enabled )
          return NO_IOMMU;
 @@ -XXX,XX +XXX,XX @@ int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)
 ...
 +        rc = dt_map_id(np, PCI_BDF(pdev->bus, devfn), "iommu-map",
 +                       "iommu-map-mask", &iommu_spec.np, iommu_spec.args);
 +        if ( rc )
 +            return (rc == -ENODEV) ? NO_IOMMU : rc;
--    rc = iommu_dt_xlate(dev, &iommu_spec);
+-    rc = iommu_dt_xlate(dev, &iommu_spec, ops);
 -    if ( rc < 0 )
 -    {
 -        iommu_fwspec_free(dev);
 -        return -EINVAL;
-+        rc = iommu_dt_xlate(dev, &iommu_spec);
++        rc = iommu_dt_xlate(dev, &iommu_spec, ops);
 +        if ( rc < 0 )
 +        {
 +            iommu_fwspec_free(dev);
 +            return -EINVAL;
 +        }
 ...
 +            (PCI_SLOT(devfn) == PCI_SLOT(pdev->devfn)) );
      return rc;
  }
 --
-.40.1
+.34.1

-[PATCH v4 6/7] xen/arm: smmuv2: Add PCI devices support for SMMUv2
+[PATCH v7 4/8] xen/arm: smmuv2: Add PCI devices support for SMMUv2
 From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
+Implement support for PCI devices in the SMMU driver. Make arm_smmu_master
+structure to hold a pointer to the device to allow it to hold PCI devices.
+Trigger iommu-map parsing when new PCI device is added. Add checks to
+assign/deassign functions to ensure PCI devices are handled correctly.
+Implement basic quarantining.
+All pci devices are automatically assigned to hardware domain if it exists
+to ensure it can probe them.
+TODO:
+Implement scratch page quarantining support.
 Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
 Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
 Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
+Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
 ---
+v6->v7:
+* use d->pci_lock in arm_smmu_assign_dev()
+* remove !is_hardware_domain and pdev->domain == d checks in assign to
+  support future dom0less use case when dom0 is using vPCI
+* remove stale todo in dev_get_dev_node
+* don't print ""
+* remove redundant dt_device_is_protected check
+* remove assign/deassing prints
+* change assign logic to remove reassign reimplementation
+* check if pdev->domain exists before assigning to it
+* explain pdev->devfn check
+* make reassign check stricter and update comment
+v5->v6:
+* check for hardware_domain == NULL (dom0less test case)
+* locking: assign pdev->domain before list_add()
+v4->v5:
+* assign device to pdev->domain (usually dom0) by default in add_device() hook
+* deassign from hwdom
+* rebase on top of ("dynamic node programming using overlay dtbo") series
+* remove TODO in comment about device prints
+* add TODO regarding locking
+* fixup after dropping ("xen/arm: Move is_protected flag to struct device")
 v3->v4:
 * add new device_is_protected check in add_device hook to match SMMUv3 and
   IPMMU-VMSA drivers
 v2->v3:
 ...
 (cherry picked from commit 0c11a7f65f044c26d87d1e27ac6283ef1f9cfb7a from
  the downstream branch spider-master from
  https://github.com/xen-troops/xen.git)
 ---
+ xen/drivers/passthrough/arm/smmu.c | 190 ++++++++++++++++++++++-------
-This is a file imported from Linux with modifications for Xen. What should be
+file changed, 147 insertions(+), 43 deletions(-)
 the coding style for Xen modifications?
 ---
  xen/drivers/passthrough/arm/smmu.c | 120 ++++++++++++++++++++++++-----
 file changed, 99 insertions(+), 21 deletions(-)
 diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/arm/smmu.c
 index XXXXXXX..XXXXXXX 100644
 --- a/xen/drivers/passthrough/arm/smmu.c
 +++ b/xen/drivers/passthrough/arm/smmu.c
-@@ -XXX,XX +XXX,XX @@ typedef enum irqreturn irqreturn_t;
+@@ -XXX,XX +XXX,XX @@ enum irqreturn {
- /* Device logger functions
-  * TODO: Handle PCI
+ typedef enum irqreturn irqreturn_t;
-  */
 -/* Device logger functions
 - * TODO: Handle PCI
 - */
 -#define dev_print(dev, lvl, fmt, ...)                        \
 -     printk(lvl "smmu: %s: " fmt, dt_node_full_name(dev_to_dt(dev)), ## __VA_ARGS__)
++/* Device logger functions */
 +#ifndef CONFIG_HAS_PCI
 +#define dev_print(dev, lvl, fmt, ...)    \
 +    printk(lvl "smmu: %s: " fmt, dev_name(dev), ## __VA_ARGS__)
 +#else
 +#define dev_print(dev, lvl, fmt, ...) ({                                \
 ...
 +#endif
  /* Xen: misc */
  #define PHYS_MASK_SHIFT        PADDR_BITS
 @@ -XXX,XX +XXX,XX @@ struct arm_smmu_master_cfg {
-     for (i = 0; idx = cfg->smendx[i], i < num; ++i)
+     for (i = 0; idx = (cfg)->smendx[i], (i) < (num); ++(i))
  struct arm_smmu_master {
 -    struct device_node        *of_node;
 +    struct device            *dev;
      struct rb_node            node;
 ...
 -    return dev_iommu_fwspec_get(&master->of_node->dev);
 +    return dev_iommu_fwspec_get(master->dev);
  }
  static void parse_driver_options(struct arm_smmu_device *smmu)
-@@ -XXX,XX +XXX,XX @@ static struct device_node *dev_get_dev_node(struct device *dev)
+@@ -XXX,XX +XXX,XX @@ static void parse_driver_options(struct arm_smmu_device *smmu)
- }
  static struct device_node *dev_get_dev_node(struct device *dev)
  {
 -#if 0 /* Xen: TODO: Add support for PCI */
 -    if (dev_is_pci(dev)) {
 -        struct pci_bus *bus = to_pci_dev(dev)->bus;
 -
 -        while (!pci_is_root_bus(bus))
 -            bus = bus->parent;
 -        return bus->bridge->parent->of_node;
 -    }
 -#endif
 -
      return dev->of_node;
  }
  static struct arm_smmu_master *find_smmu_master(struct arm_smmu_device *smmu,
 -                        struct device_node *dev_node)
 +                        struct device *dev)
  {
 ...
 -    master = find_smmu_master(smmu, dev_node);
 +    master = find_smmu_master(smmu, dev);
      if (master) {
          dev_err(dev,
-             "rejecting multiple registrations for master device %s\n",
+-            "rejecting multiple registrations for master device %s\n",
 -            dev_node->name);
-+            dev_node ? dev_node->name : "");
++            "rejecting multiple registrations for master device\n");
          return -EBUSY;
      }
      master = devm_kzalloc(dev, sizeof(*master), GFP_KERNEL);
      if (!master)
          return -ENOMEM;
 -    master->of_node = dev_node;
 +    master->dev = dev;
-+
-+    if ( device_is_protected(dev) )
+-    /* Xen: Let Xen know that the device is protected by an SMMU */
-+    {
+-    dt_device_set_protected(dev_node);
-+        dev_err(dev, "Already added to SMMU\n");
++    if ( !dev_is_pci(dev) )
-+        return -EEXIST;
++    {
 +        /* Xen: Let Xen know that the device is protected by an SMMU */
 +        dt_device_set_protected(dev_node);
 +    }
-     /* Xen: Let Xen know that the device is protected by an SMMU */
+     for (i = 0; i < fwspec->num_ids; ++i) {
-     device_set_protected(dev);
+         if (!(smmu->features & ARM_SMMU_FEAT_STREAM_MATCH) &&
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_dt_add_device_legacy(struct arm_smmu_device *smmu,
               (fwspec->ids[i] >= smmu->num_mapping_groups)) {
              dev_err(dev,
-                 "stream ID for master device %s greater than maximum allowed (%d)\n",
+-                "stream ID for master device %s greater than maximum allowed (%d)\n",
 -                dev_node->name, smmu->num_mapping_groups);
-+                dev_node ? dev_node->name : "", smmu->num_mapping_groups);
++                "SMMU stream ID %d is greater than maximum allowed (%d)\n",
 +                fwspec->ids[i], smmu->num_mapping_groups);
              return -ERANGE;
          }
          master->cfg.smendx[i] = INVALID_SMENDX;
+@@ -XXX,XX +XXX,XX @@ static int arm_smmu_dt_remove_device_legacy(struct arm_smmu_device *smmu,
+     struct device_node *dev_node = dev_get_dev_node(dev);
+     int ret;
+-    master = find_smmu_master(smmu, dev_node);
++    master = find_smmu_master(smmu, dev);
+     if (master == NULL) {
+         dev_err(dev,
+             "No registrations found for master device %s\n",
+@@ -XXX,XX +XXX,XX @@ static int arm_smmu_dt_remove_device_legacy(struct arm_smmu_device *smmu,
+     if (ret)
+         return ret;
+-    /* Protected by dt_host_lock and dtdevs_lock as caller holds these locks. */
+-    dev_node->is_protected = false;
++    if ( !dev_is_pci(dev) )
++        /* Protected by dt_host_lock and dtdevs_lock as caller holds these locks. */
++        dev_node->is_protected = false;
+     kfree(master);
+     return 0;
+@@ -XXX,XX +XXX,XX @@ static int register_smmu_master(struct arm_smmu_device *smmu,
+                          fwspec);
+ }
++/* Forward declaration */
++static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
++                   struct device *dev, u32 flag);
++static int arm_smmu_deassign_dev(struct domain *d, uint8_t devfn,
++                 struct device *dev);
++
+ /*
+  * The driver which supports generic IOMMU DT bindings must have this
+  * callback implemented.
 @@ -XXX,XX +XXX,XX @@ static int arm_smmu_dt_add_device_generic(u8 devfn, struct device *dev)
+ {
      struct arm_smmu_device *smmu;
      struct iommu_fwspec *fwspec;
++    int ret;
 +
 +#ifdef CONFIG_HAS_PCI
 +    if ( dev_is_pci(dev) )
 +    {
 +        struct pci_dev *pdev = dev_to_pci(dev);
 +        int ret;
 +
++        /* Ignore calls for phantom functions */
 +        if ( devfn != pdev->devfn )
 +            return 0;
 +
 +        ret = iommu_add_pci_sideband_ids(pdev);
 +        if ( ret < 0 )
 +            iommu_fwspec_free(dev);
 +    }
 +#endif
-+
      fwspec = dev_iommu_fwspec_get(dev);
      if (fwspec == NULL)
+@@ -XXX,XX +XXX,XX @@ static int arm_smmu_dt_add_device_generic(u8 devfn, struct device *dev)
+     if (smmu == NULL)
          return -ENXIO;
+-    return arm_smmu_dt_add_device_legacy(smmu, dev, fwspec);
++    ret = arm_smmu_dt_add_device_legacy(smmu, dev, fwspec);
++    if ( ret )
++        return ret;
++
++#ifdef CONFIG_HAS_PCI
++    if ( dev_is_pci(dev) )
++    {
++        struct pci_dev *pdev = dev_to_pci(dev);
++
++        /*
++         * During PHYSDEVOP_pci_device_add, Xen does not assign the
++         * device, so we must do it here.
++         */
++        if ( pdev->domain )
++            ret = arm_smmu_assign_dev(pdev->domain, devfn, dev, 0);
++    }
++#endif
++
++    return ret;
+ }
+ static int arm_smmu_dt_xlate_generic(struct device *dev,
 @@ -XXX,XX +XXX,XX @@ static struct arm_smmu_device *find_smmu_for_device(struct device *dev)
  {
      struct arm_smmu_device *smmu;
      struct arm_smmu_master *master = NULL;
 -    struct device_node *dev_node = dev_get_dev_node(dev);
 ...
  static int arm_smmu_add_device(struct device *dev)
  {
 @@ -XXX,XX +XXX,XX @@ static int arm_smmu_add_device(struct device *dev)
      struct arm_smmu_master_cfg *cfg;
      struct iommu_group *group;
-     void (*releasefn)(void *) = NULL;
+     void (*releasefn)(void *data) = NULL;
 -    int ret;
      smmu = find_smmu_for_device(dev);
      if (!smmu)
          return -ENODEV;
 ...
 @@ -XXX,XX +XXX,XX @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
              return -ENOMEM;
      }
 +#ifdef CONFIG_HAS_PCI
-+    if ( dev_is_pci(dev) && !is_hardware_domain(d) )
++    if ( dev_is_pci(dev) )
 +    {
 +        struct pci_dev *pdev = dev_to_pci(dev);
 +
-+        printk(XENLOG_INFO "Assigning device %04x:%02x:%02x.%u to dom%d\n",
++        /* Ignore calls for phantom functions */
-+               pdev->seg, pdev->bus, PCI_SLOT(devfn), PCI_FUNC(devfn),
++        if ( devfn != pdev->devfn )
 +               d->domain_id);
 +
 +        if ( devfn != pdev->devfn || pdev->domain == d )
 +            return 0;
 +
-+        list_move(&pdev->domain_list, &d->pdev_list);
++        ASSERT(pcidevs_locked());
 +
 +        write_lock(&pdev->domain->pci_lock);
 +        list_del(&pdev->domain_list);
 +        write_unlock(&pdev->domain->pci_lock);
 +
 +        pdev->domain = d;
++
++        write_lock(&d->pci_lock);
++        list_add(&pdev->domain_list, &d->pdev_list);
++        write_unlock(&d->pci_lock);
 +
 +        /* dom_io is used as a sentinel for quarantined devices */
 +        if ( d == dom_io )
++        {
++            struct iommu_domain *domain = dev_iommu_domain(dev);
++            if ( !iommu_quarantine )
++                return 0;
++
++            if ( domain && domain->priv )
++                arm_smmu_deassign_dev(domain->priv->cfg.domain, devfn, dev);
++
 +            return 0;
++        }
 +    }
 +#endif
 +
      if (!dev_iommu_group(dev)) {
          ret = arm_smmu_add_device(dev);
          if (ret)
 @@ -XXX,XX +XXX,XX @@ out:
      return ret;
  }
 -static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)
-+static int arm_smmu_deassign_dev(struct domain *d, u8 devfn, struct device *dev)
++static int arm_smmu_deassign_dev(struct domain *d, uint8_t devfn,
 +                 struct device *dev)
  {
      struct iommu_domain *domain = dev_iommu_domain(dev);
      struct arm_smmu_xen_domain *xen_domain;
 +#ifdef CONFIG_HAS_PCI
 +    if ( dev_is_pci(dev) )
 +    {
 +        struct pci_dev *pdev = dev_to_pci(dev);
 +
-+        printk(XENLOG_INFO "Deassigning device %04x:%02x:%02x.%u from dom%d\n",
++        /* Ignore calls for phantom functions */
 +               pdev->seg, pdev->bus, PCI_SLOT(devfn), PCI_FUNC(devfn),
 +               d->domain_id);
 +
 +        if ( devfn != pdev->devfn )
 +            return 0;
 +
 +        /* dom_io is used as a sentinel for quarantined devices */
 +        if ( d == dom_io )
 ...
 +
      xen_domain = dom_iommu(d)->arch.priv;
      if (!domain || domain->priv->cfg.domain != d) {
 @@ -XXX,XX +XXX,XX @@ static int arm_smmu_reassign_dev(struct domain *s, struct domain *t,
+ {
      int ret = 0;
-     /* Don't allow remapping on other domain than hwdom */
+-    /* Don't allow remapping on other domain than hwdom */
 -    if ( t && !is_hardware_domain(t) )
-+    if ( t && !is_hardware_domain(t) && t != dom_io )
++    /* Don't allow remapping on other domain than hwdom
 +     * or dom_io for PCI devices
 +     */
 +    if ( t && !is_hardware_domain(t) && (t != dom_io || !dev_is_pci(dev)) )
          return -EPERM;
      if (t == s)
          return 0;
 -    ret = arm_smmu_deassign_dev(s, dev);
 +    ret = arm_smmu_deassign_dev(s, devfn, dev);
      if (ret)
          return ret;
 --
-.40.1
+.34.1

-[PATCH v4 7/7] xen/arm: smmuv3: Add PCI devices support for SMMUv3
+[PATCH v7 5/8] xen/arm: smmuv3: Add PCI devices support for SMMUv3
 From: Rahul Singh <rahul.singh@arm.com>
+Implement support for PCI devices in the SMMU driver. Trigger iommu-map
+parsing when new PCI device is added. Add checks to assign/deassign
+functions to ensure PCI devices are handled correctly. Implement basic
+quarantining.
+All pci devices are automatically assigned to hardware domain if it exists
+to ensure it can probe them.
+TODO:
+Implement scratch page quarantining support.
 Signed-off-by: Rahul Singh <rahul.singh@arm.com>
 Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
+Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
 ---
+v6->v7:
+* address TODO: use d->pci_lock in arm_smmu_assign_dev()
+* remove !is_hardware_domain and pdev->domain == d checks in assign to
+  support future dom0less use case when dom0 is using vPCI
+* check if pdev->domain exists before assigning to it
+* don't print ""
+* change assign logic to remove reassign reimplementation
+* explain pdev->devfn check
+* make reassign check stricter and update comment
+v5->v6:
+* check for hardware_domain == NULL (dom0less test case)
+* locking: assign pdev->domain before list_add()
+v4->v5:
+* deassign from hwdom
+* add TODO regarding locking
+* fixup after dropping ("xen/arm: Move is_protected flag to struct device")
 v3->v4:
 * no change
 v2->v3:
 * rebase
 ...
 (cherry picked from commit 7ed6c3ab250d899fe6e893a514278e406a2893e8 from
  the downstream branch poc/pci-passthrough from
  https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
 ---
+ xen/drivers/passthrough/arm/smmu-v3.c | 117 +++++++++++++++++++++++---
-This is a file imported from Linux with modifications for Xen. What should be
+file changed, 106 insertions(+), 11 deletions(-)
 the coding style used for Xen modifications?
 ---
  xen/drivers/passthrough/arm/smmu-v3.c | 76 +++++++++++++++++++++++++--
 file changed, 72 insertions(+), 4 deletions(-)
 diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
 index XXXXXXX..XXXXXXX 100644
 --- a/xen/drivers/passthrough/arm/smmu-v3.c
 +++ b/xen/drivers/passthrough/arm/smmu-v3.c
 @@ -XXX,XX +XXX,XX @@ static bool arm_smmu_sid_in_range(struct arm_smmu_device *smmu, u32 sid)
  }
  /* Forward declaration */
  static struct arm_smmu_device *arm_smmu_get_by_dev(const struct device *dev);
-+static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
++static int arm_smmu_assign_dev(struct domain *d, u8 devfn, struct device *dev,
-+            struct device *dev, u32 flag);
++                   u32 flag);
 +static int arm_smmu_deassign_dev(struct domain *d, uint8_t devfn,
 +                 struct device *dev);
  static int arm_smmu_add_device(u8 devfn, struct device *dev)
  {
      int i, ret;
      struct arm_smmu_device *smmu;
 ...
 +#ifdef CONFIG_HAS_PCI
 +    if ( dev_is_pci(dev) )
 +    {
 +        struct pci_dev *pdev = dev_to_pci(dev);
 +        int ret;
 +
 +        /* Ignore calls for phantom functions */
 +        if ( devfn != pdev->devfn )
 +            return 0;
 +
 +        ret = iommu_add_pci_sideband_ids(pdev);
 +        if ( ret < 0 )
 ...
 +    fwspec = dev_iommu_fwspec_get(dev);
      if (!fwspec)
          return -ENODEV;
 @@ -XXX,XX +XXX,XX @@ static int arm_smmu_add_device(u8 devfn, struct device *dev)
+      */
+     arm_smmu_enable_pasid(master);
+-    if (dt_device_is_protected(dev_to_dt(dev))) {
+-        dev_err(dev, "Already added to SMMUv3\n");
+-        return -EEXIST;
+-    }
++    if ( !dev_is_pci(dev) )
++    {
++        if (dt_device_is_protected(dev_to_dt(dev))) {
++            dev_err(dev, "Already added to SMMUv3\n");
++            return -EEXIST;
++        }
+-    /* Let Xen know that the master device is protected by an IOMMU. */
+-    dt_device_set_protected(dev_to_dt(dev));
++        /* Let Xen know that the master device is protected by an IOMMU. */
++        dt_device_set_protected(dev_to_dt(dev));
++    }
      dev_info(dev, "Added master device (SMMUv3 %s StreamIds %u)\n",
              dev_name(fwspec->iommu_dev), fwspec->num_ids);
 +#ifdef CONFIG_HAS_PCI
 +    if ( dev_is_pci(dev) )
 +    {
 +        struct pci_dev *pdev = dev_to_pci(dev);
 +
-+        ret = arm_smmu_assign_dev(pdev->domain, devfn, dev, 0);
++        /*
-+        if (ret)
++         * During PHYSDEVOP_pci_device_add, Xen does not assign the
-+            goto err_free_master;
++         * device, so we must do it here.
 +         */
 +        if ( pdev->domain )
 +        {
 +            ret = arm_smmu_assign_dev(pdev->domain, devfn, dev, 0);
 +            if (ret)
 +                goto err_free_master;
 +        }
 +    }
 +#endif
 +
      return 0;
  err_free_master:
 @@ -XXX,XX +XXX,XX @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
      struct arm_smmu_domain *smmu_domain;
      struct arm_smmu_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
 +#ifdef CONFIG_HAS_PCI
-+    if ( dev_is_pci(dev) && !is_hardware_domain(d) )
++    if ( dev_is_pci(dev) )
 +    {
 +        struct pci_dev *pdev = dev_to_pci(dev);
 +
-+        printk(XENLOG_INFO "Assigning device %04x:%02x:%02x.%u to dom%d\n",
++        /* Ignore calls for phantom functions */
-+            pdev->seg, pdev->bus, PCI_SLOT(devfn),
++        if ( devfn != pdev->devfn )
-+            PCI_FUNC(devfn), d->domain_id);
++            return 0;
 +
-+        if ( devfn != pdev->devfn || pdev->domain == d )
++        ASSERT(pcidevs_locked());
-+            return 0;
++
-+
++        write_lock(&pdev->domain->pci_lock);
-+        list_move(&pdev->domain_list, &d->pdev_list);
++        list_del(&pdev->domain_list);
 +        write_unlock(&pdev->domain->pci_lock);
 +
 +        pdev->domain = d;
++
++        write_lock(&d->pci_lock);
++        list_add(&pdev->domain_list, &d->pdev_list);
++        write_unlock(&d->pci_lock);
 +
 +        /* dom_io is used as a sentinel for quarantined devices */
 +        if ( d == dom_io )
-+            return 0;
++        {
 +            struct arm_smmu_master *master = dev_iommu_priv_get(dev);
 +            if ( !iommu_quarantine )
 +                return 0;
 +
 +            if ( master && master->domain )
 +                arm_smmu_deassign_dev(master->domain->d, devfn, dev);
 +
 +            return 0;
 +        }
 +    }
 +#endif
 +
      spin_lock(&xen_domain->lock);
 ...
 +#ifdef CONFIG_HAS_PCI
 +    if ( dev_is_pci(dev) )
 +    {
 +        struct pci_dev *pdev = dev_to_pci(dev);
 +
-+        printk(XENLOG_INFO "Deassigning device %04x:%02x:%02x.%u from dom%d\n",
++        /* Ignore calls for phantom functions */
 +            pdev->seg, pdev->bus, PCI_SLOT(devfn),
 +            PCI_FUNC(devfn), d->domain_id);
 +
 +        if ( devfn != pdev->devfn )
 +            return 0;
 +
 +        /* dom_io is used as a sentinel for quarantined devices */
 +        if ( d == dom_io )
 ...
 +
      spin_lock(&xen_domain->lock);
      arm_smmu_detach_dev(master);
 @@ -XXX,XX +XXX,XX @@ static int arm_smmu_reassign_dev(struct domain *s, struct domain *t,
+ {
      int ret = 0;
-     /* Don't allow remapping on other domain than hwdom */
+-    /* Don't allow remapping on other domain than hwdom */
 -    if ( t && !is_hardware_domain(t) )
-+    if ( t && !is_hardware_domain(t) && (t != dom_io) )
++    /* Don't allow remapping on other domain than hwdom
 +     * or dom_io for PCI devices
 +     */
 +    if ( t && !is_hardware_domain(t) && (t != dom_io || !dev_is_pci(dev)) )
          return -EPERM;
      if (t == s)
          return 0;
 -    ret = arm_smmu_deassign_dev(s, dev);
 +    ret = arm_smmu_deassign_dev(s, devfn, dev);
      if (ret)
          return ret;
 --
-.40.1
+.34.1

-New patch
+[PATCH v7 6/8] xen/arm: Fix mapping for PCI bridge mmio region
+From: Rahul Singh <rahul.singh@arm.com>
+Current code skip the mapping for PCI bridge MMIO region to dom0 when
+pci_passthrough_enabled flag is set. Mapping should be skip when
+has_vpci(d) is enabled for the domain, as we need to skip the mapping
+only when VPCI handler are registered for ECAM.
+Signed-off-by: Rahul Singh <rahul.singh@arm.com>
+Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
+Acked-by: Julien Grall <jgrall@amazon.com>
+---
+This patch was originally picked up from [1]
+v6->v7:
+* add Julien's A-b
+v5->v6:
+* drop unrelated change in xen/arch/arm/domain_build.c:handle_linux_pci_domain()
+v4->v5:
+* new patch
+changes since picking up from [1]:
+* rebase on top of "dynamic node programming using overlay dtbo" series
+* replace !is_pci_passthrough_enabled() check with !IS_ENABLED(CONFIG_HAS_PCI)
+  instead of removing
+[1] https://lists.xenproject.org/archives/html/xen-devel/2023-07/msg00483.html
+---
+ xen/arch/arm/device.c | 2 +-
+file changed, 1 insertion(+), 1 deletion(-)
+diff --git a/xen/arch/arm/device.c b/xen/arch/arm/device.c
+index XXXXXXX..XXXXXXX 100644
+--- a/xen/arch/arm/device.c
++++ b/xen/arch/arm/device.c
+@@ -XXX,XX +XXX,XX @@ int handle_device(struct domain *d, struct dt_device_node *dev, p2m_type_t p2mt,
+         .d = d,
+         .p2mt = p2mt,
+         .skip_mapping = !own_device ||
+-                        (is_pci_passthrough_enabled() &&
++                        (has_vpci(d) &&
+                         (device_get_class(dev) == DEVICE_PCI_HOSTBRIDGE)),
+         .iomem_ranges = iomem_ranges,
+         .irq_ranges = irq_ranges
+--
+.34.1

-[PATCH v4 2/7] xen/arm: Move is_protected flag to struct device
+[PATCH v7 7/8] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no
-From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
+From: Stewart Hildebrand <stewart.hildebrand@amd.com>
-This flag will be re-used for PCI devices by the subsequent
+Enable the use of IOMMU + PCI in dom0 without having to specify
-patches.
+"pci-passthrough=yes". We rely on dom0 to initialize the PCI controller
 and perform a PHYSDEVOP_pci_device_add call to add each device to SMMU.
-Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
+Enable pci_init() for initializing Xen's internal PCI subsystem, and
 allow PHYSDEVOP_pci_device_add when pci-passthrough is disabled.
 is_pci_passthrough_enabled() is not an Arm-only construct, so remove the
 x86 definition of the function.
 Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
+Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
 ---
-v3->v4:
+hmm. Since
-* move is_protected flag within struct device to reduce padding
+  dec9e02f3190 ("xen: avoid generation of stub <asm/pci.h> header")
-* re-add device_is_protected checks in add_device hooks in smmu-v3.c/ipmmu-vmsa.c
+Should we also move is_pci_passthrough_enabled() back to xen/arch/arm/include/asm/pci.h ?
-* split mmu-masters check into separate patch
+Not sure if PPC/RISC-V will plan on using this check.
-v2->v3:
+v6->v7:
-* no change
+* remove x86 definition of is_pci_passthrough_enabled()
 * update comments
 * make pci_physdev_op checks stricter
-v1->v2:
+v5->v6:
-* no change
+* new patch - this effectively replaces
   ("Revert "xen/arm: Add cmdline boot option "pci-passthrough = <boolean>""")
 ---
  xen/arch/arm/pci/pci.c         | 5 +++--
  xen/arch/x86/include/asm/pci.h | 6 ------
  xen/drivers/pci/physdev.c      | 4 ++--
 files changed, 5 insertions(+), 10 deletions(-)
-downstream->v1:
+diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
 * rebase
 * s/dev_node->is_protected/dev_node->dev.is_protected/ in smmu.c
 * s/dt_device_set_protected(dev_to_dt(dev))/device_set_protected(dev)/ in smmu-v3.c
 * remove redundant device_is_protected checks in smmu-v3.c/ipmmu-vmsa.c
 (cherry picked from commit 59753aac77528a584d3950936b853ebf264b68e7 from
  the downstream branch poc/pci-passthrough from
  https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
 ---
  xen/arch/arm/domain_build.c              |  4 ++--
  xen/arch/arm/include/asm/device.h        | 14 ++++++++++++++
  xen/common/device_tree.c                 |  2 +-
  xen/drivers/passthrough/arm/ipmmu-vmsa.c |  4 ++--
  xen/drivers/passthrough/arm/smmu-v3.c    |  5 +++--
  xen/drivers/passthrough/arm/smmu.c       |  2 +-
  xen/drivers/passthrough/device_tree.c    |  8 ++++----
  xen/include/xen/device_tree.h            | 13 -------------
 files changed, 27 insertions(+), 25 deletions(-)
 diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
 index XXXXXXX..XXXXXXX 100644
---- a/xen/arch/arm/domain_build.c
+--- a/xen/arch/arm/pci/pci.c
-+++ b/xen/arch/arm/domain_build.c
++++ b/xen/arch/arm/pci/pci.c
-@@ -XXX,XX +XXX,XX @@ static int __init handle_device(struct domain *d, struct dt_device_node *dev,
+@@ -XXX,XX +XXX,XX @@
-             return res;
+ #include <xen/device_tree.h>
-         }
+ #include <xen/errno.h>
+ #include <xen/init.h>
--        if ( dt_device_is_protected(dev) )
++#include <xen/iommu.h>
-+        if ( device_is_protected(dt_to_dev(dev)) )
+ #include <xen/param.h>
-         {
+ #include <xen/pci.h>
-             dt_dprintk("%s setup iommu\n", dt_node_full_name(dev));
-             res = iommu_assign_dt_device(d, dev);
+@@ -XXX,XX +XXX,XX @@ static int __init pci_init(void)
-@@ -XXX,XX +XXX,XX @@ static int __init handle_passthrough_prop(struct kernel_info *kinfo,
+ {
-         return res;
+     /*
+      * Enable PCI passthrough when has been enabled explicitly
-     /* If xen_force, we allow assignment of devices without IOMMU protection. */
+-     * (pci-passthrough=on).
--    if ( xen_force && !dt_device_is_protected(node) )
++     * (pci-passthrough=on) or IOMMU is present and enabled.
-+    if ( xen_force && !device_is_protected(dt_to_dev(node)) )
+      */
 -    if ( !pci_passthrough_enabled )
 +    if ( !is_pci_passthrough_enabled() && !iommu_enabled )
          return 0;
-     return iommu_assign_dt_device(kinfo->d, node);
+     pci_segments_init();
-diff --git a/xen/arch/arm/include/asm/device.h b/xen/arch/arm/include/asm/device.h
+diff --git a/xen/arch/x86/include/asm/pci.h b/xen/arch/x86/include/asm/pci.h
 index XXXXXXX..XXXXXXX 100644
---- a/xen/arch/arm/include/asm/device.h
+--- a/xen/arch/x86/include/asm/pci.h
-+++ b/xen/arch/arm/include/asm/device.h
++++ b/xen/arch/x86/include/asm/pci.h
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ bool pci_ro_mmcfg_decode(unsigned long mfn, unsigned int *seg,
- #ifndef __ASM_ARM_DEVICE_H
+ extern int pci_mmcfg_config_num;
- #define __ASM_ARM_DEVICE_H
+ extern struct acpi_mcfg_allocation *pci_mmcfg_config;
-+#include <xen/types.h>
+-/* Unlike ARM, PCI passthrough is always enabled for x86. */
-+
+-static always_inline bool is_pci_passthrough_enabled(void)
  enum device_type
  {
      DEV_DT,
@@ -XXX,XX +XXX,XX @@ struct dev_archdata {
  struct device
  {
      enum device_type type;
 +    bool is_protected; /* Shows that device is protected by IOMMU */
 +    uint8_t _pad[3];
  #ifdef CONFIG_HAS_DEVICE_TREE
      struct dt_device_node *of_node; /* Used by drivers imported from Linux */
  #endif
@@ -XXX,XX +XXX,XX @@ int device_init(struct dt_device_node *dev, enum device_class class,
   */
  enum device_class device_get_class(const struct dt_device_node *dev);
 +static inline void device_set_protected(struct device *device)
 +{
 +    device->is_protected = true;
 +}
 +
 +static inline bool device_is_protected(const struct device *device)
 +{
 +    return device->is_protected;
 +}
 +
  #define DT_DEVICE_START(_name, _namestr, _class)                    \
  static const struct device_desc __dev_desc_##_name __used           \
  __section(".dev.info") = {                                          \
 diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
 index XXXXXXX..XXXXXXX 100644
 --- a/xen/common/device_tree.c
 +++ b/xen/common/device_tree.c
@@ -XXX,XX +XXX,XX @@ static unsigned long __init unflatten_dt_node(const void *fdt,
          /* By default dom0 owns the device */
          np->used_by = 0;
          /* By default the device is not protected */
 -        np->is_protected = false;
 +        np->dev.is_protected = false;
          INIT_LIST_HEAD(&np->domain_list);
          if ( new_format )
 diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
 index XXXXXXX..XXXXXXX 100644
 --- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
 +++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
@@ -XXX,XX +XXX,XX @@ static int ipmmu_add_device(u8 devfn, struct device *dev)
      if ( !to_ipmmu(dev) )
          return -ENODEV;
 -    if ( dt_device_is_protected(dev_to_dt(dev)) )
 +    if ( device_is_protected(dev) )
      {
          dev_err(dev, "Already added to IPMMU\n");
          return -EEXIST;
      }
      /* Let Xen know that the master device is protected by an IOMMU. */
 -    dt_device_set_protected(dev_to_dt(dev));
 +    device_set_protected(dev);
      dev_info(dev, "Added master device (IPMMU %s micro-TLBs %u)\n",
               dev_name(fwspec->iommu_dev), fwspec->num_ids);
 diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
 index XXXXXXX..XXXXXXX 100644
 --- a/xen/drivers/passthrough/arm/smmu-v3.c
 +++ b/xen/drivers/passthrough/arm/smmu-v3.c
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_add_device(u8 devfn, struct device *dev)
       */
      arm_smmu_enable_pasid(master);
 -    if (dt_device_is_protected(dev_to_dt(dev))) {
 +    if ( device_is_protected(dev) )
 +    {
          dev_err(dev, "Already added to SMMUv3\n");
          return -EEXIST;
      }
      /* Let Xen know that the master device is protected by an IOMMU. */
 -    dt_device_set_protected(dev_to_dt(dev));
 +    device_set_protected(dev);
      dev_info(dev, "Added master device (SMMUv3 %s StreamIds %u)\n",
              dev_name(fwspec->iommu_dev), fwspec->num_ids);
 diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/arm/smmu.c
 index XXXXXXX..XXXXXXX 100644
 --- a/xen/drivers/passthrough/arm/smmu.c
 +++ b/xen/drivers/passthrough/arm/smmu.c
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_dt_add_device_legacy(struct arm_smmu_device *smmu,
      master->of_node = dev_node;
      /* Xen: Let Xen know that the device is protected by an SMMU */
 -    dt_device_set_protected(dev_node);
 +    device_set_protected(dev);
      for (i = 0; i < fwspec->num_ids; ++i) {
          if (!(smmu->features & ARM_SMMU_FEAT_STREAM_MATCH) &&
 diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
 index XXXXXXX..XXXXXXX 100644
 --- a/xen/drivers/passthrough/device_tree.c
 +++ b/xen/drivers/passthrough/device_tree.c
@@ -XXX,XX +XXX,XX @@ int iommu_assign_dt_device(struct domain *d, struct dt_device_node *dev)
      if ( !is_iommu_enabled(d) )
          return -EINVAL;
 -    if ( !dt_device_is_protected(dev) )
 +    if ( !device_is_protected(dt_to_dev(dev)) )
          return -EINVAL;
      spin_lock(&dtdevs_lock);
@@ -XXX,XX +XXX,XX @@ int iommu_deassign_dt_device(struct domain *d, struct dt_device_node *dev)
      if ( !is_iommu_enabled(d) )
          return -EINVAL;
 -    if ( !dt_device_is_protected(dev) )
 +    if ( !device_is_protected(dt_to_dev(dev)) )
          return -EINVAL;
      spin_lock(&dtdevs_lock);
@@ -XXX,XX +XXX,XX @@ static bool_t iommu_dt_device_is_assigned(const struct dt_device_node *dev)
  {
      bool_t assigned = 0;
 -    if ( !dt_device_is_protected(dev) )
 +    if ( !device_is_protected(dt_to_dev(dev)) )
          return 0;
      spin_lock(&dtdevs_lock);
@@ -XXX,XX +XXX,XX @@ int iommu_add_dt_device(struct dt_device_node *np)
       * and iommus property, there is no need to register it again. In this case
       * simply return success early.
       */
 -    if ( dt_device_is_protected(np) )
 +    if ( device_is_protected(dev) )
          return 0;
      if ( dev_iommu_fwspec_get(dev) )
 diff --git a/xen/include/xen/device_tree.h b/xen/include/xen/device_tree.h
 index XXXXXXX..XXXXXXX 100644
 --- a/xen/include/xen/device_tree.h
 +++ b/xen/include/xen/device_tree.h
@@ -XXX,XX +XXX,XX @@ struct dt_device_node {
      struct dt_device_node *next; /* TODO: Remove it. Only use to know the last children */
      struct dt_device_node *allnext;
 -    /* IOMMU specific fields */
 -    bool is_protected;
 -
      /* HACK: Remove this if there is a need of space */
      bool_t static_evtchn_created;
@@ -XXX,XX +XXX,XX @@ static inline domid_t dt_device_used_by(const struct dt_device_node *device)
      return device->used_by;
  }
 -static inline void dt_device_set_protected(struct dt_device_node *device)
 -{
--    device->is_protected = true;
+-    return true;
 -}
 -
--static inline bool dt_device_is_protected(const struct dt_device_node *device)
+ void arch_pci_init_pdev(struct pci_dev *pdev);
--{
--    return device->is_protected;
+ static inline bool pci_check_bar(const struct pci_dev *pdev,
--}
+diff --git a/xen/drivers/pci/physdev.c b/xen/drivers/pci/physdev.c
--
+index XXXXXXX..XXXXXXX 100644
- static inline bool_t dt_property_name_is_equal(const struct dt_property *pp,
+--- a/xen/drivers/pci/physdev.c
-                                                const char *name)
++++ b/xen/drivers/pci/physdev.c
- {
+@@ -XXX,XX +XXX,XX @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
          struct pci_dev_info pdev_info;
          nodeid_t node = NUMA_NO_NODE;
 -        if ( !is_pci_passthrough_enabled() )
 +        if ( !is_pci_passthrough_enabled() && !iommu_enabled )
              return -EOPNOTSUPP;
          ret = -EFAULT;
@@ -XXX,XX +XXX,XX @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
      case PHYSDEVOP_pci_device_remove: {
          struct physdev_pci_device dev;
 -        if ( !is_pci_passthrough_enabled() )
 +        if ( !is_pci_passthrough_enabled() && !iommu_enabled )
              return -EOPNOTSUPP;
          ret = -EFAULT;
 --
-.40.1
+.34.1

-[PATCH v4 1/7] xen/arm: Improve readability of check for registered devices
+[PATCH v7 8/8] xen/arm: Map ITS doorbell register to IOMMU page tables
-From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
+From: Rahul Singh <rahul.singh@arm.com>
-Improve readability of check for devices already registered with the SMMU with
+When ITS is enabled and PCI devices that are behind an SMMU generate an
-legacy mmu-masters DT bindings by using is_protected.
+MSI interrupt, SMMU fault will be observed as there is currently no
 mapping in p2m table for the ITS translation register (GITS_TRANSLATER).
-There are 2 device tree bindings for registering a device with the SMMU:
+A mapping is required in the iommu page tables so that the device can
-* mmu-masters (legacy, SMMUv1/2 only)
+generate the MSI interrupt writing to the GITS_TRANSLATER register.
 * iommus
-A device tree may include both mmu-masters and iommus properties (although it is
+The GITS_TRANSLATER register is a 32-bit register, and there is nothing
-unnecessary to do so). When a device appears in the mmu-masters list,
+else in a page containing it, so map that page.
 np->is_protected and dev->iommu_fwspec both get set by the SMMUv1/2 driver. The
 function iommu_add_dt_device() is subsequently invoked for devices that have an
 iommus specification.
-The check as it was before this patch:
+Signed-off-by: Rahul Singh <rahul.singh@arm.com>
 Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
 Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
 ---
 This patch was originally picked up from [1], and commit description
 loosely borrowed from [2].
-  if ( dev_iommu_fwspec_get(dev) )
+Example SMMUv3 fault (qemu-system-aarch64 virt model), ITS base 0x8080000:
       return 0;
-and the new check:
+(XEN) SMMUv3: /smmuv3@9050000: event 0x10 received:
 (XEN) SMMUv3: /smmuv3@9050000:  0x0000000800000010
 (XEN) SMMUv3: /smmuv3@9050000:  0x0000008000000000
 (XEN) SMMUv3: /smmuv3@9050000:  0x0000000008090040
 (XEN) SMMUv3: /smmuv3@9050000:  0x0000000000000000
-  if ( dt_device_is_protected(np) )
+Example SMMUv2 fault (AMD/Xilinx Versal), ITS base 0xf9020000:
       return 0;
-are guarding against the same corner case: when a device has both mmu-masters
+(XEN) smmu: /axi/smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0xf9030040, fsynr=0x12, cb=0
 and iommus specifications in the device tree. The is_protected naming is more
 descriptive.
-If np->is_protected is not set (i.e. false), but dev->iommu_fwspec is set, it is
+v6->v7:
-an error condition, so return an error in this case.
+* add tlb flush after mapping
 * style: update formatting
 * revert back to printk with XENLOG_G_ERR
-Expand the comment to further clarify the corner case.
+v5->v6:
 * switch to iommu_map() interface
 * fix page_count argument
 * style fixup
 * use gprintk instead of printk
 * add my Signed-off-by
 * move to vgic_v3_its_init_virtual()
-Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
+v4->v5:
-Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
+* new patch
 [1] https://lists.xenproject.org/archives/html/xen-devel/2023-07/msg00483.html
 [2] https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-/commit/6232a0d53377009bb7fbc3c3ab81d0153734be6b
 ---
-v3->v4:
+ xen/arch/arm/vgic-v3-its.c | 20 ++++++++++++++++++++
-* new patch: this change was split from ("xen/arm: Move is_protected flag to struct device")
+file changed, 20 insertions(+)
 ---
  xen/drivers/passthrough/device_tree.c | 11 ++++++++---
 file changed, 8 insertions(+), 3 deletions(-)
-diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
+diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
 index XXXXXXX..XXXXXXX 100644
---- a/xen/drivers/passthrough/device_tree.c
+--- a/xen/arch/arm/vgic-v3-its.c
-+++ b/xen/drivers/passthrough/device_tree.c
++++ b/xen/arch/arm/vgic-v3-its.c
-@@ -XXX,XX +XXX,XX @@ int iommu_add_dt_device(struct dt_device_node *np)
+@@ -XXX,XX +XXX,XX @@ static int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr,
-         return -EINVAL;
+     register_mmio_handler(d, &vgic_its_mmio_handler, guest_addr, SZ_64K, its);
-     /*
--     * The device may already have been registered. As there is no harm in
++    if ( is_iommu_enabled(its->d) )
--     * it just return success early.
++    {
-+     * Devices that appear in the legacy mmu-masters list may have already been
++        mfn_t mfn = maddr_to_mfn(its->doorbell_address);
-+     * registered with the SMMU. In case a device has both a mmu-masters entry
++        unsigned int flush_flags = 0;
-+     * and iommus property, there is no need to register it again. In this case
++        int ret = iommu_map(its->d, _dfn(mfn_x(mfn)), mfn, 1, IOMMUF_writable,
-+     * simply return success early.
++                            &flush_flags);
       */
 -    if ( dev_iommu_fwspec_get(dev) )
 +    if ( dt_device_is_protected(np) )
          return 0;
 +    if ( dev_iommu_fwspec_get(dev) )
 +        return -EEXIST;
 +
-     /*
++        if ( ret < 0 )
-      * According to the Documentation/devicetree/bindings/iommu/iommu.txt
++        {
-      * from Linux.
++            printk(XENLOG_G_ERR
 +                    "GICv3: Map ITS translation register for %pd failed.\n",
 +                    its->d);
 +            return ret;
 +        }
 +
 +        ret = iommu_iotlb_flush(its->d, _dfn(mfn_x(mfn)), 1, flush_flags);
 +        if ( ret < 0 )
 +            return ret;
 +    }
 +
      /* Register the virtual ITS to be able to clean it up later. */
      list_add_tail(&its->vits_list, &d->arch.vgic.vits_list);
 --
-.40.1
+.34.1

This series introduces SMMU handling for PCIe passthrough on ARM. These patches
are independent from (and don't depend on) the vPCI reference counting/locking
work in progress, and should be able to be upstreamed independently.

v3->v4:
* split a change from ("xen/arm: Move is_protected flag to struct device") into
  a new separate patch
* see individual patches for further details

v2->v3:
* drop "pci/arm: Use iommu_add_dt_pci_device()"
* drop "RFC: pci/arm: don't do iommu call for phantom functions"
* move invocation of sideband ID mapping function to add_device()
  platform_ops/iommu_ops hook

v1->v2:
* phantom device handling
* shuffle around iommu_add_dt_pci_device()

Oleksandr Andrushchenko (1):
  xen/arm: smmuv2: Add PCI devices support for SMMUv2

Oleksandr Tyshchenko (4):
  xen/arm: Improve readability of check for registered devices
  xen/arm: Move is_protected flag to struct device
  iommu/arm: Add iommu_dt_xlate()
  iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API

Rahul Singh (1):
  xen/arm: smmuv3: Add PCI devices support for SMMUv3

Stewart Hildebrand (1):
  iommu/arm: iommu_add_dt_pci_sideband_ids phantom handling

xen/arch/arm/domain_build.c              |   4 +-
 xen/arch/arm/include/asm/device.h        |  14 ++
 xen/common/device_tree.c                 |   2 +-
 xen/drivers/passthrough/arm/ipmmu-vmsa.c |   4 +-
 xen/drivers/passthrough/arm/smmu-v3.c    |  81 ++++++++-
 xen/drivers/passthrough/arm/smmu.c       | 122 +++++++++++---
 xen/drivers/passthrough/device_tree.c    | 205 ++++++++++++++++++++---
 xen/include/xen/device_tree.h            |  38 +++--
 xen/include/xen/iommu.h                  |  22 ++-
 9 files changed, 423 insertions(+), 69 deletions(-)

-- 
2.40.1

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

Improve readability of check for devices already registered with the SMMU with
legacy mmu-masters DT bindings by using is_protected.

There are 2 device tree bindings for registering a device with the SMMU:
* mmu-masters (legacy, SMMUv1/2 only)
* iommus

A device tree may include both mmu-masters and iommus properties (although it is
unnecessary to do so). When a device appears in the mmu-masters list,
np->is_protected and dev->iommu_fwspec both get set by the SMMUv1/2 driver. The
function iommu_add_dt_device() is subsequently invoked for devices that have an
iommus specification.

The check as it was before this patch:

if ( dev_iommu_fwspec_get(dev) )
      return 0;

and the new check:

if ( dt_device_is_protected(np) )
      return 0;

are guarding against the same corner case: when a device has both mmu-masters
and iommus specifications in the device tree. The is_protected naming is more
descriptive.

If np->is_protected is not set (i.e. false), but dev->iommu_fwspec is set, it is
an error condition, so return an error in this case.

Expand the comment to further clarify the corner case.

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
---
v3->v4:
* new patch: this change was split from ("xen/arm: Move is_protected flag to struct device")
---
 xen/drivers/passthrough/device_tree.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

This flag will be re-used for PCI devices by the subsequent
patches.

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
---
v3->v4:
* move is_protected flag within struct device to reduce padding
* re-add device_is_protected checks in add_device hooks in smmu-v3.c/ipmmu-vmsa.c
* split mmu-masters check into separate patch

v2->v3:
* no change

v1->v2:
* no change

downstream->v1:
* rebase
* s/dev_node->is_protected/dev_node->dev.is_protected/ in smmu.c
* s/dt_device_set_protected(dev_to_dt(dev))/device_set_protected(dev)/ in smmu-v3.c
* remove redundant device_is_protected checks in smmu-v3.c/ipmmu-vmsa.c

(cherry picked from commit 59753aac77528a584d3950936b853ebf264b68e7 from
 the downstream branch poc/pci-passthrough from
 https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
---
 xen/arch/arm/domain_build.c              |  4 ++--
 xen/arch/arm/include/asm/device.h        | 14 ++++++++++++++
 xen/common/device_tree.c                 |  2 +-
 xen/drivers/passthrough/arm/ipmmu-vmsa.c |  4 ++--
 xen/drivers/passthrough/arm/smmu-v3.c    |  5 +++--
 xen/drivers/passthrough/arm/smmu.c       |  2 +-
 xen/drivers/passthrough/device_tree.c    |  8 ++++----
 xen/include/xen/device_tree.h            | 13 -------------
 8 files changed, 27 insertions(+), 25 deletions(-)

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index XXXXXXX..XXXXXXX 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -XXX,XX +XXX,XX @@ static int __init handle_device(struct domain *d, struct dt_device_node *dev,
             return res;
         }
 
-        if ( dt_device_is_protected(dev) )
+        if ( device_is_protected(dt_to_dev(dev)) )
         {
             dt_dprintk("%s setup iommu\n", dt_node_full_name(dev));
             res = iommu_assign_dt_device(d, dev);
@@ -XXX,XX +XXX,XX @@ static int __init handle_passthrough_prop(struct kernel_info *kinfo,
         return res;
 
     /* If xen_force, we allow assignment of devices without IOMMU protection. */
-    if ( xen_force && !dt_device_is_protected(node) )
+    if ( xen_force && !device_is_protected(dt_to_dev(node)) )
         return 0;
 
     return iommu_assign_dt_device(kinfo->d, node);
diff --git a/xen/arch/arm/include/asm/device.h b/xen/arch/arm/include/asm/device.h
index XXXXXXX..XXXXXXX 100644
--- a/xen/arch/arm/include/asm/device.h
+++ b/xen/arch/arm/include/asm/device.h
@@ -XXX,XX +XXX,XX @@
 #ifndef __ASM_ARM_DEVICE_H
 #define __ASM_ARM_DEVICE_H
 
+#include <xen/types.h>
+
 enum device_type
 {
     DEV_DT,
@@ -XXX,XX +XXX,XX @@ struct dev_archdata {
 struct device
 {
     enum device_type type;
+    bool is_protected; /* Shows that device is protected by IOMMU */
+    uint8_t _pad[3];
 #ifdef CONFIG_HAS_DEVICE_TREE
     struct dt_device_node *of_node; /* Used by drivers imported from Linux */
 #endif
@@ -XXX,XX +XXX,XX @@ int device_init(struct dt_device_node *dev, enum device_class class,
  */
 enum device_class device_get_class(const struct dt_device_node *dev);
 
+static inline void device_set_protected(struct device *device)
+{
+    device->is_protected = true;
+}
+
+static inline bool device_is_protected(const struct device *device)
+{
+    return device->is_protected;
+}
+
 #define DT_DEVICE_START(_name, _namestr, _class)                    \
 static const struct device_desc __dev_desc_##_name __used           \
 __section(".dev.info") = {                                          \
diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
index XXXXXXX..XXXXXXX 100644
--- a/xen/common/device_tree.c
+++ b/xen/common/device_tree.c
@@ -XXX,XX +XXX,XX @@ static unsigned long __init unflatten_dt_node(const void *fdt,
         /* By default dom0 owns the device */
         np->used_by = 0;
         /* By default the device is not protected */
-        np->is_protected = false;
+        np->dev.is_protected = false;
         INIT_LIST_HEAD(&np->domain_list);
 
         if ( new_format )
diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
index XXXXXXX..XXXXXXX 100644
--- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
+++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
@@ -XXX,XX +XXX,XX @@ static int ipmmu_add_device(u8 devfn, struct device *dev)
     if ( !to_ipmmu(dev) )
         return -ENODEV;
 
-    if ( dt_device_is_protected(dev_to_dt(dev)) )
+    if ( device_is_protected(dev) )
     {
         dev_err(dev, "Already added to IPMMU\n");
         return -EEXIST;
     }
 
     /* Let Xen know that the master device is protected by an IOMMU. */
-    dt_device_set_protected(dev_to_dt(dev));
+    device_set_protected(dev);
 
     dev_info(dev, "Added master device (IPMMU %s micro-TLBs %u)\n",
              dev_name(fwspec->iommu_dev), fwspec->num_ids);
diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
index XXXXXXX..XXXXXXX 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.c
+++ b/xen/drivers/passthrough/arm/smmu-v3.c
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_add_device(u8 devfn, struct device *dev)
 	 */
 	arm_smmu_enable_pasid(master);
 
-	if (dt_device_is_protected(dev_to_dt(dev))) {
+	if ( device_is_protected(dev) )
+	{
 		dev_err(dev, "Already added to SMMUv3\n");
 		return -EEXIST;
 	}
 
 	/* Let Xen know that the master device is protected by an IOMMU. */
-	dt_device_set_protected(dev_to_dt(dev));
+	device_set_protected(dev);
 
 	dev_info(dev, "Added master device (SMMUv3 %s StreamIds %u)\n",
 			dev_name(fwspec->iommu_dev), fwspec->num_ids);
diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/arm/smmu.c
index XXXXXXX..XXXXXXX 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_dt_add_device_legacy(struct arm_smmu_device *smmu,
 	master->of_node = dev_node;
 
 	/* Xen: Let Xen know that the device is protected by an SMMU */
-	dt_device_set_protected(dev_node);
+	device_set_protected(dev);
 
 	for (i = 0; i < fwspec->num_ids; ++i) {
 		if (!(smmu->features & ARM_SMMU_FEAT_STREAM_MATCH) &&
diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
index XXXXXXX..XXXXXXX 100644
--- a/xen/drivers/passthrough/device_tree.c
+++ b/xen/drivers/passthrough/device_tree.c
@@ -XXX,XX +XXX,XX @@ int iommu_assign_dt_device(struct domain *d, struct dt_device_node *dev)
     if ( !is_iommu_enabled(d) )
         return -EINVAL;
 
-    if ( !dt_device_is_protected(dev) )
+    if ( !device_is_protected(dt_to_dev(dev)) )
         return -EINVAL;
 
     spin_lock(&dtdevs_lock);
@@ -XXX,XX +XXX,XX @@ int iommu_deassign_dt_device(struct domain *d, struct dt_device_node *dev)
     if ( !is_iommu_enabled(d) )
         return -EINVAL;
 
-    if ( !dt_device_is_protected(dev) )
+    if ( !device_is_protected(dt_to_dev(dev)) )
         return -EINVAL;
 
     spin_lock(&dtdevs_lock);
@@ -XXX,XX +XXX,XX @@ static bool_t iommu_dt_device_is_assigned(const struct dt_device_node *dev)
 {
     bool_t assigned = 0;
 
-    if ( !dt_device_is_protected(dev) )
+    if ( !device_is_protected(dt_to_dev(dev)) )
         return 0;
 
     spin_lock(&dtdevs_lock);
@@ -XXX,XX +XXX,XX @@ int iommu_add_dt_device(struct dt_device_node *np)
      * and iommus property, there is no need to register it again. In this case
      * simply return success early.
      */
-    if ( dt_device_is_protected(np) )
+    if ( device_is_protected(dev) )
         return 0;
 
     if ( dev_iommu_fwspec_get(dev) )
diff --git a/xen/include/xen/device_tree.h b/xen/include/xen/device_tree.h
index XXXXXXX..XXXXXXX 100644
--- a/xen/include/xen/device_tree.h
+++ b/xen/include/xen/device_tree.h
@@ -XXX,XX +XXX,XX @@ struct dt_device_node {
     struct dt_device_node *next; /* TODO: Remove it. Only use to know the last children */
     struct dt_device_node *allnext;
 
-    /* IOMMU specific fields */
-    bool is_protected;
-
     /* HACK: Remove this if there is a need of space */
     bool_t static_evtchn_created;
 
@@ -XXX,XX +XXX,XX @@ static inline domid_t dt_device_used_by(const struct dt_device_node *device)
     return device->used_by;
 }
 
-static inline void dt_device_set_protected(struct dt_device_node *device)
-{
-    device->is_protected = true;
-}
-
-static inline bool dt_device_is_protected(const struct dt_device_node *device)
-{
-    return device->is_protected;
-}
-
 static inline bool_t dt_property_name_is_equal(const struct dt_property *pp,
                                                const char *name)
 {
-- 
2.40.1

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

Move code for processing DT IOMMU specifier to a separate helper.
This helper will be re-used for adding PCI devices by the subsequent
patches as we will need exact the same actions for processing
DT PCI-IOMMU specifier.

While at it introduce NO_IOMMU to avoid magic "1".

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
---
v3->v4:
* make dt_phandle_args *iommu_spec const
* move !ops->add_device check to helper

v2->v3:
* no change

v1->v2:
* no change

downstream->v1:
* trivial rebase
* s/dt_iommu_xlate/iommu_dt_xlate/

(cherry picked from commit c26bab0415ca303df86aba1d06ef8edc713734d3 from
 the downstream branch poc/pci-passthrough from
 https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
---
 xen/drivers/passthrough/device_tree.c | 47 ++++++++++++++++++---------
 1 file changed, 31 insertions(+), 16 deletions(-)

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

The main purpose of this patch is to add a way to register PCI device
(which is behind the IOMMU) using the generic PCI-IOMMU DT bindings [1]
before assigning that device to a domain.

This behaves similarly to the existing iommu_add_dt_device API, except it
handles PCI devices, and it is to be invoked from the add_device hook in the
SMMU driver.

The function of_map_id to translate an ID through a downstream mapping
(which is also suitable for mapping Requester ID) was borrowed from Linux
(v5.10-rc6) and updated according to the Xen code base.

[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/pci-iommu.txt

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
---
v3->v4:
* wrap #include <asm/acpi.h> and if ( acpi_disabled ) in #ifdef CONFIG_ACPI
* fix Michal's remarks about style, parenthesis, and print formats
* remove !ops->dt_xlate check since it is already in iommu_dt_xlate helper
* rename s/iommu_dt_pci_map_id/dt_map_id/ because it is generic, not specific
  to iommu
* update commit description

v2->v3:
* new patch title (was: iommu/arm: Introduce iommu_add_dt_pci_device API)
* renamed function
  from: iommu_add_dt_pci_device
  to: iommu_add_dt_pci_sideband_ids
* removed stale ops->add_device check
* iommu.h: add empty stub iommu_add_dt_pci_sideband_ids for !HAS_DEVICE_TREE
* iommu.h: add iommu_add_pci_sideband_ids helper
* iommu.h: don't wrap prototype in #ifdef CONFIG_HAS_PCI
* s/iommu_fwspec_free(pci_to_dev(pdev))/iommu_fwspec_free(dev)/

v1->v2:
* remove extra devfn parameter since pdev fully describes the device
* remove ops->add_device() call from iommu_add_dt_pci_device(). Instead, rely on
  the existing iommu call in iommu_add_device().
* move the ops->add_device and ops->dt_xlate checks earlier

downstream->v1:
* rebase
* add const qualifier to struct dt_device_node *np arg in dt_map_id()
* add const qualifier to struct dt_device_node *np declaration in iommu_add_pci_device()
* use stdint.h types instead of u8/u32/etc...
* rename functions:
  s/dt_iommu_xlate/iommu_dt_xlate/
  s/dt_map_id/iommu_dt_pci_map_id/
  s/iommu_add_pci_device/iommu_add_dt_pci_device/
* add device_is_protected check in iommu_add_dt_pci_device
* wrap prototypes in CONFIG_HAS_PCI

(cherry picked from commit 734e3bf6ee77e7947667ab8fa96c25b349c2e1da from
 the downstream branch poc/pci-passthrough from
 https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
---
 xen/drivers/passthrough/device_tree.c | 134 ++++++++++++++++++++++++++
 xen/include/xen/device_tree.h         |  25 +++++
 xen/include/xen/iommu.h               |  22 ++++-
 3 files changed, 180 insertions(+), 1 deletion(-)

diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
index XXXXXXX..XXXXXXX 100644
--- a/xen/drivers/passthrough/device_tree.c
+++ b/xen/drivers/passthrough/device_tree.c
@@ -XXX,XX +XXX,XX @@ static int iommu_dt_xlate(struct device *dev,
     return ops->dt_xlate(dev, iommu_spec);
 }
 
+#ifdef CONFIG_HAS_PCI
+int dt_map_id(const struct dt_device_node *np, uint32_t id,
+              const char *map_name, const char *map_mask_name,
+              struct dt_device_node **target, uint32_t *id_out)
+{
+    uint32_t map_mask, masked_id, map_len;
+    const __be32 *map = NULL;
+
+    if ( !np || !map_name || (!target && !id_out) )
+        return -EINVAL;
+
+    map = dt_get_property(np, map_name, &map_len);
+    if ( !map )
+    {
+        if ( target )
+            return -ENODEV;
+
+        /* Otherwise, no map implies no translation */
+        *id_out = id;
+        return 0;
+    }
+
+    if ( !map_len || (map_len % (4 * sizeof(*map))) )
+    {
+        printk(XENLOG_ERR "%s: Error: Bad %s length: %u\n", np->full_name,
+               map_name, map_len);
+        return -EINVAL;
+    }
+
+    /* The default is to select all bits. */
+    map_mask = 0xffffffff;
+
+    /*
+     * Can be overridden by "{iommu,msi}-map-mask" property.
+     * If df_property_read_u32() fails, the default is used.
+     */
+    if ( map_mask_name )
+        dt_property_read_u32(np, map_mask_name, &map_mask);
+
+    masked_id = map_mask & id;
+    for ( ; (int)map_len > 0; map_len -= 4 * sizeof(*map), map += 4 )
+    {
+        struct dt_device_node *phandle_node;
+        uint32_t id_base = be32_to_cpup(map + 0);
+        uint32_t phandle = be32_to_cpup(map + 1);
+        uint32_t out_base = be32_to_cpup(map + 2);
+        uint32_t id_len = be32_to_cpup(map + 3);
+
+        if ( id_base & ~map_mask )
+        {
+            printk(XENLOG_ERR "%s: Invalid %s translation - %s-mask (0x%"PRIx32") ignores id-base (0x%"PRIx32")\n",
+                   np->full_name, map_name, map_name, map_mask, id_base);
+            return -EFAULT;
+        }
+
+        if ( (masked_id < id_base) || (masked_id >= (id_base + id_len)) )
+            continue;
+
+        phandle_node = dt_find_node_by_phandle(phandle);
+        if ( !phandle_node )
+            return -ENODEV;
+
+        if ( target )
+        {
+            if ( !*target )
+                *target = phandle_node;
+
+            if ( *target != phandle_node )
+                continue;
+        }
+
+        if ( id_out )
+            *id_out = masked_id - id_base + out_base;
+
+        printk(XENLOG_DEBUG "%s: %s, using mask %08"PRIx32", id-base: %08"PRIx32", out-base: %08"PRIx32", length: %08"PRIx32", id: %08"PRIx32" -> %08"PRIx32"\n",
+               np->full_name, map_name, map_mask, id_base, out_base, id_len, id,
+               masked_id - id_base + out_base);
+        return 0;
+    }
+
+    printk(XENLOG_ERR "%s: no %s translation for id 0x%"PRIx32" on %s\n",
+           np->full_name, map_name, id, (target && *target) ? (*target)->full_name : NULL);
+
+    /*
+     * NOTE: Linux bypasses translation without returning an error here,
+     * but should we behave in the same way on Xen? Restrict for now.
+     */
+    return -EFAULT;
+}
+
+int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)
+{
+    const struct iommu_ops *ops = iommu_get_ops();
+    struct dt_phandle_args iommu_spec = { .args_count = 1 };
+    struct device *dev = pci_to_dev(pdev);
+    const struct dt_device_node *np;
+    int rc = NO_IOMMU;
+
+    if ( !iommu_enabled )
+        return NO_IOMMU;
+
+    if ( !ops )
+        return -EINVAL;
+
+    if ( device_is_protected(dev) )
+        return 0;
+
+    if ( dev_iommu_fwspec_get(dev) )
+        return -EEXIST;
+
+    np = pci_find_host_bridge_node(pdev);
+    if ( !np )
+        return -ENODEV;
+
+    /*
+     * According to the Documentation/devicetree/bindings/pci/pci-iommu.txt
+     * from Linux.
+     */
+    rc = dt_map_id(np, PCI_BDF(pdev->bus, pdev->devfn), "iommu-map",
+                   "iommu-map-mask", &iommu_spec.np, iommu_spec.args);
+    if ( rc )
+        return (rc == -ENODEV) ? NO_IOMMU : rc;
+
+    rc = iommu_dt_xlate(dev, &iommu_spec);
+    if ( rc < 0 )
+    {
+        iommu_fwspec_free(dev);
+        return -EINVAL;
+    }
+
+    return rc;
+}
+#endif /* CONFIG_HAS_PCI */
+
 int iommu_add_dt_device(struct dt_device_node *np)
 {
     const struct iommu_ops *ops = iommu_get_ops();
diff --git a/xen/include/xen/device_tree.h b/xen/include/xen/device_tree.h
index XXXXXXX..XXXXXXX 100644
--- a/xen/include/xen/device_tree.h
+++ b/xen/include/xen/device_tree.h
@@ -XXX,XX +XXX,XX @@ int dt_count_phandle_with_args(const struct dt_device_node *np,
  */
 int dt_get_pci_domain_nr(struct dt_device_node *node);
 
+#ifdef CONFIG_HAS_PCI
+/**
+ * dt_map_id - Translate an ID through a downstream mapping.
+ * @np: root complex device node.
+ * @id: device ID to map.
+ * @map_name: property name of the map to use.
+ * @map_mask_name: optional property name of the mask to use.
+ * @target: optional pointer to a target device node.
+ * @id_out: optional pointer to receive the translated ID.
+ *
+ * Given a device ID, look up the appropriate implementation-defined
+ * platform ID and/or the target device which receives transactions on that
+ * ID, as per the "iommu-map" and "msi-map" bindings. Either of @target or
+ * @id_out may be NULL if only the other is required. If @target points to
+ * a non-NULL device node pointer, only entries targeting that node will be
+ * matched; if it points to a NULL value, it will receive the device node of
+ * the first matching target phandle, with a reference held.
+ *
+ * Return: 0 on success or a standard error code on failure.
+ */
+int dt_map_id(const struct dt_device_node *np, uint32_t id,
+              const char *map_name, const char *map_mask_name,
+              struct dt_device_node **target, uint32_t *id_out);
+#endif /* CONFIG_HAS_PCI */
+
 struct dt_device_node *dt_find_node_by_phandle(dt_phandle handle);
 
 #ifdef CONFIG_DEVICE_TREE_DEBUG
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index XXXXXXX..XXXXXXX 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -XXX,XX +XXX,XX @@
 #include <xen/spinlock.h>
 #include <public/domctl.h>
 #include <public/hvm/ioreq.h>
+#ifdef CONFIG_ACPI
+#include <asm/acpi.h>
+#endif
 #include <asm/device.h>
 
 TYPE_SAFE(uint64_t, dfn);
@@ -XXX,XX +XXX,XX @@ int iommu_dt_domain_init(struct domain *d);
 int iommu_release_dt_devices(struct domain *d);
 
 /*
- * Helper to add master device to the IOMMU using generic IOMMU DT bindings.
+ * Helpers to add master device to the IOMMU using generic (PCI-)IOMMU
+ * DT bindings.
  *
  * Return values:
  *  0 : device is protected by an IOMMU
@@ -XXX,XX +XXX,XX @@ int iommu_release_dt_devices(struct domain *d);
  *      (IOMMU is not enabled/present or device is not connected to it).
  */
 int iommu_add_dt_device(struct dt_device_node *np);
+int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev);
 
 int iommu_do_dt_domctl(struct xen_domctl *, struct domain *,
                        XEN_GUEST_HANDLE_PARAM(xen_domctl_t));
 
+#else /* !HAS_DEVICE_TREE */
+static inline int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)
+{
+    return 0;
+}
 #endif /* HAS_DEVICE_TREE */
 
+static inline int iommu_add_pci_sideband_ids(struct pci_dev *pdev)
+{
+    int ret = 0;
+#ifdef CONFIG_ACPI
+    if ( acpi_disabled )
+#endif
+        ret = iommu_add_dt_pci_sideband_ids(pdev);
+    return ret;
+}
+
 struct page_info;
 
 /*
-- 
2.40.1

Handle phantom functions in iommu_add_dt_pci_sideband_ids(). Each phantom
function will have a unique requestor ID (RID)/BDF. On ARM, we need to
map/translate the RID/BDF to an AXI stream ID for each phantom function
according to the pci-iommu device tree mapping [1]. The RID/BDF -> AXI stream ID
mapping in DT could allow phantom devices (i.e. devices with phantom functions)
to use different AXI stream IDs based on the (phantom) function.

[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/pci-iommu.txt

Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
---
v3->v4:
* s/iommu_dt_pci_map_id/dt_map_id/

v2->v3:
* new patch title (was: iommu/arm: iommu_add_dt_pci_device phantom handling)
* rework loop to reduce duplication
* s/iommu_fwspec_free(pci_to_dev(pdev))/iommu_fwspec_free(dev)/

v1->v2:
* new patch
---
 xen/drivers/passthrough/device_tree.c | 33 ++++++++++++++++-----------
 1 file changed, 20 insertions(+), 13 deletions(-)

From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
---
v3->v4:
* add new device_is_protected check in add_device hook to match SMMUv3 and
  IPMMU-VMSA drivers

v2->v3:
* invoke iommu_add_pci_sideband_ids() from add_device hook

v1->v2:
* ignore add_device/assign_device/reassign_device calls for phantom functions
  (i.e. devfn != pdev->devfn)

downstream->v1:
* wrap unused function in #ifdef 0
* remove the remove_device() stub since it was submitted separately to the list
  [XEN][PATCH v6 12/19] xen/smmu: Add remove_device callback for smmu_iommu ops
  https://lists.xenproject.org/archives/html/xen-devel/2023-05/msg00204.html
* arm_smmu_(de)assign_dev: return error instead of crashing system
* update condition in arm_smmu_reassign_dev
* style fixup
* add && !is_hardware_domain(d) into condition in arm_smmu_assign_dev()

(cherry picked from commit 0c11a7f65f044c26d87d1e27ac6283ef1f9cfb7a from
 the downstream branch spider-master from
 https://github.com/xen-troops/xen.git)
---

This is a file imported from Linux with modifications for Xen. What should be
the coding style for Xen modifications?
---
 xen/drivers/passthrough/arm/smmu.c | 120 ++++++++++++++++++++++++-----
 1 file changed, 99 insertions(+), 21 deletions(-)

diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/arm/smmu.c
index XXXXXXX..XXXXXXX 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -XXX,XX +XXX,XX @@ typedef enum irqreturn irqreturn_t;
 /* Device logger functions
  * TODO: Handle PCI
  */
-#define dev_print(dev, lvl, fmt, ...)						\
-	 printk(lvl "smmu: %s: " fmt, dt_node_full_name(dev_to_dt(dev)), ## __VA_ARGS__)
+#ifndef CONFIG_HAS_PCI
+#define dev_print(dev, lvl, fmt, ...)    \
+    printk(lvl "smmu: %s: " fmt, dev_name(dev), ## __VA_ARGS__)
+#else
+#define dev_print(dev, lvl, fmt, ...) ({                                \
+    if ( !dev_is_pci((dev)) )                                           \
+        printk(lvl "smmu: %s: " fmt, dev_name((dev)), ## __VA_ARGS__);  \
+    else                                                                \
+    {                                                                   \
+        struct pci_dev *pdev = dev_to_pci((dev));                       \
+        printk(lvl "smmu: %pp: " fmt, &pdev->sbdf, ## __VA_ARGS__);     \
+    }                                                                   \
+})
+#endif
 
 #define dev_dbg(dev, fmt, ...) dev_print(dev, XENLOG_DEBUG, fmt, ## __VA_ARGS__)
 #define dev_notice(dev, fmt, ...) dev_print(dev, XENLOG_INFO, fmt, ## __VA_ARGS__)
@@ -XXX,XX +XXX,XX @@ static void __iomem *devm_ioremap_resource(struct device *dev,
  * Xen: PCI functions
  * TODO: It should be implemented when PCI will be supported
  */
+#if 0 /* unused */
 #define to_pci_dev(dev)	(NULL)
 static inline int pci_for_each_dma_alias(struct pci_dev *pdev,
 					 int (*fn) (struct pci_dev *pdev,
@@ -XXX,XX +XXX,XX @@ static inline int pci_for_each_dma_alias(struct pci_dev *pdev,
 	BUG();
 	return 0;
 }
+#endif
 
 /* Xen: misc */
 #define PHYS_MASK_SHIFT		PADDR_BITS
@@ -XXX,XX +XXX,XX @@ struct arm_smmu_master_cfg {
 	for (i = 0; idx = cfg->smendx[i], i < num; ++i)
 
 struct arm_smmu_master {
-	struct device_node		*of_node;
+	struct device			*dev;
 	struct rb_node			node;
 	struct arm_smmu_master_cfg	cfg;
 };
@@ -XXX,XX +XXX,XX @@ arm_smmu_get_fwspec(struct arm_smmu_master_cfg *cfg)
 {
 	struct arm_smmu_master *master = container_of(cfg,
 			                                      struct arm_smmu_master, cfg);
-	return dev_iommu_fwspec_get(&master->of_node->dev);
+	return dev_iommu_fwspec_get(master->dev);
 }
 
 static void parse_driver_options(struct arm_smmu_device *smmu)
@@ -XXX,XX +XXX,XX @@ static struct device_node *dev_get_dev_node(struct device *dev)
 }
 
 static struct arm_smmu_master *find_smmu_master(struct arm_smmu_device *smmu,
-						struct device_node *dev_node)
+						struct device *dev)
 {
 	struct rb_node *node = smmu->masters.rb_node;
 
@@ -XXX,XX +XXX,XX @@ static struct arm_smmu_master *find_smmu_master(struct arm_smmu_device *smmu,
 
 		master = container_of(node, struct arm_smmu_master, node);
 
-		if (dev_node < master->of_node)
+		if (dev < master->dev)
 			node = node->rb_left;
-		else if (dev_node > master->of_node)
+		else if (dev > master->dev)
 			node = node->rb_right;
 		else
 			return master;
@@ -XXX,XX +XXX,XX @@ static int insert_smmu_master(struct arm_smmu_device *smmu,
 			= container_of(*new, struct arm_smmu_master, node);
 
 		parent = *new;
-		if (master->of_node < this->of_node)
+		if (master->dev < this->dev)
 			new = &((*new)->rb_left);
-		else if (master->of_node > this->of_node)
+		else if (master->dev > this->dev)
 			new = &((*new)->rb_right);
 		else
 			return -EEXIST;
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_dt_add_device_legacy(struct arm_smmu_device *smmu,
 	struct arm_smmu_master *master;
 	struct device_node *dev_node = dev_get_dev_node(dev);
 
-	master = find_smmu_master(smmu, dev_node);
+	master = find_smmu_master(smmu, dev);
 	if (master) {
 		dev_err(dev,
 			"rejecting multiple registrations for master device %s\n",
-			dev_node->name);
+			dev_node ? dev_node->name : "");
 		return -EBUSY;
 	}
 
 	master = devm_kzalloc(dev, sizeof(*master), GFP_KERNEL);
 	if (!master)
 		return -ENOMEM;
-	master->of_node = dev_node;
+	master->dev = dev;
+
+	if ( device_is_protected(dev) )
+	{
+		dev_err(dev, "Already added to SMMU\n");
+		return -EEXIST;
+	}
 
 	/* Xen: Let Xen know that the device is protected by an SMMU */
 	device_set_protected(dev);
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_dt_add_device_legacy(struct arm_smmu_device *smmu,
 		     (fwspec->ids[i] >= smmu->num_mapping_groups)) {
 			dev_err(dev,
 				"stream ID for master device %s greater than maximum allowed (%d)\n",
-				dev_node->name, smmu->num_mapping_groups);
+				dev_node ? dev_node->name : "", smmu->num_mapping_groups);
 			return -ERANGE;
 		}
 		master->cfg.smendx[i] = INVALID_SMENDX;
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_dt_add_device_generic(u8 devfn, struct device *dev)
 	struct arm_smmu_device *smmu;
 	struct iommu_fwspec *fwspec;
 
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+		int ret;
+
+		if ( devfn != pdev->devfn )
+			return 0;
+
+		ret = iommu_add_pci_sideband_ids(pdev);
+		if ( ret < 0 )
+			iommu_fwspec_free(dev);
+	}
+#endif
+
 	fwspec = dev_iommu_fwspec_get(dev);
 	if (fwspec == NULL)
 		return -ENXIO;
@@ -XXX,XX +XXX,XX @@ static struct arm_smmu_device *find_smmu_for_device(struct device *dev)
 {
 	struct arm_smmu_device *smmu;
 	struct arm_smmu_master *master = NULL;
-	struct device_node *dev_node = dev_get_dev_node(dev);
 
 	spin_lock(&arm_smmu_devices_lock);
 	list_for_each_entry(smmu, &arm_smmu_devices, list) {
-		master = find_smmu_master(smmu, dev_node);
+		master = find_smmu_master(smmu, dev);
 		if (master)
 			break;
 	}
@@ -XXX,XX +XXX,XX @@ static bool arm_smmu_capable(enum iommu_cap cap)
 }
 #endif
 
+#if 0 /* Not used */
 static int __arm_smmu_get_pci_sid(struct pci_dev *pdev, u16 alias, void *data)
 {
 	*((u16 *)data) = alias;
@@ -XXX,XX +XXX,XX @@ static void __arm_smmu_release_pci_iommudata(void *data)
 {
 	kfree(data);
 }
+#endif
 
 static int arm_smmu_add_device(struct device *dev)
 {
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_add_device(struct device *dev)
 	struct arm_smmu_master_cfg *cfg;
 	struct iommu_group *group;
 	void (*releasefn)(void *) = NULL;
-	int ret;
 
 	smmu = find_smmu_for_device(dev);
 	if (!smmu)
 		return -ENODEV;
 
+	/* There is no need to distinguish here, thanks to PCI-IOMMU DT bindings */
+#if 0
 	if (dev_is_pci(dev)) {
 		struct pci_dev *pdev = to_pci_dev(dev);
 		struct iommu_fwspec *fwspec;
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_add_device(struct device *dev)
 				       &fwspec->ids[0]);
 		releasefn = __arm_smmu_release_pci_iommudata;
 		cfg->smmu = smmu;
-	} else {
+	} else
+#endif
+	{
 		struct arm_smmu_master *master;
 
-		master = find_smmu_master(smmu, dev->of_node);
+		master = find_smmu_master(smmu, dev);
 		if (!master) {
 			return -ENODEV;
 		}
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
 			return -ENOMEM;
 	}
 
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) && !is_hardware_domain(d) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+
+		printk(XENLOG_INFO "Assigning device %04x:%02x:%02x.%u to dom%d\n",
+		       pdev->seg, pdev->bus, PCI_SLOT(devfn), PCI_FUNC(devfn),
+		       d->domain_id);
+
+		if ( devfn != pdev->devfn || pdev->domain == d )
+			return 0;
+
+		list_move(&pdev->domain_list, &d->pdev_list);
+		pdev->domain = d;
+
+		/* dom_io is used as a sentinel for quarantined devices */
+		if ( d == dom_io )
+			return 0;
+	}
+#endif
+
 	if (!dev_iommu_group(dev)) {
 		ret = arm_smmu_add_device(dev);
 		if (ret)
@@ -XXX,XX +XXX,XX @@ out:
 	return ret;
 }
 
-static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)
+static int arm_smmu_deassign_dev(struct domain *d, u8 devfn, struct device *dev)
 {
 	struct iommu_domain *domain = dev_iommu_domain(dev);
 	struct arm_smmu_xen_domain *xen_domain;
 
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+
+		printk(XENLOG_INFO "Deassigning device %04x:%02x:%02x.%u from dom%d\n",
+		       pdev->seg, pdev->bus, PCI_SLOT(devfn), PCI_FUNC(devfn),
+		       d->domain_id);
+
+		if ( devfn != pdev->devfn )
+			return 0;
+
+		/* dom_io is used as a sentinel for quarantined devices */
+		if ( d == dom_io )
+			return 0;
+	}
+#endif
+
 	xen_domain = dom_iommu(d)->arch.priv;
 
 	if (!domain || domain->priv->cfg.domain != d) {
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_reassign_dev(struct domain *s, struct domain *t,
 	int ret = 0;
 
 	/* Don't allow remapping on other domain than hwdom */
-	if ( t && !is_hardware_domain(t) )
+	if ( t && !is_hardware_domain(t) && t != dom_io )
 		return -EPERM;
 
 	if (t == s)
 		return 0;
 
-	ret = arm_smmu_deassign_dev(s, dev);
+	ret = arm_smmu_deassign_dev(s, devfn, dev);
 	if (ret)
 		return ret;
 
-- 
2.40.1

From: Rahul Singh <rahul.singh@arm.com>

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
---
v3->v4:
* no change

v2->v3:
* rebase
* invoke iommu_add_pci_sideband_ids() from add_device hook

v1->v2:
* ignore add_device/assign_device/reassign_device calls for phantom functions
  (i.e. devfn != pdev->devfn)

downstream->v1:
* rebase
* move 2 replacements of s/dt_device_set_protected(dev_to_dt(dev))/device_set_protected(dev)/
  from this commit to ("xen/arm: Move is_protected flag to struct device")
  so as to not break ability to bisect
* adjust patch title (remove stray space)
* arm_smmu_(de)assign_dev: return error instead of crashing system
* remove arm_smmu_remove_device() stub
* update condition in arm_smmu_reassign_dev
* style fixup

(cherry picked from commit 7ed6c3ab250d899fe6e893a514278e406a2893e8 from
 the downstream branch poc/pci-passthrough from
 https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
---

This is a file imported from Linux with modifications for Xen. What should be
the coding style used for Xen modifications?
---
 xen/drivers/passthrough/arm/smmu-v3.c | 76 +++++++++++++++++++++++++--
 1 file changed, 72 insertions(+), 4 deletions(-)

diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
index XXXXXXX..XXXXXXX 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.c
+++ b/xen/drivers/passthrough/arm/smmu-v3.c
@@ -XXX,XX +XXX,XX @@ static bool arm_smmu_sid_in_range(struct arm_smmu_device *smmu, u32 sid)
 }
 /* Forward declaration */
 static struct arm_smmu_device *arm_smmu_get_by_dev(const struct device *dev);
+static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
+			struct device *dev, u32 flag);
 
 static int arm_smmu_add_device(u8 devfn, struct device *dev)
 {
 	int i, ret;
 	struct arm_smmu_device *smmu;
 	struct arm_smmu_master *master;
-	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+	struct iommu_fwspec *fwspec;
+
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+		int ret;
+
+		if ( devfn != pdev->devfn )
+			return 0;
+
+		ret = iommu_add_pci_sideband_ids(pdev);
+		if ( ret < 0 )
+			iommu_fwspec_free(dev);
+	}
+#endif
 
+	fwspec = dev_iommu_fwspec_get(dev);
 	if (!fwspec)
 		return -ENODEV;
 
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_add_device(u8 devfn, struct device *dev)
 	dev_info(dev, "Added master device (SMMUv3 %s StreamIds %u)\n",
 			dev_name(fwspec->iommu_dev), fwspec->num_ids);
 
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+
+		ret = arm_smmu_assign_dev(pdev->domain, devfn, dev, 0);
+		if (ret)
+			goto err_free_master;
+	}
+#endif
+
 	return 0;
 
 err_free_master:
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
 	struct arm_smmu_domain *smmu_domain;
 	struct arm_smmu_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
 
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) && !is_hardware_domain(d) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+
+		printk(XENLOG_INFO "Assigning device %04x:%02x:%02x.%u to dom%d\n",
+			pdev->seg, pdev->bus, PCI_SLOT(devfn),
+			PCI_FUNC(devfn), d->domain_id);
+
+		if ( devfn != pdev->devfn || pdev->domain == d )
+			return 0;
+
+		list_move(&pdev->domain_list, &d->pdev_list);
+		pdev->domain = d;
+
+		/* dom_io is used as a sentinel for quarantined devices */
+		if ( d == dom_io )
+			return 0;
+	}
+#endif
+
 	spin_lock(&xen_domain->lock);
 
 	/*
@@ -XXX,XX +XXX,XX @@ out:
 	return ret;
 }
 
-static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)
+static int arm_smmu_deassign_dev(struct domain *d, uint8_t devfn, struct device *dev)
 {
 	struct iommu_domain *io_domain = arm_smmu_get_domain(d, dev);
 	struct arm_smmu_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)
 		return -ESRCH;
 	}
 
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+
+		printk(XENLOG_INFO "Deassigning device %04x:%02x:%02x.%u from dom%d\n",
+			pdev->seg, pdev->bus, PCI_SLOT(devfn),
+			PCI_FUNC(devfn), d->domain_id);
+
+		if ( devfn != pdev->devfn )
+			return 0;
+
+		/* dom_io is used as a sentinel for quarantined devices */
+		if ( d == dom_io )
+			return 0;
+	}
+#endif
+
 	spin_lock(&xen_domain->lock);
 
 	arm_smmu_detach_dev(master);
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_reassign_dev(struct domain *s, struct domain *t,
 	int ret = 0;
 
 	/* Don't allow remapping on other domain than hwdom */
-	if ( t && !is_hardware_domain(t) )
+	if ( t && !is_hardware_domain(t) && (t != dom_io) )
 		return -EPERM;
 
 	if (t == s)
 		return 0;
 
-	ret = arm_smmu_deassign_dev(s, dev);
+	ret = arm_smmu_deassign_dev(s, devfn, dev);
 	if (ret)
 		return ret;
 
-- 
2.40.1

This series introduces SMMU handling for PCIe passthrough on ARM. These patches
should be able to be upstreamed independently from the vPCI series [1]. See [2]
for notes about test cases.

[1] https://lists.xenproject.org/archives/html/xen-devel/2023-10/msg00660.html
[2] https://lists.xenproject.org/archives/html/xen-devel/2023-06/msg01135.html

v6->v7:
* drop ("xen/arm: don't pass iommu properties to hwdom for iommu-map")

v5->v6:
* don't revert ("xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"")
* add ("xen/arm: enable dom0 to use PCI devices with pci-passthrough=no")

v4->v5:
* drop ("xen/arm: Improve readability of check for registered devices")
* drop ("xen/arm: Move is_protected flag to struct device")
* add ("xen/arm: don't pass iommu properties to hwdom for iommu-map")
* add ("xen/arm: Fix mapping for PCI bridge mmio region")
* revert ("xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"")
* add ("xen/arm: Map ITS doorbell register to IOMMU page tables.")
* fix test case #1 with PCI device in dom0

v3->v4:
* split a change from ("xen/arm: Move is_protected flag to struct device") into
  a new separate patch
* see individual patches for further details

Oleksandr Andrushchenko (1):
  xen/arm: smmuv2: Add PCI devices support for SMMUv2

Oleksandr Tyshchenko (2):
  iommu/arm: Add iommu_dt_xlate()
  iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API

Rahul Singh (3):
  xen/arm: smmuv3: Add PCI devices support for SMMUv3
  xen/arm: Fix mapping for PCI bridge mmio region
  xen/arm: Map ITS doorbell register to IOMMU page tables

Stewart Hildebrand (2):
  iommu/arm: iommu_add_dt_pci_sideband_ids phantom handling
  xen/arm: enable dom0 to use PCI devices with pci-passthrough=no

xen/arch/arm/device.c                 |   2 +-
 xen/arch/arm/pci/pci.c                |   5 +-
 xen/arch/arm/vgic-v3-its.c            |  20 +++
 xen/arch/x86/include/asm/pci.h        |   6 -
 xen/common/device-tree/device-tree.c  |  91 ++++++++++++
 xen/drivers/passthrough/arm/smmu-v3.c | 117 ++++++++++++++--
 xen/drivers/passthrough/arm/smmu.c    | 190 ++++++++++++++++++++------
 xen/drivers/passthrough/device_tree.c |  97 ++++++++++---
 xen/drivers/pci/physdev.c             |   4 +-
 xen/include/xen/device_tree.h         |  23 ++++
 xen/include/xen/iommu.h               |  31 ++++-
 11 files changed, 502 insertions(+), 84 deletions(-)

-- 
2.34.1

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

While at it introduce NO_IOMMU to avoid magic "1".

v5->v6:
* pass ops parameter to iommu_dt_xlate()
* add Julien's R-b

v4->v5:
* rebase on top of "dynamic node programming using overlay dtbo" series
* move #define NO_IOMMU 1 to header
* s/these/this/ inside comment

v3->v4:
* make dt_phandle_args *iommu_spec const
* move !ops->add_device check to helper

v2->v3:
* no change

v1->v2:
* no change

downstream->v1:
* trivial rebase
* s/dt_iommu_xlate/iommu_dt_xlate/

(cherry picked from commit c26bab0415ca303df86aba1d06ef8edc713734d3 from
 the downstream branch poc/pci-passthrough from
 https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
---
 xen/drivers/passthrough/device_tree.c | 48 +++++++++++++++++----------
 xen/include/xen/iommu.h               |  3 ++
 2 files changed, 33 insertions(+), 18 deletions(-)

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

The main purpose of this patch is to add a way to register PCI device
(which is behind the IOMMU) using the generic PCI-IOMMU DT bindings [1]
before assigning that device to a domain.

This behaves similarly to the existing iommu_add_dt_device API, except it
handles PCI devices, and it is to be invoked from the add_device hook in the
SMMU driver.

The function dt_map_id to translate an ID through a downstream mapping
(which is also suitable for mapping Requester ID) was borrowed from Linux
(v5.10-rc6) and updated according to the Xen code base.

[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/pci-iommu.txt

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
Regarding pci_for_each_dma_alias question: getting host bridge node
directly seems like a simpler solution with the same result. AFAIU
with pci_for_each_dma_alias in linux we would arrive to the host brige
node anyway, but also try to call dt_map_id for each device along the
way. I am not sure why exactly it is done this way in linux, as
according to the pci-iommu.txt, iommu-map node can only be present in
the PCI root.

v6->v7:
* put iommu_add_pci_sideband_ids under ifdef
* remove ifdef CONFIG_APCI
* style: add newline for symmetry

v5->v6:
* pass ops to iommu_dt_xlate()

v4->v5:
* style: add newlines after variable declarations and before return in iommu.h
* drop device_is_protected() check in iommu_add_dt_pci_sideband_ids()
* rebase on top of ("dynamic node programming using overlay dtbo") series
* fix typo in commit message
* remove #ifdef around dt_map_id() prototype
* move dt_map_id() to xen/common/device_tree.c
* add function name in error prints
* use dprintk for debug prints
* use GENMASK and #include <xen/bitops.h>
* fix typo in comment
* remove unnecessary (int) cast in loop condition
* assign *id_out and return success in case of no translation in dt_map_id()
* don't initialize local variable unnecessarily
* return error in case of ACPI/no DT in iommu_add_{dt_}pci_sideband_ids()

v3->v4:
* wrap #include <asm/acpi.h> and if ( acpi_disabled ) in #ifdef CONFIG_ACPI
* fix Michal's remarks about style, parenthesis, and print formats
* remove !ops->dt_xlate check since it is already in iommu_dt_xlate helper
* rename s/iommu_dt_pci_map_id/dt_map_id/ because it is generic, not specific
  to iommu
* update commit description

(cherry picked from commit 734e3bf6ee77e7947667ab8fa96c25b349c2e1da from
 the downstream branch poc/pci-passthrough from
 https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
---
 xen/common/device-tree/device-tree.c  | 91 +++++++++++++++++++++++++++
 xen/drivers/passthrough/device_tree.c | 42 +++++++++++++
 xen/include/xen/device_tree.h         | 23 +++++++
 xen/include/xen/iommu.h               | 28 ++++++++-
 4 files changed, 183 insertions(+), 1 deletion(-)

diff --git a/xen/common/device-tree/device-tree.c b/xen/common/device-tree/device-tree.c
index XXXXXXX..XXXXXXX 100644
--- a/xen/common/device-tree/device-tree.c
+++ b/xen/common/device-tree/device-tree.c
@@ -XXX,XX +XXX,XX @@
  * published by the Free Software Foundation.
  */
 
+#include <xen/bitops.h>
 #include <xen/types.h>
 #include <xen/init.h>
 #include <xen/guest_access.h>
@@ -XXX,XX +XXX,XX @@ int dt_get_pci_domain_nr(struct dt_device_node *node)
     return (u16)domain;
 }
 
+int dt_map_id(const struct dt_device_node *np, uint32_t id,
+              const char *map_name, const char *map_mask_name,
+              struct dt_device_node **target, uint32_t *id_out)
+{
+    uint32_t map_mask, masked_id, map_len;
+    const __be32 *map = NULL;
+
+    if ( !np || !map_name || (!target && !id_out) )
+        return -EINVAL;
+
+    map = dt_get_property(np, map_name, &map_len);
+    if ( !map )
+    {
+        if ( target )
+            return -ENODEV;
+
+        /* Otherwise, no map implies no translation */
+        *id_out = id;
+        return 0;
+    }
+
+    if ( !map_len || (map_len % (4 * sizeof(*map))) )
+    {
+        printk(XENLOG_ERR "%s(): %s: Error: Bad %s length: %u\n", __func__,
+               np->full_name, map_name, map_len);
+        return -EINVAL;
+    }
+
+    /* The default is to select all bits. */
+    map_mask = GENMASK(31, 0);
+
+    /*
+     * Can be overridden by "{iommu,msi}-map-mask" property.
+     * If dt_property_read_u32() fails, the default is used.
+     */
+    if ( map_mask_name )
+        dt_property_read_u32(np, map_mask_name, &map_mask);
+
+    masked_id = map_mask & id;
+    for ( ; map_len > 0; map_len -= 4 * sizeof(*map), map += 4 )
+    {
+        struct dt_device_node *phandle_node;
+        uint32_t id_base = be32_to_cpup(map + 0);
+        uint32_t phandle = be32_to_cpup(map + 1);
+        uint32_t out_base = be32_to_cpup(map + 2);
+        uint32_t id_len = be32_to_cpup(map + 3);
+
+        if ( id_base & ~map_mask )
+        {
+            printk(XENLOG_ERR "%s(): %s: Invalid %s translation - %s-mask (0x%"PRIx32") ignores id-base (0x%"PRIx32")\n",
+                   __func__, np->full_name, map_name, map_name, map_mask,
+                   id_base);
+            return -EFAULT;
+        }
+
+        if ( (masked_id < id_base) || (masked_id >= (id_base + id_len)) )
+            continue;
+
+        phandle_node = dt_find_node_by_phandle(phandle);
+        if ( !phandle_node )
+            return -ENODEV;
+
+        if ( target )
+        {
+            if ( !*target )
+                *target = phandle_node;
+
+            if ( *target != phandle_node )
+                continue;
+        }
+
+        if ( id_out )
+            *id_out = masked_id - id_base + out_base;
+
+        dprintk(XENLOG_DEBUG, "%s: %s, using mask %08"PRIx32", id-base: %08"PRIx32", out-base: %08"PRIx32", length: %08"PRIx32", id: %08"PRIx32" -> %08"PRIx32"\n",
+               np->full_name, map_name, map_mask, id_base, out_base, id_len, id,
+               masked_id - id_base + out_base);
+        return 0;
+    }
+
+    dprintk(XENLOG_DEBUG, "%s: no %s translation for id 0x%"PRIx32" on %s\n",
+           np->full_name, map_name, id,
+           (target && *target) ? (*target)->full_name : NULL);
+
+    if ( id_out )
+        *id_out = id;
+
+    return 0;
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
index XXXXXXX..XXXXXXX 100644
--- a/xen/drivers/passthrough/device_tree.c
+++ b/xen/drivers/passthrough/device_tree.c
@@ -XXX,XX +XXX,XX @@ static int iommu_dt_xlate(struct device *dev,
     return ops->dt_xlate(dev, iommu_spec);
 }
 
+#ifdef CONFIG_HAS_PCI
+int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)
+{
+    const struct iommu_ops *ops = iommu_get_ops();
+    struct dt_phandle_args iommu_spec = { .args_count = 1 };
+    struct device *dev = pci_to_dev(pdev);
+    const struct dt_device_node *np;
+    int rc;
+
+    if ( !iommu_enabled )
+        return NO_IOMMU;
+
+    if ( !ops )
+        return -EINVAL;
+
+    if ( dev_iommu_fwspec_get(dev) )
+        return -EEXIST;
+
+    np = pci_find_host_bridge_node(pdev);
+    if ( !np )
+        return -ENODEV;
+
+    /*
+     * According to the Documentation/devicetree/bindings/pci/pci-iommu.txt
+     * from Linux.
+     */
+    rc = dt_map_id(np, PCI_BDF(pdev->bus, pdev->devfn), "iommu-map",
+                   "iommu-map-mask", &iommu_spec.np, iommu_spec.args);
+    if ( rc )
+        return (rc == -ENODEV) ? NO_IOMMU : rc;
+
+    rc = iommu_dt_xlate(dev, &iommu_spec, ops);
+    if ( rc < 0 )
+    {
+        iommu_fwspec_free(dev);
+        return -EINVAL;
+    }
+
+    return rc;
+}
+#endif /* CONFIG_HAS_PCI */
+
 int iommu_remove_dt_device(struct dt_device_node *np)
 {
     const struct iommu_ops *ops = iommu_get_ops();
diff --git a/xen/include/xen/device_tree.h b/xen/include/xen/device_tree.h
index XXXXXXX..XXXXXXX 100644
--- a/xen/include/xen/device_tree.h
+++ b/xen/include/xen/device_tree.h
@@ -XXX,XX +XXX,XX @@ int dt_count_phandle_with_args(const struct dt_device_node *np,
  */
 int dt_get_pci_domain_nr(struct dt_device_node *node);
 
+/**
+ * dt_map_id - Translate an ID through a downstream mapping.
+ * @np: root complex device node.
+ * @id: device ID to map.
+ * @map_name: property name of the map to use.
+ * @map_mask_name: optional property name of the mask to use.
+ * @target: optional pointer to a target device node.
+ * @id_out: optional pointer to receive the translated ID.
+ *
+ * Given a device ID, look up the appropriate implementation-defined
+ * platform ID and/or the target device which receives transactions on that
+ * ID, as per the "iommu-map" and "msi-map" bindings. Either of @target or
+ * @id_out may be NULL if only the other is required. If @target points to
+ * a non-NULL device node pointer, only entries targeting that node will be
+ * matched; if it points to a NULL value, it will receive the device node of
+ * the first matching target phandle, with a reference held.
+ *
+ * Return: 0 on success or a standard error code on failure.
+ */
+int dt_map_id(const struct dt_device_node *np, uint32_t id,
+              const char *map_name, const char *map_mask_name,
+              struct dt_device_node **target, uint32_t *id_out);
+
 struct dt_device_node *dt_find_node_by_phandle(dt_phandle handle);
 
 #ifdef CONFIG_DEVICE_TREE_DEBUG
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index XXXXXXX..XXXXXXX 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -XXX,XX +XXX,XX @@
 #include <xen/errno.h>
 #include <public/domctl.h>
 #include <public/hvm/ioreq.h>
+#include <xen/acpi.h>
 #include <asm/device.h>
 
 TYPE_SAFE(uint64_t, dfn);
@@ -XXX,XX +XXX,XX @@ int iommu_dt_domain_init(struct domain *d);
 int iommu_release_dt_devices(struct domain *d);
 
 /*
- * Helper to add master device to the IOMMU using generic IOMMU DT bindings.
+ * Helpers to add master device to the IOMMU using generic (PCI-)IOMMU
+ * DT bindings.
  *
  * Return values:
  *  0 : device is protected by an IOMMU
@@ -XXX,XX +XXX,XX @@ int iommu_release_dt_devices(struct domain *d);
  *      (IOMMU is not enabled/present or device is not connected to it).
  */
 int iommu_add_dt_device(struct dt_device_node *np);
+int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev);
 
 int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
                        XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl);
@@ -XXX,XX +XXX,XX @@ int iommu_remove_dt_device(struct dt_device_node *np);
 /* Error code for reporting no IOMMU is present */
 #define NO_IOMMU    1
 
+#else /* !HAS_DEVICE_TREE */
+static inline int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)
+{
+    return -ENOSYS;
+}
+
 #endif /* HAS_DEVICE_TREE */
 
+#ifdef CONFIG_HAS_PCI
+static inline int iommu_add_pci_sideband_ids(struct pci_dev *pdev)
+{
+    int ret = -ENOSYS;
+
+    if ( acpi_disabled )
+        ret = iommu_add_dt_pci_sideband_ids(pdev);
+
+    return ret;
+}
+#else /* !HAS_PCI */
+static inline int iommu_add_pci_sideband_ids(struct pci_dev *pdev)
+{
+    return -ENOSYS;
+}
+#endif
+
 struct page_info;
 
 /*
-- 
2.34.1

From: Stewart Hildebrand <stewart.hildebrand@amd.com>

[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/pci-iommu.txt

Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
---
v6->v7:
* no change

v5->v6:
* no change

v4->v5:
* no change

v3->v4:
* s/iommu_dt_pci_map_id/dt_map_id/

v2->v3:
* new patch title (was: iommu/arm: iommu_add_dt_pci_device phantom handling)
* rework loop to reduce duplication
* s/iommu_fwspec_free(pci_to_dev(pdev))/iommu_fwspec_free(dev)/

v1->v2:
* new patch

---
TODO: investigate Jan's comment [2]
[2] https://lore.kernel.org/xen-devel/806a2978-19fb-4d31-ab6a-35ea7317c8de@suse.com/
---
 xen/drivers/passthrough/device_tree.c | 33 ++++++++++++++++-----------
 1 file changed, 20 insertions(+), 13 deletions(-)

From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>

Implement support for PCI devices in the SMMU driver. Make arm_smmu_master
structure to hold a pointer to the device to allow it to hold PCI devices.
Trigger iommu-map parsing when new PCI device is added. Add checks to
assign/deassign functions to ensure PCI devices are handled correctly.
Implement basic quarantining.

All pci devices are automatically assigned to hardware domain if it exists
to ensure it can probe them.

TODO:
Implement scratch page quarantining support.

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
v6->v7:
* use d->pci_lock in arm_smmu_assign_dev()
* remove !is_hardware_domain and pdev->domain == d checks in assign to
  support future dom0less use case when dom0 is using vPCI
* remove stale todo in dev_get_dev_node
* don't print ""
* remove redundant dt_device_is_protected check
* remove assign/deassing prints
* change assign logic to remove reassign reimplementation
* check if pdev->domain exists before assigning to it
* explain pdev->devfn check
* make reassign check stricter and update comment

v5->v6:
* check for hardware_domain == NULL (dom0less test case)
* locking: assign pdev->domain before list_add()

v4->v5:
* assign device to pdev->domain (usually dom0) by default in add_device() hook
* deassign from hwdom
* rebase on top of ("dynamic node programming using overlay dtbo") series
* remove TODO in comment about device prints
* add TODO regarding locking
* fixup after dropping ("xen/arm: Move is_protected flag to struct device")

v3->v4:
* add new device_is_protected check in add_device hook to match SMMUv3 and
  IPMMU-VMSA drivers

v2->v3:
* invoke iommu_add_pci_sideband_ids() from add_device hook

v1->v2:
* ignore add_device/assign_device/reassign_device calls for phantom functions
  (i.e. devfn != pdev->devfn)

(cherry picked from commit 0c11a7f65f044c26d87d1e27ac6283ef1f9cfb7a from
 the downstream branch spider-master from
 https://github.com/xen-troops/xen.git)
---
 xen/drivers/passthrough/arm/smmu.c | 190 ++++++++++++++++++++++-------
 1 file changed, 147 insertions(+), 43 deletions(-)

diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/arm/smmu.c
index XXXXXXX..XXXXXXX 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -XXX,XX +XXX,XX @@ enum irqreturn {
 
 typedef enum irqreturn irqreturn_t;
 
-/* Device logger functions
- * TODO: Handle PCI
- */
-#define dev_print(dev, lvl, fmt, ...)						\
-	 printk(lvl "smmu: %s: " fmt, dt_node_full_name(dev_to_dt(dev)), ## __VA_ARGS__)
+/* Device logger functions */
+#ifndef CONFIG_HAS_PCI
+#define dev_print(dev, lvl, fmt, ...)    \
+    printk(lvl "smmu: %s: " fmt, dev_name(dev), ## __VA_ARGS__)
+#else
+#define dev_print(dev, lvl, fmt, ...) ({                                \
+    if ( !dev_is_pci((dev)) )                                           \
+        printk(lvl "smmu: %s: " fmt, dev_name((dev)), ## __VA_ARGS__);  \
+    else                                                                \
+    {                                                                   \
+        struct pci_dev *pdev = dev_to_pci((dev));                       \
+        printk(lvl "smmu: %pp: " fmt, &pdev->sbdf, ## __VA_ARGS__);     \
+    }                                                                   \
+})
+#endif
 
 #define dev_dbg(dev, fmt, ...) dev_print(dev, XENLOG_DEBUG, fmt, ## __VA_ARGS__)
 #define dev_notice(dev, fmt, ...) dev_print(dev, XENLOG_INFO, fmt, ## __VA_ARGS__)
@@ -XXX,XX +XXX,XX @@ static void __iomem *devm_ioremap_resource(struct device *dev,
  * Xen: PCI functions
  * TODO: It should be implemented when PCI will be supported
  */
+#if 0 /* unused */
 #define to_pci_dev(dev)	(NULL)
 static inline int pci_for_each_dma_alias(struct pci_dev *pdev,
 					 int (*fn) (struct pci_dev *pdev,
@@ -XXX,XX +XXX,XX @@ static inline int pci_for_each_dma_alias(struct pci_dev *pdev,
 	BUG();
 	return 0;
 }
+#endif
 
 /* Xen: misc */
 #define PHYS_MASK_SHIFT		PADDR_BITS
@@ -XXX,XX +XXX,XX @@ struct arm_smmu_master_cfg {
 	for (i = 0; idx = (cfg)->smendx[i], (i) < (num); ++(i))
 
 struct arm_smmu_master {
-	struct device_node		*of_node;
+	struct device			*dev;
 	struct rb_node			node;
 	struct arm_smmu_master_cfg	cfg;
 };
@@ -XXX,XX +XXX,XX @@ arm_smmu_get_fwspec(struct arm_smmu_master_cfg *cfg)
 {
 	struct arm_smmu_master *master = container_of(cfg,
 			                                      struct arm_smmu_master, cfg);
-	return dev_iommu_fwspec_get(&master->of_node->dev);
+	return dev_iommu_fwspec_get(master->dev);
 }
 
 static void parse_driver_options(struct arm_smmu_device *smmu)
@@ -XXX,XX +XXX,XX @@ static void parse_driver_options(struct arm_smmu_device *smmu)
 
 static struct device_node *dev_get_dev_node(struct device *dev)
 {
-#if 0 /* Xen: TODO: Add support for PCI */
-	if (dev_is_pci(dev)) {
-		struct pci_bus *bus = to_pci_dev(dev)->bus;
-
-		while (!pci_is_root_bus(bus))
-			bus = bus->parent;
-		return bus->bridge->parent->of_node;
-	}
-#endif
-
 	return dev->of_node;
 }
 
 static struct arm_smmu_master *find_smmu_master(struct arm_smmu_device *smmu,
-						struct device_node *dev_node)
+						struct device *dev)
 {
 	struct rb_node *node = smmu->masters.rb_node;
 
@@ -XXX,XX +XXX,XX @@ static struct arm_smmu_master *find_smmu_master(struct arm_smmu_device *smmu,
 
 		master = container_of(node, struct arm_smmu_master, node);
 
-		if (dev_node < master->of_node)
+		if (dev < master->dev)
 			node = node->rb_left;
-		else if (dev_node > master->of_node)
+		else if (dev > master->dev)
 			node = node->rb_right;
 		else
 			return master;
@@ -XXX,XX +XXX,XX @@ static int insert_smmu_master(struct arm_smmu_device *smmu,
 			= container_of(*new, struct arm_smmu_master, node);
 
 		parent = *new;
-		if (master->of_node < this->of_node)
+		if (master->dev < this->dev)
 			new = &((*new)->rb_left);
-		else if (master->of_node > this->of_node)
+		else if (master->dev > this->dev)
 			new = &((*new)->rb_right);
 		else
 			return -EEXIST;
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_dt_add_device_legacy(struct arm_smmu_device *smmu,
 	struct arm_smmu_master *master;
 	struct device_node *dev_node = dev_get_dev_node(dev);
 
-	master = find_smmu_master(smmu, dev_node);
+	master = find_smmu_master(smmu, dev);
 	if (master) {
 		dev_err(dev,
-			"rejecting multiple registrations for master device %s\n",
-			dev_node->name);
+			"rejecting multiple registrations for master device\n");
 		return -EBUSY;
 	}
 
 	master = devm_kzalloc(dev, sizeof(*master), GFP_KERNEL);
 	if (!master)
 		return -ENOMEM;
-	master->of_node = dev_node;
+	master->dev = dev;
 
-	/* Xen: Let Xen know that the device is protected by an SMMU */
-	dt_device_set_protected(dev_node);
+	if ( !dev_is_pci(dev) )
+	{
+		/* Xen: Let Xen know that the device is protected by an SMMU */
+		dt_device_set_protected(dev_node);
+	}
 
 	for (i = 0; i < fwspec->num_ids; ++i) {
 		if (!(smmu->features & ARM_SMMU_FEAT_STREAM_MATCH) &&
 		     (fwspec->ids[i] >= smmu->num_mapping_groups)) {
 			dev_err(dev,
-				"stream ID for master device %s greater than maximum allowed (%d)\n",
-				dev_node->name, smmu->num_mapping_groups);
+				"SMMU stream ID %d is greater than maximum allowed (%d)\n",
+				fwspec->ids[i], smmu->num_mapping_groups);
 			return -ERANGE;
 		}
 		master->cfg.smendx[i] = INVALID_SMENDX;
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_dt_remove_device_legacy(struct arm_smmu_device *smmu,
 	struct device_node *dev_node = dev_get_dev_node(dev);
 	int ret;
 
-	master = find_smmu_master(smmu, dev_node);
+	master = find_smmu_master(smmu, dev);
 	if (master == NULL) {
 		dev_err(dev,
 			"No registrations found for master device %s\n",
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_dt_remove_device_legacy(struct arm_smmu_device *smmu,
 	if (ret)
 		return ret;
 
-	/* Protected by dt_host_lock and dtdevs_lock as caller holds these locks. */
-	dev_node->is_protected = false;
+	if ( !dev_is_pci(dev) )
+		/* Protected by dt_host_lock and dtdevs_lock as caller holds these locks. */
+		dev_node->is_protected = false;
 
 	kfree(master);
 	return 0;
@@ -XXX,XX +XXX,XX @@ static int register_smmu_master(struct arm_smmu_device *smmu,
 					     fwspec);
 }
 
+/* Forward declaration */
+static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
+			       struct device *dev, u32 flag);
+static int arm_smmu_deassign_dev(struct domain *d, uint8_t devfn,
+				 struct device *dev);
+
 /*
  * The driver which supports generic IOMMU DT bindings must have this
  * callback implemented.
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_dt_add_device_generic(u8 devfn, struct device *dev)
 {
 	struct arm_smmu_device *smmu;
 	struct iommu_fwspec *fwspec;
+	int ret;
+
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+		int ret;
+
+		/* Ignore calls for phantom functions */
+		if ( devfn != pdev->devfn )
+			return 0;
+
+		ret = iommu_add_pci_sideband_ids(pdev);
+		if ( ret < 0 )
+			iommu_fwspec_free(dev);
+	}
+#endif
 
 	fwspec = dev_iommu_fwspec_get(dev);
 	if (fwspec == NULL)
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_dt_add_device_generic(u8 devfn, struct device *dev)
 	if (smmu == NULL)
 		return -ENXIO;
 
-	return arm_smmu_dt_add_device_legacy(smmu, dev, fwspec);
+	ret = arm_smmu_dt_add_device_legacy(smmu, dev, fwspec);
+	if ( ret )
+		return ret;
+
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+
+		/*
+		 * During PHYSDEVOP_pci_device_add, Xen does not assign the
+		 * device, so we must do it here.
+		 */
+		if ( pdev->domain )
+			ret = arm_smmu_assign_dev(pdev->domain, devfn, dev, 0);
+	}
+#endif
+
+	return ret;
 }
 
 static int arm_smmu_dt_xlate_generic(struct device *dev,
@@ -XXX,XX +XXX,XX @@ static struct arm_smmu_device *find_smmu_for_device(struct device *dev)
 {
 	struct arm_smmu_device *smmu;
 	struct arm_smmu_master *master = NULL;
-	struct device_node *dev_node = dev_get_dev_node(dev);
 
 	spin_lock(&arm_smmu_devices_lock);
 	list_for_each_entry(smmu, &arm_smmu_devices, list) {
-		master = find_smmu_master(smmu, dev_node);
+		master = find_smmu_master(smmu, dev);
 		if (master)
 			break;
 	}
@@ -XXX,XX +XXX,XX @@ static bool arm_smmu_capable(enum iommu_cap cap)
 }
 #endif
 
+#if 0 /* Not used */
 static int __arm_smmu_get_pci_sid(struct pci_dev *pdev, u16 alias, void *data)
 {
 	*((u16 *)data) = alias;
@@ -XXX,XX +XXX,XX @@ static void __arm_smmu_release_pci_iommudata(void *data)
 {
 	kfree(data);
 }
+#endif
 
 static int arm_smmu_add_device(struct device *dev)
 {
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_add_device(struct device *dev)
 	struct arm_smmu_master_cfg *cfg;
 	struct iommu_group *group;
 	void (*releasefn)(void *data) = NULL;
-	int ret;
 
 	smmu = find_smmu_for_device(dev);
 	if (!smmu)
 		return -ENODEV;
 
+	/* There is no need to distinguish here, thanks to PCI-IOMMU DT bindings */
+#if 0
 	if (dev_is_pci(dev)) {
 		struct pci_dev *pdev = to_pci_dev(dev);
 		struct iommu_fwspec *fwspec;
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_add_device(struct device *dev)
 				       &fwspec->ids[0]);
 		releasefn = __arm_smmu_release_pci_iommudata;
 		cfg->smmu = smmu;
-	} else {
+	} else
+#endif
+	{
 		struct arm_smmu_master *master;
 
-		master = find_smmu_master(smmu, dev->of_node);
+		master = find_smmu_master(smmu, dev);
 		if (!master) {
 			return -ENODEV;
 		}
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
 			return -ENOMEM;
 	}
 
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+
+		/* Ignore calls for phantom functions */
+		if ( devfn != pdev->devfn )
+			return 0;
+
+		ASSERT(pcidevs_locked());
+
+		write_lock(&pdev->domain->pci_lock);
+		list_del(&pdev->domain_list);
+		write_unlock(&pdev->domain->pci_lock);
+
+		pdev->domain = d;
+
+		write_lock(&d->pci_lock);
+		list_add(&pdev->domain_list, &d->pdev_list);
+		write_unlock(&d->pci_lock);
+
+		/* dom_io is used as a sentinel for quarantined devices */
+		if ( d == dom_io )
+		{
+			struct iommu_domain *domain = dev_iommu_domain(dev);
+			if ( !iommu_quarantine )
+				return 0;
+
+			if ( domain && domain->priv )
+				arm_smmu_deassign_dev(domain->priv->cfg.domain, devfn, dev);
+
+			return 0;
+		}
+	}
+#endif
+
 	if (!dev_iommu_group(dev)) {
 		ret = arm_smmu_add_device(dev);
 		if (ret)
@@ -XXX,XX +XXX,XX @@ out:
 	return ret;
 }
 
-static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)
+static int arm_smmu_deassign_dev(struct domain *d, uint8_t devfn,
+				 struct device *dev)
 {
 	struct iommu_domain *domain = dev_iommu_domain(dev);
 	struct arm_smmu_xen_domain *xen_domain;
 
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+
+		/* Ignore calls for phantom functions */
+		if ( devfn != pdev->devfn )
+			return 0;
+
+		/* dom_io is used as a sentinel for quarantined devices */
+		if ( d == dom_io )
+			return 0;
+	}
+#endif
+
 	xen_domain = dom_iommu(d)->arch.priv;
 
 	if (!domain || domain->priv->cfg.domain != d) {
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_reassign_dev(struct domain *s, struct domain *t,
 {
 	int ret = 0;
 
-	/* Don't allow remapping on other domain than hwdom */
-	if ( t && !is_hardware_domain(t) )
+	/* Don't allow remapping on other domain than hwdom
+	 * or dom_io for PCI devices
+	 */
+	if ( t && !is_hardware_domain(t) && (t != dom_io || !dev_is_pci(dev)) )
 		return -EPERM;
 
 	if (t == s)
 		return 0;
 
-	ret = arm_smmu_deassign_dev(s, dev);
+	ret = arm_smmu_deassign_dev(s, devfn, dev);
 	if (ret)
 		return ret;
 
-- 
2.34.1

From: Rahul Singh <rahul.singh@arm.com>

Implement support for PCI devices in the SMMU driver. Trigger iommu-map
parsing when new PCI device is added. Add checks to assign/deassign
functions to ensure PCI devices are handled correctly. Implement basic
quarantining.

All pci devices are automatically assigned to hardware domain if it exists
to ensure it can probe them.

TODO:
Implement scratch page quarantining support.

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
v6->v7:
* address TODO: use d->pci_lock in arm_smmu_assign_dev()
* remove !is_hardware_domain and pdev->domain == d checks in assign to
  support future dom0less use case when dom0 is using vPCI
* check if pdev->domain exists before assigning to it
* don't print ""
* change assign logic to remove reassign reimplementation
* explain pdev->devfn check
* make reassign check stricter and update comment

v5->v6:
* check for hardware_domain == NULL (dom0less test case)
* locking: assign pdev->domain before list_add()

v4->v5:
* deassign from hwdom
* add TODO regarding locking
* fixup after dropping ("xen/arm: Move is_protected flag to struct device")

v3->v4:
* no change

v2->v3:
* rebase
* invoke iommu_add_pci_sideband_ids() from add_device hook

v1->v2:
* ignore add_device/assign_device/reassign_device calls for phantom functions
  (i.e. devfn != pdev->devfn)

(cherry picked from commit 7ed6c3ab250d899fe6e893a514278e406a2893e8 from
 the downstream branch poc/pci-passthrough from
 https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
---
 xen/drivers/passthrough/arm/smmu-v3.c | 117 +++++++++++++++++++++++---
 1 file changed, 106 insertions(+), 11 deletions(-)

diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
index XXXXXXX..XXXXXXX 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.c
+++ b/xen/drivers/passthrough/arm/smmu-v3.c
@@ -XXX,XX +XXX,XX @@ static bool arm_smmu_sid_in_range(struct arm_smmu_device *smmu, u32 sid)
 }
 /* Forward declaration */
 static struct arm_smmu_device *arm_smmu_get_by_dev(const struct device *dev);
+static int arm_smmu_assign_dev(struct domain *d, u8 devfn, struct device *dev,
+			       u32 flag);
+static int arm_smmu_deassign_dev(struct domain *d, uint8_t devfn,
+				 struct device *dev);
 
 static int arm_smmu_add_device(u8 devfn, struct device *dev)
 {
 	int i, ret;
 	struct arm_smmu_device *smmu;
 	struct arm_smmu_master *master;
-	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+	struct iommu_fwspec *fwspec;
+
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+		int ret;
+				
+		/* Ignore calls for phantom functions */
+		if ( devfn != pdev->devfn )
+			return 0;
+
+		ret = iommu_add_pci_sideband_ids(pdev);
+		if ( ret < 0 )
+			iommu_fwspec_free(dev);
+	}
+#endif
 
+	fwspec = dev_iommu_fwspec_get(dev);
 	if (!fwspec)
 		return -ENODEV;
 
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_add_device(u8 devfn, struct device *dev)
 	 */
 	arm_smmu_enable_pasid(master);
 
-	if (dt_device_is_protected(dev_to_dt(dev))) {
-		dev_err(dev, "Already added to SMMUv3\n");
-		return -EEXIST;
-	}
+	if ( !dev_is_pci(dev) )
+	{
+		if (dt_device_is_protected(dev_to_dt(dev))) {
+			dev_err(dev, "Already added to SMMUv3\n");
+			return -EEXIST;
+		}
 
-	/* Let Xen know that the master device is protected by an IOMMU. */
-	dt_device_set_protected(dev_to_dt(dev));
+		/* Let Xen know that the master device is protected by an IOMMU. */
+		dt_device_set_protected(dev_to_dt(dev));
+	}
 
 	dev_info(dev, "Added master device (SMMUv3 %s StreamIds %u)\n",
 			dev_name(fwspec->iommu_dev), fwspec->num_ids);
 
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+
+		/*
+		 * During PHYSDEVOP_pci_device_add, Xen does not assign the
+		 * device, so we must do it here.
+		 */
+		if ( pdev->domain )
+		{
+			ret = arm_smmu_assign_dev(pdev->domain, devfn, dev, 0);
+			if (ret)
+				goto err_free_master;
+		}
+	}
+#endif
+
 	return 0;
 
 err_free_master:
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
 	struct arm_smmu_domain *smmu_domain;
 	struct arm_smmu_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
 
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+
+		/* Ignore calls for phantom functions */
+		if ( devfn != pdev->devfn )
+			return 0;
+
+		ASSERT(pcidevs_locked());
+
+		write_lock(&pdev->domain->pci_lock);
+		list_del(&pdev->domain_list);
+		write_unlock(&pdev->domain->pci_lock);
+
+		pdev->domain = d;
+
+		write_lock(&d->pci_lock);
+		list_add(&pdev->domain_list, &d->pdev_list);
+		write_unlock(&d->pci_lock);
+
+		/* dom_io is used as a sentinel for quarantined devices */
+		if ( d == dom_io )
+		{
+			struct arm_smmu_master *master = dev_iommu_priv_get(dev);
+			if ( !iommu_quarantine )
+				return 0;
+
+			if ( master && master->domain )
+				arm_smmu_deassign_dev(master->domain->d, devfn, dev);
+
+			return 0;
+		}
+	}
+#endif
+
 	spin_lock(&xen_domain->lock);
 
 	/*
@@ -XXX,XX +XXX,XX @@ out:
 	return ret;
 }
 
-static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)
+static int arm_smmu_deassign_dev(struct domain *d, uint8_t devfn, struct device *dev)
 {
 	struct iommu_domain *io_domain = arm_smmu_get_domain(d, dev);
 	struct arm_smmu_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)
 		return -ESRCH;
 	}
 
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+
+		/* Ignore calls for phantom functions */
+		if ( devfn != pdev->devfn )
+			return 0;
+
+		/* dom_io is used as a sentinel for quarantined devices */
+		if ( d == dom_io )
+			return 0;
+	}
+#endif
+
 	spin_lock(&xen_domain->lock);
 
 	arm_smmu_detach_dev(master);
@@ -XXX,XX +XXX,XX @@ static int arm_smmu_reassign_dev(struct domain *s, struct domain *t,
 {
 	int ret = 0;
 
-	/* Don't allow remapping on other domain than hwdom */
-	if ( t && !is_hardware_domain(t) )
+	/* Don't allow remapping on other domain than hwdom
+	 * or dom_io for PCI devices
+	 */
+	if ( t && !is_hardware_domain(t) && (t != dom_io || !dev_is_pci(dev)) )
 		return -EPERM;
 
 	if (t == s)
 		return 0;
 
-	ret = arm_smmu_deassign_dev(s, dev);
+	ret = arm_smmu_deassign_dev(s, devfn, dev);
 	if (ret)
 		return ret;
 
-- 
2.34.1

From: Rahul Singh <rahul.singh@arm.com>

Current code skip the mapping for PCI bridge MMIO region to dom0 when
pci_passthrough_enabled flag is set. Mapping should be skip when
has_vpci(d) is enabled for the domain, as we need to skip the mapping
only when VPCI handler are registered for ECAM.

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
Acked-by: Julien Grall <jgrall@amazon.com>
---
This patch was originally picked up from [1]
v6->v7:
* add Julien's A-b

v5->v6:
* drop unrelated change in xen/arch/arm/domain_build.c:handle_linux_pci_domain()

v4->v5:
* new patch

changes since picking up from [1]:
* rebase on top of "dynamic node programming using overlay dtbo" series
* replace !is_pci_passthrough_enabled() check with !IS_ENABLED(CONFIG_HAS_PCI)
  instead of removing

[1] https://lists.xenproject.org/archives/html/xen-devel/2023-07/msg00483.html
---
 xen/arch/arm/device.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/xen/arch/arm/device.c b/xen/arch/arm/device.c
index XXXXXXX..XXXXXXX 100644
--- a/xen/arch/arm/device.c
+++ b/xen/arch/arm/device.c
@@ -XXX,XX +XXX,XX @@ int handle_device(struct domain *d, struct dt_device_node *dev, p2m_type_t p2mt,
         .d = d,
         .p2mt = p2mt,
         .skip_mapping = !own_device ||
-                        (is_pci_passthrough_enabled() &&
+                        (has_vpci(d) &&
                         (device_get_class(dev) == DEVICE_PCI_HOSTBRIDGE)),
         .iomem_ranges = iomem_ranges,
         .irq_ranges = irq_ranges
-- 
2.34.1

From: Stewart Hildebrand <stewart.hildebrand@amd.com>

Enable the use of IOMMU + PCI in dom0 without having to specify
"pci-passthrough=yes". We rely on dom0 to initialize the PCI controller
and perform a PHYSDEVOP_pci_device_add call to add each device to SMMU.

Enable pci_init() for initializing Xen's internal PCI subsystem, and
allow PHYSDEVOP_pci_device_add when pci-passthrough is disabled.

is_pci_passthrough_enabled() is not an Arm-only construct, so remove the
x86 definition of the function.

Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
hmm. Since
  dec9e02f3190 ("xen: avoid generation of stub <asm/pci.h> header")
Should we also move is_pci_passthrough_enabled() back to xen/arch/arm/include/asm/pci.h ?
Not sure if PPC/RISC-V will plan on using this check.

v6->v7:
* remove x86 definition of is_pci_passthrough_enabled()
* update comments
* make pci_physdev_op checks stricter

v5->v6:
* new patch - this effectively replaces
  ("Revert "xen/arm: Add cmdline boot option "pci-passthrough = <boolean>""")
---
 xen/arch/arm/pci/pci.c         | 5 +++--
 xen/arch/x86/include/asm/pci.h | 6 ------
 xen/drivers/pci/physdev.c      | 4 ++--
 3 files changed, 5 insertions(+), 10 deletions(-)

diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
index XXXXXXX..XXXXXXX 100644
--- a/xen/arch/arm/pci/pci.c
+++ b/xen/arch/arm/pci/pci.c
@@ -XXX,XX +XXX,XX @@
 #include <xen/device_tree.h>
 #include <xen/errno.h>
 #include <xen/init.h>
+#include <xen/iommu.h>
 #include <xen/param.h>
 #include <xen/pci.h>
 
@@ -XXX,XX +XXX,XX @@ static int __init pci_init(void)
 {
     /*
      * Enable PCI passthrough when has been enabled explicitly
-     * (pci-passthrough=on).
+     * (pci-passthrough=on) or IOMMU is present and enabled.
      */
-    if ( !pci_passthrough_enabled )
+    if ( !is_pci_passthrough_enabled() && !iommu_enabled )
         return 0;
 
     pci_segments_init();
diff --git a/xen/arch/x86/include/asm/pci.h b/xen/arch/x86/include/asm/pci.h
index XXXXXXX..XXXXXXX 100644
--- a/xen/arch/x86/include/asm/pci.h
+++ b/xen/arch/x86/include/asm/pci.h
@@ -XXX,XX +XXX,XX @@ bool pci_ro_mmcfg_decode(unsigned long mfn, unsigned int *seg,
 extern int pci_mmcfg_config_num;
 extern struct acpi_mcfg_allocation *pci_mmcfg_config;
 
-/* Unlike ARM, PCI passthrough is always enabled for x86. */
-static always_inline bool is_pci_passthrough_enabled(void)
-{
-    return true;
-}
-
 void arch_pci_init_pdev(struct pci_dev *pdev);
 
 static inline bool pci_check_bar(const struct pci_dev *pdev,
diff --git a/xen/drivers/pci/physdev.c b/xen/drivers/pci/physdev.c
index XXXXXXX..XXXXXXX 100644
--- a/xen/drivers/pci/physdev.c
+++ b/xen/drivers/pci/physdev.c
@@ -XXX,XX +XXX,XX @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         struct pci_dev_info pdev_info;
         nodeid_t node = NUMA_NO_NODE;
 
-        if ( !is_pci_passthrough_enabled() )
+        if ( !is_pci_passthrough_enabled() && !iommu_enabled )
             return -EOPNOTSUPP;
 
         ret = -EFAULT;
@@ -XXX,XX +XXX,XX @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
     case PHYSDEVOP_pci_device_remove: {
         struct physdev_pci_device dev;
 
-        if ( !is_pci_passthrough_enabled() )
+        if ( !is_pci_passthrough_enabled() && !iommu_enabled )
             return -EOPNOTSUPP;
 
         ret = -EFAULT;
-- 
2.34.1

From: Rahul Singh <rahul.singh@arm.com>

When ITS is enabled and PCI devices that are behind an SMMU generate an
MSI interrupt, SMMU fault will be observed as there is currently no
mapping in p2m table for the ITS translation register (GITS_TRANSLATER).

A mapping is required in the iommu page tables so that the device can
generate the MSI interrupt writing to the GITS_TRANSLATER register.

The GITS_TRANSLATER register is a 32-bit register, and there is nothing
else in a page containing it, so map that page.

Example SMMUv3 fault (qemu-system-aarch64 virt model), ITS base 0x8080000:

(XEN) SMMUv3: /smmuv3@9050000: event 0x10 received:
(XEN) SMMUv3: /smmuv3@9050000:  0x0000000800000010
(XEN) SMMUv3: /smmuv3@9050000:  0x0000008000000000
(XEN) SMMUv3: /smmuv3@9050000:  0x0000000008090040
(XEN) SMMUv3: /smmuv3@9050000:  0x0000000000000000

Example SMMUv2 fault (AMD/Xilinx Versal), ITS base 0xf9020000:

(XEN) smmu: /axi/smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0xf9030040, fsynr=0x12, cb=0

v6->v7:
* add tlb flush after mapping
* style: update formatting
* revert back to printk with XENLOG_G_ERR

v5->v6:
* switch to iommu_map() interface
* fix page_count argument
* style fixup
* use gprintk instead of printk
* add my Signed-off-by
* move to vgic_v3_its_init_virtual()

v4->v5:
* new patch

[1] https://lists.xenproject.org/archives/html/xen-devel/2023-07/msg00483.html
[2] https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-/commit/6232a0d53377009bb7fbc3c3ab81d0153734be6b
---
 xen/arch/arm/vgic-v3-its.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index XXXXXXX..XXXXXXX 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -XXX,XX +XXX,XX @@ static int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr,
 
     register_mmio_handler(d, &vgic_its_mmio_handler, guest_addr, SZ_64K, its);
 
+    if ( is_iommu_enabled(its->d) )
+    {
+        mfn_t mfn = maddr_to_mfn(its->doorbell_address);
+        unsigned int flush_flags = 0;
+        int ret = iommu_map(its->d, _dfn(mfn_x(mfn)), mfn, 1, IOMMUF_writable,
+                            &flush_flags);
+
+        if ( ret < 0 )
+        {
+            printk(XENLOG_G_ERR
+                    "GICv3: Map ITS translation register for %pd failed.\n",
+                    its->d);
+            return ret;
+        }
+
+        ret = iommu_iotlb_flush(its->d, _dfn(mfn_x(mfn)), 1, flush_flags);
+        if ( ret < 0 )
+            return ret;
+    }
+
     /* Register the virtual ITS to be able to clean it up later. */
     list_add_tail(&its->vits_list, &d->arch.vgic.vits_list);
 
-- 
2.34.1