When the IOMMU address space width is smaller than the physical
address width, a MMIO region of a device can fail to map because the
region is outside the supported IOVA ranges of the VM. In this case,
PCI peer-to-peer transactions on BARs are not supported.

This can occur with the 39-bit IOMMU address space width, as can be
the case on consumer processors or when using a vIOMMU device with
default settings.

The current error message is unclear. Change the error report to a
warning because it is a non fatal condition for the VM, clarify the
induced limitations for the user and provide advice on how to possibly
resolve this issue: setting the CPU address space width of the IOMMU
address space width accordingly.

Signed-off-by: Cédric Le Goater <clg@redhat.com>
---
 hw/vfio/common.c | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)

diff --git a/hw/vfio/common.c b/hw/vfio/common.c
index 3ca5dbf883ed2262e36952fcc47e717ff4154f12..561be7f57cf903e6e2bcbbb708be9e4d4ee8941c 100644
--- a/hw/vfio/common.c
+++ b/hw/vfio/common.c
@@ -569,6 +569,20 @@ static bool vfio_get_section_iova_range(VFIOContainerBase *bcontainer,
     return true;
 }
 
+static void vfio_container_p2p_error_append(VFIOContainerBase *bcontainer,
+                                            Error **errp)
+{
+    error_append_hint(errp, "PCI peer-to-peer transactions on BARs "
+                      "are not supported. ");
+    if (vfio_viommu_preset(bcontainer)) {
+        error_append_hint(errp, "Try setting the vIOMMU \"aw-bits\" "
+                          "property to match CPU address space width\n");
+    } else {
+        error_append_hint(errp, "Try setting the CPU \"phys-bits\" "
+                          "property to match IOMMU address space width\n");
+    }
+}
+
 static void vfio_listener_region_add(MemoryListener *listener,
                                      MemoryRegionSection *section)
 {
@@ -685,9 +699,13 @@ static void vfio_listener_region_add(MemoryListener *listener,
                    "0x%"HWADDR_PRIx", %p) = %d (%s)",
                    bcontainer, iova, int128_get64(llsize), vaddr, ret,
                    strerror(-ret));
+        /*
+         * MMIO region mapping failures are not fatal but in this case
+         * PCI peer-to-peer transactions are broken.
+         */
         if (memory_region_is_ram_device(section->mr)) {
-            /* Allow unexpected mappings not to be fatal for RAM devices */
-            error_report_err(err);
+            vfio_container_p2p_error_append(bcontainer, &err);
+            warn_report_err(err);
             return;
         }
         goto fail;
-- 
2.48.1