From: Vikram Garhwal <fnu.vikram@xilinx.com>

Signed-off-by: Vikram Garhwal <fnu.vikram@xilinx.com>
Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
Signed-off-by: Henry Wang <xin.wang2@amd.com>
---
v4:
- No change.
v3:
- No change.
v2:
- Update the content based on the changes in this version.
---
 docs/misc/arm/overlay.txt | 99 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 99 insertions(+)
 create mode 100644 docs/misc/arm/overlay.txt

diff --git a/docs/misc/arm/overlay.txt b/docs/misc/arm/overlay.txt
new file mode 100644
index 0000000000..811a6de369
--- /dev/null
+++ b/docs/misc/arm/overlay.txt
@@ -0,0 +1,99 @@
+# Device Tree Overlays support in Xen
+
+Xen now supports dynamic device assignment to running domains,
+i.e. adding/removing nodes (using .dtbo) to/from Xen device tree, and
+attaching/detaching them to/from a running domain with given $domid.
+
+Dynamic node assignment works in two steps:
+
+## Add/Remove device tree overlay to/from Xen device tree
+
+1. Xen tools check the dtbo given and parse all other user provided arguments
+2. Xen tools pass the dtbo to Xen hypervisor via hypercall.
+3. Xen hypervisor applies/removes the dtbo to/from Xen device tree.
+
+## Attach/Detach device from the DT overlay to/from domain
+
+1. Xen tools check the dtbo given and parse all other user provided arguments
+2. Xen tools pass the dtbo to Xen hypervisor via hypercall.
+3. Xen hypervisor attach/detach the device to/from the user-provided $domid by
+   mapping/unmapping node resources in the DT overlay.
+
+# Examples
+
+Here are a few examples on how to use it.
+
+## Dom0 device add
+
+For assigning a device tree overlay to Dom0, user should firstly properly
+prepare the DT overlay. More information about device tree overlays can be
+found in [1]. Then, in Dom0, enter the following:
+
+    (dom0) xl dt-overlay add overlay.dtbo
+
+This will allocate the devices mentioned in overlay.dtbo to Xen device tree.
+
+To assign the newly added device from the dtbo to Dom0:
+
+    (dom0) xl dt-overlay attach overlay.dtbo 0
+
+Next, if the user wants to add the same device tree overlay to dom0
+Linux, execute the following:
+
+    (dom0) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay
+    (dom0) cat overlay.dtbo > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo
+
+Finally if needed, the relevant Linux kernel drive can be loaded using:
+
+    (dom0) modprobe module_name.ko
+
+## Dom0 device remove
+
+For removing the device from Dom0, first detach the device from Dom0:
+
+    (dom0) xl dt-overlay detach overlay.dtbo 0
+
+NOTE: The user is expected to unload any Linux kernel modules which
+might be accessing the devices in overlay.dtbo before detach the device.
+Detaching devices without unloading the modules might result in a crash.
+
+Then remove the overlay from Xen device tree:
+
+    (dom0) xl dt-overlay remove overlay.dtbo
+
+## DomU device add/remove
+
+All the nodes in dtbo will be assigned to a domain; the user will need
+to prepare the dtb for the domU. For example, the `interrupt-parent` property
+of the DomU overlay should be changed to the Xen hardcoded value `0xfde8`.
+Below assumes the properly written DomU dtbo is `overlay_domu.dtbo`.
+
+User will need to create the DomU with below properties properly configured
+in the xl config file:
+- `iomem`
+- `passthrough` (if IOMMU is needed)
+
+User will also need to modprobe the relevant drivers.
+
+Example for domU device add:
+
+    (dom0) xl dt-overlay add overlay.dtbo            # If not executed before
+    (dom0) xl dt-overlay attach overlay.dtbo $domid
+    (dom0) xl console $domid                         # To access $domid console
+
+Next, if the user needs to modify/prepare the overlay.dtbo suitable for
+the domU:
+
+    (domU) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay
+    (domU) cat overlay_domu.dtbo > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo
+
+Finally, if needed, the relevant Linux kernel drive can be probed:
+
+    (domU) modprobe module_name.ko
+
+Example for domU overlay remove:
+
+    (dom0) xl dt-overlay detach overlay.dtbo $domid
+    (dom0) xl dt-overlay remove overlay.dtbo
+
+[1] https://www.kernel.org/doc/Documentation/devicetree/overlay-notes.txt
-- 
2.34.1

Hi Henry,

On 23/05/2024 08:40, Henry Wang wrote:
> From: Vikram Garhwal <fnu.vikram@xilinx.com>
> 
> Signed-off-by: Vikram Garhwal <fnu.vikram@xilinx.com>
> Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
> Signed-off-by: Henry Wang <xin.wang2@amd.com>
> ---
> v4:
> - No change.
> v3:
> - No change.
> v2:
> - Update the content based on the changes in this version.
> ---
>   docs/misc/arm/overlay.txt | 99 +++++++++++++++++++++++++++++++++++++++
>   1 file changed, 99 insertions(+)
>   create mode 100644 docs/misc/arm/overlay.txt
> 
> diff --git a/docs/misc/arm/overlay.txt b/docs/misc/arm/overlay.txt
> new file mode 100644
> index 0000000000..811a6de369
> --- /dev/null
> +++ b/docs/misc/arm/overlay.txt
> @@ -0,0 +1,99 @@
> +# Device Tree Overlays support in Xen
> +
> +Xen now supports dynamic device assignment to running domains,

This reads as we "support" the feature. I would prefer if we write "Xen 
expirementally supports..." or similar.

> +i.e. adding/removing nodes (using .dtbo) to/from Xen device tree, and
> +attaching/detaching them to/from a running domain with given $domid.
> +
> +Dynamic node assignment works in two steps:
> +
> +## Add/Remove device tree overlay to/from Xen device tree
> +
> +1. Xen tools check the dtbo given and parse all other user provided arguments
> +2. Xen tools pass the dtbo to Xen hypervisor via hypercall.
> +3. Xen hypervisor applies/removes the dtbo to/from Xen device tree.
> +
> +## Attach/Detach device from the DT overlay to/from domain
> +
> +1. Xen tools check the dtbo given and parse all other user provided arguments
> +2. Xen tools pass the dtbo to Xen hypervisor via hypercall.
> +3. Xen hypervisor attach/detach the device to/from the user-provided $domid by
> +   mapping/unmapping node resources in the DT overlay.
> +
> +# Examples
> +
> +Here are a few examples on how to use it.
> +
> +## Dom0 device add
> +
> +For assigning a device tree overlay to Dom0, user should firstly properly
> +prepare the DT overlay. More information about device tree overlays can be
> +found in [1]. Then, in Dom0, enter the following:
> +
> +    (dom0) xl dt-overlay add overlay.dtbo
> +
> +This will allocate the devices mentioned in overlay.dtbo to Xen device tree.
> +
> +To assign the newly added device from the dtbo to Dom0:
> +
> +    (dom0) xl dt-overlay attach overlay.dtbo 0
> +
> +Next, if the user wants to add the same device tree overlay to dom0
> +Linux, execute the following:
> +
> +    (dom0) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay
> +    (dom0) cat overlay.dtbo > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo
> +
> +Finally if needed, the relevant Linux kernel drive can be loaded using:
> +
> +    (dom0) modprobe module_name.ko
> +
> +## Dom0 device remove
> +
> +For removing the device from Dom0, first detach the device from Dom0:
> +
> +    (dom0) xl dt-overlay detach overlay.dtbo 0
> +
> +NOTE: The user is expected to unload any Linux kernel modules which
> +might be accessing the devices in overlay.dtbo before detach the device.
> +Detaching devices without unloading the modules might result in a crash.
> +
> +Then remove the overlay from Xen device tree:
> +
> +    (dom0) xl dt-overlay remove overlay.dtbo
> +
> +## DomU device add/remove
> +
> +All the nodes in dtbo will be assigned to a domain; the user will need
> +to prepare the dtb for the domU. For example, the `interrupt-parent` property
> +of the DomU overlay should be changed to the Xen hardcoded value `0xfde8`.
> +Below assumes the properly written DomU dtbo is `overlay_domu.dtbo`.
> +
> +User will need to create the DomU with below properties properly configured
> +in the xl config file:
> +- `iomem`

I don't quite understand how the user can specify the MMIO region if the 
device is attached after the domain is created.

> +- `passthrough` (if IOMMU is needed)
> +
> +User will also need to modprobe the relevant drivers.
> +
> +Example for domU device add:
> +
> +    (dom0) xl dt-overlay add overlay.dtbo            # If not executed before
> +    (dom0) xl dt-overlay attach overlay.dtbo $domid

Can how clarify how the MMIO will be mapped? Is it direct mapped? If so, 
couldn't this result to clash with other part of the address space (e.g. 
RAM?).

> +    (dom0) xl console $domid                         # To access $domid console
> +
> +Next, if the user needs to modify/prepare the overlay.dtbo suitable for
> +the domU:
> +
> +    (domU) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay
> +    (domU) cat overlay_domu.dtbo > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo
> +
> +Finally, if needed, the relevant Linux kernel drive can be probed:
> +
> +    (domU) modprobe module_name.ko
> +
> +Example for domU overlay remove:
> +
> +    (dom0) xl dt-overlay detach overlay.dtbo $domid
> +    (dom0) xl dt-overlay remove overlay.dtbo

I assume we have safety check in place to ensure we can't remove the 
device if it is already attached. Is that correct?

Cheers,

-- 
Julien Grall

On Thu, 23 May 2024, Julien Grall wrote:
> Hi Henry,
> 
> On 23/05/2024 08:40, Henry Wang wrote:
> > From: Vikram Garhwal <fnu.vikram@xilinx.com>
> > 
> > Signed-off-by: Vikram Garhwal <fnu.vikram@xilinx.com>
> > Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > Signed-off-by: Henry Wang <xin.wang2@amd.com>
> > ---
> > v4:
> > - No change.
> > v3:
> > - No change.
> > v2:
> > - Update the content based on the changes in this version.
> > ---
> >   docs/misc/arm/overlay.txt | 99 +++++++++++++++++++++++++++++++++++++++
> >   1 file changed, 99 insertions(+)
> >   create mode 100644 docs/misc/arm/overlay.txt
> > 
> > diff --git a/docs/misc/arm/overlay.txt b/docs/misc/arm/overlay.txt
> > new file mode 100644
> > index 0000000000..811a6de369
> > --- /dev/null
> > +++ b/docs/misc/arm/overlay.txt
> > @@ -0,0 +1,99 @@
> > +# Device Tree Overlays support in Xen
> > +
> > +Xen now supports dynamic device assignment to running domains,
> 
> This reads as we "support" the feature. I would prefer if we write "Xen
> expirementally supports..." or similar.

Done


> > +i.e. adding/removing nodes (using .dtbo) to/from Xen device tree, and
> > +attaching/detaching them to/from a running domain with given $domid.
> > +
> > +Dynamic node assignment works in two steps:
> > +
> > +## Add/Remove device tree overlay to/from Xen device tree
> > +
> > +1. Xen tools check the dtbo given and parse all other user provided
> > arguments
> > +2. Xen tools pass the dtbo to Xen hypervisor via hypercall.
> > +3. Xen hypervisor applies/removes the dtbo to/from Xen device tree.
> > +
> > +## Attach/Detach device from the DT overlay to/from domain
> > +
> > +1. Xen tools check the dtbo given and parse all other user provided
> > arguments
> > +2. Xen tools pass the dtbo to Xen hypervisor via hypercall.
> > +3. Xen hypervisor attach/detach the device to/from the user-provided $domid
> > by
> > +   mapping/unmapping node resources in the DT overlay.
> > +
> > +# Examples
> > +
> > +Here are a few examples on how to use it.
> > +
> > +## Dom0 device add
> > +
> > +For assigning a device tree overlay to Dom0, user should firstly properly
> > +prepare the DT overlay. More information about device tree overlays can be
> > +found in [1]. Then, in Dom0, enter the following:
> > +
> > +    (dom0) xl dt-overlay add overlay.dtbo
> > +
> > +This will allocate the devices mentioned in overlay.dtbo to Xen device
> > tree.
> > +
> > +To assign the newly added device from the dtbo to Dom0:
> > +
> > +    (dom0) xl dt-overlay attach overlay.dtbo 0
> > +
> > +Next, if the user wants to add the same device tree overlay to dom0
> > +Linux, execute the following:
> > +
> > +    (dom0) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay
> > +    (dom0) cat overlay.dtbo >
> > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo
> > +
> > +Finally if needed, the relevant Linux kernel drive can be loaded using:
> > +
> > +    (dom0) modprobe module_name.ko
> > +
> > +## Dom0 device remove
> > +
> > +For removing the device from Dom0, first detach the device from Dom0:
> > +
> > +    (dom0) xl dt-overlay detach overlay.dtbo 0
> > +
> > +NOTE: The user is expected to unload any Linux kernel modules which
> > +might be accessing the devices in overlay.dtbo before detach the device.
> > +Detaching devices without unloading the modules might result in a crash.
> > +
> > +Then remove the overlay from Xen device tree:
> > +
> > +    (dom0) xl dt-overlay remove overlay.dtbo
> > +
> > +## DomU device add/remove
> > +
> > +All the nodes in dtbo will be assigned to a domain; the user will need
> > +to prepare the dtb for the domU. For example, the `interrupt-parent`
> > property
> > +of the DomU overlay should be changed to the Xen hardcoded value `0xfde8`.
> > +Below assumes the properly written DomU dtbo is `overlay_domu.dtbo`.
> > +
> > +User will need to create the DomU with below properties properly configured
> > +in the xl config file:
> > +- `iomem`
> 
> I don't quite understand how the user can specify the MMIO region if the
> device is attached after the domain is created.

I think this was meant for a domain about to be created (not already
running). I clarified.


> 
> > +- `passthrough` (if IOMMU is needed)
> > +
> > +User will also need to modprobe the relevant drivers.
> > +
> > +Example for domU device add:
> > +
> > +    (dom0) xl dt-overlay add overlay.dtbo            # If not executed
> > before
> > +    (dom0) xl dt-overlay attach overlay.dtbo $domid
> 
> Can how clarify how the MMIO will be mapped? Is it direct mapped? If so,
> couldn't this result to clash with other part of the address space (e.g.
> RAM?).

Yes, it is reusing the same code as dom0, which makes the code nice but
it doesn't support non-1:1 mappings. I think those should be done via
the xen,reg property. My suggestion would be this:

- if xen,reg is present, use it
- if xen,reg is not present, fall back to 1:1 mapping based on reg

For the next version of the series, I'd just document the current
limitation of the implementation. I added this to patch #4:

diff --git a/xen/common/dt-overlay.c b/xen/common/dt-overlay.c
index 0f8b25ccb4..c2b03865a7 100644
--- a/xen/common/dt-overlay.c
+++ b/xen/common/dt-overlay.c
@@ -845,7 +845,7 @@ static long handle_attach_overlay_nodes(struct domain *d,
                                         uint32_t overlay_fdt_size)
 {
     int rc;
-    unsigned int j;
+    unsigned int j, len;
     struct overlay_track *entry;
 
     rc = check_overlay_fdt(overlay_fdt, overlay_fdt_size);
@@ -888,6 +888,12 @@ static long handle_attach_overlay_nodes(struct domain *d,
             goto out;
         }
 
+        if ( dt_get_property(overlay_node, "xen,reg", &len) )
+        {
+            printk(XENLOG_ERR "xen,reg not supported yet in overlay\n");
+            rc = -EOPNOTSUPP;
+            goto out;
+        }
         write_lock(&dt_host_lock);
         rc = handle_device(d, overlay_node, p2m_mmio_direct_c,
                            entry->iomem_ranges, entry->irq_ranges);



> > +    (dom0) xl console $domid                         # To access $domid
> > console
> > +
> > +Next, if the user needs to modify/prepare the overlay.dtbo suitable for
> > +the domU:
> > +
> > +    (domU) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay
> > +    (domU) cat overlay_domu.dtbo >
> > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo
> > +
> > +Finally, if needed, the relevant Linux kernel drive can be probed:
> > +
> > +    (domU) modprobe module_name.ko
> > +
> > +Example for domU overlay remove:
> > +
> > +    (dom0) xl dt-overlay detach overlay.dtbo $domid
> > +    (dom0) xl dt-overlay remove overlay.dtbo
> 
> I assume we have safety check in place to ensure we can't remove the device if
> it is already attached. Is that correct?

I'll remove this part of the doc.