From: Vikram Garhwal <fnu.vikram@xilinx.com>

Signed-off-by: Vikram Garhwal <fnu.vikram@xilinx.com>
Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
Signed-off-by: Henry Wang <xin.wang2@amd.com>
---
 docs/misc/arm/overlay.txt | 82 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)
 create mode 100644 docs/misc/arm/overlay.txt

diff --git a/docs/misc/arm/overlay.txt b/docs/misc/arm/overlay.txt
new file mode 100644
index 0000000000..0a2dee951a
--- /dev/null
+++ b/docs/misc/arm/overlay.txt
@@ -0,0 +1,82 @@
+# Device Tree Overlays support in Xen
+
+Xen experimentally supports dynamic device assignment to running
+domains, i.e. adding/removing nodes (using .dtbo) to/from Xen device
+tree, and attaching them to a running domain with given $domid.
+
+Dynamic node assignment works in two steps:
+
+## Add/Remove device tree overlay to/from Xen device tree
+
+1. Xen tools check the dtbo given and parse all other user provided arguments
+2. Xen tools pass the dtbo to Xen hypervisor via hypercall.
+3. Xen hypervisor applies/removes the dtbo to/from Xen device tree.
+
+## Attach device from the DT overlay to domain
+
+1. Xen tools check the dtbo given and parse all other user provided arguments
+2. Xen tools pass the dtbo to Xen hypervisor via hypercall.
+3. Xen hypervisor attach the device to the user-provided $domid by
+   mapping node resources in the DT overlay.
+
+# Examples
+
+Here are a few examples on how to use it.
+
+## Dom0 device add
+
+For assigning a device tree overlay to Dom0, user should firstly properly
+prepare the DT overlay. More information about device tree overlays can be
+found in [1]. Then, in Dom0, enter the following:
+
+    (dom0) xl dt-overlay add overlay.dtbo
+
+This will allocate the devices mentioned in overlay.dtbo to Xen device tree.
+
+To assign the newly added device from the dtbo to Dom0:
+
+    (dom0) xl dt-overlay attach overlay.dtbo 0
+
+Next, if the user wants to add the same device tree overlay to dom0
+Linux, execute the following:
+
+    (dom0) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay
+    (dom0) cat overlay.dtbo > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo
+
+Finally if needed, the relevant Linux kernel drive can be loaded using:
+
+    (dom0) modprobe module_name.ko
+
+## DomU device add/remove
+
+All the nodes in dtbo will be assigned to a domain; the user will need
+to prepare the dtb for the domU. For example, the `interrupt-parent`
+property of the DomU overlay should be changed to the Xen hardcoded
+value `0xfde8`, and the xen,reg property should be added to specify the
+address mappings. If xen,reg is not present, it is assumed 1:1 mapping.
+Below assumes the properly written DomU dtbo is `overlay_domu.dtbo`.
+
+For new domains to be created, the user will need to create the DomU
+with below properties properly configured in the xl config file:
+- `iomem`
+- `passthrough` (if IOMMU is needed)
+
+User will also need to modprobe the relevant drivers. For already
+running domains, the user can use the xl dt-overlay attach command,
+example:
+
+    (dom0) xl dt-overlay add overlay.dtbo            # If not executed before
+    (dom0) xl dt-overlay attach overlay.dtbo $domid
+    (dom0) xl console $domid                         # To access $domid console
+
+Next, if the user needs to modify/prepare the overlay.dtbo suitable for
+the domU:
+
+    (domU) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay
+    (domU) cat overlay_domu.dtbo > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo
+
+Finally, if needed, the relevant Linux kernel drive can be probed:
+
+    (domU) modprobe module_name.ko
+
+[1] https://www.kernel.org/doc/Documentation/devicetree/overlay-notes.txt
-- 
2.25.1

Hi Stefano,

On 24/05/2024 03:18, Stefano Stabellini wrote:
> From: Vikram Garhwal <fnu.vikram@xilinx.com>
> 
> Signed-off-by: Vikram Garhwal <fnu.vikram@xilinx.com>
> Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
> Signed-off-by: Henry Wang <xin.wang2@amd.com>
> ---
>   docs/misc/arm/overlay.txt | 82 +++++++++++++++++++++++++++++++++++++++
>   1 file changed, 82 insertions(+)
>   create mode 100644 docs/misc/arm/overlay.txt
> 
> diff --git a/docs/misc/arm/overlay.txt b/docs/misc/arm/overlay.txt
> new file mode 100644
> index 0000000000..0a2dee951a
> --- /dev/null
> +++ b/docs/misc/arm/overlay.txt
> @@ -0,0 +1,82 @@
> +# Device Tree Overlays support in Xen
> +
> +Xen experimentally supports dynamic device assignment to running
> +domains, i.e. adding/removing nodes (using .dtbo) to/from Xen device
> +tree, and attaching them to a running domain with given $domid.
> +
> +Dynamic node assignment works in two steps:
> +
> +## Add/Remove device tree overlay to/from Xen device tree
> +
> +1. Xen tools check the dtbo given and parse all other user provided arguments
> +2. Xen tools pass the dtbo to Xen hypervisor via hypercall.
> +3. Xen hypervisor applies/removes the dtbo to/from Xen device tree.
> +
> +## Attach device from the DT overlay to domain
> +
> +1. Xen tools check the dtbo given and parse all other user provided arguments
> +2. Xen tools pass the dtbo to Xen hypervisor via hypercall.
> +3. Xen hypervisor attach the device to the user-provided $domid by
> +   mapping node resources in the DT overlay.
> +
> +# Examples
> +
> +Here are a few examples on how to use it.
> +
> +## Dom0 device add
> +
> +For assigning a device tree overlay to Dom0, user should firstly properly
> +prepare the DT overlay. More information about device tree overlays can be
> +found in [1]. Then, in Dom0, enter the following:
> +
> +    (dom0) xl dt-overlay add overlay.dtbo
> +
> +This will allocate the devices mentioned in overlay.dtbo to Xen device tree.
> +
> +To assign the newly added device from the dtbo to Dom0:
> +
> +    (dom0) xl dt-overlay attach overlay.dtbo 0
> +
> +Next, if the user wants to add the same device tree overlay to dom0
> +Linux, execute the following:
> +
> +    (dom0) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay
> +    (dom0) cat overlay.dtbo > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo
> +
> +Finally if needed, the relevant Linux kernel drive can be loaded using:
> +
> +    (dom0) modprobe module_name.ko
> +
> +## DomU device add/remove
> +
> +All the nodes in dtbo will be assigned to a domain; the user will need
> +to prepare the dtb for the domU.

s/dtb/dtbo/? But I am little bit confused with the wording. I think you 
may want to add *different dtbo* so it clarifies from the start (this 
only becomes obvious at the end of the section) that the user is not 
meant to use the same for all the commands.

  For example, the `interrupt-parent`
> +property of the DomU overlay should be changed to the Xen hardcoded
> +value `0xfde8`, and the xen,reg property should be added to specify the
> +address mappings. 
> If xen,reg is not present, it is assumed 1:1 mapping.

Repeating an earlier comment here. I think xen,reg should be mandatory 
for non-direct mapped domain.

Also, can you clarify what is the expect property layout for xen,reg?

> +Below assumes the properly written DomU dtbo is `overlay_domu.dtbo`.
> +
> +For new domains to be created, the user will need to create the DomU
> +with below properties properly configured in the xl config file:
> +- `iomem`

I looked at your reply in v4 and I am afraid I still don't understand 
why we are mentioning 'iomem'. If we want to use the commands below, 
then the domain needs to be created in advance. So you can't yet know 
'iomem'.

You could avoid "xl dt-overlay attach" but then you need the user to 
specify both "irqs" and "iomem". From a user point of view, it would be 
easier to add a new propery in the configuration file listing the 
overlays. Something like:

dt_overlays = [ "overlay.dtbo", ... ]

Anyway, that somewhat separate. For now, I think we want to drop 'iomem' 
from the list and reword this paragraph to say that the 'passthrough' 
property needs to be set if you plan to use DT overlay and devices 
requiring the IOMMU.

> +- `passthrough` (if IOMMU is needed)

This property is required at the start because we don't support enabling 
the IOMMU lazily.

> +
> +User will also need to modprobe the relevant drivers. For already
> +running domains, the user can use the xl dt-overlay attach command,
> +example:
> +
> +    (dom0) xl dt-overlay add overlay.dtbo            # If not executed before
> +    (dom0) xl dt-overlay attach overlay.dtbo $domid
> +    (dom0) xl console $domid                         # To access $domid console
> +
> +Next, if the user needs to modify/prepare the overlay.dtbo suitable for
> +the domU:
> +
> +    (domU) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay
> +    (domU) cat overlay_domu.dtbo > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo
> +
> +Finally, if needed, the relevant Linux kernel drive can be probed:
> +
> +    (domU) modprobe module_name.ko
> +
> +[1] https://www.kernel.org/doc/Documentation/devicetree/overlay-notes.txt

Cheers,

-- 
Julien Grall

On Fri, 24 May 2024, Julien Grall wrote:
> Hi Stefano,
> 
> On 24/05/2024 03:18, Stefano Stabellini wrote:
> > From: Vikram Garhwal <fnu.vikram@xilinx.com>
> > 
> > Signed-off-by: Vikram Garhwal <fnu.vikram@xilinx.com>
> > Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > Signed-off-by: Henry Wang <xin.wang2@amd.com>
> > ---
> >   docs/misc/arm/overlay.txt | 82 +++++++++++++++++++++++++++++++++++++++
> >   1 file changed, 82 insertions(+)
> >   create mode 100644 docs/misc/arm/overlay.txt
> > 
> > diff --git a/docs/misc/arm/overlay.txt b/docs/misc/arm/overlay.txt
> > new file mode 100644
> > index 0000000000..0a2dee951a
> > --- /dev/null
> > +++ b/docs/misc/arm/overlay.txt
> > @@ -0,0 +1,82 @@
> > +# Device Tree Overlays support in Xen
> > +
> > +Xen experimentally supports dynamic device assignment to running
> > +domains, i.e. adding/removing nodes (using .dtbo) to/from Xen device
> > +tree, and attaching them to a running domain with given $domid.
> > +
> > +Dynamic node assignment works in two steps:
> > +
> > +## Add/Remove device tree overlay to/from Xen device tree
> > +
> > +1. Xen tools check the dtbo given and parse all other user provided
> > arguments
> > +2. Xen tools pass the dtbo to Xen hypervisor via hypercall.
> > +3. Xen hypervisor applies/removes the dtbo to/from Xen device tree.
> > +
> > +## Attach device from the DT overlay to domain
> > +
> > +1. Xen tools check the dtbo given and parse all other user provided
> > arguments
> > +2. Xen tools pass the dtbo to Xen hypervisor via hypercall.
> > +3. Xen hypervisor attach the device to the user-provided $domid by
> > +   mapping node resources in the DT overlay.
> > +
> > +# Examples
> > +
> > +Here are a few examples on how to use it.
> > +
> > +## Dom0 device add
> > +
> > +For assigning a device tree overlay to Dom0, user should firstly properly
> > +prepare the DT overlay. More information about device tree overlays can be
> > +found in [1]. Then, in Dom0, enter the following:
> > +
> > +    (dom0) xl dt-overlay add overlay.dtbo
> > +
> > +This will allocate the devices mentioned in overlay.dtbo to Xen device
> > tree.
> > +
> > +To assign the newly added device from the dtbo to Dom0:
> > +
> > +    (dom0) xl dt-overlay attach overlay.dtbo 0
> > +
> > +Next, if the user wants to add the same device tree overlay to dom0
> > +Linux, execute the following:
> > +
> > +    (dom0) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay
> > +    (dom0) cat overlay.dtbo >
> > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo
> > +
> > +Finally if needed, the relevant Linux kernel drive can be loaded using:
> > +
> > +    (dom0) modprobe module_name.ko
> > +
> > +## DomU device add/remove
> > +
> > +All the nodes in dtbo will be assigned to a domain; the user will need
> > +to prepare the dtb for the domU.
> 
> s/dtb/dtbo/? 

Yes, done


> But I am little bit confused with the wording. I think you may
> want to add *different dtbo* so it clarifies from the start (this only becomes
> obvious at the end of the section) that the user is not meant to use the same
> for all the commands.

Yes it was confusing, I tried to clarify it. This is a doc so we could
improve it further after the freeze.


>  For example, the `interrupt-parent`
> > +property of the DomU overlay should be changed to the Xen hardcoded
> > +value `0xfde8`, and the xen,reg property should be added to specify the
> > +address mappings. If xen,reg is not present, it is assumed 1:1 mapping.
> 
> Repeating an earlier comment here. I think xen,reg should be mandatory for
> non-direct mapped domain.

That's OK


> Also, can you clarify what is the expect property layout for xen,reg?

I'll add a pointer


> > +Below assumes the properly written DomU dtbo is `overlay_domu.dtbo`.
> > +
> > +For new domains to be created, the user will need to create the DomU
> > +with below properties properly configured in the xl config file:
> > +- `iomem`
> 
> I looked at your reply in v4 and I am afraid I still don't understand why we
> are mentioning 'iomem'. If we want to use the commands below, then the domain
> needs to be created in advance. So you can't yet know 'iomem'.
> 
> You could avoid "xl dt-overlay attach" but then you need the user to specify
> both "irqs" and "iomem". From a user point of view, it would be easier to add
> a new propery in the configuration file listing the overlays. Something like:
> 
> dt_overlays = [ "overlay.dtbo", ... ]
> 
> Anyway, that somewhat separate. For now, I think we want to drop 'iomem' from
> the list and reword this paragraph to say that the 'passthrough' property
> needs to be set if you plan to use DT overlay and devices requiring the IOMMU.

OK, I did that


> 
> > +- `passthrough` (if IOMMU is needed)
> 
> This property is required at the start because we don't support enabling the
> IOMMU lazily.

Yeah. I think it is much clearer now.


> > +
> > +User will also need to modprobe the relevant drivers. For already
> > +running domains, the user can use the xl dt-overlay attach command,
> > +example:
> > +
> > +    (dom0) xl dt-overlay add overlay.dtbo            # If not executed
> > before
> > +    (dom0) xl dt-overlay attach overlay.dtbo $domid
> > +    (dom0) xl console $domid                         # To access $domid
> > console
> > +
> > +Next, if the user needs to modify/prepare the overlay.dtbo suitable for
> > +the domU:
> > +
> > +    (domU) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay
> > +    (domU) cat overlay_domu.dtbo >
> > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo
> > +
> > +Finally, if needed, the relevant Linux kernel drive can be probed:
> > +
> > +    (domU) modprobe module_name.ko
> > +
> > +[1] https://www.kernel.org/doc/Documentation/devicetree/overlay-notes.txt