[v3] xen/dom0less: Calculate guest DTB size based on MAX_VIRT_CPUS

[PATCH V3] xen/dom0less: Calculate guest DTB size based on MAX_VIRT_CPUS

Posted by Oleksandr Tyshchenko 1 month, 3 weeks ago

Creating a dom0less guest with a high vCPU count (e.g., >32) fails
because the fixed 4KiB device tree buffer (DOMU_DTB_SIZE) overflows
during creation.

The FDT nodes for each vCPU are the primary consumer of space,
and the previous fixed-size buffer was insufficient.

This patch replaces the fixed size with a formula that calculates
the required buffer size based on a fixed baseline plus a scalable
portion for each potential vCPU up to the MAX_VIRT_CPUS limit.

Please note, the change to DOMU_DTB_SIZE formula would result in
a smaller buffer size of 3072 bytes compared to the original 4096 bytes
on Arm32 platforms where MAX_VIRT_CPUS is 8.

***

The following tests were done to confirm that the proposed formula
fits:

1. Arm64 testing with varying vCPU counts (MAX_VIRT_CPUS=128),
   final compacted FDT size:

   - 1 vCPU: 1586 bytes (with 18432 byte buffer)
   - 2 vCPUs: 1698 bytes
   - 32 vCPUs: 5058 bytes
   - 128 vCPUs: 15810 bytes

2. Arm64 testing with simulated Arm32 conditions (MAX_VIRT_CPUS=8),
   final compacted FDT size:

   - 1 vCPU: 1586 bytes (with 3072 byte buffer)
   - 8 vCPUs: 2370 bytes

3. Arm32 testing (MAX_VIRT_CPUS=8),
   final compacted FDT size:

   - 8 vCPUs: 1127 bytes (with 3072 byte buffer)

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Reviewed-by: Grygorii Strashko <grygorii_strashko@epam.com>
Reviewed-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
Tested-by: Harry Ramsey <harry.ramsey@arm.com>
---
V1: https://patchew.org/Xen/20251202193246.3357821-1-oleksandr._5Ftyshchenko@epam.com/
V2: https://patchew.org/Xen/20251203185817.3722903-1-oleksandr._5Ftyshchenko@epam.com/

  V2:
   - update commit subj/desc
   - use a formula that accounts MAX_VIRT_CPUS
   - add BUILD_BUG_ON

  V3:
   - add R-b and T-b
   - add more info to commmit desc
---
---
 xen/common/device-tree/dom0less-build.c | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/xen/common/device-tree/dom0less-build.c b/xen/common/device-tree/dom0less-build.c
index 2600350a3c..0c271d4ca3 100644
--- a/xen/common/device-tree/dom0less-build.c
+++ b/xen/common/device-tree/dom0less-build.c
@@ -439,15 +439,25 @@ static int __init domain_handle_dtb_boot_module(struct domain *d,
 
 /*
  * The max size for DT is 2MB. However, the generated DT is small (not including
- * domU passthrough DT nodes whose size we account separately), 4KB are enough
- * for now, but we might have to increase it in the future.
+ * domU passthrough DT nodes whose size we account separately). The size is
+ * calculated from a fixed baseline plus a scalable portion for each potential
+ * vCPU node up to the system limit (MAX_VIRT_CPUS), as the vCPU nodes are
+ * the primary consumer of space.
+ *
+ * The baseline of 2KiB is a safe buffer for all non-vCPU FDT content.
+ * Empirical testing with the maximum number of other device tree nodes shows
+ * a final compacted base size of ~1.5KiB. The 128 bytes per vCPU is derived
+ * from a worst-case analysis of the FDT construction-time size for a single
+ * vCPU node.
  */
-#define DOMU_DTB_SIZE 4096
+#define DOMU_DTB_SIZE (2048 + (MAX_VIRT_CPUS * 128))
 static int __init prepare_dtb_domU(struct domain *d, struct kernel_info *kinfo)
 {
     int addrcells, sizecells;
     int ret, fdt_size = DOMU_DTB_SIZE;
 
+    BUILD_BUG_ON(DOMU_DTB_SIZE > SZ_2M);
+
     kinfo->phandle_intc = GUEST_PHANDLE_GIC;
 
 #ifdef CONFIG_GRANT_TABLE
-- 
2.34.1

Re: [PATCH V3] xen/dom0less: Calculate guest DTB size based on MAX_VIRT_CPUS

Posted by Stefano Stabellini 1 month, 3 weeks ago

On Wed, 17 Dec 2025, Oleksandr Tyshchenko wrote:
> Creating a dom0less guest with a high vCPU count (e.g., >32) fails
> because the fixed 4KiB device tree buffer (DOMU_DTB_SIZE) overflows
> during creation.
> 
> The FDT nodes for each vCPU are the primary consumer of space,
> and the previous fixed-size buffer was insufficient.
> 
> This patch replaces the fixed size with a formula that calculates
> the required buffer size based on a fixed baseline plus a scalable
> portion for each potential vCPU up to the MAX_VIRT_CPUS limit.
> 
> Please note, the change to DOMU_DTB_SIZE formula would result in
> a smaller buffer size of 3072 bytes compared to the original 4096 bytes
> on Arm32 platforms where MAX_VIRT_CPUS is 8.

I am OK with this patch I would only ask to retain the minimum size of
4KB due to the possible presence of passthrough device nodes.


> ***
> 
> The following tests were done to confirm that the proposed formula
> fits:
> 
> 1. Arm64 testing with varying vCPU counts (MAX_VIRT_CPUS=128),
>    final compacted FDT size:
> 
>    - 1 vCPU: 1586 bytes (with 18432 byte buffer)
>    - 2 vCPUs: 1698 bytes
>    - 32 vCPUs: 5058 bytes
>    - 128 vCPUs: 15810 bytes
> 
> 2. Arm64 testing with simulated Arm32 conditions (MAX_VIRT_CPUS=8),
>    final compacted FDT size:
> 
>    - 1 vCPU: 1586 bytes (with 3072 byte buffer)
>    - 8 vCPUs: 2370 bytes
> 
> 3. Arm32 testing (MAX_VIRT_CPUS=8),
>    final compacted FDT size:
> 
>    - 8 vCPUs: 1127 bytes (with 3072 byte buffer)
> 
> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> Reviewed-by: Grygorii Strashko <grygorii_strashko@epam.com>
> Reviewed-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> Tested-by: Harry Ramsey <harry.ramsey@arm.com>
>
> ---
> V1: https://patchew.org/Xen/20251202193246.3357821-1-oleksandr._5Ftyshchenko@epam.com/
> V2: https://patchew.org/Xen/20251203185817.3722903-1-oleksandr._5Ftyshchenko@epam.com/
> 
>   V2:
>    - update commit subj/desc
>    - use a formula that accounts MAX_VIRT_CPUS
>    - add BUILD_BUG_ON
> 
>   V3:
>    - add R-b and T-b
>    - add more info to commmit desc
> ---
> ---
>  xen/common/device-tree/dom0less-build.c | 16 +++++++++++++---
>  1 file changed, 13 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/common/device-tree/dom0less-build.c b/xen/common/device-tree/dom0less-build.c
> index 2600350a3c..0c271d4ca3 100644
> --- a/xen/common/device-tree/dom0less-build.c
> +++ b/xen/common/device-tree/dom0less-build.c
> @@ -439,15 +439,25 @@ static int __init domain_handle_dtb_boot_module(struct domain *d,
>  
>  /*
>   * The max size for DT is 2MB. However, the generated DT is small (not including
> - * domU passthrough DT nodes whose size we account separately), 4KB are enough
> - * for now, but we might have to increase it in the future.
> + * domU passthrough DT nodes whose size we account separately). The size is
> + * calculated from a fixed baseline plus a scalable portion for each potential
> + * vCPU node up to the system limit (MAX_VIRT_CPUS), as the vCPU nodes are
> + * the primary consumer of space.
> + *
> + * The baseline of 2KiB is a safe buffer for all non-vCPU FDT content.
> + * Empirical testing with the maximum number of other device tree nodes shows
> + * a final compacted base size of ~1.5KiB. The 128 bytes per vCPU is derived
> + * from a worst-case analysis of the FDT construction-time size for a single
> + * vCPU node.
>   */
> -#define DOMU_DTB_SIZE 4096
> +#define DOMU_DTB_SIZE (2048 + (MAX_VIRT_CPUS * 128))

Something along the lines of:

#define DOMU_DTB_SIZE MIN(4096, (2048 + (MAX_VIRT_CPUS * 128)))


>  static int __init prepare_dtb_domU(struct domain *d, struct kernel_info *kinfo)
>  {
>      int addrcells, sizecells;
>      int ret, fdt_size = DOMU_DTB_SIZE;
>  
> +    BUILD_BUG_ON(DOMU_DTB_SIZE > SZ_2M);
> +
>      kinfo->phandle_intc = GUEST_PHANDLE_GIC;
>  
>  #ifdef CONFIG_GRANT_TABLE
> -- 
> 2.34.1
>

Re: [PATCH V3] xen/dom0less: Calculate guest DTB size based on MAX_VIRT_CPUS

Posted by Oleksandr Tyshchenko 1 month, 3 weeks ago


On 18.12.25 02:05, Stefano Stabellini wrote:

Hello Stefano

> On Wed, 17 Dec 2025, Oleksandr Tyshchenko wrote:
>> Creating a dom0less guest with a high vCPU count (e.g., >32) fails
>> because the fixed 4KiB device tree buffer (DOMU_DTB_SIZE) overflows
>> during creation.
>>
>> The FDT nodes for each vCPU are the primary consumer of space,
>> and the previous fixed-size buffer was insufficient.
>>
>> This patch replaces the fixed size with a formula that calculates
>> the required buffer size based on a fixed baseline plus a scalable
>> portion for each potential vCPU up to the MAX_VIRT_CPUS limit.
>>
>> Please note, the change to DOMU_DTB_SIZE formula would result in
>> a smaller buffer size of 3072 bytes compared to the original 4096 bytes
>> on Arm32 platforms where MAX_VIRT_CPUS is 8.
> 
> I am OK with this patch I would only ask to retain the minimum size of
> 4KB due to the possible presence of passthrough device nodes.

I think there might be cases when even 4KB would not be enough to cover 
the whole dtb with passthrough device nodes. But the existing code 
should already handle that, so if a partial device tree is provided, 
then it will be accounted separately:

      /* Account for domU passthrough DT size */
      if ( kinfo->dtb )
          fdt_size += kinfo->dtb->size;


[snip]

Re: [PATCH V3] xen/dom0less: Calculate guest DTB size based on MAX_VIRT_CPUS

Posted by Stefano Stabellini 1 month, 3 weeks ago

On Thu, 17 Dec 2025, Oleksandr Tyshchenko wrote:
> On 18.12.25 02:05, Stefano Stabellini wrote:
> Hello Stefano
> 
> > On Wed, 17 Dec 2025, Oleksandr Tyshchenko wrote:
> >> Creating a dom0less guest with a high vCPU count (e.g., >32) fails
> >> because the fixed 4KiB device tree buffer (DOMU_DTB_SIZE) overflows
> >> during creation.
> >>
> >> The FDT nodes for each vCPU are the primary consumer of space,
> >> and the previous fixed-size buffer was insufficient.
> >>
> >> This patch replaces the fixed size with a formula that calculates
> >> the required buffer size based on a fixed baseline plus a scalable
> >> portion for each potential vCPU up to the MAX_VIRT_CPUS limit.
> >>
> >> Please note, the change to DOMU_DTB_SIZE formula would result in
> >> a smaller buffer size of 3072 bytes compared to the original 4096 bytes
> >> on Arm32 platforms where MAX_VIRT_CPUS is 8.
> > 
> > I am OK with this patch I would only ask to retain the minimum size of
> > 4KB due to the possible presence of passthrough device nodes.
> 
> I think there might be cases when even 4KB would not be enough to cover 
> the whole dtb with passthrough device nodes. But the existing code 
> should already handle that, so if a partial device tree is provided, 
> then it will be accounted separately:
> 
>       /* Account for domU passthrough DT size */
>       if ( kinfo->dtb )
>           fdt_size += kinfo->dtb->size;

I forgot about this. In that case:

Acked-by: Stefano Stabellini <sstabellini@kernel.org>

Re: [PATCH V3] xen/dom0less: Calculate guest DTB size based on MAX_VIRT_CPUS

Posted by Orzel, Michal 1 month, 3 weeks ago


On 18/12/2025 08:44, Oleksandr Tyshchenko wrote:
> 
> 
> On 18.12.25 02:05, Stefano Stabellini wrote:
> 
> Hello Stefano
> 
>> On Wed, 17 Dec 2025, Oleksandr Tyshchenko wrote:
>>> Creating a dom0less guest with a high vCPU count (e.g., >32) fails
>>> because the fixed 4KiB device tree buffer (DOMU_DTB_SIZE) overflows
>>> during creation.
>>>
>>> The FDT nodes for each vCPU are the primary consumer of space,
>>> and the previous fixed-size buffer was insufficient.
>>>
>>> This patch replaces the fixed size with a formula that calculates
>>> the required buffer size based on a fixed baseline plus a scalable
>>> portion for each potential vCPU up to the MAX_VIRT_CPUS limit.
>>>
>>> Please note, the change to DOMU_DTB_SIZE formula would result in
>>> a smaller buffer size of 3072 bytes compared to the original 4096 bytes
>>> on Arm32 platforms where MAX_VIRT_CPUS is 8.
>>
>> I am OK with this patch I would only ask to retain the minimum size of
>> 4KB due to the possible presence of passthrough device nodes.
> 
> I think there might be cases when even 4KB would not be enough to cover 
> the whole dtb with passthrough device nodes. But the existing code 
> should already handle that, so if a partial device tree is provided, 
> then it will be accounted separately:
> 
>       /* Account for domU passthrough DT size */
>       if ( kinfo->dtb )
>           fdt_size += kinfo->dtb->size;
I agree with Oleksandr:
Acked-by: Michal Orzel <michal.orzel@amd.com>

~Michal