xen/arch/arm/domain_build.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
Xen fails to construct the hardware domain's device tree with
FDT_ERR_NOSPACE (-3) when the host memory map is highly fragmented
(e.g., numerous reserved memory regions).
This occurs because DOM0_FDT_EXTRA_SIZE underestimates the space
required for the generated extra /memory node. make_memory_node()
aggregates all reserved regions into a single reg property. With
NR_MEM_BANKS (256) and 64-bit address/size cells, this property
can grow up to 4KB (256 * 16), easily overflowing the allocated
buffer.
Fix this by increasing DOM0_FDT_EXTRA_SIZE to account for
the worst-case size: NR_MEM_BANKS * 16 bytes.
Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
---
Just to be clear, I have not seen a real-world issue with this.
The issue was observed during testing of limit conditions.
With this patch applied, Xen successfully boots the hardware domain,
exposing 256 reserved memory regions to it (using a synthetically
generated configuration).
---
---
xen/arch/arm/domain_build.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index e8795745dd..7f9f0f5510 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -100,9 +100,11 @@ int __init parse_arch_dom0_param(const char *s, const char *e)
/*
* Amount of extra space required to dom0's device tree. No new nodes
* are added (yet) but one terminating reserve map entry (16 bytes) is
- * added.
+ * added. Plus space for an extra memory node to cover all possible reserved
+ * memory regions (2 addr cells + 2 size cells).
*/
-#define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry))
+#define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry) + \
+ (NR_MEM_BANKS * 16))
unsigned int __init dom0_max_vcpus(void)
{
--
2.34.1
On 26/03/2026 14:15, Oleksandr Tyshchenko wrote:
> Xen fails to construct the hardware domain's device tree with
> FDT_ERR_NOSPACE (-3) when the host memory map is highly fragmented
> (e.g., numerous reserved memory regions).
>
> This occurs because DOM0_FDT_EXTRA_SIZE underestimates the space
> required for the generated extra /memory node. make_memory_node()
Where does this extra /memory node come from? If this is for normal reserved
memory regions, they should be present in the host dtb and therefore accounted
by fdt_totalsize (the host dtb should have reserved regions described in /memory
and /reserved-memory. Are you trying to account for static shm regions?
> aggregates all reserved regions into a single reg property. With
> NR_MEM_BANKS (256) and 64-bit address/size cells, this property
> can grow up to 4KB (256 * 16), easily overflowing the allocated
> buffer.
>
> Fix this by increasing DOM0_FDT_EXTRA_SIZE to account for
> the worst-case size: NR_MEM_BANKS * 16 bytes.
>
> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> ---
> Just to be clear, I have not seen a real-world issue with this.
> The issue was observed during testing of limit conditions.
> With this patch applied, Xen successfully boots the hardware domain,
> exposing 256 reserved memory regions to it (using a synthetically
> generated configuration).
> ---
> ---
> xen/arch/arm/domain_build.c | 6 ++++--
> 1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index e8795745dd..7f9f0f5510 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -100,9 +100,11 @@ int __init parse_arch_dom0_param(const char *s, const char *e)
> /*
> * Amount of extra space required to dom0's device tree. No new nodes
This comment would want to be updated because since its introduction things have
changed. Even the 128 came up as a result of adding /hypervisor node.
> * are added (yet) but one terminating reserve map entry (16 bytes) is
> - * added.
> + * added. Plus space for an extra memory node to cover all possible reserved
> + * memory regions (2 addr cells + 2 size cells).
> */
> -#define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry))
> +#define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry) + \
> + (NR_MEM_BANKS * 16))
>
> unsigned int __init dom0_max_vcpus(void)
> {
~Michal
On 3/26/26 18:50, Orzel, Michal wrote:
Hello Michal
>
>
> On 26/03/2026 14:15, Oleksandr Tyshchenko wrote:
>> Xen fails to construct the hardware domain's device tree with
>> FDT_ERR_NOSPACE (-3) when the host memory map is highly fragmented
>> (e.g., numerous reserved memory regions).
>>
>> This occurs because DOM0_FDT_EXTRA_SIZE underestimates the space
>> required for the generated extra /memory node. make_memory_node()
> Where does this extra /memory node come from? If this is for normal reserved
> memory regions, they should be present in the host dtb and therefore accounted
> by fdt_totalsize (the host dtb should have reserved regions described in /memory
> and /reserved-memory. Are you trying to account for static shm regions?
I might have misunderstood something, but here is my analysis:
The extra /memory node is generated by Xen itself in handle_node() ->
make_memory_node() (please refer to the if ( reserved_mem->nr_banks > 0
) check).
Even though the normal reserved memory regions are present in the host
DTB (and thus accounted for in fdt_totalsize), Xen generates a new
/memory node specifically for the hardware domain to describe these
regions as reserved but present in the memory map. And since this node
is generated at runtime (it is not a direct copy from the host DTB),
its size must be covered by DOM0_FDT_EXTRA_SIZE.
For the instance, 10 reserved regions:
(XEN) RAM: 0000000040000000 - 000000007fffffff
(XEN)
(XEN) MODULE[0]: 0000000043200000 - 000000004330afff Xen
(XEN) MODULE[1]: 0000000043400000 - 0000000043402fff Device Tree
(XEN) MODULE[2]: 0000000042e00000 - 000000004316907f Ramdisk
(XEN) MODULE[3]: 0000000040400000 - 0000000042d2ffff Kernel
(XEN) RESVD[0]: 0000000040009000 - 0000000040009fff
(XEN) RESVD[1]: 0000000040008000 - 0000000040008fff
(XEN) RESVD[2]: 0000000040007000 - 0000000040007fff
(XEN) RESVD[3]: 0000000040006000 - 0000000040006fff
(XEN) RESVD[4]: 0000000040005000 - 0000000040005fff
(XEN) RESVD[5]: 0000000040004000 - 0000000040004fff
(XEN) RESVD[6]: 0000000040003000 - 0000000040003fff
(XEN) RESVD[7]: 0000000040002000 - 0000000040002fff
(XEN) RESVD[8]: 0000000040001000 - 0000000040001fff
(XEN) RESVD[9]: 0000000040000000 - 0000000040000fff
...
From make_memory_node():
(XEN) Create memory node
(XEN) Bank 0: 0x50000000->0x70000000
(XEN) (reg size 4, nr cells 4)
(XEN) Create memory node
(XEN) Bank 0: 0x40009000->0x4000a000
(XEN) Bank 1: 0x40008000->0x40009000
(XEN) Bank 2: 0x40007000->0x40008000
(XEN) Bank 3: 0x40006000->0x40007000
(XEN) Bank 4: 0x40005000->0x40006000
(XEN) Bank 5: 0x40004000->0x40005000
(XEN) Bank 6: 0x40003000->0x40004000
(XEN) Bank 7: 0x40002000->0x40003000
(XEN) Bank 8: 0x40001000->0x40002000
(XEN) Bank 9: 0x40000000->0x40001000
(XEN) (reg size 4, nr cells 40)
>
>> aggregates all reserved regions into a single reg property. With
>> NR_MEM_BANKS (256) and 64-bit address/size cells, this property
>> can grow up to 4KB (256 * 16), easily overflowing the allocated
>> buffer.
>>
>> Fix this by increasing DOM0_FDT_EXTRA_SIZE to account for
>> the worst-case size: NR_MEM_BANKS * 16 bytes.
>>
>> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
>> ---
>> Just to be clear, I have not seen a real-world issue with this.
>> The issue was observed during testing of limit conditions.
>> With this patch applied, Xen successfully boots the hardware domain,
>> exposing 256 reserved memory regions to it (using a synthetically
>> generated configuration).
>> ---
>> ---
>> xen/arch/arm/domain_build.c | 6 ++++--
>> 1 file changed, 4 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>> index e8795745dd..7f9f0f5510 100644
>> --- a/xen/arch/arm/domain_build.c
>> +++ b/xen/arch/arm/domain_build.c
>> @@ -100,9 +100,11 @@ int __init parse_arch_dom0_param(const char *s, const char *e)
>> /*
>> * Amount of extra space required to dom0's device tree. No new nodes
> This comment would want to be updated because since its introduction things have
> changed. Even the 128 came up as a result of adding /hypervisor node.
You are right. I suggest the following wording:
Amount of extra space required to dom0's device tree.
This covers nodes generated by Xen, which are not directly copied
from the host DTB. It is calculated as:
- Space for /hypervisor node (128 bytes).
- One terminating reserve map entry (16 bytes).
- Space for a generated memory node covering all possible reserved
memory regions (NR_MEM_BANKS * 16 bytes).
>
>> * are added (yet) but one terminating reserve map entry (16 bytes) is
>> - * added.
>> + * added. Plus space for an extra memory node to cover all possible reserved
>> + * memory regions (2 addr cells + 2 size cells).
>> */
>> -#define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry))
>> +#define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry) + \
>> + (NR_MEM_BANKS * 16))
>>
>> unsigned int __init dom0_max_vcpus(void)
>> {
>
> ~Michal
>
On 26/03/2026 20:03, Oleksandr Tyshchenko wrote:
>
>
> On 3/26/26 18:50, Orzel, Michal wrote:
>
> Hello Michal
>
>>
>>
>> On 26/03/2026 14:15, Oleksandr Tyshchenko wrote:
>>> Xen fails to construct the hardware domain's device tree with
>>> FDT_ERR_NOSPACE (-3) when the host memory map is highly fragmented
>>> (e.g., numerous reserved memory regions).
>>>
>>> This occurs because DOM0_FDT_EXTRA_SIZE underestimates the space
>>> required for the generated extra /memory node. make_memory_node()
>> Where does this extra /memory node come from? If this is for normal reserved
>> memory regions, they should be present in the host dtb and therefore accounted
>> by fdt_totalsize (the host dtb should have reserved regions described in /memory
>> and /reserved-memory. Are you trying to account for static shm regions?
>
>
> I might have misunderstood something, but here is my analysis:
>
> The extra /memory node is generated by Xen itself in handle_node() ->
> make_memory_node() (please refer to the if ( reserved_mem->nr_banks > 0
> ) check).
>
> Even though the normal reserved memory regions are present in the host
> DTB (and thus accounted for in fdt_totalsize), Xen generates a new
> /memory node specifically for the hardware domain to describe these
> regions as reserved but present in the memory map. And since this node
> is generated at runtime (it is not a direct copy from the host DTB),
> its size must be covered by DOM0_FDT_EXTRA_SIZE.
Yes, but the original DTB should also have these reserved regions described in
/memory nodes, thus taking up some space that is already accounted in
fdt_totalsize. Are you trying to say that in host DTB, these reserved ranges fit
nicely into e.g. a single /memory node range (i.e. a single reg pair covering
most of the RAM)? I can see that it might be possible but the commit msg needs
to be clear about it. As of now, it reads as if the problem occured always when
there are multiple reserved memory regions. That's not true if a host DTB
generates one /memory per one /reserved.
Another issue is with the static shm nodes. User specifies the regions in the
domain configuration and Xen creates *additional* nodes under /reserved and
/memory that afaict we don't account for.
>
> For the instance, 10 reserved regions:
>
> (XEN) RAM: 0000000040000000 - 000000007fffffff
> (XEN)
> (XEN) MODULE[0]: 0000000043200000 - 000000004330afff Xen
> (XEN) MODULE[1]: 0000000043400000 - 0000000043402fff Device Tree
> (XEN) MODULE[2]: 0000000042e00000 - 000000004316907f Ramdisk
> (XEN) MODULE[3]: 0000000040400000 - 0000000042d2ffff Kernel
> (XEN) RESVD[0]: 0000000040009000 - 0000000040009fff
> (XEN) RESVD[1]: 0000000040008000 - 0000000040008fff
> (XEN) RESVD[2]: 0000000040007000 - 0000000040007fff
> (XEN) RESVD[3]: 0000000040006000 - 0000000040006fff
> (XEN) RESVD[4]: 0000000040005000 - 0000000040005fff
> (XEN) RESVD[5]: 0000000040004000 - 0000000040004fff
> (XEN) RESVD[6]: 0000000040003000 - 0000000040003fff
> (XEN) RESVD[7]: 0000000040002000 - 0000000040002fff
> (XEN) RESVD[8]: 0000000040001000 - 0000000040001fff
> (XEN) RESVD[9]: 0000000040000000 - 0000000040000fff
> ...
>
> From make_memory_node():
>
> (XEN) Create memory node
> (XEN) Bank 0: 0x50000000->0x70000000
> (XEN) (reg size 4, nr cells 4)
>
>
>
> (XEN) Create memory node
> (XEN) Bank 0: 0x40009000->0x4000a000
> (XEN) Bank 1: 0x40008000->0x40009000
> (XEN) Bank 2: 0x40007000->0x40008000
> (XEN) Bank 3: 0x40006000->0x40007000
> (XEN) Bank 4: 0x40005000->0x40006000
> (XEN) Bank 5: 0x40004000->0x40005000
> (XEN) Bank 6: 0x40003000->0x40004000
> (XEN) Bank 7: 0x40002000->0x40003000
> (XEN) Bank 8: 0x40001000->0x40002000
> (XEN) Bank 9: 0x40000000->0x40001000
> (XEN) (reg size 4, nr cells 40)
>
>>
>>> aggregates all reserved regions into a single reg property. With
>>> NR_MEM_BANKS (256) and 64-bit address/size cells, this property
>>> can grow up to 4KB (256 * 16), easily overflowing the allocated
>>> buffer.
>>>
>>> Fix this by increasing DOM0_FDT_EXTRA_SIZE to account for
>>> the worst-case size: NR_MEM_BANKS * 16 bytes.
>>>
>>> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
>>> ---
>>> Just to be clear, I have not seen a real-world issue with this.
>>> The issue was observed during testing of limit conditions.
>>> With this patch applied, Xen successfully boots the hardware domain,
>>> exposing 256 reserved memory regions to it (using a synthetically
>>> generated configuration).
>>> ---
>>> ---
>>> xen/arch/arm/domain_build.c | 6 ++++--
>>> 1 file changed, 4 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>> index e8795745dd..7f9f0f5510 100644
>>> --- a/xen/arch/arm/domain_build.c
>>> +++ b/xen/arch/arm/domain_build.c
>>> @@ -100,9 +100,11 @@ int __init parse_arch_dom0_param(const char *s, const char *e)
>>> /*
>>> * Amount of extra space required to dom0's device tree. No new nodes
>> This comment would want to be updated because since its introduction things have
>> changed. Even the 128 came up as a result of adding /hypervisor node.
>
> You are right. I suggest the following wording:
>
> Amount of extra space required to dom0's device tree.
> This covers nodes generated by Xen, which are not directly copied
> from the host DTB. It is calculated as:
> - Space for /hypervisor node (128 bytes).
> - One terminating reserve map entry (16 bytes).
> - Space for a generated memory node covering all possible reserved
> memory regions (NR_MEM_BANKS * 16 bytes).
>
>
>>
>>> * are added (yet) but one terminating reserve map entry (16 bytes) is
>>> - * added.
>>> + * added. Plus space for an extra memory node to cover all possible reserved
>>> + * memory regions (2 addr cells + 2 size cells).
>>> */
>>> -#define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry))
>>> +#define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry) + \
>>> + (NR_MEM_BANKS * 16))
>>>
>>> unsigned int __init dom0_max_vcpus(void)
>>> {
>>
>> ~Michal
>>
~Michal
On 3/27/26 09:30, Orzel, Michal wrote:
Hello Michal
>
>
> On 26/03/2026 20:03, Oleksandr Tyshchenko wrote:
>>
>>
>> On 3/26/26 18:50, Orzel, Michal wrote:
>>
>> Hello Michal
>>
>>>
>>>
>>> On 26/03/2026 14:15, Oleksandr Tyshchenko wrote:
>>>> Xen fails to construct the hardware domain's device tree with
>>>> FDT_ERR_NOSPACE (-3) when the host memory map is highly fragmented
>>>> (e.g., numerous reserved memory regions).
>>>>
>>>> This occurs because DOM0_FDT_EXTRA_SIZE underestimates the space
>>>> required for the generated extra /memory node. make_memory_node()
>>> Where does this extra /memory node come from? If this is for normal reserved
>>> memory regions, they should be present in the host dtb and therefore accounted
>>> by fdt_totalsize (the host dtb should have reserved regions described in /memory
>>> and /reserved-memory. Are you trying to account for static shm regions?
>>
>>
>> I might have misunderstood something, but here is my analysis:
>>
>> The extra /memory node is generated by Xen itself in handle_node() ->
>> make_memory_node() (please refer to the if ( reserved_mem->nr_banks > 0
>> ) check).
>>
>> Even though the normal reserved memory regions are present in the host
>> DTB (and thus accounted for in fdt_totalsize), Xen generates a new
>> /memory node specifically for the hardware domain to describe these
>> regions as reserved but present in the memory map. And since this node
>> is generated at runtime (it is not a direct copy from the host DTB),
>> its size must be covered by DOM0_FDT_EXTRA_SIZE.
> Yes, but the original DTB should also have these reserved regions described in
> /memory nodes, thus taking up some space that is already accounted in
> fdt_totalsize. Are you trying to say that in host DTB, these reserved ranges fit
> nicely into e.g. a single /memory node range (i.e. a single reg pair covering
> most of the RAM)?
yes
I can see that it might be possible but the commit msg needs
> to be clear about it. As of now, it reads as if the problem occured always when
> there are multiple reserved memory regions. That's not true if a host DTB
> generates one /memory per one /reserved.
Yes, you are correct that the total size depends on how the host DTB is
structured compared to how Xen regenerates it at runtime. So, the issue
can arise if host DTB represents RAM using a single, large reg entry or
a few entries.
***
I will update the commit message to clarify that, something like below:
Xen fails to construct the hardware domain's device tree with
FDT_ERR_NOSPACE (-3) when the host memory map is highly fragmented
(e.g., numerous reserved memory regions) and the host DTB represents
RAM compactly (e.g., a single reg pair or just a few).
This occurs because DOM0_FDT_EXTRA_SIZE underestimates the space
required for the generated extra /memory node. While the host DTB
might represent RAM compactly, make_memory_node() aggregates
all reserved regions into a single reg property.
With NR_MEM_BANKS (256) and 64-bit address/size cells, this property
can grow up to 4KB (256 * 16), easily exceeding the space originally
occupied by the host DTB's nodes plus the current padding, thereby
overflowing the allocated buffer.
>
> Another issue is with the static shm nodes. User specifies the regions in the
> domain configuration and Xen creates *additional* nodes under /reserved and
> /memory that afaict we don't account for.
Yes, you are right.
Since these SHM sub-nodes and properties are generated purely from the
Xen domain configuration and are not present in the host DTB, they have
zero space allocated for them in fdt_totalsize.
So we need to redefine the macro. I propose the following formula that
separates the range data (16 bytes per bank in /memory) from the node
overhead (160 bytes per SHM region):
#define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry) + \
(NR_MEM_BANKS * 16) + \
(NR_SHMEM_BANKS * 160))
***
I will append the following text for the commit message:
Additionally, SHM regions require space for discrete sub-nodes under
/reserved-memory. Each of the up to NR_SHMEM_BANKS (32) regions triggers
the creation of a sub-node with properties (compatible, reg, xen,id, and
xen,offset). These runtime-generated sub-nodes require approximately 160
bytes each, further consuming the allocated buffer space.
Fix this by increasing DOM0_FDT_EXTRA_SIZE to account for both
fragmented reg properties (NR_MEM_BANKS * 16) under extra /memory
node and the injection of discrete SHM sub-nodes under /reserved-memory
node (NR_SHMEM_BANKS * 160).
Please tell me, does it look ok?
>
>>
>> For the instance, 10 reserved regions:
>>
>> (XEN) RAM: 0000000040000000 - 000000007fffffff
>> (XEN)
>> (XEN) MODULE[0]: 0000000043200000 - 000000004330afff Xen
>> (XEN) MODULE[1]: 0000000043400000 - 0000000043402fff Device Tree
>> (XEN) MODULE[2]: 0000000042e00000 - 000000004316907f Ramdisk
>> (XEN) MODULE[3]: 0000000040400000 - 0000000042d2ffff Kernel
>> (XEN) RESVD[0]: 0000000040009000 - 0000000040009fff
>> (XEN) RESVD[1]: 0000000040008000 - 0000000040008fff
>> (XEN) RESVD[2]: 0000000040007000 - 0000000040007fff
>> (XEN) RESVD[3]: 0000000040006000 - 0000000040006fff
>> (XEN) RESVD[4]: 0000000040005000 - 0000000040005fff
>> (XEN) RESVD[5]: 0000000040004000 - 0000000040004fff
>> (XEN) RESVD[6]: 0000000040003000 - 0000000040003fff
>> (XEN) RESVD[7]: 0000000040002000 - 0000000040002fff
>> (XEN) RESVD[8]: 0000000040001000 - 0000000040001fff
>> (XEN) RESVD[9]: 0000000040000000 - 0000000040000fff
>> ...
>>
>> From make_memory_node():
>>
>> (XEN) Create memory node
>> (XEN) Bank 0: 0x50000000->0x70000000
>> (XEN) (reg size 4, nr cells 4)
>>
>>
>>
>> (XEN) Create memory node
>> (XEN) Bank 0: 0x40009000->0x4000a000
>> (XEN) Bank 1: 0x40008000->0x40009000
>> (XEN) Bank 2: 0x40007000->0x40008000
>> (XEN) Bank 3: 0x40006000->0x40007000
>> (XEN) Bank 4: 0x40005000->0x40006000
>> (XEN) Bank 5: 0x40004000->0x40005000
>> (XEN) Bank 6: 0x40003000->0x40004000
>> (XEN) Bank 7: 0x40002000->0x40003000
>> (XEN) Bank 8: 0x40001000->0x40002000
>> (XEN) Bank 9: 0x40000000->0x40001000
>> (XEN) (reg size 4, nr cells 40)
>>
>>>
>>>> aggregates all reserved regions into a single reg property. With
>>>> NR_MEM_BANKS (256) and 64-bit address/size cells, this property
>>>> can grow up to 4KB (256 * 16), easily overflowing the allocated
>>>> buffer.
>>>>
>>>> Fix this by increasing DOM0_FDT_EXTRA_SIZE to account for
>>>> the worst-case size: NR_MEM_BANKS * 16 bytes.
>>>>
>>>> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
>>>> ---
>>>> Just to be clear, I have not seen a real-world issue with this.
>>>> The issue was observed during testing of limit conditions.
>>>> With this patch applied, Xen successfully boots the hardware domain,
>>>> exposing 256 reserved memory regions to it (using a synthetically
>>>> generated configuration).
>>>> ---
>>>> ---
>>>> xen/arch/arm/domain_build.c | 6 ++++--
>>>> 1 file changed, 4 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>>> index e8795745dd..7f9f0f5510 100644
>>>> --- a/xen/arch/arm/domain_build.c
>>>> +++ b/xen/arch/arm/domain_build.c
>>>> @@ -100,9 +100,11 @@ int __init parse_arch_dom0_param(const char *s, const char *e)
>>>> /*
>>>> * Amount of extra space required to dom0's device tree. No new nodes
>>> This comment would want to be updated because since its introduction things have
>>> changed. Even the 128 came up as a result of adding /hypervisor node.
>>
>> You are right. I suggest the following wording:
>>
>> Amount of extra space required to dom0's device tree.
>> This covers nodes generated by Xen, which are not directly copied
>> from the host DTB. It is calculated as:
>> - Space for /hypervisor node (128 bytes).
>> - One terminating reserve map entry (16 bytes).
>> - Space for a generated memory node covering all possible reserved
>> memory regions (NR_MEM_BANKS * 16 bytes).
>>
>>
>>>
>>>> * are added (yet) but one terminating reserve map entry (16 bytes) is
>>>> - * added.
>>>> + * added. Plus space for an extra memory node to cover all possible reserved
>>>> + * memory regions (2 addr cells + 2 size cells).
>>>> */
>>>> -#define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry))
>>>> +#define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry) + \
>>>> + (NR_MEM_BANKS * 16))
>>>>
>>>> unsigned int __init dom0_max_vcpus(void)
>>>> {
>>>
>>> ~Michal
>>>
>
> ~Michal
>
>
On 27/03/2026 13:23, Oleksandr Tyshchenko wrote: > > > On 3/27/26 09:30, Orzel, Michal wrote: > > Hello Michal > >> >> >> On 26/03/2026 20:03, Oleksandr Tyshchenko wrote: >>> >>> >>> On 3/26/26 18:50, Orzel, Michal wrote: >>> >>> Hello Michal >>> >>>> >>>> >>>> On 26/03/2026 14:15, Oleksandr Tyshchenko wrote: >>>>> Xen fails to construct the hardware domain's device tree with >>>>> FDT_ERR_NOSPACE (-3) when the host memory map is highly fragmented >>>>> (e.g., numerous reserved memory regions). >>>>> >>>>> This occurs because DOM0_FDT_EXTRA_SIZE underestimates the space >>>>> required for the generated extra /memory node. make_memory_node() >>>> Where does this extra /memory node come from? If this is for normal reserved >>>> memory regions, they should be present in the host dtb and therefore accounted >>>> by fdt_totalsize (the host dtb should have reserved regions described in /memory >>>> and /reserved-memory. Are you trying to account for static shm regions? >>> >>> >>> I might have misunderstood something, but here is my analysis: >>> >>> The extra /memory node is generated by Xen itself in handle_node() -> >>> make_memory_node() (please refer to the if ( reserved_mem->nr_banks > 0 >>> ) check). >>> >>> Even though the normal reserved memory regions are present in the host >>> DTB (and thus accounted for in fdt_totalsize), Xen generates a new >>> /memory node specifically for the hardware domain to describe these >>> regions as reserved but present in the memory map. And since this node >>> is generated at runtime (it is not a direct copy from the host DTB), >>> its size must be covered by DOM0_FDT_EXTRA_SIZE. >> Yes, but the original DTB should also have these reserved regions described in >> /memory nodes, thus taking up some space that is already accounted in >> fdt_totalsize. Are you trying to say that in host DTB, these reserved ranges fit >> nicely into e.g. a single /memory node range (i.e. a single reg pair covering >> most of the RAM)? > > yes > > > I can see that it might be possible but the commit msg needs >> to be clear about it. As of now, it reads as if the problem occured always when >> there are multiple reserved memory regions. That's not true if a host DTB >> generates one /memory per one /reserved. > > Yes, you are correct that the total size depends on how the host DTB is > structured compared to how Xen regenerates it at runtime. So, the issue > can arise if host DTB represents RAM using a single, large reg entry or > a few entries. > > *** > > I will update the commit message to clarify that, something like below: > > Xen fails to construct the hardware domain's device tree with > FDT_ERR_NOSPACE (-3) when the host memory map is highly fragmented > (e.g., numerous reserved memory regions) and the host DTB represents > RAM compactly (e.g., a single reg pair or just a few). > > This occurs because DOM0_FDT_EXTRA_SIZE underestimates the space > required for the generated extra /memory node. While the host DTB > might represent RAM compactly, make_memory_node() aggregates > all reserved regions into a single reg property. > With NR_MEM_BANKS (256) and 64-bit address/size cells, this property > can grow up to 4KB (256 * 16), easily exceeding the space originally > occupied by the host DTB's nodes plus the current padding, thereby > overflowing the allocated buffer. This reads better. > > >> >> Another issue is with the static shm nodes. User specifies the regions in the >> domain configuration and Xen creates *additional* nodes under /reserved and >> /memory that afaict we don't account for. > > Yes, you are right. > > Since these SHM sub-nodes and properties are generated purely from the > Xen domain configuration and are not present in the host DTB, they have > zero space allocated for them in fdt_totalsize. > > So we need to redefine the macro. I propose the following formula that > separates the range data (16 bytes per bank in /memory) from the node > overhead (160 bytes per SHM region): What is included in these 160 bytes? Did you manually check all fdt functions inside make_shm_resv_memory_node? > > #define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry) + \ > (NR_MEM_BANKS * 16) + \ > (NR_SHMEM_BANKS * 160)) I think you only accounted for shm nodes under /reserved-memory. As any other reserved memory node, they are also added to /memory reg property (see DT_MEM_NODE_REG_RANGE_SIZE). ~Michal
On 3/31/26 11:12, Orzel, Michal wrote:
Hello Michal
>
>
> On 27/03/2026 13:23, Oleksandr Tyshchenko wrote:
>>
>>
>> On 3/27/26 09:30, Orzel, Michal wrote:
>>
>> Hello Michal
>>
>>>
>>>
>>> On 26/03/2026 20:03, Oleksandr Tyshchenko wrote:
>>>>
>>>>
>>>> On 3/26/26 18:50, Orzel, Michal wrote:
>>>>
>>>> Hello Michal
>>>>
>>>>>
>>>>>
>>>>> On 26/03/2026 14:15, Oleksandr Tyshchenko wrote:
>>>>>> Xen fails to construct the hardware domain's device tree with
>>>>>> FDT_ERR_NOSPACE (-3) when the host memory map is highly fragmented
>>>>>> (e.g., numerous reserved memory regions).
>>>>>>
>>>>>> This occurs because DOM0_FDT_EXTRA_SIZE underestimates the space
>>>>>> required for the generated extra /memory node. make_memory_node()
>>>>> Where does this extra /memory node come from? If this is for normal reserved
>>>>> memory regions, they should be present in the host dtb and therefore accounted
>>>>> by fdt_totalsize (the host dtb should have reserved regions described in /memory
>>>>> and /reserved-memory. Are you trying to account for static shm regions?
>>>>
>>>>
>>>> I might have misunderstood something, but here is my analysis:
>>>>
>>>> The extra /memory node is generated by Xen itself in handle_node() ->
>>>> make_memory_node() (please refer to the if ( reserved_mem->nr_banks > 0
>>>> ) check).
>>>>
>>>> Even though the normal reserved memory regions are present in the host
>>>> DTB (and thus accounted for in fdt_totalsize), Xen generates a new
>>>> /memory node specifically for the hardware domain to describe these
>>>> regions as reserved but present in the memory map. And since this node
>>>> is generated at runtime (it is not a direct copy from the host DTB),
>>>> its size must be covered by DOM0_FDT_EXTRA_SIZE.
>>> Yes, but the original DTB should also have these reserved regions described in
>>> /memory nodes, thus taking up some space that is already accounted in
>>> fdt_totalsize. Are you trying to say that in host DTB, these reserved ranges fit
>>> nicely into e.g. a single /memory node range (i.e. a single reg pair covering
>>> most of the RAM)?
>>
>> yes
>>
>>
>> I can see that it might be possible but the commit msg needs
>>> to be clear about it. As of now, it reads as if the problem occured always when
>>> there are multiple reserved memory regions. That's not true if a host DTB
>>> generates one /memory per one /reserved.
>>
>> Yes, you are correct that the total size depends on how the host DTB is
>> structured compared to how Xen regenerates it at runtime. So, the issue
>> can arise if host DTB represents RAM using a single, large reg entry or
>> a few entries.
>>
>> ***
>>
>> I will update the commit message to clarify that, something like below:
>>
>> Xen fails to construct the hardware domain's device tree with
>> FDT_ERR_NOSPACE (-3) when the host memory map is highly fragmented
>> (e.g., numerous reserved memory regions) and the host DTB represents
>> RAM compactly (e.g., a single reg pair or just a few).
>>
>> This occurs because DOM0_FDT_EXTRA_SIZE underestimates the space
>> required for the generated extra /memory node. While the host DTB
>> might represent RAM compactly, make_memory_node() aggregates
>> all reserved regions into a single reg property.
>> With NR_MEM_BANKS (256) and 64-bit address/size cells, this property
>> can grow up to 4KB (256 * 16), easily exceeding the space originally
>> occupied by the host DTB's nodes plus the current padding, thereby
>> overflowing the allocated buffer.
> This reads better.
ok
>
>>
>>
>>>
>>> Another issue is with the static shm nodes. User specifies the regions in the
>>> domain configuration and Xen creates *additional* nodes under /reserved and
>>> /memory that afaict we don't account for.
>>
>> Yes, you are right.
>>
>> Since these SHM sub-nodes and properties are generated purely from the
>> Xen domain configuration and are not present in the host DTB, they have
>> zero space allocated for them in fdt_totalsize.
>>
>> So we need to redefine the macro. I propose the following formula that
>> separates the range data (16 bytes per bank in /memory) from the node
>> overhead (160 bytes per SHM region):
> What is included in these 160 bytes? Did you manually check all fdt functions
> inside make_shm_resv_memory_node?
According to my calculations (which, of course, might be not precise):
- FDT_BEGIN_NODE + xen-shmem@ffffffffffffffff\0 (27b padded to 28): 32 bytes
- compatible (12b header + 21b string padded to 24): 36 bytes
- reg (12b header + 16b payload [4 cells]): 28 bytes
- xen,id (12b header + 16b max string [15 chars + \0]): 28 bytes
- xen,offset (12b header + 8b payload): 20 bytes
- FDT_END_NODE: 4 bytes
Total exact node payload: 148 bytes. I also added 12-byte margin (so it
gets rounded up to the nearest 16-byte boundary).
>
>>
>> #define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry) + \
>> (NR_MEM_BANKS * 16) + \
>> (NR_SHMEM_BANKS * 160))
> I think you only accounted for shm nodes under /reserved-memory. As any other
> reserved memory node, they are also added to /memory reg property (see
> DT_MEM_NODE_REG_RANGE_SIZE).
You are right, and I completely missed this in my original calculation.
I mistakenly believed (NR_MEM_BANKS * 16) would cover the entire
capacity of the /memory node's reg.
The shm_mem_node_fill_reg_range() appends the shared memory banks
directly into the main /memory node's reg. Each SHM bank adds 16 bytes
(4 cells = 16 bytes) to the main memory node.
So, I will refine the macro to explicitly reflect both the 160-byte
discrete sub-node and the 16-byte extra to the /memory node:
#define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry) + \
(NR_MEM_BANKS * 16) + \
(NR_SHMEM_BANKS * (160 + 16)))
Or wait, we can actually drop the SHM overhead entirely when
CONFIG_STATIC_SHM=n:
#define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry) + \
(NR_MEM_BANKS * 16) + \
(IS_ENABLED(CONFIG_STATIC_SHM) ? \
(NR_SHMEM_BANKS * (160 + 16)) : 0))
>
> ~Michal
>
On 31/03/2026 16:10, Oleksandr Tyshchenko wrote: > > > On 3/31/26 11:12, Orzel, Michal wrote: > > Hello Michal > > >> >> >> On 27/03/2026 13:23, Oleksandr Tyshchenko wrote: >>> >>> >>> On 3/27/26 09:30, Orzel, Michal wrote: >>> >>> Hello Michal >>> >>>> >>>> >>>> On 26/03/2026 20:03, Oleksandr Tyshchenko wrote: >>>>> >>>>> >>>>> On 3/26/26 18:50, Orzel, Michal wrote: >>>>> >>>>> Hello Michal >>>>> >>>>>> >>>>>> >>>>>> On 26/03/2026 14:15, Oleksandr Tyshchenko wrote: >>>>>>> Xen fails to construct the hardware domain's device tree with >>>>>>> FDT_ERR_NOSPACE (-3) when the host memory map is highly fragmented >>>>>>> (e.g., numerous reserved memory regions). >>>>>>> >>>>>>> This occurs because DOM0_FDT_EXTRA_SIZE underestimates the space >>>>>>> required for the generated extra /memory node. make_memory_node() >>>>>> Where does this extra /memory node come from? If this is for normal reserved >>>>>> memory regions, they should be present in the host dtb and therefore accounted >>>>>> by fdt_totalsize (the host dtb should have reserved regions described in /memory >>>>>> and /reserved-memory. Are you trying to account for static shm regions? >>>>> >>>>> >>>>> I might have misunderstood something, but here is my analysis: >>>>> >>>>> The extra /memory node is generated by Xen itself in handle_node() -> >>>>> make_memory_node() (please refer to the if ( reserved_mem->nr_banks > 0 >>>>> ) check). >>>>> >>>>> Even though the normal reserved memory regions are present in the host >>>>> DTB (and thus accounted for in fdt_totalsize), Xen generates a new >>>>> /memory node specifically for the hardware domain to describe these >>>>> regions as reserved but present in the memory map. And since this node >>>>> is generated at runtime (it is not a direct copy from the host DTB), >>>>> its size must be covered by DOM0_FDT_EXTRA_SIZE. >>>> Yes, but the original DTB should also have these reserved regions described in >>>> /memory nodes, thus taking up some space that is already accounted in >>>> fdt_totalsize. Are you trying to say that in host DTB, these reserved ranges fit >>>> nicely into e.g. a single /memory node range (i.e. a single reg pair covering >>>> most of the RAM)? >>> >>> yes >>> >>> >>> I can see that it might be possible but the commit msg needs >>>> to be clear about it. As of now, it reads as if the problem occured always when >>>> there are multiple reserved memory regions. That's not true if a host DTB >>>> generates one /memory per one /reserved. >>> >>> Yes, you are correct that the total size depends on how the host DTB is >>> structured compared to how Xen regenerates it at runtime. So, the issue >>> can arise if host DTB represents RAM using a single, large reg entry or >>> a few entries. >>> >>> *** >>> >>> I will update the commit message to clarify that, something like below: >>> >>> Xen fails to construct the hardware domain's device tree with >>> FDT_ERR_NOSPACE (-3) when the host memory map is highly fragmented >>> (e.g., numerous reserved memory regions) and the host DTB represents >>> RAM compactly (e.g., a single reg pair or just a few). >>> >>> This occurs because DOM0_FDT_EXTRA_SIZE underestimates the space >>> required for the generated extra /memory node. While the host DTB >>> might represent RAM compactly, make_memory_node() aggregates >>> all reserved regions into a single reg property. >>> With NR_MEM_BANKS (256) and 64-bit address/size cells, this property >>> can grow up to 4KB (256 * 16), easily exceeding the space originally >>> occupied by the host DTB's nodes plus the current padding, thereby >>> overflowing the allocated buffer. >> This reads better. > > ok > > >> >>> >>> >>>> >>>> Another issue is with the static shm nodes. User specifies the regions in the >>>> domain configuration and Xen creates *additional* nodes under /reserved and >>>> /memory that afaict we don't account for. >>> >>> Yes, you are right. >>> >>> Since these SHM sub-nodes and properties are generated purely from the >>> Xen domain configuration and are not present in the host DTB, they have >>> zero space allocated for them in fdt_totalsize. >>> >>> So we need to redefine the macro. I propose the following formula that >>> separates the range data (16 bytes per bank in /memory) from the node >>> overhead (160 bytes per SHM region): >> What is included in these 160 bytes? Did you manually check all fdt functions >> inside make_shm_resv_memory_node? > > According to my calculations (which, of course, might be not precise): > > - FDT_BEGIN_NODE + xen-shmem@ffffffffffffffff\0 (27b padded to 28): 32 bytes > - compatible (12b header + 21b string padded to 24): 36 bytes > - reg (12b header + 16b payload [4 cells]): 28 bytes > - xen,id (12b header + 16b max string [15 chars + \0]): 28 bytes > - xen,offset (12b header + 8b payload): 20 bytes > - FDT_END_NODE: 4 bytes > Total exact node payload: 148 bytes. I also added 12-byte margin (so it > gets rounded up to the nearest 16-byte boundary). > >> >>> >>> #define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry) + \ >>> (NR_MEM_BANKS * 16) + \ >>> (NR_SHMEM_BANKS * 160)) >> I think you only accounted for shm nodes under /reserved-memory. As any other >> reserved memory node, they are also added to /memory reg property (see >> DT_MEM_NODE_REG_RANGE_SIZE). > > You are right, and I completely missed this in my original calculation. > I mistakenly believed (NR_MEM_BANKS * 16) would cover the entire > capacity of the /memory node's reg. > > The shm_mem_node_fill_reg_range() appends the shared memory banks > directly into the main /memory node's reg. Each SHM bank adds 16 bytes > (4 cells = 16 bytes) to the main memory node. > > So, I will refine the macro to explicitly reflect both the 160-byte > discrete sub-node and the 16-byte extra to the /memory node: > > #define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry) + \ > (NR_MEM_BANKS * 16) + \ > (NR_SHMEM_BANKS * (160 + 16))) > > Or wait, we can actually drop the SHM overhead entirely when > CONFIG_STATIC_SHM=n: > > #define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry) + \ > (NR_MEM_BANKS * 16) + \ > (IS_ENABLED(CONFIG_STATIC_SHM) ? \ > (NR_SHMEM_BANKS * (160 + 16)) : 0)) Yes, the CONFIG was my next question. I'm ok with this solution. ~Michal
© 2016 - 2026 Red Hat, Inc.