The domain configuration may request more vcpus than are present in the system.
For dom0, the function dom0_max_vcpus() was used to clamp down to physically
available vcpus. Here we are introducing a generalized version,
dom_max_vcpus(), that takes a boot domain and sets the max vcpus based on the
lesser of the requested max and the available vcpus.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/domain-builder/Makefile      |  1 +
 xen/arch/x86/domain-builder/domain.c      | 38 +++++++++++++++++++++++
 xen/arch/x86/include/asm/domain-builder.h |  1 +
 xen/arch/x86/setup.c                      |  4 +--
 4 files changed, 42 insertions(+), 2 deletions(-)
 create mode 100644 xen/arch/x86/domain-builder/domain.c

diff --git a/xen/arch/x86/domain-builder/Makefile b/xen/arch/x86/domain-builder/Makefile
index b10cd56b286b..67024b5cb213 100644
--- a/xen/arch/x86/domain-builder/Makefile
+++ b/xen/arch/x86/domain-builder/Makefile
@@ -1,2 +1,3 @@
 obj-$(CONFIG_DOMAIN_BUILDER) += fdt.init.o
 obj-y += core.init.o
+obj-y += domain.init.o
diff --git a/xen/arch/x86/domain-builder/domain.c b/xen/arch/x86/domain-builder/domain.c
new file mode 100644
index 000000000000..f2277b9e3cf3
--- /dev/null
+++ b/xen/arch/x86/domain-builder/domain.c
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2024, Apertus Solutions, LLC
+ */
+
+#include <xen/cpumask.h>
+#include <xen/domain.h>
+#include <xen/init.h>
+#include <xen/sched.h>
+
+#include <asm/bootinfo.h>
+
+unsigned int __init dom_max_vcpus(struct boot_domain *bd)
+{
+    unsigned int limit = bd->mode & BUILD_MODE_PARAVIRT ?
+                                    MAX_VIRT_CPUS : HVM_MAX_VCPUS;
+
+    if ( bd->capabilities & BUILD_CAPS_CONTROL )
+        limit = dom0_max_vcpus();
+    else
+        limit = min(limit,
+                    (uint32_t)cpumask_weight(cpupool_valid_cpus(cpupool0)));
+
+    if ( bd->max_vcpus == 0 || bd->max_vcpus > limit )
+        bd->max_vcpus = limit;
+
+    return bd->max_vcpus;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/x86/include/asm/domain-builder.h b/xen/arch/x86/include/asm/domain-builder.h
index 7518b6ddf3b9..f37f73e2255b 100644
--- a/xen/arch/x86/include/asm/domain-builder.h
+++ b/xen/arch/x86/include/asm/domain-builder.h
@@ -8,5 +8,6 @@ int __init builder_get_cmdline(
     struct boot_info *bi, int offset, char *cmdline, size_t size);
 
 void builder_init(struct boot_info *bi);
+unsigned int dom_max_vcpus(struct boot_domain *bd);
 
 #endif
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 6a939ccede3f..86bbd7c72ccd 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1010,18 +1010,18 @@ static struct domain *__init create_dom0(struct boot_info *bi)
     char *cmdline = NULL;
     size_t cmdline_size;
     unsigned int create_flags = 0;
+    struct boot_domain *bd = &bi->domains[0];
     struct xen_domctl_createdomain dom0_cfg = {
         .flags = IS_ENABLED(CONFIG_TBOOT) ? XEN_DOMCTL_CDF_s3_integrity : 0,
         .max_evtchn_port = -1,
         .max_grant_frames = -1,
         .max_maptrack_frames = -1,
         .grant_opts = XEN_DOMCTL_GRANT_version(opt_gnttab_max_version),
-        .max_vcpus = dom0_max_vcpus(),
+        .max_vcpus = dom_max_vcpus(bd),
         .arch = {
             .misc_flags = opt_dom0_msr_relaxed ? XEN_X86_MSR_RELAXED : 0,
         },
     };
-    struct boot_domain *bd = &bi->domains[0];
     struct domain *d;
 
     if ( opt_dom0_pvh ||
-- 
2.30.2

On 2025-04-19 18:07, Daniel P. Smith wrote:
> The domain configuration may request more vcpus than are present in the system.
> For dom0, the function dom0_max_vcpus() was used to clamp down to physically
> available vcpus. Here we are introducing a generalized version,
> dom_max_vcpus(), that takes a boot domain and sets the max vcpus based on the
> lesser of the requested max and the available vcpus.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---

> diff --git a/xen/arch/x86/domain-builder/domain.c b/xen/arch/x86/domain-builder/domain.c
> new file mode 100644
> index 000000000000..f2277b9e3cf3
> --- /dev/null
> +++ b/xen/arch/x86/domain-builder/domain.c
> @@ -0,0 +1,38 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2024, Apertus Solutions, LLC
> + */
> +
> +#include <xen/cpumask.h>
> +#include <xen/domain.h>
> +#include <xen/init.h>
> +#include <xen/sched.h>
> +
> +#include <asm/bootinfo.h>
> +
> +unsigned int __init dom_max_vcpus(struct boot_domain *bd)
> +{
> +    unsigned int limit = bd->mode & BUILD_MODE_PARAVIRT ?
> +                                    MAX_VIRT_CPUS : HVM_MAX_VCPUS;
> +
> +    if ( bd->capabilities & BUILD_CAPS_CONTROL )

I added xen/include/public/bootfdt.h with DOMAIN_CAPS_CONTROL and the 
other capabilities to provide common values.

> +        limit = dom0_max_vcpus();

dom0_max_vcpus() applies Xen's dom0_max_vcpus command line option.  That 
is desirable for a traditional dom0.  For a disaggregated, Hyperlaunch 
system, I'm not sure it's appropriate.  Considering there can multiple 
control domains, it's more questionable.

Might it be better to only apply Xen "dom0" command line options to 
non-hyperlaunch dom0?  Or a domain with all of 
BUILD_CAPS_CONTROL/HARDWARE/XENSTORE?

I guess it could stay as-is, but it seems unusual.

Regards,
Jason

On 4/22/25 16:36, Jason Andryuk wrote:
> On 2025-04-19 18:07, Daniel P. Smith wrote:
>> The domain configuration may request more vcpus than are present in 
>> the system.
>> For dom0, the function dom0_max_vcpus() was used to clamp down to 
>> physically
>> available vcpus. Here we are introducing a generalized version,
>> dom_max_vcpus(), that takes a boot domain and sets the max vcpus based 
>> on the
>> lesser of the requested max and the available vcpus.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
> 
>> diff --git a/xen/arch/x86/domain-builder/domain.c b/xen/arch/x86/ 
>> domain-builder/domain.c
>> new file mode 100644
>> index 000000000000..f2277b9e3cf3
>> --- /dev/null
>> +++ b/xen/arch/x86/domain-builder/domain.c
>> @@ -0,0 +1,38 @@
>> +/* SPDX-License-Identifier: GPL-2.0-only */
>> +/*
>> + * Copyright (C) 2024, Apertus Solutions, LLC
>> + */
>> +
>> +#include <xen/cpumask.h>
>> +#include <xen/domain.h>
>> +#include <xen/init.h>
>> +#include <xen/sched.h>
>> +
>> +#include <asm/bootinfo.h>
>> +
>> +unsigned int __init dom_max_vcpus(struct boot_domain *bd)
>> +{
>> +    unsigned int limit = bd->mode & BUILD_MODE_PARAVIRT ?
>> +                                    MAX_VIRT_CPUS : HVM_MAX_VCPUS;
>> +
>> +    if ( bd->capabilities & BUILD_CAPS_CONTROL )
> 
> I added xen/include/public/bootfdt.h with DOMAIN_CAPS_CONTROL and the 
> other capabilities to provide common values.
> 
>> +        limit = dom0_max_vcpus();
> 
> dom0_max_vcpus() applies Xen's dom0_max_vcpus command line option.  That 
> is desirable for a traditional dom0.  For a disaggregated, Hyperlaunch 
> system, I'm not sure it's appropriate.  Considering there can multiple 
> control domains, it's more questionable.
> 
> Might it be better to only apply Xen "dom0" command line options to non- 
> hyperlaunch dom0?  Or a domain with all of BUILD_CAPS_CONTROL/HARDWARE/ 
> XENSTORE?
> 
> I guess it could stay as-is, but it seems unusual.

The larger issue is that the cmdline params are going to need to be 
addressed. I see three approaches, first would be to only apply the 
params when there is a single domain with ctrl/hw/xs all set, or the 
second approach would be to change the params to support multiple domain 
statements. Though the second approach has the issue of the need to 
decide how to support the legacy params

v/r,
dps

On Tue Apr 22, 2025 at 9:36 PM BST, Jason Andryuk wrote:
> On 2025-04-19 18:07, Daniel P. Smith wrote:
>> +        limit = dom0_max_vcpus();
>
> dom0_max_vcpus() applies Xen's dom0_max_vcpus command line option.  That 
> is desirable for a traditional dom0.  For a disaggregated, Hyperlaunch 
> system, I'm not sure it's appropriate.  Considering there can multiple 
> control domains, it's more questionable.
>
> Might it be better to only apply Xen "dom0" command line options to 
> non-hyperlaunch dom0?  Or a domain with all of 
> BUILD_CAPS_CONTROL/HARDWARE/XENSTORE?

Alternatively, why not apply it to the hardware domain instead? That's
guaranteed to be (at most) one, and will still function appropriately
when doing non-DTB based boots.

I'll make this adjustment while rebasing this rfc against my latest
hlaunch series.

>
> I guess it could stay as-is, but it seems unusual.

And would probably be particularly weird when it applies to all your
control domains and _not to your hardware domain, which incidentally is
the one domain with domid 0.

>
> Regards,
> Jason

Cheers,
Alejandro