Later patches will keep refactoring create_dom0()
until it can create arbitrary domains. This is one
small step in that direction.

Signed-off-by: Alejandro Vallejo <alejandro.garciavallejo@amd.com>
---
 xen/arch/x86/setup.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index c6890669b9..6943ffba79 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1054,7 +1054,8 @@ static struct domain *__init create_dom0(struct boot_info *bi)
     if ( IS_ERR(d) )
         panic("Error creating d%u: %ld\n", bd->domid, PTR_ERR(d));
 
-    init_dom0_cpuid_policy(d);
+    if ( pv_shim || d->cdf & (CDF_privileged | CDF_hardware) )
+        init_dom0_cpuid_policy(d);
 
     if ( alloc_dom0_vcpu0(d) == NULL )
         panic("Error creating %pdv0\n", d);

base-commit: 55719030b0bb0069fc8b57cd808dc98dc9d39add
-- 
2.43.0

On Thu, Jul 17, 2025 at 07:58:24PM +0200, Alejandro Vallejo wrote:
> Later patches will keep refactoring create_dom0()
> until it can create arbitrary domains. This is one
> small step in that direction.
> 
> Signed-off-by: Alejandro Vallejo <alejandro.garciavallejo@amd.com>
> ---
>  xen/arch/x86/setup.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index c6890669b9..6943ffba79 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -1054,7 +1054,8 @@ static struct domain *__init create_dom0(struct boot_info *bi)
>      if ( IS_ERR(d) )
>          panic("Error creating d%u: %ld\n", bd->domid, PTR_ERR(d));
>  
> -    init_dom0_cpuid_policy(d);
> +    if ( pv_shim || d->cdf & (CDF_privileged | CDF_hardware) )

You possibly want this to be:

(d->cdf & (CDF_privileged | CDF_hardware)) == (CDF_privileged | CDF_hardware)

To ensure the contents of dom0_cpuid_cmdline is only applied to dom0,
and not to the hardware or control domains.  I assume it should be
possible to pass a different set of cpuid options for the hardware vs
the control domains.

Thanks, Roger.

On Wed Jul 23, 2025 at 9:18 AM CEST, Roger Pau Monné wrote:
> On Thu, Jul 17, 2025 at 07:58:24PM +0200, Alejandro Vallejo wrote:
>> Later patches will keep refactoring create_dom0()
>> until it can create arbitrary domains. This is one
>> small step in that direction.
>> 
>> Signed-off-by: Alejandro Vallejo <alejandro.garciavallejo@amd.com>
>> ---
>>  xen/arch/x86/setup.c | 3 ++-
>>  1 file changed, 2 insertions(+), 1 deletion(-)
>> 
>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>> index c6890669b9..6943ffba79 100644
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
>> @@ -1054,7 +1054,8 @@ static struct domain *__init create_dom0(struct boot_info *bi)
>>      if ( IS_ERR(d) )
>>          panic("Error creating d%u: %ld\n", bd->domid, PTR_ERR(d));
>>  
>> -    init_dom0_cpuid_policy(d);
>> +    if ( pv_shim || d->cdf & (CDF_privileged | CDF_hardware) )
>
> You possibly want this to be:
>
> (d->cdf & (CDF_privileged | CDF_hardware)) == (CDF_privileged | CDF_hardware)
>
> To ensure the contents of dom0_cpuid_cmdline is only applied to dom0,
> and not to the hardware or control domains.  I assume it should be
> possible to pass a different set of cpuid options for the hardware vs
> the control domains.
>
> Thanks, Roger.

Why only a hwdom+ctldom, surely a single hwdom should get it too. I can see
the argument for a ctldom not getting it. For our use case having dom0
disaggregated is of the essence, so what happens with a hwdom that is not a
ctldom is fairly important.

That said, I'm thinking moving in a different direction and have a generic
init_cpuid_policy() that internally checks for hw or control, or leave the
default policy or something else.

This would remove the conditional and allow much finer selection. e.g: a domain
brought up through a "nomigrate" DTB node (TBD: nonexisting binding atm) would
get itsc reported, just as dom0 does today.

Cheers,
Alejandro

On Fri, Jul 25, 2025 at 12:02:18PM +0200, Alejandro Vallejo wrote:
> On Wed Jul 23, 2025 at 9:18 AM CEST, Roger Pau Monné wrote:
> > On Thu, Jul 17, 2025 at 07:58:24PM +0200, Alejandro Vallejo wrote:
> >> Later patches will keep refactoring create_dom0()
> >> until it can create arbitrary domains. This is one
> >> small step in that direction.
> >> 
> >> Signed-off-by: Alejandro Vallejo <alejandro.garciavallejo@amd.com>
> >> ---
> >>  xen/arch/x86/setup.c | 3 ++-
> >>  1 file changed, 2 insertions(+), 1 deletion(-)
> >> 
> >> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> >> index c6890669b9..6943ffba79 100644
> >> --- a/xen/arch/x86/setup.c
> >> +++ b/xen/arch/x86/setup.c
> >> @@ -1054,7 +1054,8 @@ static struct domain *__init create_dom0(struct boot_info *bi)
> >>      if ( IS_ERR(d) )
> >>          panic("Error creating d%u: %ld\n", bd->domid, PTR_ERR(d));
> >>  
> >> -    init_dom0_cpuid_policy(d);
> >> +    if ( pv_shim || d->cdf & (CDF_privileged | CDF_hardware) )
> >
> > You possibly want this to be:
> >
> > (d->cdf & (CDF_privileged | CDF_hardware)) == (CDF_privileged | CDF_hardware)
> >
> > To ensure the contents of dom0_cpuid_cmdline is only applied to dom0,
> > and not to the hardware or control domains.  I assume it should be
> > possible to pass a different set of cpuid options for the hardware vs
> > the control domains.
> >
> > Thanks, Roger.
> 
> Why only a hwdom+ctldom, surely a single hwdom should get it too.

hm, not really I think: a late hardware domain would get any custom
cpuid options from the toolstack that created it, or in the
hyperlaunch case from the provided configuration, but not from the
dom0-cpuid command line option I would expect.  Otherwise you have two
different sources of cpuid options, the inheritance from dom0-cpuid,
plus whatever is provided from the hardware domain configuration.

> I can see
> the argument for a ctldom not getting it. For our use case having dom0
> disaggregated is of the essence, so what happens with a hwdom that is not a
> ctldom is fairly important.
> 
> That said, I'm thinking moving in a different direction and have a generic
> init_cpuid_policy() that internally checks for hw or control, or leave the
> default policy or something else.

Right, so instead of introducing init_{hwdom,ctrldom}_cpuid_policy()
equivalents you would rather generalize init_dom0_cpuid_policy() so
it's used by all domains.

> This would remove the conditional and allow much finer selection. e.g: a domain
> brought up through a "nomigrate" DTB node (TBD: nonexisting binding atm) would
> get itsc reported, just as dom0 does today.

We might want to enforce such no migration attribute at the hypervisor
level (by adding a new domain field to signal it possibly?), as this
is all toolstack knowledge ATM.

On a related tangent: domains brought up using hyperlaunch will have a
config file, capable of expressing options like cpuid features I
expect, at which point ITSC could be set in the config file and Xen
won't need to do any guessing?

Thanks, Roger.

On 7/25/25 06:56, Roger Pau Monné wrote:
> On Fri, Jul 25, 2025 at 12:02:18PM +0200, Alejandro Vallejo wrote:
>> On Wed Jul 23, 2025 at 9:18 AM CEST, Roger Pau Monné wrote:
>>> On Thu, Jul 17, 2025 at 07:58:24PM +0200, Alejandro Vallejo wrote:
>>>> Later patches will keep refactoring create_dom0()
>>>> until it can create arbitrary domains. This is one
>>>> small step in that direction.
>>>>
>>>> Signed-off-by: Alejandro Vallejo <alejandro.garciavallejo@amd.com>
>>>> ---
>>>>   xen/arch/x86/setup.c | 3 ++-
>>>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>>>> index c6890669b9..6943ffba79 100644
>>>> --- a/xen/arch/x86/setup.c
>>>> +++ b/xen/arch/x86/setup.c
>>>> @@ -1054,7 +1054,8 @@ static struct domain *__init create_dom0(struct boot_info *bi)
>>>>       if ( IS_ERR(d) )
>>>>           panic("Error creating d%u: %ld\n", bd->domid, PTR_ERR(d));
>>>>   
>>>> -    init_dom0_cpuid_policy(d);
>>>> +    if ( pv_shim || d->cdf & (CDF_privileged | CDF_hardware) )
>>>
>>> You possibly want this to be:
>>>
>>> (d->cdf & (CDF_privileged | CDF_hardware)) == (CDF_privileged | CDF_hardware)
>>>
>>> To ensure the contents of dom0_cpuid_cmdline is only applied to dom0,
>>> and not to the hardware or control domains.  I assume it should be
>>> possible to pass a different set of cpuid options for the hardware vs
>>> the control domains.
>>>
>>> Thanks, Roger.
>>
>> Why only a hwdom+ctldom, surely a single hwdom should get it too.
> 
> hm, not really I think: a late hardware domain would get any custom
> cpuid options from the toolstack that created it, or in the
> hyperlaunch case from the provided configuration, but not from the
> dom0-cpuid command line option I would expect.  Otherwise you have two
> different sources of cpuid options, the inheritance from dom0-cpuid,
> plus whatever is provided from the hardware domain configuration.

Yes, this has been a sticking point for me and never got any good 
answers thus far. Should the dom0 related xen command line options only 
apply when not booting via hyperlaunch. If the answer is no, then you're 
in this area with some dom0 options that really are applicable to hwdom 
vs ctldom and vice-a-versa. Some could even be suggested to apply to 
both. And then, I don't believe there really is a consensus one which 
options apply to which domains. Over the years working on this, I have 
been an advocate that commandline adjustments allow for quicker 
troubleshooting by the user/administrator. In the last version of the 
multidomain construction RFC, I am growing more and more to advocate for 
my initial proposition, that dom0 options only apply when not using 
hyperlaunch.

>> I can see
>> the argument for a ctldom not getting it. For our use case having dom0
>> disaggregated is of the essence, so what happens with a hwdom that is not a
>> ctldom is fairly important.
>>
>> That said, I'm thinking moving in a different direction and have a generic
>> init_cpuid_policy() that internally checks for hw or control, or leave the
>> default policy or something else.
> 
> Right, so instead of introducing init_{hwdom,ctrldom}_cpuid_policy()
> equivalents you would rather generalize init_dom0_cpuid_policy() so
> it's used by all domains.
> 
>> This would remove the conditional and allow much finer selection. e.g: a domain
>> brought up through a "nomigrate" DTB node (TBD: nonexisting binding atm) would
>> get itsc reported, just as dom0 does today.
> 
> We might want to enforce such no migration attribute at the hypervisor
> level (by adding a new domain field to signal it possibly?), as this
> is all toolstack knowledge ATM.
> 
> On a related tangent: domains brought up using hyperlaunch will have a
> config file, capable of expressing options like cpuid features I
> expect, at which point ITSC could be set in the config file and Xen
> won't need to do any guessing?

The goal has never been to support all the configuration options that a 
full toolstack can adjust for the VM. I have always felt it should be a 
light-weight toolstack to allow essential configuration. I still 
advocate for the boot domain that would have capabilities like parsing 
the AML, walking PCI device tree, etc. It would then handle any more 
complicated initialization/configuration of the VMs before exiting and 
allowing all the domains to start

v/r,
dps

On 29.07.2025 23:29, Daniel P. Smith wrote:
> On 7/25/25 06:56, Roger Pau Monné wrote:
>> On Fri, Jul 25, 2025 at 12:02:18PM +0200, Alejandro Vallejo wrote:
>>> On Wed Jul 23, 2025 at 9:18 AM CEST, Roger Pau Monné wrote:
>>>> On Thu, Jul 17, 2025 at 07:58:24PM +0200, Alejandro Vallejo wrote:
>>>>> Later patches will keep refactoring create_dom0()
>>>>> until it can create arbitrary domains. This is one
>>>>> small step in that direction.
>>>>>
>>>>> Signed-off-by: Alejandro Vallejo <alejandro.garciavallejo@amd.com>
>>>>> ---
>>>>>   xen/arch/x86/setup.c | 3 ++-
>>>>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>>>>> index c6890669b9..6943ffba79 100644
>>>>> --- a/xen/arch/x86/setup.c
>>>>> +++ b/xen/arch/x86/setup.c
>>>>> @@ -1054,7 +1054,8 @@ static struct domain *__init create_dom0(struct boot_info *bi)
>>>>>       if ( IS_ERR(d) )
>>>>>           panic("Error creating d%u: %ld\n", bd->domid, PTR_ERR(d));
>>>>>   
>>>>> -    init_dom0_cpuid_policy(d);
>>>>> +    if ( pv_shim || d->cdf & (CDF_privileged | CDF_hardware) )
>>>>
>>>> You possibly want this to be:
>>>>
>>>> (d->cdf & (CDF_privileged | CDF_hardware)) == (CDF_privileged | CDF_hardware)
>>>>
>>>> To ensure the contents of dom0_cpuid_cmdline is only applied to dom0,
>>>> and not to the hardware or control domains.  I assume it should be
>>>> possible to pass a different set of cpuid options for the hardware vs
>>>> the control domains.
>>>>
>>>> Thanks, Roger.
>>>
>>> Why only a hwdom+ctldom, surely a single hwdom should get it too.
>>
>> hm, not really I think: a late hardware domain would get any custom
>> cpuid options from the toolstack that created it, or in the
>> hyperlaunch case from the provided configuration, but not from the
>> dom0-cpuid command line option I would expect.  Otherwise you have two
>> different sources of cpuid options, the inheritance from dom0-cpuid,
>> plus whatever is provided from the hardware domain configuration.
> 
> Yes, this has been a sticking point for me and never got any good 
> answers thus far. Should the dom0 related xen command line options only 
> apply when not booting via hyperlaunch. If the answer is no, then you're 
> in this area with some dom0 options that really are applicable to hwdom 
> vs ctldom and vice-a-versa. Some could even be suggested to apply to 
> both. And then, I don't believe there really is a consensus one which 
> options apply to which domains. Over the years working on this, I have 
> been an advocate that commandline adjustments allow for quicker 
> troubleshooting by the user/administrator. In the last version of the 
> multidomain construction RFC, I am growing more and more to advocate for 
> my initial proposition, that dom0 options only apply when not using 
> hyperlaunch.

With the hyperlaunch plans, is there something that's still properly
"Dom0", perhaps under certain conditions? That (and only that) is
where I would see respective command line options to apply. IOW no
more than one specific domain (i.e. in particular not to both hwdom
and ctldom, when they're separate). In cases when respective options
are entirely ignored, I think some kind of warning would want issuing.

Jan

On Wed Jul 30, 2025 at 9:48 AM CEST, Jan Beulich wrote:
> On 29.07.2025 23:29, Daniel P. Smith wrote:
>> On 7/25/25 06:56, Roger Pau Monné wrote:
>>> On Fri, Jul 25, 2025 at 12:02:18PM +0200, Alejandro Vallejo wrote:
>>>> On Wed Jul 23, 2025 at 9:18 AM CEST, Roger Pau Monné wrote:
>>>>> On Thu, Jul 17, 2025 at 07:58:24PM +0200, Alejandro Vallejo wrote:
>>>>>> Later patches will keep refactoring create_dom0()
>>>>>> until it can create arbitrary domains. This is one
>>>>>> small step in that direction.
>>>>>>
>>>>>> Signed-off-by: Alejandro Vallejo <alejandro.garciavallejo@amd.com>
>>>>>> ---
>>>>>>   xen/arch/x86/setup.c | 3 ++-
>>>>>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>>>>>> index c6890669b9..6943ffba79 100644
>>>>>> --- a/xen/arch/x86/setup.c
>>>>>> +++ b/xen/arch/x86/setup.c
>>>>>> @@ -1054,7 +1054,8 @@ static struct domain *__init create_dom0(struct boot_info *bi)
>>>>>>       if ( IS_ERR(d) )
>>>>>>           panic("Error creating d%u: %ld\n", bd->domid, PTR_ERR(d));
>>>>>>   
>>>>>> -    init_dom0_cpuid_policy(d);
>>>>>> +    if ( pv_shim || d->cdf & (CDF_privileged | CDF_hardware) )
>>>>>
>>>>> You possibly want this to be:
>>>>>
>>>>> (d->cdf & (CDF_privileged | CDF_hardware)) == (CDF_privileged | CDF_hardware)
>>>>>
>>>>> To ensure the contents of dom0_cpuid_cmdline is only applied to dom0,
>>>>> and not to the hardware or control domains.  I assume it should be
>>>>> possible to pass a different set of cpuid options for the hardware vs
>>>>> the control domains.
>>>>>
>>>>> Thanks, Roger.
>>>>
>>>> Why only a hwdom+ctldom, surely a single hwdom should get it too.
>>>
>>> hm, not really I think: a late hardware domain would get any custom
>>> cpuid options from the toolstack that created it, or in the
>>> hyperlaunch case from the provided configuration, but not from the
>>> dom0-cpuid command line option I would expect.  Otherwise you have two
>>> different sources of cpuid options, the inheritance from dom0-cpuid,
>>> plus whatever is provided from the hardware domain configuration.
>> 
>> Yes, this has been a sticking point for me and never got any good 
>> answers thus far. Should the dom0 related xen command line options only 
>> apply when not booting via hyperlaunch. If the answer is no, then you're 
>> in this area with some dom0 options that really are applicable to hwdom 
>> vs ctldom and vice-a-versa. Some could even be suggested to apply to 
>> both. And then, I don't believe there really is a consensus one which 
>> options apply to which domains. Over the years working on this, I have 
>> been an advocate that commandline adjustments allow for quicker 
>> troubleshooting by the user/administrator.

>> In the last version of the multidomain construction RFC, I am growing more
>> and more to advocate for my initial proposition, that dom0 options only
>> apply when not using  hyperlaunch.

I agree. It simplifies everything a ton, and it's far less confusing to know
ultimate settings, which in a predefined initial system definition is important.

>
> With the hyperlaunch plans, is there something that's still properly
> "Dom0", perhaps under certain conditions? That (and only that) is
> where I would see respective command line options to apply. IOW no
> more than one specific domain (i.e. in particular not to both hwdom
> and ctldom, when they're separate). In cases when respective options
> are entirely ignored, I think some kind of warning would want issuing.
>
> Jan

The problem is that lines are blurred. A ctldtdom + hwdom + xsdom with domid0
is clearly a dom0. Is it still a dom0 when there's no xenstore? What about when
it's not privileged? What about a ctldom + hwdom + xsdom with domid3? What about
dom0_mem options when some domains have already been constructed and available
memory is less than total host memory?

Also if a domain is or isn't dom0 depending on whether a certain other domain
exists makes things confusing. You have a DTB+commandline and get a behaviour,
then add a domain and you get another behaviour on the first one, even when you
didn't touch its configuration.

My general view after a while experimenting with the full series is to _not_ use
the dom0 command line, as Daniel mentions. The simplifying effect of not looking
at (e.g) dom0_mem is staggering.

There's exceptions. nmi=dom0 should be renamed to nmi=hwdom (if anything,
because that's exactly what it does even with late hwdom), but anything with
dom0_X ought to be ignored. Which implies first and foremost moving its uses
outside domain construction and general use.

All dom0_ options ought to be parsed and used from __init functions before
construct_dom0(), and construct_dom0 ought to depend strictly on information
in boot_domain + domain.

Only then we'll have sanity.

Cheers
Alejandro

On 30.07.2025 11:48, Alejandro Vallejo wrote:
> On Wed Jul 30, 2025 at 9:48 AM CEST, Jan Beulich wrote:
>> On 29.07.2025 23:29, Daniel P. Smith wrote:
>>> On 7/25/25 06:56, Roger Pau Monné wrote:
>>>> On Fri, Jul 25, 2025 at 12:02:18PM +0200, Alejandro Vallejo wrote:
>>>>> On Wed Jul 23, 2025 at 9:18 AM CEST, Roger Pau Monné wrote:
>>>>>> On Thu, Jul 17, 2025 at 07:58:24PM +0200, Alejandro Vallejo wrote:
>>>>>>> Later patches will keep refactoring create_dom0()
>>>>>>> until it can create arbitrary domains. This is one
>>>>>>> small step in that direction.
>>>>>>>
>>>>>>> Signed-off-by: Alejandro Vallejo <alejandro.garciavallejo@amd.com>
>>>>>>> ---
>>>>>>>   xen/arch/x86/setup.c | 3 ++-
>>>>>>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>>>>>>
>>>>>>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>>>>>>> index c6890669b9..6943ffba79 100644
>>>>>>> --- a/xen/arch/x86/setup.c
>>>>>>> +++ b/xen/arch/x86/setup.c
>>>>>>> @@ -1054,7 +1054,8 @@ static struct domain *__init create_dom0(struct boot_info *bi)
>>>>>>>       if ( IS_ERR(d) )
>>>>>>>           panic("Error creating d%u: %ld\n", bd->domid, PTR_ERR(d));
>>>>>>>   
>>>>>>> -    init_dom0_cpuid_policy(d);
>>>>>>> +    if ( pv_shim || d->cdf & (CDF_privileged | CDF_hardware) )
>>>>>>
>>>>>> You possibly want this to be:
>>>>>>
>>>>>> (d->cdf & (CDF_privileged | CDF_hardware)) == (CDF_privileged | CDF_hardware)
>>>>>>
>>>>>> To ensure the contents of dom0_cpuid_cmdline is only applied to dom0,
>>>>>> and not to the hardware or control domains.  I assume it should be
>>>>>> possible to pass a different set of cpuid options for the hardware vs
>>>>>> the control domains.
>>>>>>
>>>>>> Thanks, Roger.
>>>>>
>>>>> Why only a hwdom+ctldom, surely a single hwdom should get it too.
>>>>
>>>> hm, not really I think: a late hardware domain would get any custom
>>>> cpuid options from the toolstack that created it, or in the
>>>> hyperlaunch case from the provided configuration, but not from the
>>>> dom0-cpuid command line option I would expect.  Otherwise you have two
>>>> different sources of cpuid options, the inheritance from dom0-cpuid,
>>>> plus whatever is provided from the hardware domain configuration.
>>>
>>> Yes, this has been a sticking point for me and never got any good 
>>> answers thus far. Should the dom0 related xen command line options only 
>>> apply when not booting via hyperlaunch. If the answer is no, then you're 
>>> in this area with some dom0 options that really are applicable to hwdom 
>>> vs ctldom and vice-a-versa. Some could even be suggested to apply to 
>>> both. And then, I don't believe there really is a consensus one which 
>>> options apply to which domains. Over the years working on this, I have 
>>> been an advocate that commandline adjustments allow for quicker 
>>> troubleshooting by the user/administrator.
> 
>>> In the last version of the multidomain construction RFC, I am growing more
>>> and more to advocate for my initial proposition, that dom0 options only
>>> apply when not using  hyperlaunch.
> 
> I agree. It simplifies everything a ton, and it's far less confusing to know
> ultimate settings, which in a predefined initial system definition is important.
> 
>>
>> With the hyperlaunch plans, is there something that's still properly
>> "Dom0", perhaps under certain conditions? That (and only that) is
>> where I would see respective command line options to apply. IOW no
>> more than one specific domain (i.e. in particular not to both hwdom
>> and ctldom, when they're separate). In cases when respective options
>> are entirely ignored, I think some kind of warning would want issuing.
> 
> The problem is that lines are blurred. A ctldtdom + hwdom + xsdom with domid0
> is clearly a dom0. Is it still a dom0 when there's no xenstore? What about when
> it's not privileged? What about a ctldom + hwdom + xsdom with domid3? What about
> dom0_mem options when some domains have already been constructed and available
> memory is less than total host memory?

Well, this is what needs determining before we actually move in any (unclear)
direction. And we need to keep in mind that people used to infer certain
things from domain ID being 0. 

> Also if a domain is or isn't dom0 depending on whether a certain other domain
> exists makes things confusing. You have a DTB+commandline and get a behaviour,
> then add a domain and you get another behaviour on the first one, even when you
> didn't touch its configuration.
> 
> My general view after a while experimenting with the full series is to _not_ use
> the dom0 command line, as Daniel mentions. The simplifying effect of not looking
> at (e.g) dom0_mem is staggering.

Which likely would imply not to create any domain with ID 0.

Jan

> There's exceptions. nmi=dom0 should be renamed to nmi=hwdom (if anything,
> because that's exactly what it does even with late hwdom), but anything with
> dom0_X ought to be ignored. Which implies first and foremost moving its uses
> outside domain construction and general use.
> 
> All dom0_ options ought to be parsed and used from __init functions before
> construct_dom0(), and construct_dom0 ought to depend strictly on information
> in boot_domain + domain.
> 
> Only then we'll have sanity.
> 
> Cheers
> Alejandro

On Wed Jul 30, 2025 at 11:58 AM CEST, Jan Beulich wrote:
> On 30.07.2025 11:48, Alejandro Vallejo wrote:
>> On Wed Jul 30, 2025 at 9:48 AM CEST, Jan Beulich wrote:
>>> On 29.07.2025 23:29, Daniel P. Smith wrote:
>>>> On 7/25/25 06:56, Roger Pau Monné wrote:
>>>>> On Fri, Jul 25, 2025 at 12:02:18PM +0200, Alejandro Vallejo wrote:
>>>>>> On Wed Jul 23, 2025 at 9:18 AM CEST, Roger Pau Monné wrote:
>>>>>>> On Thu, Jul 17, 2025 at 07:58:24PM +0200, Alejandro Vallejo wrote:
>>>>>>>> Later patches will keep refactoring create_dom0()
>>>>>>>> until it can create arbitrary domains. This is one
>>>>>>>> small step in that direction.
>>>>>>>>
>>>>>>>> Signed-off-by: Alejandro Vallejo <alejandro.garciavallejo@amd.com>
>>>>>>>> ---
>>>>>>>>   xen/arch/x86/setup.c | 3 ++-
>>>>>>>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>>>>>>>
>>>>>>>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>>>>>>>> index c6890669b9..6943ffba79 100644
>>>>>>>> --- a/xen/arch/x86/setup.c
>>>>>>>> +++ b/xen/arch/x86/setup.c
>>>>>>>> @@ -1054,7 +1054,8 @@ static struct domain *__init create_dom0(struct boot_info *bi)
>>>>>>>>       if ( IS_ERR(d) )
>>>>>>>>           panic("Error creating d%u: %ld\n", bd->domid, PTR_ERR(d));
>>>>>>>>   
>>>>>>>> -    init_dom0_cpuid_policy(d);
>>>>>>>> +    if ( pv_shim || d->cdf & (CDF_privileged | CDF_hardware) )
>>>>>>>
>>>>>>> You possibly want this to be:
>>>>>>>
>>>>>>> (d->cdf & (CDF_privileged | CDF_hardware)) == (CDF_privileged | CDF_hardware)
>>>>>>>
>>>>>>> To ensure the contents of dom0_cpuid_cmdline is only applied to dom0,
>>>>>>> and not to the hardware or control domains.  I assume it should be
>>>>>>> possible to pass a different set of cpuid options for the hardware vs
>>>>>>> the control domains.
>>>>>>>
>>>>>>> Thanks, Roger.
>>>>>>
>>>>>> Why only a hwdom+ctldom, surely a single hwdom should get it too.
>>>>>
>>>>> hm, not really I think: a late hardware domain would get any custom
>>>>> cpuid options from the toolstack that created it, or in the
>>>>> hyperlaunch case from the provided configuration, but not from the
>>>>> dom0-cpuid command line option I would expect.  Otherwise you have two
>>>>> different sources of cpuid options, the inheritance from dom0-cpuid,
>>>>> plus whatever is provided from the hardware domain configuration.
>>>>
>>>> Yes, this has been a sticking point for me and never got any good 
>>>> answers thus far. Should the dom0 related xen command line options only 
>>>> apply when not booting via hyperlaunch. If the answer is no, then you're 
>>>> in this area with some dom0 options that really are applicable to hwdom 
>>>> vs ctldom and vice-a-versa. Some could even be suggested to apply to 
>>>> both. And then, I don't believe there really is a consensus one which 
>>>> options apply to which domains. Over the years working on this, I have 
>>>> been an advocate that commandline adjustments allow for quicker 
>>>> troubleshooting by the user/administrator.
>> 
>>>> In the last version of the multidomain construction RFC, I am growing more
>>>> and more to advocate for my initial proposition, that dom0 options only
>>>> apply when not using  hyperlaunch.
>> 
>> I agree. It simplifies everything a ton, and it's far less confusing to know
>> ultimate settings, which in a predefined initial system definition is important.
>> 
>>>
>>> With the hyperlaunch plans, is there something that's still properly
>>> "Dom0", perhaps under certain conditions? That (and only that) is
>>> where I would see respective command line options to apply. IOW no
>>> more than one specific domain (i.e. in particular not to both hwdom
>>> and ctldom, when they're separate). In cases when respective options
>>> are entirely ignored, I think some kind of warning would want issuing.
>> 
>> The problem is that lines are blurred. A ctldtdom + hwdom + xsdom with domid0
>> is clearly a dom0. Is it still a dom0 when there's no xenstore? What about when
>> it's not privileged? What about a ctldom + hwdom + xsdom with domid3? What about
>> dom0_mem options when some domains have already been constructed and available
>> memory is less than total host memory?
>
> Well, this is what needs determining before we actually move in any (unclear)
> direction. And we need to keep in mind that people used to infer certain
> things from domain ID being 0. 
>
>> Also if a domain is or isn't dom0 depending on whether a certain other domain
>> exists makes things confusing. You have a DTB+commandline and get a behaviour,
>> then add a domain and you get another behaviour on the first one, even when you
>> didn't touch its configuration.
>> 
>> My general view after a while experimenting with the full series is to _not_ use
>> the dom0 command line, as Daniel mentions. The simplifying effect of not looking
>> at (e.g) dom0_mem is staggering.
>
> Which likely would imply not to create any domain with ID 0.
>
> Jan

I'm currently of that opinion as well, but that doesn't preclude that the
codebase must stop assuming there _is_ a dom0. There might not be any, or it
might not have domid 0. The scheduler, the io bitmap setter and some late hwdom
code still assume domids.

I have some patches to fix at least _that_ side of the situation, but I haven't
had time to test them in any credible form.

The fact that dom0 construction logic should not be looking at opt_* variables
still stands, IMO.

Cheers,
Alejandro

On 17.07.2025 19:58, Alejandro Vallejo wrote:
> Later patches will keep refactoring create_dom0()
> until it can create arbitrary domains. This is one
> small step in that direction.
> 
> Signed-off-by: Alejandro Vallejo <alejandro.garciavallejo@amd.com>

As with the other patch - we first need to settle on principles. And
then ...

> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -1054,7 +1054,8 @@ static struct domain *__init create_dom0(struct boot_info *bi)
>      if ( IS_ERR(d) )
>          panic("Error creating d%u: %ld\n", bd->domid, PTR_ERR(d));
>  
> -    init_dom0_cpuid_policy(d);
> +    if ( pv_shim || d->cdf & (CDF_privileged | CDF_hardware) )
> +        init_dom0_cpuid_policy(d);

... we also want to avoid confusing naming (i.e. the "dom0" in here).

Finally, nit: The & expression wants parenthesizing against the ||.

Jan

On Thu, Jul 17, 2025 at 07:58:24PM +0200, Alejandro Vallejo wrote:
> Later patches will keep refactoring create_dom0()
> until it can create arbitrary domains. This is one
> small step in that direction.
> 
> Signed-off-by: Alejandro Vallejo <alejandro.garciavallejo@amd.com>

Reviewed-by: Denis Mukhin <dmukhin@ford.com> 

> ---
>  xen/arch/x86/setup.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index c6890669b9..6943ffba79 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -1054,7 +1054,8 @@ static struct domain *__init create_dom0(struct boot_info *bi)
>      if ( IS_ERR(d) )
>          panic("Error creating d%u: %ld\n", bd->domid, PTR_ERR(d));
> 
> -    init_dom0_cpuid_policy(d);
> +    if ( pv_shim || d->cdf & (CDF_privileged | CDF_hardware) )
> +        init_dom0_cpuid_policy(d);
> 
>      if ( alloc_dom0_vcpu0(d) == NULL )
>          panic("Error creating %pdv0\n", d);
> 
> base-commit: 55719030b0bb0069fc8b57cd808dc98dc9d39add
> --
> 2.43.0
> 
>