The code building PVH Dom0 made use of sequences of P2M changes
which are disallowed as of XSA-378. First of all population of the
first Mb of memory needs to be redone. Then, largely as a
workaround, checking introduced by XSA-378 needs to be slightly
relaxed.

Note that with these adjustments I get Dom0 to start booting on my
development system, but the Dom0 kernel then gets stuck. Since it
was the first time for me to try PVH Dom0 in this context (see
below for why I was hesitant), I cannot tell yet whether this is
due further fallout from the XSA, or some further unrelated
problem. Dom0's BSP makes it all the way through to entering the
idle loop while all APs are still in VPF_down.

[And there was another rather basic issue to fight first (patch
 was submitted separately as RFC): vPCI wasn't aware of hidden PCI
 devices, hitting an ASSERT(). Obviously I couldn't afford not
 having a functioning serial console.]

In the course I ran into an oom condition while populating Dom0's
RAM. Hence next some re-work of dom0_compute_nr_pages(). In turn
in the course of putting that together I did notice that PV Dom0,
when run in shadow mode, wouldn't have its shadow allocation
properly set.

Finally make debug key '0' work at least partially for PVH Dom0
and limit the amount of log output for offline (down) vCPU-s.

1: P2M: relax guarding of MMIO entries
2: P2M: relax permissions of PVH Dom0's MMIO entries
3: PVH: improve Dom0 memory size calculation
4: PV: properly set shadow allocation for Dom0
5: PVH: actually show Dom0's register state from debug key '0'
6: HVM: skip offline vCPU-s when dumping VMCBs/VMCSes

Jan

On 02.09.2021 10:30, Jan Beulich wrote:
> The code building PVH Dom0 made use of sequences of P2M changes
> which are disallowed as of XSA-378. First of all population of the
> first Mb of memory needs to be redone. Then, largely as a
> workaround, checking introduced by XSA-378 needs to be slightly
> relaxed.
> 
> Note that with these adjustments I get Dom0 to start booting on my
> development system, but the Dom0 kernel then gets stuck. Since it
> was the first time for me to try PVH Dom0 in this context (see
> below for why I was hesitant), I cannot tell yet whether this is
> due further fallout from the XSA, or some further unrelated
> problem. Dom0's BSP makes it all the way through to entering the
> idle loop while all APs are still in VPF_down.

This last part of the mystery is now solved: By cloning from my PV
.config, I've inherited the X86_X2APIC=n that I have there. Yet
ACPI's MADT gets populated with only x2APIC entries when building
Dom0, which such a kernel would mostly ignore (except for logging).
IOW in a way this was indeed a missing select, except that what's
needed wouldn't quite work yet:

--- a/arch/x86/xen/Kconfig
+++ b/arch/x86/xen/Kconfig
@@ -83,6 +83,6 @@ config XEN_PVH
 	bool "Xen PVH guest support"
 	depends on XEN && XEN_PVHVM && ACPI
 	select PVH
-	def_bool n
+	select X86_X2APIC if XEN_DOM0
 	help
 	  Support for running as a Xen PVH guest.

This is because, as mentioned, XEN_DOM0 gets turned off when XEN_PV
is off. Whereas x2APIC isn't strictly needed for DomU afaics
(MADT gets populated with LAPIC entries only), so the "select" also
shouldn't be unconditional.

While likely no-one will really care, I'd like to note that this
effectively means a 32-bit Linux PVH Dom0 would be impossible, as
X86_X2APIC depends on X86_64. This may want reflecting in the
eventual adjustment to the XEN_DOM0 dependencies.

Btw, I've meanwhile also checked timers: They're active and get
updated while Dom0 is in this funny idle state.

Jan

On 02.09.21 13:42, Jan Beulich wrote:
> On 02.09.2021 10:30, Jan Beulich wrote:
>> The code building PVH Dom0 made use of sequences of P2M changes
>> which are disallowed as of XSA-378. First of all population of the
>> first Mb of memory needs to be redone. Then, largely as a
>> workaround, checking introduced by XSA-378 needs to be slightly
>> relaxed.
>>
>> Note that with these adjustments I get Dom0 to start booting on my
>> development system, but the Dom0 kernel then gets stuck. Since it
>> was the first time for me to try PVH Dom0 in this context (see
>> below for why I was hesitant), I cannot tell yet whether this is
>> due further fallout from the XSA, or some further unrelated
>> problem. Dom0's BSP makes it all the way through to entering the
>> idle loop while all APs are still in VPF_down.
> 
> This last part of the mystery is now solved: By cloning from my PV
> .config, I've inherited the X86_X2APIC=n that I have there. Yet
> ACPI's MADT gets populated with only x2APIC entries when building
> Dom0, which such a kernel would mostly ignore (except for logging).
> IOW in a way this was indeed a missing select, except that what's
> needed wouldn't quite work yet:
> 
> --- a/arch/x86/xen/Kconfig
> +++ b/arch/x86/xen/Kconfig
> @@ -83,6 +83,6 @@ config XEN_PVH
>   	bool "Xen PVH guest support"
>   	depends on XEN && XEN_PVHVM && ACPI
>   	select PVH
> -	def_bool n
> +	select X86_X2APIC if XEN_DOM0
>   	help
>   	  Support for running as a Xen PVH guest.
> 
> This is because, as mentioned, XEN_DOM0 gets turned off when XEN_PV
> is off. Whereas x2APIC isn't strictly needed for DomU afaics
> (MADT gets populated with LAPIC entries only), so the "select" also
> shouldn't be unconditional.

Correct.

We should rename XEN_DOM0 to XEN_DOM0_PV, and add a "real" XEN_DOM0.

> While likely no-one will really care, I'd like to note that this
> effectively means a 32-bit Linux PVH Dom0 would be impossible, as
> X86_X2APIC depends on X86_64. This may want reflecting in the
> eventual adjustment to the XEN_DOM0 dependencies.

Indeed.


Juergen

On 02.09.2021 14:15, Juergen Gross wrote:
> On 02.09.21 13:42, Jan Beulich wrote:
>> On 02.09.2021 10:30, Jan Beulich wrote:
>>> The code building PVH Dom0 made use of sequences of P2M changes
>>> which are disallowed as of XSA-378. First of all population of the
>>> first Mb of memory needs to be redone. Then, largely as a
>>> workaround, checking introduced by XSA-378 needs to be slightly
>>> relaxed.
>>>
>>> Note that with these adjustments I get Dom0 to start booting on my
>>> development system, but the Dom0 kernel then gets stuck. Since it
>>> was the first time for me to try PVH Dom0 in this context (see
>>> below for why I was hesitant), I cannot tell yet whether this is
>>> due further fallout from the XSA, or some further unrelated
>>> problem. Dom0's BSP makes it all the way through to entering the
>>> idle loop while all APs are still in VPF_down.
>>
>> This last part of the mystery is now solved: By cloning from my PV
>> .config, I've inherited the X86_X2APIC=n that I have there. Yet
>> ACPI's MADT gets populated with only x2APIC entries when building
>> Dom0, which such a kernel would mostly ignore (except for logging).
>> IOW in a way this was indeed a missing select, except that what's
>> needed wouldn't quite work yet:
>>
>> --- a/arch/x86/xen/Kconfig
>> +++ b/arch/x86/xen/Kconfig
>> @@ -83,6 +83,6 @@ config XEN_PVH
>>   	bool "Xen PVH guest support"
>>   	depends on XEN && XEN_PVHVM && ACPI
>>   	select PVH
>> -	def_bool n
>> +	select X86_X2APIC if XEN_DOM0
>>   	help
>>   	  Support for running as a Xen PVH guest.
>>
>> This is because, as mentioned, XEN_DOM0 gets turned off when XEN_PV
>> is off. Whereas x2APIC isn't strictly needed for DomU afaics
>> (MADT gets populated with LAPIC entries only), so the "select" also
>> shouldn't be unconditional.
> 
> Correct.
> 
> We should rename XEN_DOM0 to XEN_DOM0_PV, and add a "real" XEN_DOM0.

Actually, as I have just found, the lack of XEN_DOM0 _is_ a problem:
xen_initial_domain() gets hardcoded to 0 without it. I'll have to
make a(nother) patch along the lines of what you suggest; hvc-xen
and "earlyprintk=xen" also don't really work together, because the
first thing xenboot_write_console() does is a !xen_pv_domain() check.

Jan