From: Tu Dinh <ngoc-tu.dinh@vates.tech>

Intel model-specific last branch records (LBRs) were replaced by
architectural LBRs (see Chapter 20 of Intel SDM volume 3B). This
patchset implements virtual LBRs for HVM guests using Intel's "load
guest IA32_LBR_CTL" and "clear IA32_LBR_CTL" VMX controls.

Add the necessary CPUID and VMX feature checks into Xen. Note that in
this patchset, MSR_IA32_LASTBRANCH_DEPTH is only allowed to be equal to
that of the host's.

Tu Dinh (4):
  x86: Add Intel architectural LBR featureset bits
  x86: Add architectural LBR declarations
  x86: Adjust arch LBR CPU policy
  x86/vmx: Virtualize architectural LBRs

 tools/libs/guest/xg_cpuid_x86.c             |   2 +-
 tools/misc/xen-cpuid.c                      |   3 +
 xen/arch/x86/cpu-policy.c                   |  39 +++
 xen/arch/x86/cpu/common.c                   |   7 +
 xen/arch/x86/hvm/vmx/vmcs.c                 |  11 +-
 xen/arch/x86/hvm/vmx/vmx.c                  | 269 ++++++++++++++------
 xen/arch/x86/include/asm/hvm/vmx/vmcs.h     |   8 +
 xen/arch/x86/include/asm/msr-index.h        |  11 +
 xen/include/public/arch-x86/cpufeatureset.h |  28 +-
 xen/include/xen/lib/x86/cpu-policy.h        |  38 ++-
 xen/lib/x86/cpuid.c                         |   6 +
 11 files changed, 339 insertions(+), 83 deletions(-)

-- 
2.43.0



Ngoc Tu Dinh | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

On 18.11.2024 09:49, ngoc-tu.dinh@vates.tech wrote:
> From: Tu Dinh <ngoc-tu.dinh@vates.tech>
> 
> Intel model-specific last branch records (LBRs) were replaced by
> architectural LBRs (see Chapter 20 of Intel SDM volume 3B). This
> patchset implements virtual LBRs for HVM guests using Intel's "load
> guest IA32_LBR_CTL" and "clear IA32_LBR_CTL" VMX controls.
> 
> Add the necessary CPUID and VMX feature checks into Xen. Note that in
> this patchset, MSR_IA32_LASTBRANCH_DEPTH is only allowed to be equal to
> that of the host's.
> 
> Tu Dinh (4):
>   x86: Add Intel architectural LBR featureset bits
>   x86: Add architectural LBR declarations
>   x86: Adjust arch LBR CPU policy
>   x86/vmx: Virtualize architectural LBRs
> 
>  tools/libs/guest/xg_cpuid_x86.c             |   2 +-
>  tools/misc/xen-cpuid.c                      |   3 +
>  xen/arch/x86/cpu-policy.c                   |  39 +++
>  xen/arch/x86/cpu/common.c                   |   7 +
>  xen/arch/x86/hvm/vmx/vmcs.c                 |  11 +-
>  xen/arch/x86/hvm/vmx/vmx.c                  | 269 ++++++++++++++------
>  xen/arch/x86/include/asm/hvm/vmx/vmcs.h     |   8 +
>  xen/arch/x86/include/asm/msr-index.h        |  11 +
>  xen/include/public/arch-x86/cpufeatureset.h |  28 +-
>  xen/include/xen/lib/x86/cpu-policy.h        |  38 ++-
>  xen/lib/x86/cpuid.c                         |   6 +
>  11 files changed, 339 insertions(+), 83 deletions(-)

Looking over just the files touched: No change to XSAVE logic at all?

Jan

On 18/11/2024 09:52, Jan Beulich wrote:
> Looking over just the files touched: No change to XSAVE logic at all?

XSAVE is hidden behind a new IA32_XSS bit. I'll try to implement that next.


Ngoc Tu Dinh | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

On 18/11/2024 9:13 am, Tu Dinh wrote:
> On 18/11/2024 09:52, Jan Beulich wrote:
>> Looking over just the files touched: No change to XSAVE logic at all?
> XSAVE is hidden behind a new IA32_XSS bit. I'll try to implement that next.

It's rather more severe than that.

Without XSAVE support, Xen can't context-switch the LBR state when vCPUs
are scheduled in and out.  (In patch 4 you seem to have copied the
legacy way, which is extremely expensive.)

Architecturally, ARCH_LBR depends on XSAVES so OSes can context switch
it easily(ish) per thread.

There's also a reason why we haven't got this working yet.  There are a
couple of areas of prerequisite work which need addressing before XSS
can be enabled properly.

If you're willing to tackle this, then I can explain what needs doing,
and in roughly which order.

~Andrew

Hi Andrew,

On 18/11/2024 10:52, Andrew Cooper wrote:
> On 18/11/2024 9:13 am, Tu Dinh wrote:
>> On 18/11/2024 09:52, Jan Beulich wrote:
>>> Looking over just the files touched: No change to XSAVE logic at all?
>> XSAVE is hidden behind a new IA32_XSS bit. I'll try to implement that next.
> 
> It's rather more severe than that.
> 
> Without XSAVE support, Xen can't context-switch the LBR state when vCPUs
> are scheduled in and out.  (In patch 4 you seem to have copied the
> legacy way, which is extremely expensive.)
> 
> Architecturally, ARCH_LBR depends on XSAVES so OSes can context switch
> it easily(ish) per thread.
> 
> There's also a reason why we haven't got this working yet.  There are a
> couple of areas of prerequisite work which need addressing before XSS
> can be enabled properly.
> 
> If you're willing to tackle this, then I can explain what needs doing,
> and in roughly which order.
> 
> ~Andrew

Following the community call yesterday, I'd like to clarify my 
understanding of the issue:

- Firstly, virtual XSS support for architectural LBR must be enabled. I 
noticed that XSS is already implemented, just not enabled; barring the 
LBR format issues below, are there any other issues with the current XSS 
implementation?

- There are LBR format differences between some cores of the same CPU 
(e.g. in Intel hybrid CPUs: P-cores use effective IP while E-cores use 
linear IP). These differences are expected to be handled by 
XSAVES/XRSTORS. However, Xen would have to make sure that LBR MSRs are 
saved/restored by XSS instead of by manually poking MSRs.

- A related issue is handling the compressed XSAVE format for migration 
streams. Xen currently expands/compacts XSAVE format manually during 
migration; are there any concerns with arch LBR breaking the XSAVE 
migration logic?

My understanding is that as long as we don't manually poke the LBR state 
component, and that LBR state size remains consistent across hybrid 
cores in the same CPU (which it should be for XSAVE compatibility), 
there should be no concern with the XSAVE state itself. However, Xen 
must check CPU features of both sides during migration to make sure that 
XSAVE states are compatible, which is more complex in migrations 
involving hosts with hybrid CPUs.

Please tell me if I'm missing any potential issues.

Thanks,


Ngoc Tu Dinh | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

On 06.12.2024 15:16, Tu Dinh wrote:
> On 18/11/2024 10:52, Andrew Cooper wrote:
>> On 18/11/2024 9:13 am, Tu Dinh wrote:
>>> On 18/11/2024 09:52, Jan Beulich wrote:
>>>> Looking over just the files touched: No change to XSAVE logic at all?
>>> XSAVE is hidden behind a new IA32_XSS bit. I'll try to implement that next.
>>
>> It's rather more severe than that.
>>
>> Without XSAVE support, Xen can't context-switch the LBR state when vCPUs
>> are scheduled in and out.  (In patch 4 you seem to have copied the
>> legacy way, which is extremely expensive.)
>>
>> Architecturally, ARCH_LBR depends on XSAVES so OSes can context switch
>> it easily(ish) per thread.
>>
>> There's also a reason why we haven't got this working yet.  There are a
>> couple of areas of prerequisite work which need addressing before XSS
>> can be enabled properly.
>>
>> If you're willing to tackle this, then I can explain what needs doing,
>> and in roughly which order.
> 
> Following the community call yesterday, I'd like to clarify my 
> understanding of the issue:
> 
> - Firstly, virtual XSS support for architectural LBR must be enabled. I 
> noticed that XSS is already implemented, just not enabled; barring the 
> LBR format issues below, are there any other issues with the current XSS 
> implementation?

At the very least, as with any code that exists be is effectively unused
(and iiuc even DCE-ed by the compiler in at least some of the cases) it'll
need to be determined whether what we have actually works. It seems more
likely than not that some corrections here and there may be needed.

> - There are LBR format differences between some cores of the same CPU 
> (e.g. in Intel hybrid CPUs: P-cores use effective IP while E-cores use 
> linear IP). These differences are expected to be handled by 
> XSAVES/XRSTORS. However, Xen would have to make sure that LBR MSRs are 
> saved/restored by XSS instead of by manually poking MSRs.

How could XSAVES/XRSTORS take care of format differences? Something that
was saved on one type of core and is to be restored on the other type of
core would require CS.base at the time the LBR was originally written
to be able to convert effective <=> linear IP.

> - A related issue is handling the compressed XSAVE format for migration 
> streams. Xen currently expands/compacts XSAVE format manually during 
> migration; are there any concerns with arch LBR breaking the XSAVE 
> migration logic?

That ought to be fine, I think, barring the first point I made above.

Jan

On 18/11/2024 10:52, Andrew Cooper wrote:
> There's also a reason why we haven't got this working yet.  There are a
> couple of areas of prerequisite work which need addressing before XSS
> can be enabled properly.
> 
> If you're willing to tackle this, then I can explain what needs doing,
> and in roughly which order.

I would appreciate explanations of the pending XSS issues.


Ngoc Tu Dinh | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech