This v2 is essentially a resend of the v1 series. I took over this work
from Weijiang, so I added my Signed-off-by and incremented the version
number. This repost is to seek more feedback on this work, which is a
dependency for CET KVM support. In turn, CET KVM support is a dependency
for both FRED KVM support and CET AMD support.

==Background==

This series spins off from CET KVM virtualization enabling series [1].
The purpose is to get these preparation work resolved ahead of KVM part
landing. There was a discussion about introducing CET supervisor state
support [2] [3].

CET supervisor state, i.e., IA32_PL{0,1,2}_SSP, are xsave-managed MSRs,
it can be enabled via IA32_XSS[bit 12]. KVM relies on host side CET
supervisor state support to fully enable guest CET MSR contents storage.
The benefits are: 1) No need to manually save/restore the 3 MSRs when
vCPU fpu context is sched in/out. 2) Omit manually swapping the three
MSRs at VM-Exit/VM-Entry for guest/host. 3) Make guest CET user/supervisor
states managed in a consistent manner within host kernel FPU framework.

==Solution==

This series tries to:
1) Fix existing issue regarding enabling guest supervisor states support.
2) Add CET supervisor state support in core kernel.
3) Introduce new FPU config for guest fpstate setup.

With the preparation work landed, for guest fpstate, we have xstate_bv[12]
== xcomp_bv[12] == 1 and CET supervisor state is saved/reloaded when
xsaves/xrstors executes on guest fpstate.
For non-guest/normal fpstate, we have xstate_bv[12] == xcomp_bv[12] == 0,
then HW can optimize xsaves/xrstors operations.

==Performance==

We measured context-switching performance with the benchmark [4] in following
three cases.

case 1: the baseline. i.e., this series isn't applied
case 2: baseline + this series. CET-S space is allocated for guest fpu only.
case 3: baseline + allocate CET-S space for all tasks. Hardware init
        optimization avoids writing out CET-S space on each XSAVES.

the data are as follows

case |IA32_XSS[12] | Space | RFBM[12] | Drop%	
-----+-------------+-------+----------+------
  1  |	   0	   | None  |	0     |  0.0%
  2  |	   1	   | None  |	0     |  0.2%
  3  |	   1	   | 24B?  |	1     |  0.2%

Case 2 and 3 have no difference in performnace. But case 2 is preferred because
it can save 24B of CET-S space for all non-vCPU threads with just a one-line
change in patch 3:

+	fpu_kernel_cfg.default_features &= ~XFEATURE_MASK_KERNEL_DYNAMIC;

Patch 4 and 5 have their own merits. Regardless of the approach we take, using
different FPU configuration settings for the guest and the kernel improves
readability, decouples them from each other, and arguably enhances
extensibility.

==Changelog==

v1->v2:
 - rebase onto the latest kvm-x86/next
 - Add performance data to the cover-letter
 - v1: https://lore.kernel.org/kvm/73802bff-833c-4233-9a5b-88af0d062c82@intel.com/

==Organization==

Patch1: Preserve guest supervisor xfeatures in __state_perm.
Patch2: Enable CET supervisor xstate support.
Patch3: Introduce kernel dynamic xfeature set.
Patch4: Initialize fpu_guest_cfg settings.
Patch5: Create guest fpstate with fpu_guest_cfg.
Patch6: Check invalid fpstate config before executes xsaves.

[1]: https://lore.kernel.org/all/20240219074733.122080-1-weijiang.yang@intel.com/
[2]: https://lore.kernel.org/all/ZM1jV3UPL0AMpVDI@google.com/
[3]: https://lore.kernel.org/all/2597a87b-1248-b8ce-ce60-94074bc67ea4@intel.com/
[4]: https://github.com/antonblanchard/will-it-scale/blob/master/tests/context_switch1.c

Sean Christopherson (1):
  x86/fpu/xstate: Always preserve non-user xfeatures/flags in
    __state_perm

Yang Weijiang (5):
  x86/fpu/xstate: Add CET supervisor mode state support
  x86/fpu/xstate: Introduce XFEATURE_MASK_KERNEL_DYNAMIC xfeature set
  x86/fpu/xstate: Introduce fpu_guest_cfg for guest FPU configuration
  x86/fpu/xstate: Create guest fpstate with guest specific config
  x86/fpu/xstate: Warn if CET supervisor state is detected in normal
    fpstate

 arch/x86/include/asm/fpu/types.h  | 16 ++++++++--
 arch/x86/include/asm/fpu/xstate.h | 11 ++++---
 arch/x86/kernel/fpu/core.c        | 53 ++++++++++++++++++++++++-------
 arch/x86/kernel/fpu/xstate.c      | 35 +++++++++++++++-----
 arch/x86/kernel/fpu/xstate.h      |  2 ++
 5 files changed, 90 insertions(+), 27 deletions(-)

-- 
2.46.1

On Tue, Nov 26, 2024 at 06:17:04PM +0800, Chao Gao wrote:
>This v2 is essentially a resend of the v1 series. I took over this work
>from Weijiang, so I added my Signed-off-by and incremented the version
>number. This repost is to seek more feedback on this work, which is a
>dependency for CET KVM support. In turn, CET KVM support is a dependency
>for both FRED KVM support and CET AMD support.
>
>==Background==
>
>This series spins off from CET KVM virtualization enabling series [1].
>The purpose is to get these preparation work resolved ahead of KVM part
>landing. There was a discussion about introducing CET supervisor state
>support [2] [3].

Gentle ping.

On Tue, Nov 26, 2024 at 06:17:04PM +0800, Chao Gao wrote:
>This v2 is essentially a resend of the v1 series. I took over this work
>from Weijiang, so I added my Signed-off-by and incremented the version
>number. This repost is to seek more feedback on this work, which is a
>dependency for CET KVM support. In turn, CET KVM support is a dependency
>for both FRED KVM support and CET AMD support.

This series is primarily for the CET KVM series. Merging it through the tip
tree means this code will not have an actual user until the CET KVM series
is merged. A good proposal from Rick is that x86 maintainers can ack this
series, and then it can be picked up by the KVM maintainers along with the
CET KVM series. Dave, Paolo and Sean, are you okay with this approach?

On Tue, Nov 26, 2024, Chao Gao wrote:
> On Tue, Nov 26, 2024 at 06:17:04PM +0800, Chao Gao wrote:
> >This v2 is essentially a resend of the v1 series. I took over this work
> >from Weijiang, so I added my Signed-off-by and incremented the version
> >number. This repost is to seek more feedback on this work, which is a
> >dependency for CET KVM support. In turn, CET KVM support is a dependency
> >for both FRED KVM support and CET AMD support.
> 
> This series is primarily for the CET KVM series. Merging it through the tip
> tree means this code will not have an actual user until the CET KVM series
> is merged. A good proposal from Rick is that x86 maintainers can ack this
> series, and then it can be picked up by the KVM maintainers along with the
> CET KVM series. Dave, Paolo and Sean, are you okay with this approach?

Boris indicated off-list that he would prefer to take this through tip and give
KVM an immutable branch.  I'm a-ok with either approach.

On 1/10/2025 5:26 PM, Sean Christopherson wrote:
> On Tue, Nov 26, 2024, Chao Gao wrote:
>> On Tue, Nov 26, 2024 at 06:17:04PM +0800, Chao Gao wrote:
>>> This v2 is essentially a resend of the v1 series. I took over this work
>> >from Weijiang, so I added my Signed-off-by and incremented the version
>>> number. This repost is to seek more feedback on this work, which is a
>>> dependency for CET KVM support. In turn, CET KVM support is a dependency
>>> for both FRED KVM support and CET AMD support.
>>
>> This series is primarily for the CET KVM series. Merging it through the tip
>> tree means this code will not have an actual user until the CET KVM series
>> is merged. A good proposal from Rick is that x86 maintainers can ack this
>> series, and then it can be picked up by the KVM maintainers along with the
>> CET KVM series. Dave, Paolo and Sean, are you okay with this approach?
> 
> Boris indicated off-list that he would prefer to take this through tip and give
> KVM an immutable branch.  I'm a-ok with either approach.
> 

Hi Sean and Boris,

At this point because KVM is the only user of this feature, would it
make more sense to take this patch set through KVM x86 tree?

Thanks!
     Xin

On Tue, Mar 04, 2025, Xin Li wrote:
> On 1/10/2025 5:26 PM, Sean Christopherson wrote:
> > On Tue, Nov 26, 2024, Chao Gao wrote:
> > > On Tue, Nov 26, 2024 at 06:17:04PM +0800, Chao Gao wrote:
> > > > This v2 is essentially a resend of the v1 series. I took over this work
> > > >from Weijiang, so I added my Signed-off-by and incremented the version
> > > > number. This repost is to seek more feedback on this work, which is a
> > > > dependency for CET KVM support. In turn, CET KVM support is a dependency
> > > > for both FRED KVM support and CET AMD support.
> > > 
> > > This series is primarily for the CET KVM series. Merging it through the tip
> > > tree means this code will not have an actual user until the CET KVM series
> > > is merged. A good proposal from Rick is that x86 maintainers can ack this
> > > series, and then it can be picked up by the KVM maintainers along with the
> > > CET KVM series. Dave, Paolo and Sean, are you okay with this approach?
> > 
> > Boris indicated off-list that he would prefer to take this through tip and give
> > KVM an immutable branch.  I'm a-ok with either approach.
> > 
> 
> Hi Sean and Boris,
> 
> At this point because KVM is the only user of this feature, would it
> make more sense to take this patch set through KVM x86 tree?

Which tree it goes through is largely irrelevant, it needs explicit acceptance
from the tip tree folks no matter what.

On 1/10/2025 5:26 PM, Sean Christopherson wrote:
> On Tue, Nov 26, 2024, Chao Gao wrote:
>> On Tue, Nov 26, 2024 at 06:17:04PM +0800, Chao Gao wrote:
>>> This v2 is essentially a resend of the v1 series. I took over this work
>> >from Weijiang, so I added my Signed-off-by and incremented the version
>>> number. This repost is to seek more feedback on this work, which is a
>>> dependency for CET KVM support. In turn, CET KVM support is a dependency
>>> for both FRED KVM support and CET AMD support.
>>
>> This series is primarily for the CET KVM series. Merging it through the tip
>> tree means this code will not have an actual user until the CET KVM series
>> is merged. A good proposal from Rick is that x86 maintainers can ack this
>> series, and then it can be picked up by the KVM maintainers along with the
>> CET KVM series. Dave, Paolo and Sean, are you okay with this approach?
> 
> Boris indicated off-list that he would prefer to take this through tip and give
> KVM an immutable branch.  I'm a-ok with either approach.
> 

So is the plan to merge this patch set in the v6.14 cycle?

Thanks!
     Xin