Hello,

This series introduce support for confidential computing along with a
AMD SEV implementation. It also bundles some of the functional
requirements (ASID scheme, ABI, ...) which could be separated if needed.

(I bundled everything in this serie to have a complete coherent serie)

This work receives funding by the Hyper Open X consortium (France 2030).

# Concepts

A confidential guest is a bit special as :
 - its memory is by default encrypted or not directly accessible by the
   hypervisor, thus other domains/dom0 as well; it must be explicitely
   shared by the guest itself
 - so its page-tables are also not accessible

# Implementation

Confidential computing is exposed in a uniform way regardless of actual
implementation (SEV, TDX, RME, ...) through the coco_op hypercall (mostly
for use by the Dom0 toolstack). This interface provides a way to query
informations on the coco platform (support status, features (un)safety,
...), and prepare initial guest memory.
Only HVM domains have support for confidential computing.
(in the future, we may want to have attestation support)

In order to create a confidential computing domain, the process is follow : 
 - create a HVM/PVH domain with XEN_DOMCTL_CDF_coco
 - populate initial memory as usual
 - apply coco_prepare_initial_mem on all initial pages
   (under SEV, this will encrypt memory)

Under xl, it is exposed through the `coco` parameter ("coco = 1").

Xen hypercalls usually use virtual addresses as parameter, which causes issues
when issuing them from a confidential guest (where its memory is usually not
available to the hypervisor e.g encryted). This problem is solved by introducing
a new experimental hypercall ABI ("Physical Address ABI") which don't use virtual
addresses anymore, of course it needs explicit guest support.

## SEV Implementation

Currently, only plain SEV (no ES) is implemented. I would prefer to use SEV-ES
as plain SEV has several flaws like having non-trivial emulation paths and the
hypervisor can break the guest encryption by manipulating its registers (unlike
SEV-ES where the hypervisor has a very controlled view on guest registers).

# Series organization

The first part introduce some non-coco/SEV specific bits.

The second part introduce a physical address ABI, that is required to
make proper hypercalls under a confidential computing guest. That's
something that we should discuss more, but for now, it allows some
minimal guest hypercall support in confidential computing guest.

A patch introduce a ASID management rework (based on Vaishali's work)
required to make SEV work, as in this case the ASID is tied to the guest
encryption key. Which also includes a rework on TLB flushing logic.

Then a general confidential computing infrastructure (not SEV-specific)
along with the AMD SEV implementation.

And some extra patches to workaround some limitations (DF_FLUSH support
and temporary debug tools).

You can find Linux branches with early SEV support (more or less working)
https://github.com/xcp-ng/linux/tree/xen-sev-6.6/
https://github.com/xcp-ng/linux/tree/xen-sev-6.14/

Teddy Astie (16):
  x86/msr: Introduce SYSCFG_MEM_ENCRYPT MSR.
  x86/svm: Move svm_domain structure to svm.h
  x86/hvm: Add support for physical address ABI
  x86/public: Expose physaddr_abi through Xen HVM CPUID leaf
  docs/x86: Document HVM Physical Addresss ABI
  vmx: Introduce vcpu single context VPID invalidation
  x86/hvm: Introduce Xen-wide ASID allocator
  x86/crypto: Introduce AMD PSP driver for SEV
  common: Introduce confidential computing infrastructure
  xl/coco: Introduce confidential computing support
  x86/svm: Introduce NPCTRL VMCB bits
  x86/cpufeature: Introduce SME and SEV-related CPU features
  x86/coco: Introduce AMD-SEV support
  sev/emulate: Handle some non-emulable HVM paths
  HACK: coco: Leak ASID for coco guests
  HACK: Add sev_console hypercall

 docs/guest-guide/x86/hypercall-abi.rst      |   4 +
 tools/include/libxl.h                       |   5 +
 tools/include/xenctrl.h                     |   4 +
 tools/include/xenguest.h                    |   1 +
 tools/libs/ctrl/xc_domain.c                 |  36 +
 tools/libs/guest/Makefile.common            |   2 +
 tools/libs/guest/xg_dom_boot.c              |  33 +
 tools/libs/guest/xg_dom_coco.c              |  35 +
 tools/libs/guest/xg_dom_coco.h              |  39 +
 tools/libs/guest/xg_dom_x86.c               |   1 +
 tools/libs/light/libxl_cpuid.c              |   1 +
 tools/libs/light/libxl_create.c             |   4 +
 tools/libs/light/libxl_dom.c                |   1 +
 tools/libs/light/libxl_types.idl            |   1 +
 tools/libs/util/libxlu_disk_l.c             |  13 +-
 tools/libs/util/libxlu_disk_l.h             |   7 +-
 tools/misc/xen-cpuid.c                      |   1 +
 tools/ocaml/libs/xc/xenctrl.ml              |   1 +
 tools/ocaml/libs/xc/xenctrl.mli             |   1 +
 tools/xl/xl_parse.c                         |   2 +
 xen/arch/x86/Makefile                       |   1 +
 xen/arch/x86/coco/Makefile                  |   1 +
 xen/arch/x86/coco/sev.c                     | 262 ++++++
 xen/arch/x86/cpu/amd.c                      |  10 +
 xen/arch/x86/cpu/common.c                   |   2 +
 xen/arch/x86/cpuid.c                        |   7 +
 xen/arch/x86/domain.c                       |   4 +
 xen/arch/x86/flushtlb.c                     |   7 +-
 xen/arch/x86/hvm/Kconfig                    |  10 +
 xen/arch/x86/hvm/asid.c                     | 170 ++--
 xen/arch/x86/hvm/emulate.c                  | 139 +++-
 xen/arch/x86/hvm/hvm.c                      |  55 +-
 xen/arch/x86/hvm/hypercall.c                |  17 +-
 xen/arch/x86/hvm/nestedhvm.c                |   7 +-
 xen/arch/x86/hvm/svm/asid.c                 |  77 +-
 xen/arch/x86/hvm/svm/nestedsvm.c            |   2 +-
 xen/arch/x86/hvm/svm/svm.c                  |  43 +-
 xen/arch/x86/hvm/svm/svm.h                  |   4 -
 xen/arch/x86/hvm/svm/vmcb.c                 |  17 +-
 xen/arch/x86/hvm/vmx/vmcs.c                 |   6 +-
 xen/arch/x86/hvm/vmx/vmx.c                  |  68 +-
 xen/arch/x86/hvm/vmx/vvmx.c                 |   5 +-
 xen/arch/x86/include/asm/coco.h             |   8 +
 xen/arch/x86/include/asm/cpufeature.h       |   4 +
 xen/arch/x86/include/asm/hvm/asid.h         |  26 +-
 xen/arch/x86/include/asm/hvm/domain.h       |   2 +
 xen/arch/x86/include/asm/hvm/hvm.h          |  15 +-
 xen/arch/x86/include/asm/hvm/svm/sev.h      |  14 +
 xen/arch/x86/include/asm/hvm/svm/svm.h      |  32 +
 xen/arch/x86/include/asm/hvm/svm/vmcb.h     |  22 +-
 xen/arch/x86/include/asm/hvm/vcpu.h         |  10 +-
 xen/arch/x86/include/asm/hvm/vmx/vmx.h      |  19 +-
 xen/arch/x86/include/asm/msr-index.h        |   1 +
 xen/arch/x86/include/asm/psp-sev.h          | 655 +++++++++++++++
 xen/arch/x86/mm/hap/hap.c                   |   7 +-
 xen/arch/x86/mm/p2m.c                       |   7 +-
 xen/arch/x86/mm/paging.c                    |   2 +-
 xen/arch/x86/mm/shadow/hvm.c                |   1 +
 xen/arch/x86/mm/shadow/multi.c              |   1 +
 xen/common/Kconfig                          |   5 +
 xen/common/Makefile                         |   1 +
 xen/common/coco.c                           | 140 ++++
 xen/common/domain.c                         |  41 +-
 xen/drivers/Kconfig                         |   2 +
 xen/drivers/Makefile                        |   1 +
 xen/drivers/crypto/Kconfig                  |  10 +
 xen/drivers/crypto/Makefile                 |   1 +
 xen/drivers/crypto/asp.c                    | 830 ++++++++++++++++++++
 xen/include/hypercall-defs.c                |   4 +
 xen/include/public/arch-x86/cpufeatureset.h |   5 +
 xen/include/public/arch-x86/cpuid.h         |   2 +
 xen/include/public/domctl.h                 |   5 +-
 xen/include/public/hvm/coco.h               |  65 ++
 xen/include/public/xen.h                    |   2 +
 xen/include/xen/coco.h                      |  88 +++
 xen/include/xen/lib/x86/cpu-policy.h        |   9 +-
 xen/include/xen/sched.h                     |  14 +
 77 files changed, 2859 insertions(+), 298 deletions(-)
 create mode 100644 tools/libs/guest/xg_dom_coco.c
 create mode 100644 tools/libs/guest/xg_dom_coco.h
 create mode 100644 xen/arch/x86/coco/Makefile
 create mode 100644 xen/arch/x86/coco/sev.c
 create mode 100644 xen/arch/x86/include/asm/coco.h
 create mode 100644 xen/arch/x86/include/asm/hvm/svm/sev.h
 create mode 100644 xen/arch/x86/include/asm/psp-sev.h
 create mode 100644 xen/common/coco.c
 create mode 100644 xen/drivers/crypto/Kconfig
 create mode 100644 xen/drivers/crypto/Makefile
 create mode 100644 xen/drivers/crypto/asp.c
 create mode 100644 xen/include/public/hvm/coco.h
 create mode 100644 xen/include/xen/coco.h

-- 
2.49.0



Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

On 16.05.25 11:31, Teddy Astie wrote:
> Hello,
> 
> This series introduce support for confidential computing along with a
> AMD SEV implementation. It also bundles some of the functional
> requirements (ASID scheme, ABI, ...) which could be separated if needed.
> 
> (I bundled everything in this serie to have a complete coherent serie)
> 
> This work receives funding by the Hyper Open X consortium (France 2030).
> 
> # Concepts
> 
> A confidential guest is a bit special as :
>   - its memory is by default encrypted or not directly accessible by the
>     hypervisor, thus other domains/dom0 as well; it must be explicitely
>     shared by the guest itself
>   - so its page-tables are also not accessible
> 
> # Implementation
> 
> Confidential computing is exposed in a uniform way regardless of actual
> implementation (SEV, TDX, RME, ...) through the coco_op hypercall (mostly
> for use by the Dom0 toolstack). This interface provides a way to query
> informations on the coco platform (support status, features (un)safety,
> ...), and prepare initial guest memory.
> Only HVM domains have support for confidential computing.
> (in the future, we may want to have attestation support)
> 
> In order to create a confidential computing domain, the process is follow :
>   - create a HVM/PVH domain with XEN_DOMCTL_CDF_coco
>   - populate initial memory as usual
>   - apply coco_prepare_initial_mem on all initial pages
>     (under SEV, this will encrypt memory)
> 
> Under xl, it is exposed through the `coco` parameter ("coco = 1").

Wouldn't it make sense to allow specifying the kind of domain
(SEV, SEV-ES, SEV-SNP, TDX) like KVM does?

It might not be needed right now, but in future this could be needed
(e.g. when allowing migration between hosts with different SEV
features).

I don't think this is important during RFC phase, but the final
configuration and hypervisor interfaces of this series should allow
that.


Juergen

Le 16/05/2025 à 12:54, Jürgen Groß a écrit :
> On 16.05.25 11:31, Teddy Astie wrote:
>>
>> In order to create a confidential computing domain, the process is 
>> follow :
>>   - create a HVM/PVH domain with XEN_DOMCTL_CDF_coco
>>   - populate initial memory as usual
>>   - apply coco_prepare_initial_mem on all initial pages
>>     (under SEV, this will encrypt memory)
>>
>> Under xl, it is exposed through the `coco` parameter ("coco = 1").
> 
> Wouldn't it make sense to allow specifying the kind of domain
> (SEV, SEV-ES, SEV-SNP, TDX) like KVM does?
> 

Yes, I was thinking of exposing it through in a optional arch-specific 
parameter for specifying some SEV-specific parameters (enable SNP, ...).

And by default rely on what the platform provides with a "best default" 
configuration.
(AFAICT it's not possible to have both SEV (AMD-specific) and TDX 
(Intel-specific), or at least not yet)

> It might not be needed right now, but in future this could be needed
> (e.g. when allowing migration between hosts with different SEV
> features).
> 
> I don't think this is important during RFC phase, but the final
> configuration and hypervisor interfaces of this series should allow
> that.
> 
> 
> Juergen

Teddy


Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech