[edk2-devel] [PATCH v12 00/46] SEV-ES guest support

Lendacky, Thomas posted 46 patches 2 weeks ago
Failed in applying to current master (apply log)
MdeModulePkg/MdeModulePkg.dec                 |    9 +
OvmfPkg/OvmfPkg.dec                           |    9 +
UefiCpuPkg/UefiCpuPkg.dec                     |   17 +
OvmfPkg/OvmfPkgIa32.dsc                       |    6 +
OvmfPkg/OvmfPkgIa32X64.dsc                    |    6 +
OvmfPkg/OvmfPkgX64.dsc                        |    6 +
OvmfPkg/OvmfXen.dsc                           |    1 +
UefiCpuPkg/UefiCpuPkg.dsc                     |    2 +
UefiPayloadPkg/UefiPayloadPkgIa32.dsc         |    2 +
UefiPayloadPkg/UefiPayloadPkgIa32X64.dsc      |    2 +
OvmfPkg/OvmfPkgX64.fdf                        |    9 +
MdeModulePkg/Core/DxeIplPeim/DxeIpl.inf       |    2 +
MdePkg/Library/BaseLib/BaseLib.inf            |    4 +
OvmfPkg/Library/VmgExitLib/VmgExitLib.inf     |   36 +
OvmfPkg/PlatformPei/PlatformPei.inf           |    9 +
.../FvbServicesRuntimeDxe.inf                 |    2 +
OvmfPkg/ResetVector/ResetVector.inf           |    8 +
OvmfPkg/Sec/SecMain.inf                       |    4 +
.../DxeCpuExceptionHandlerLib.inf             |    1 +
.../PeiCpuExceptionHandlerLib.inf             |    1 +
.../SecPeiCpuExceptionHandlerLib.inf          |    1 +
.../SmmCpuExceptionHandlerLib.inf             |    1 +
.../Xcode5SecPeiCpuExceptionHandlerLib.inf    |    1 +
UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf |    4 +
UefiCpuPkg/Library/MpInitLib/PeiMpInitLib.inf |    4 +
.../Library/VmgExitLibNull/VmgExitLibNull.inf |   27 +
.../Core/DxeIplPeim/X64/VirtualMemory.h       |   12 +-
MdePkg/Include/Library/BaseLib.h              |   31 +
MdePkg/Include/Register/Amd/Fam17Msr.h        |   46 +
MdePkg/Include/Register/Amd/Ghcb.h            |  166 ++
.../IndustryStandard/InstructionParsing.h     |   83 +
OvmfPkg/Include/Library/MemEncryptSevLib.h    |   12 +
.../QemuFlash.h                               |   13 +
UefiCpuPkg/CpuDxe/CpuGdt.h                    |    4 +-
UefiCpuPkg/Include/Library/VmgExitLib.h       |  103 +
UefiCpuPkg/Library/MpInitLib/MpLib.h          |   68 +-
.../Core/DxeIplPeim/Ia32/DxeLoadFunc.c        |    4 +-
.../Core/DxeIplPeim/X64/DxeLoadFunc.c         |   11 +-
.../Core/DxeIplPeim/X64/VirtualMemory.c       |   57 +-
.../MemEncryptSevLibInternal.c                |   75 +-
OvmfPkg/Library/VmgExitLib/VmgExitLib.c       |  159 ++
OvmfPkg/Library/VmgExitLib/VmgExitVcHandler.c | 1716 +++++++++++++++++
OvmfPkg/PlatformPei/AmdSev.c                  |   89 +
OvmfPkg/PlatformPei/MemDetect.c               |   43 +
.../QemuFlash.c                               |   23 +-
.../QemuFlashDxe.c                            |   40 +
.../QemuFlashSmm.c                            |   16 +
OvmfPkg/Sec/SecMain.c                         |  188 +-
UefiCpuPkg/CpuDxe/CpuGdt.c                    |    8 +-
.../CpuExceptionCommon.c                      |   10 +-
.../PeiDxeSmmCpuException.c                   |   20 +-
.../SecPeiCpuException.c                      |   19 +
UefiCpuPkg/Library/MpInitLib/DxeMpLib.c       |  120 +-
UefiCpuPkg/Library/MpInitLib/MpLib.c          |  337 +++-
UefiCpuPkg/Library/MpInitLib/PeiMpLib.c       |   19 +
.../Library/VmgExitLibNull/VmgExitLibNull.c   |  121 ++
UefiCpuPkg/PiSmmCpuDxeSmm/X64/SmmFuncsArch.c  |    2 +-
Maintainers.txt                               |   10 +
MdeModulePkg/MdeModulePkg.uni                 |    8 +
MdePkg/Library/BaseLib/Ia32/VmgExit.nasm      |   37 +
MdePkg/Library/BaseLib/Ia32/XGetBv.nasm       |   31 +
MdePkg/Library/BaseLib/X64/VmgExit.nasm       |   32 +
MdePkg/Library/BaseLib/X64/XGetBv.nasm        |   34 +
OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm  |  100 +
OvmfPkg/ResetVector/Ia32/PageTables64.asm     |  351 +++-
OvmfPkg/ResetVector/ResetVector.nasmb         |   20 +
.../X64/ExceptionHandlerAsm.nasm              |   17 +
.../X64/Xcode5ExceptionHandlerAsm.nasm        |   17 +
UefiCpuPkg/Library/MpInitLib/Ia32/MpEqu.inc   |    2 +-
.../Library/MpInitLib/Ia32/MpFuncs.nasm       |   15 +
UefiCpuPkg/Library/MpInitLib/X64/MpEqu.inc    |    4 +-
UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm |  370 +++-
.../Library/VmgExitLibNull/VmgExitLibNull.uni |   15 +
.../ResetVector/Vtf0/Ia16/Real16ToFlat32.asm  |    9 +
UefiCpuPkg/UefiCpuPkg.uni                     |   11 +
75 files changed, 4772 insertions(+), 100 deletions(-)
create mode 100644 OvmfPkg/Library/VmgExitLib/VmgExitLib.inf
create mode 100644 UefiCpuPkg/Library/VmgExitLibNull/VmgExitLibNull.inf
create mode 100644 MdePkg/Include/Register/Amd/Ghcb.h
create mode 100644 OvmfPkg/Include/IndustryStandard/InstructionParsing.h
create mode 100644 UefiCpuPkg/Include/Library/VmgExitLib.h
create mode 100644 OvmfPkg/Library/VmgExitLib/VmgExitLib.c
create mode 100644 OvmfPkg/Library/VmgExitLib/VmgExitVcHandler.c
create mode 100644 UefiCpuPkg/Library/VmgExitLibNull/VmgExitLibNull.c
create mode 100644 MdePkg/Library/BaseLib/Ia32/VmgExit.nasm
create mode 100644 MdePkg/Library/BaseLib/Ia32/XGetBv.nasm
create mode 100644 MdePkg/Library/BaseLib/X64/VmgExit.nasm
create mode 100644 MdePkg/Library/BaseLib/X64/XGetBv.nasm
create mode 100644 OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
create mode 100644 UefiCpuPkg/Library/VmgExitLibNull/VmgExitLibNull.uni

[edk2-devel] [PATCH v12 00/46] SEV-ES guest support

Posted by Lendacky, Thomas 2 weeks ago
From: Tom Lendacky <thomas.lendacky@amd.com>

This patch series provides support for running EDK2/OVMF under SEV-ES.

Secure Encrypted Virtualization - Encrypted State (SEV-ES) expands on the
SEV support to protect the guest register state from the hypervisor. See
"AMD64 Architecture Programmer's Manual Volume 2: System Programming",
section "15.35 Encrypted State (SEV-ES)" [1].

In order to allow a hypervisor to perform functions on behalf of a guest,
there is architectural support for notifying a guest's operating system
when certain types of VMEXITs are about to occur. This allows the guest to
selectively share information with the hypervisor to satisfy the requested
function. The notification is performed using a new exception, the VMM
Communication exception (#VC). The information is shared through the
Guest-Hypervisor Communication Block (GHCB) using the VMGEXIT instruction.
The GHCB format and the protocol for using it is documented in "SEV-ES
Guest-Hypervisor Communication Block Standardization" [2].

The main areas of the EDK2 code that are updated to support SEV-ES are
around the exception handling support and the AP boot support.

Exception support is required starting in Sec, continuing through Pei
and into Dxe in order to handle #VC exceptions that are generated.  Each
AP requires it's own GHCB page as well as a page to hold values specific
to that AP.

AP booting poses some interesting challenges. The INIT-SIPI-SIPI sequence
is typically used to boot the APs. However, the hypervisor is not allowed
to update the guest registers. The GHCB document [2] talks about how SMP
booting under SEV-ES is performed.

Since the GHCB page must be a shared (unencrypted) page, the processor
must be running in long mode in order for the guest and hypervisor to
communicate with each other. As a result, SEV-ES is only supported under
the X64 architecture.

[1] https://www.amd.com/system/files/TechDocs/24593.pdf
[2] https://developer.amd.com/wp-content/resources/56421.pdf

---

These patches are based on commit:
6074f57e5b19 ("MdePkg/Include/IndustryStandard: Main CXL header")

A version of the tree can be found at:
https://github.com/AMDESE/ovmf/tree/sev-es-v20

Cc: Andrew Fish <afish@apple.com>
Cc: Anthony Perard <anthony.perard@citrix.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Cc: Benjamin You <benjamin.you@intel.com>
Cc: Dandan Bi <dandan.bi@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Guo Dong <guo.dong@intel.com>
Cc: Hao A Wu <hao.a.wu@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Julien Grall <julien@xen.org>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Leif Lindholm <leif@nuviainc.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Maurice Ma <maurice.ma@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ray Ni <ray.ni@intel.com>

Changes since v11:
- Make the XGETBV and VMGEXIT .nasm files buildable for all environments
  and remove the updates that add these instructions to GccInline.c

Changes since v10:
- Fix conflicts around GccInline.c file after moving to latest commit
- Fix conflicts with OVMF PCD values after moving to latest commit

Changes since v9:
- Fixed bit field declarations in the GHCB structure to use UINT32
  and not UINT64.
- Fixed a warning produced by VS2019 in the instruction parsing code
  by expliciting casting a bit shift to an INT64.
- Sorted section entries in the OVMF VmgExitLib INF file.
- Moved the new Maintainers.txt entry so entries remain sorted.
- Documentation style fixes for return values.
- Miscellaneous code style fixes.

Changes since v8:
- Move IOIO exit info definitions into Ghcb.h file
  - Add a macro for calculating IO instruction bytes (IOIO_DATA_BYTES)
- Exception handler support for debug registers
  - Moved the DRx register saving changes into the UefiCpuPkg patch for
    base #VC support in CpuExceptionHandlerLib.
- OvmfPkg VmgExitLib
  - Remove the .uni file
  - Update .inf file:
    - New file location for VmgExitVcHandler.c
    - Add additional Packages and LibraryClasses
- Introduce a header file to hold the #VC instruction parsing related
  definitions
  - Include additional #defines for instruction decoding to replace
    hard coded values for things like instruction prefixes and escapes.
- Replace hardcoded CPUID values with values from existing header files
  and use existing CR4 definition for accessing CR4 data.
- Change the type used for obtaining data addresses in the instruction
  parsing
  - Switch from INTN to UINT64 and use compiler conversions and casting
    to perform the correct address calculation
- ResetVector code:
  - Revert some inadvertant changes introduced in v7 for reserving the
    SEV-ES work area memory and for checking the status of SEV-ES.
- AP Booting
  - Provide support for non-broadcast INIT-SIPI-SIPI AP boot (minimize
    code duplication by creating a function to set the AP jump table
    vector address).
- Fix file/directory entry in maintainer changes.
- Various coding style fixes
  - Commenting, if statements, etc.
- Various documentation style fixes

Changes since v7:
- Reserve the SEV-ES workarea when S3 is enabled
- Fix warnings issued by the Visual Studio compiler
- Create a NULL VmgExitLib instance that is used for VMGEXIT
  related operations as well as #VC handling. Then create the full
  VmgExitLib support only in OvmfPkg - where it will be used. This
  removes a bunch of implementation code from platforms that will
  not be using the functionality.
- Remove single use interfaces from the VmgExitLib (VmgMmioWrite
  and VmgSetApJumpTable)

Changes since v6:
- Add function comments to all functions, including local functions
- Add function parameter direction to all functions (in/out)
- Add support for MMIO MOVZX/MOVSX instructions
- Ensure the per-CPU variable page remains encrypted
- Coding-style fixes as identified by Ecc

Changes since v5:
- Remove extraneous VmgExitLib usage
- Miscellaneous changes to address feedback (coding style, etc.)

Changes since v4:
- Move the SEV-ES protocol negotiation out of the SEC exception handler
  and into the SecMain.c file. As a result:
  - Move the SecGhcb related PCDs out of UefiCpuPkg and into OvmfPkg
  - Combine SecAMDSevVcHandler.c and PeiDxeAMDSevVcHandler.c into a
    single AMDSevVcHandler.c
- Consolidate VmgExitLib usage into common LibraryClasses sections
- Add documentation comments to the VmgExitLib functions

Changes since v3:
- Remove the need for the MP library finalization routine. The AP
  jump table address will be held by the hypervisor rather than
  communicated via the GHCB MSR. This removes some fragility around
  the UEFI to OS transition.
- Rename the SEV-ES RIP reset area to SEV-ES workarea and use it to
  communicate the SEV-ES status, so that SEC CPU exception handling is
  only established for an SEV-ES guest.
- Fix SMM build breakageAdd around QemuFlashPtrWrite().
- Fix SMM build breakage by adding VC exception support the SMM CPU
  exception handling.
- Add memory fencing around the invocation of AsmVmgExit().
- Clarify comments around the SEV-ES AP reset RIP values and usage.
- Move some PCD definitions from MdeModulePkg to UefiCpuPkg.
- Remove the 16-bit code selector definition from MdeModulePkg

Changes since v2:
- Added a way to locate the SEV-ES fixed AP RIP address for starting
  AP's to avoid updating the actual flash image (build time location
  that is identified with a GUID value).
- Create a VmgExit library to replace static inline functions.
- Move some PCDs to the appropriate packages
- Add support for writing to QEMU flash under SEV-ES
- Add additional MMIO opcode support
- Cleaned up the GHCB MSR CPUID protocol support

Changes since v1:
- Patches reworked to be more specific to the component/area being updated
  and order of definition/usage
- Created a library for VMGEXIT-related functions to replace use of inline
  functions
- Allocation method for GDT changed from AllocatePool to AllocatePages
- Early caching only enabled for SEV-ES guests
- Ensure AP loop mode set to halt loop mode for SEV-ES guests
- Reserved SEC GHCB-related memory areas when S3 is enabled

Tom Lendacky (46):
  MdeModulePkg: Create PCDs to be used in support of SEV-ES
  UefiCpuPkg: Create PCD to be used in support of SEV-ES
  MdePkg: Add the MSR definition for the GHCB register
  MdePkg: Add a structure definition for the GHCB
  MdeModulePkg/DxeIplPeim: Support GHCB pages when creating page tables
  MdePkg/BaseLib: Add support for the XGETBV instruction
  MdePkg/BaseLib: Add support for the VMGEXIT instruction
  UefiCpuPkg: Implement library support for VMGEXIT
  OvmfPkg: Prepare OvmfPkg to use the VmgExitLib library
  UefiPayloadPkg: Prepare UefiPayloadPkg to use the VmgExitLib library
  UefiCpuPkg/CpuExceptionHandler: Add base support for the #VC exception
  OvmfPkg/VmgExitLib: Implement library support for VmgExitLib in OVMF
  OvmfPkg/VmgExitLib: Add support for IOIO_PROT NAE events
  OvmfPkg/VmgExitLib: Support string IO for IOIO_PROT NAE events
  OvmfPkg/VmgExitLib: Add support for CPUID NAE events
  OvmfPkg/VmgExitLib: Add support for MSR_PROT NAE events
  OvmfPkg/VmgExitLib: Add support for NPF NAE events (MMIO)
  OvmfPkg/VmgExitLib: Add support for WBINVD NAE events
  OvmfPkg/VmgExitLib: Add support for RDTSC NAE events
  OvmfPkg/VmgExitLib: Add support for RDPMC NAE events
  OvmfPkg/VmgExitLib: Add support for INVD NAE events
  OvmfPkg/VmgExitLib: Add support for VMMCALL NAE events
  OvmfPkg/VmgExitLib: Add support for RDTSCP NAE events
  OvmfPkg/VmgExitLib: Add support for MONITOR/MONITORX NAE events
  OvmfPkg/VmgExitLib: Add support for MWAIT/MWAITX NAE events
  OvmfPkg/VmgExitLib: Add support for DR7 Read/Write NAE events
  OvmfPkg/MemEncryptSevLib: Add an SEV-ES guest indicator function
  OvmfPkg: Add support to perform SEV-ES initialization
  OvmfPkg: Create a GHCB page for use during Sec phase
  OvmfPkg/PlatformPei: Reserve GHCB-related areas if S3 is supported
  OvmfPkg: Create GHCB pages for use during Pei and Dxe phase
  OvmfPkg/PlatformPei: Move early GDT into ram when SEV-ES is enabled
  UefiCpuPkg: Create an SEV-ES workarea PCD
  OvmfPkg: Reserve a page in memory for the SEV-ES usage
  OvmfPkg/PlatformPei: Reserve SEV-ES work area if S3 is supported
  OvmfPkg/ResetVector: Add support for a 32-bit SEV check
  OvmfPkg/Sec: Add #VC exception handling for Sec phase
  OvmfPkg/Sec: Enable cache early to speed up booting
  OvmfPkg/QemuFlashFvbServicesRuntimeDxe: Bypass flash detection with
    SEV-ES
  UefiCpuPkg: Add a 16-bit protected mode code segment descriptor
  UefiCpuPkg/MpInitLib: Add CPU MP data flag to indicate if SEV-ES is
    enabled
  UefiCpuPkg: Allow AP booting under SEV-ES
  OvmfPkg: Use the SEV-ES work area for the SEV-ES AP reset vector
  OvmfPkg: Move the GHCB allocations into reserved memory
  UefiCpuPkg/MpInitLib: Prepare SEV-ES guest APs for OS use
  Maintainers.txt: Add reviewers for the OvmfPkg SEV-related files

 MdeModulePkg/MdeModulePkg.dec                 |    9 +
 OvmfPkg/OvmfPkg.dec                           |    9 +
 UefiCpuPkg/UefiCpuPkg.dec                     |   17 +
 OvmfPkg/OvmfPkgIa32.dsc                       |    6 +
 OvmfPkg/OvmfPkgIa32X64.dsc                    |    6 +
 OvmfPkg/OvmfPkgX64.dsc                        |    6 +
 OvmfPkg/OvmfXen.dsc                           |    1 +
 UefiCpuPkg/UefiCpuPkg.dsc                     |    2 +
 UefiPayloadPkg/UefiPayloadPkgIa32.dsc         |    2 +
 UefiPayloadPkg/UefiPayloadPkgIa32X64.dsc      |    2 +
 OvmfPkg/OvmfPkgX64.fdf                        |    9 +
 MdeModulePkg/Core/DxeIplPeim/DxeIpl.inf       |    2 +
 MdePkg/Library/BaseLib/BaseLib.inf            |    4 +
 OvmfPkg/Library/VmgExitLib/VmgExitLib.inf     |   36 +
 OvmfPkg/PlatformPei/PlatformPei.inf           |    9 +
 .../FvbServicesRuntimeDxe.inf                 |    2 +
 OvmfPkg/ResetVector/ResetVector.inf           |    8 +
 OvmfPkg/Sec/SecMain.inf                       |    4 +
 .../DxeCpuExceptionHandlerLib.inf             |    1 +
 .../PeiCpuExceptionHandlerLib.inf             |    1 +
 .../SecPeiCpuExceptionHandlerLib.inf          |    1 +
 .../SmmCpuExceptionHandlerLib.inf             |    1 +
 .../Xcode5SecPeiCpuExceptionHandlerLib.inf    |    1 +
 UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf |    4 +
 UefiCpuPkg/Library/MpInitLib/PeiMpInitLib.inf |    4 +
 .../Library/VmgExitLibNull/VmgExitLibNull.inf |   27 +
 .../Core/DxeIplPeim/X64/VirtualMemory.h       |   12 +-
 MdePkg/Include/Library/BaseLib.h              |   31 +
 MdePkg/Include/Register/Amd/Fam17Msr.h        |   46 +
 MdePkg/Include/Register/Amd/Ghcb.h            |  166 ++
 .../IndustryStandard/InstructionParsing.h     |   83 +
 OvmfPkg/Include/Library/MemEncryptSevLib.h    |   12 +
 .../QemuFlash.h                               |   13 +
 UefiCpuPkg/CpuDxe/CpuGdt.h                    |    4 +-
 UefiCpuPkg/Include/Library/VmgExitLib.h       |  103 +
 UefiCpuPkg/Library/MpInitLib/MpLib.h          |   68 +-
 .../Core/DxeIplPeim/Ia32/DxeLoadFunc.c        |    4 +-
 .../Core/DxeIplPeim/X64/DxeLoadFunc.c         |   11 +-
 .../Core/DxeIplPeim/X64/VirtualMemory.c       |   57 +-
 .../MemEncryptSevLibInternal.c                |   75 +-
 OvmfPkg/Library/VmgExitLib/VmgExitLib.c       |  159 ++
 OvmfPkg/Library/VmgExitLib/VmgExitVcHandler.c | 1716 +++++++++++++++++
 OvmfPkg/PlatformPei/AmdSev.c                  |   89 +
 OvmfPkg/PlatformPei/MemDetect.c               |   43 +
 .../QemuFlash.c                               |   23 +-
 .../QemuFlashDxe.c                            |   40 +
 .../QemuFlashSmm.c                            |   16 +
 OvmfPkg/Sec/SecMain.c                         |  188 +-
 UefiCpuPkg/CpuDxe/CpuGdt.c                    |    8 +-
 .../CpuExceptionCommon.c                      |   10 +-
 .../PeiDxeSmmCpuException.c                   |   20 +-
 .../SecPeiCpuException.c                      |   19 +
 UefiCpuPkg/Library/MpInitLib/DxeMpLib.c       |  120 +-
 UefiCpuPkg/Library/MpInitLib/MpLib.c          |  337 +++-
 UefiCpuPkg/Library/MpInitLib/PeiMpLib.c       |   19 +
 .../Library/VmgExitLibNull/VmgExitLibNull.c   |  121 ++
 UefiCpuPkg/PiSmmCpuDxeSmm/X64/SmmFuncsArch.c  |    2 +-
 Maintainers.txt                               |   10 +
 MdeModulePkg/MdeModulePkg.uni                 |    8 +
 MdePkg/Library/BaseLib/Ia32/VmgExit.nasm      |   37 +
 MdePkg/Library/BaseLib/Ia32/XGetBv.nasm       |   31 +
 MdePkg/Library/BaseLib/X64/VmgExit.nasm       |   32 +
 MdePkg/Library/BaseLib/X64/XGetBv.nasm        |   34 +
 OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm  |  100 +
 OvmfPkg/ResetVector/Ia32/PageTables64.asm     |  351 +++-
 OvmfPkg/ResetVector/ResetVector.nasmb         |   20 +
 .../X64/ExceptionHandlerAsm.nasm              |   17 +
 .../X64/Xcode5ExceptionHandlerAsm.nasm        |   17 +
 UefiCpuPkg/Library/MpInitLib/Ia32/MpEqu.inc   |    2 +-
 .../Library/MpInitLib/Ia32/MpFuncs.nasm       |   15 +
 UefiCpuPkg/Library/MpInitLib/X64/MpEqu.inc    |    4 +-
 UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm |  370 +++-
 .../Library/VmgExitLibNull/VmgExitLibNull.uni |   15 +
 .../ResetVector/Vtf0/Ia16/Real16ToFlat32.asm  |    9 +
 UefiCpuPkg/UefiCpuPkg.uni                     |   11 +
 75 files changed, 4772 insertions(+), 100 deletions(-)
 create mode 100644 OvmfPkg/Library/VmgExitLib/VmgExitLib.inf
 create mode 100644 UefiCpuPkg/Library/VmgExitLibNull/VmgExitLibNull.inf
 create mode 100644 MdePkg/Include/Register/Amd/Ghcb.h
 create mode 100644 OvmfPkg/Include/IndustryStandard/InstructionParsing.h
 create mode 100644 UefiCpuPkg/Include/Library/VmgExitLib.h
 create mode 100644 OvmfPkg/Library/VmgExitLib/VmgExitLib.c
 create mode 100644 OvmfPkg/Library/VmgExitLib/VmgExitVcHandler.c
 create mode 100644 UefiCpuPkg/Library/VmgExitLibNull/VmgExitLibNull.c
 create mode 100644 MdePkg/Library/BaseLib/Ia32/VmgExit.nasm
 create mode 100644 MdePkg/Library/BaseLib/Ia32/XGetBv.nasm
 create mode 100644 MdePkg/Library/BaseLib/X64/VmgExit.nasm
 create mode 100644 MdePkg/Library/BaseLib/X64/XGetBv.nasm
 create mode 100644 OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
 create mode 100644 UefiCpuPkg/Library/VmgExitLibNull/VmgExitLibNull.uni

-- 
2.27.0


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#63304): https://edk2.groups.io/g/devel/message/63304
Mute This Topic: https://groups.io/mt/75824926/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v12 00/46] SEV-ES guest support

Posted by Laszlo Ersek 2 weeks ago
Hi Tom,

On 07/27/20 17:25, Lendacky, Thomas wrote:
> From: Tom Lendacky <thomas.lendacky@amd.com>
> 
> This patch series provides support for running EDK2/OVMF under SEV-ES.
> 
> Secure Encrypted Virtualization - Encrypted State (SEV-ES) expands on the
> SEV support to protect the guest register state from the hypervisor. See
> "AMD64 Architecture Programmer's Manual Volume 2: System Programming",
> section "15.35 Encrypted State (SEV-ES)" [1].
> 
> In order to allow a hypervisor to perform functions on behalf of a guest,
> there is architectural support for notifying a guest's operating system
> when certain types of VMEXITs are about to occur. This allows the guest to
> selectively share information with the hypervisor to satisfy the requested
> function. The notification is performed using a new exception, the VMM
> Communication exception (#VC). The information is shared through the
> Guest-Hypervisor Communication Block (GHCB) using the VMGEXIT instruction.
> The GHCB format and the protocol for using it is documented in "SEV-ES
> Guest-Hypervisor Communication Block Standardization" [2].
> 
> The main areas of the EDK2 code that are updated to support SEV-ES are
> around the exception handling support and the AP boot support.
> 
> Exception support is required starting in Sec, continuing through Pei
> and into Dxe in order to handle #VC exceptions that are generated.  Each
> AP requires it's own GHCB page as well as a page to hold values specific
> to that AP.
> 
> AP booting poses some interesting challenges. The INIT-SIPI-SIPI sequence
> is typically used to boot the APs. However, the hypervisor is not allowed
> to update the guest registers. The GHCB document [2] talks about how SMP
> booting under SEV-ES is performed.
> 
> Since the GHCB page must be a shared (unencrypted) page, the processor
> must be running in long mode in order for the guest and hypervisor to
> communicate with each other. As a result, SEV-ES is only supported under
> the X64 architecture.
> 
> [1] https://www.amd.com/system/files/TechDocs/24593.pdf
> [2] https://developer.amd.com/wp-content/resources/56421.pdf
> 
> ---
> 
> These patches are based on commit:
> 6074f57e5b19 ("MdePkg/Include/IndustryStandard: Main CXL header")
> 
> A version of the tree can be found at:
> https://github.com/AMDESE/ovmf/tree/sev-es-v20
> 
> Cc: Andrew Fish <afish@apple.com>
> Cc: Anthony Perard <anthony.perard@citrix.com>
> Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
> Cc: Benjamin You <benjamin.you@intel.com>
> Cc: Dandan Bi <dandan.bi@intel.com>
> Cc: Eric Dong <eric.dong@intel.com>
> Cc: Guo Dong <guo.dong@intel.com>
> Cc: Hao A Wu <hao.a.wu@intel.com>
> Cc: Jian J Wang <jian.j.wang@intel.com>
> Cc: Jordan Justen <jordan.l.justen@intel.com>
> Cc: Julien Grall <julien@xen.org>
> Cc: Laszlo Ersek <lersek@redhat.com>
> Cc: Leif Lindholm <leif@nuviainc.com>
> Cc: Liming Gao <liming.gao@intel.com>
> Cc: Maurice Ma <maurice.ma@intel.com>
> Cc: Michael D Kinney <michael.d.kinney@intel.com>
> Cc: Ray Ni <ray.ni@intel.com>
> 
> Changes since v11:
> - Make the XGETBV and VMGEXIT .nasm files buildable for all environments
>   and remove the updates that add these instructions to GccInline.c

Patches 40-46 (inclusive) seem to be missing from my mailbox (and the
list archive on groups.io lacks them too, apparently).

Did you get rate-limited by some component when sending the series, perhaps?

Thanks
Laszlo


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#63351): https://edk2.groups.io/g/devel/message/63351
Mute This Topic: https://groups.io/mt/75824926/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v12 00/46] SEV-ES guest support

Posted by Laszlo Ersek 2 weeks ago
On 07/27/20 19:41, Laszlo Ersek wrote:
> Hi Tom,
> 
> On 07/27/20 17:25, Lendacky, Thomas wrote:
>> From: Tom Lendacky <thomas.lendacky@amd.com>
>>
>> This patch series provides support for running EDK2/OVMF under SEV-ES.
>>
>> Secure Encrypted Virtualization - Encrypted State (SEV-ES) expands on the
>> SEV support to protect the guest register state from the hypervisor. See
>> "AMD64 Architecture Programmer's Manual Volume 2: System Programming",
>> section "15.35 Encrypted State (SEV-ES)" [1].
>>
>> In order to allow a hypervisor to perform functions on behalf of a guest,
>> there is architectural support for notifying a guest's operating system
>> when certain types of VMEXITs are about to occur. This allows the guest to
>> selectively share information with the hypervisor to satisfy the requested
>> function. The notification is performed using a new exception, the VMM
>> Communication exception (#VC). The information is shared through the
>> Guest-Hypervisor Communication Block (GHCB) using the VMGEXIT instruction.
>> The GHCB format and the protocol for using it is documented in "SEV-ES
>> Guest-Hypervisor Communication Block Standardization" [2].
>>
>> The main areas of the EDK2 code that are updated to support SEV-ES are
>> around the exception handling support and the AP boot support.
>>
>> Exception support is required starting in Sec, continuing through Pei
>> and into Dxe in order to handle #VC exceptions that are generated.  Each
>> AP requires it's own GHCB page as well as a page to hold values specific
>> to that AP.
>>
>> AP booting poses some interesting challenges. The INIT-SIPI-SIPI sequence
>> is typically used to boot the APs. However, the hypervisor is not allowed
>> to update the guest registers. The GHCB document [2] talks about how SMP
>> booting under SEV-ES is performed.
>>
>> Since the GHCB page must be a shared (unencrypted) page, the processor
>> must be running in long mode in order for the guest and hypervisor to
>> communicate with each other. As a result, SEV-ES is only supported under
>> the X64 architecture.
>>
>> [1] https://www.amd.com/system/files/TechDocs/24593.pdf
>> [2] https://developer.amd.com/wp-content/resources/56421.pdf
>>
>> ---
>>
>> These patches are based on commit:
>> 6074f57e5b19 ("MdePkg/Include/IndustryStandard: Main CXL header")
>>
>> A version of the tree can be found at:
>> https://github.com/AMDESE/ovmf/tree/sev-es-v20
>>
>> Cc: Andrew Fish <afish@apple.com>
>> Cc: Anthony Perard <anthony.perard@citrix.com>
>> Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
>> Cc: Benjamin You <benjamin.you@intel.com>
>> Cc: Dandan Bi <dandan.bi@intel.com>
>> Cc: Eric Dong <eric.dong@intel.com>
>> Cc: Guo Dong <guo.dong@intel.com>
>> Cc: Hao A Wu <hao.a.wu@intel.com>
>> Cc: Jian J Wang <jian.j.wang@intel.com>
>> Cc: Jordan Justen <jordan.l.justen@intel.com>
>> Cc: Julien Grall <julien@xen.org>
>> Cc: Laszlo Ersek <lersek@redhat.com>
>> Cc: Leif Lindholm <leif@nuviainc.com>
>> Cc: Liming Gao <liming.gao@intel.com>
>> Cc: Maurice Ma <maurice.ma@intel.com>
>> Cc: Michael D Kinney <michael.d.kinney@intel.com>
>> Cc: Ray Ni <ray.ni@intel.com>
>>
>> Changes since v11:
>> - Make the XGETBV and VMGEXIT .nasm files buildable for all environments
>>   and remove the updates that add these instructions to GccInline.c
> 
> Patches 40-46 (inclusive) seem to be missing from my mailbox (and the
> list archive on groups.io lacks them too, apparently).
> 
> Did you get rate-limited by some component when sending the series, perhaps?

On a second / closer look, that seems quite likely, because my INBOX
does have all the (directly delivered) patches; only my list folder is
missing the tail.

Thanks
Laszlo


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#63352): https://edk2.groups.io/g/devel/message/63352
Mute This Topic: https://groups.io/mt/75824926/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v12 00/46] SEV-ES guest support

Posted by Lendacky, Thomas 2 weeks ago
On 7/27/20 12:43 PM, Laszlo Ersek wrote:
> On 07/27/20 19:41, Laszlo Ersek wrote:
>> Hi Tom,

Hi Laszlo,

>>
>> On 07/27/20 17:25, Lendacky, Thomas wrote:
>>> From: Tom Lendacky <thomas.lendacky@amd.com>
>>>
>>> This patch series provides support for running EDK2/OVMF under SEV-ES.
>>>
>>> Secure Encrypted Virtualization - Encrypted State (SEV-ES) expands on the
>>> SEV support to protect the guest register state from the hypervisor. See
>>> "AMD64 Architecture Programmer's Manual Volume 2: System Programming",
>>> section "15.35 Encrypted State (SEV-ES)" [1].
>>>
>>> In order to allow a hypervisor to perform functions on behalf of a guest,
>>> there is architectural support for notifying a guest's operating system
>>> when certain types of VMEXITs are about to occur. This allows the guest to
>>> selectively share information with the hypervisor to satisfy the requested
>>> function. The notification is performed using a new exception, the VMM
>>> Communication exception (#VC). The information is shared through the
>>> Guest-Hypervisor Communication Block (GHCB) using the VMGEXIT instruction.
>>> The GHCB format and the protocol for using it is documented in "SEV-ES
>>> Guest-Hypervisor Communication Block Standardization" [2].
>>>
>>> The main areas of the EDK2 code that are updated to support SEV-ES are
>>> around the exception handling support and the AP boot support.
>>>
>>> Exception support is required starting in Sec, continuing through Pei
>>> and into Dxe in order to handle #VC exceptions that are generated.  Each
>>> AP requires it's own GHCB page as well as a page to hold values specific
>>> to that AP.
>>>
>>> AP booting poses some interesting challenges. The INIT-SIPI-SIPI sequence
>>> is typically used to boot the APs. However, the hypervisor is not allowed
>>> to update the guest registers. The GHCB document [2] talks about how SMP
>>> booting under SEV-ES is performed.
>>>
>>> Since the GHCB page must be a shared (unencrypted) page, the processor
>>> must be running in long mode in order for the guest and hypervisor to
>>> communicate with each other. As a result, SEV-ES is only supported under
>>> the X64 architecture.
>>>
>>> [1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.amd.com%2Fsystem%2Ffiles%2FTechDocs%2F24593.pdf&amp;data=02%7C01%7Cthomas.lendacky%40amd.com%7Cb388414fce2747ca9fef08d832549c1e%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637314686421485283&amp;sdata=q1XqA0IvSWYmCHqRFQ704595cECr6MH6MXIJBoKAkog%3D&amp;reserved=0
>>> [2] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.amd.com%2Fwp-content%2Fresources%2F56421.pdf&amp;data=02%7C01%7Cthomas.lendacky%40amd.com%7Cb388414fce2747ca9fef08d832549c1e%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637314686421485283&amp;sdata=EfLCdVJ9EcGQxBRv79NLLrZ8vyiCxepLbkNM%2FsQMOf0%3D&amp;reserved=0
>>>
>>> ---
>>>
>>> These patches are based on commit:
>>> 6074f57e5b19 ("MdePkg/Include/IndustryStandard: Main CXL header")
>>>
>>> A version of the tree can be found at:
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAMDESE%2Fovmf%2Ftree%2Fsev-es-v20&amp;data=02%7C01%7Cthomas.lendacky%40amd.com%7Cb388414fce2747ca9fef08d832549c1e%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637314686421485283&amp;sdata=auEp%2Fd6Ju4V3BYG5UW%2BzwJzDersXm%2BuziQnB%2FPLDnYg%3D&amp;reserved=0
>>>
>>> Cc: Andrew Fish <afish@apple.com>
>>> Cc: Anthony Perard <anthony.perard@citrix.com>
>>> Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
>>> Cc: Benjamin You <benjamin.you@intel.com>
>>> Cc: Dandan Bi <dandan.bi@intel.com>
>>> Cc: Eric Dong <eric.dong@intel.com>
>>> Cc: Guo Dong <guo.dong@intel.com>
>>> Cc: Hao A Wu <hao.a.wu@intel.com>
>>> Cc: Jian J Wang <jian.j.wang@intel.com>
>>> Cc: Jordan Justen <jordan.l.justen@intel.com>
>>> Cc: Julien Grall <julien@xen.org>
>>> Cc: Laszlo Ersek <lersek@redhat.com>
>>> Cc: Leif Lindholm <leif@nuviainc.com>
>>> Cc: Liming Gao <liming.gao@intel.com>
>>> Cc: Maurice Ma <maurice.ma@intel.com>
>>> Cc: Michael D Kinney <michael.d.kinney@intel.com>
>>> Cc: Ray Ni <ray.ni@intel.com>
>>>
>>> Changes since v11:
>>> - Make the XGETBV and VMGEXIT .nasm files buildable for all environments
>>>   and remove the updates that add these instructions to GccInline.c
>>
>> Patches 40-46 (inclusive) seem to be missing from my mailbox (and the
>> list archive on groups.io lacks them too, apparently).
>>
>> Did you get rate-limited by some component when sending the series, perhaps?
> 
> On a second / closer look, that seems quite likely, because my INBOX
> does have all the (directly delivered) patches; only my list folder is
> missing the tail.

Yes, I always get an error when I reach 40 emails:

Reported error: 550 5.0.350 Remote server returned an error -> 500 We have
received more than 40 messages in 30 minutes from you. To guard against
autoresponder mail loops, we must reject additional messages from you
temporarily. Please try again later.

So I usually have to wait a few hours and resend the last ones so they
make it to groups.io.

Thanks,
Tom

> 
> Thanks
> Laszlo
> 

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#63353): https://edk2.groups.io/g/devel/message/63353
Mute This Topic: https://groups.io/mt/75824926/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v12 00/46] SEV-ES guest support

Posted by Laszlo Ersek 2 weeks ago
On 07/27/20 19:49, Tom Lendacky wrote:

> Yes, I always get an error when I reach 40 emails:
> 
> Reported error: 550 5.0.350 Remote server returned an error -> 500 We have
> received more than 40 messages in 30 minutes from you. To guard against
> autoresponder mail loops, we must reject additional messages from you
> temporarily. Please try again later.
> 
> So I usually have to wait a few hours and resend the last ones so they
> make it to groups.io.

Sigh.

Phil found one way to deal with this non-interactively: see the
"sendemail.smtpBatchSize" and "sendemail.smtpReloginDelay" git-config
options. From git-send-email(1):

       --batch-size=<num>
           Some email servers (e.g. smtp.163.com) limit the number
           emails to be sent per session (connection) and this will
           lead to a failure when sending many messages. With this
           option, send-email will disconnect after sending $<num>
           messages and wait for a few seconds (see --relogin-delay)
           and reconnect, to work around such a limit. You may want to
           use some form of credential helper to avoid having to
           retype your password every time this happens. Defaults to
           the sendemail.smtpBatchSize configuration variable.

       --relogin-delay=<int>
           Waiting $<int> seconds before reconnecting to SMTP server.
           Used together with --batch-size option. Defaults to the
           sendemail.smtpReloginDelay configuration variable.

That way the "second batch" should need no manual work, hopefully.

Thanks
Laszlo


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#63399): https://edk2.groups.io/g/devel/message/63399
Mute This Topic: https://groups.io/mt/75824926/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v12 00/46] SEV-ES guest support

Posted by Lendacky, Thomas 2 weeks ago
On 7/28/20 5:07 AM, Laszlo Ersek wrote:
> On 07/27/20 19:49, Tom Lendacky wrote:
> 
>> Yes, I always get an error when I reach 40 emails:
>>
>> Reported error: 550 5.0.350 Remote server returned an error -> 500 We have
>> received more than 40 messages in 30 minutes from you. To guard against
>> autoresponder mail loops, we must reject additional messages from you
>> temporarily. Please try again later.
>>
>> So I usually have to wait a few hours and resend the last ones so they
>> make it to groups.io.
> 
> Sigh.
> 
> Phil found one way to deal with this non-interactively: see the
> "sendemail.smtpBatchSize" and "sendemail.smtpReloginDelay" git-config
> options. From git-send-email(1):
> 
>        --batch-size=<num>
>            Some email servers (e.g. smtp.163.com) limit the number
>            emails to be sent per session (connection) and this will
>            lead to a failure when sending many messages. With this
>            option, send-email will disconnect after sending $<num>
>            messages and wait for a few seconds (see --relogin-delay)
>            and reconnect, to work around such a limit. You may want to
>            use some form of credential helper to avoid having to
>            retype your password every time this happens. Defaults to
>            the sendemail.smtpBatchSize configuration variable.
> 
>        --relogin-delay=<int>
>            Waiting $<int> seconds before reconnecting to SMTP server.
>            Used together with --batch-size option. Defaults to the
>            sendemail.smtpReloginDelay configuration variable.
> 
> That way the "second batch" should need no manual work, hopefully.

I've tried with and without batch-size and relogin-delay. My initial
submissions were without these parameters and I hit the 40 email limit. I
later tried a batch-size of 1 with a relogin-delay of 1 and that still
results in the 40 email limit. I haven't tried any combinations of a
larger batch-size.

I don't think groups.io has a per-session limit, I think it is actually
tracking individual emails over a period of time, so I don't think these
parameters do anything for that.

Thanks,
Tom

> 
> Thanks
> Laszlo
> 

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#63403): https://edk2.groups.io/g/devel/message/63403
Mute This Topic: https://groups.io/mt/75824926/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v12 00/46] SEV-ES guest support

Posted by Laszlo Ersek 2 weeks ago
On 07/28/20 14:52, Tom Lendacky wrote:
> On 7/28/20 5:07 AM, Laszlo Ersek wrote:
>> On 07/27/20 19:49, Tom Lendacky wrote:
>>
>>> Yes, I always get an error when I reach 40 emails:
>>>
>>> Reported error: 550 5.0.350 Remote server returned an error -> 500 We have
>>> received more than 40 messages in 30 minutes from you. To guard against
>>> autoresponder mail loops, we must reject additional messages from you
>>> temporarily. Please try again later.
>>>
>>> So I usually have to wait a few hours and resend the last ones so they
>>> make it to groups.io.
>>
>> Sigh.
>>
>> Phil found one way to deal with this non-interactively: see the
>> "sendemail.smtpBatchSize" and "sendemail.smtpReloginDelay" git-config
>> options. From git-send-email(1):
>>
>>        --batch-size=<num>
>>            Some email servers (e.g. smtp.163.com) limit the number
>>            emails to be sent per session (connection) and this will
>>            lead to a failure when sending many messages. With this
>>            option, send-email will disconnect after sending $<num>
>>            messages and wait for a few seconds (see --relogin-delay)
>>            and reconnect, to work around such a limit. You may want to
>>            use some form of credential helper to avoid having to
>>            retype your password every time this happens. Defaults to
>>            the sendemail.smtpBatchSize configuration variable.
>>
>>        --relogin-delay=<int>
>>            Waiting $<int> seconds before reconnecting to SMTP server.
>>            Used together with --batch-size option. Defaults to the
>>            sendemail.smtpReloginDelay configuration variable.
>>
>> That way the "second batch" should need no manual work, hopefully.
> 
> I've tried with and without batch-size and relogin-delay. My initial
> submissions were without these parameters and I hit the 40 email limit. I
> later tried a batch-size of 1 with a relogin-delay of 1 and that still
> results in the 40 email limit. I haven't tried any combinations of a
> larger batch-size.
> 
> I don't think groups.io has a per-session limit, I think it is actually
> tracking individual emails over a period of time, so I don't think these
> parameters do anything for that.

Thank you for explaining.

I've now filed a request with <support@groups.io> to raise the limit (on
this list only) to 200 messages per 30 minutes. That should allow people
to post large series *and* to respond to other (unrelated)
email threads as well, in the normal course of their workday.

Thanks
Laszlo


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#63405): https://edk2.groups.io/g/devel/message/63405
Mute This Topic: https://groups.io/mt/75824926/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-