Hi Peter,
> On 18 Dec 2023, at 10:32, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> This patchset adds support for emulating the Arm architectural features
> FEAT_NV and FEAT_NV2 which allow nested virtualization, i.e. where a
> hypervisor can run a guest which thinks it is running at EL2.
>
> Nominally FEAT_NV is sufficient for this and FEAT_NV2 merely improves
> the performance in the nested-virt setup, but in practice hypervisors
> such as KVM are going to require FEAT_NV2 and not bother to support
> the FEAT_NV-only case, so I have implemented them one after the other
> in this single patchset.
>
> The feature is essentially a collection of changes that allow the
> hypervisor to lie to the guest so that it thinks it is running in EL2
> when it's really at EL1. The best summary of what all the changes are
> is in section D8.11 "Nested virtualization" in the Arm ARM, but the
> short summary is:
> * EL2 system registers etc trap to EL2 rather than UNDEFing
> * ERET traps to EL2
> * the CurrentEL register reports "EL2" when NV is enabled
> * on exception entry, SPSR_EL1.M may report "EL2" as the EL the
> exception was taken from
> * when HCR_EL1.NV1 is also set, then there are some extra tweaks
> (NV1 == 1 means "guest thinks it is running with HCR_EL2.E2H == 0")
> * some AT S1 address translation insns can be trapped to EL2
> and FEAT_NV2 adds:
> * accesses to some system registers are transformed into memory
> accesses instead of trapping to EL2
> * accesses to a few EL2 system registers are redirected to the
> equivalent EL1 registers
>
> This patchset is sufficient that you can run an L0 guest kernel that
> has support for FEAT_NV/FEAT_NV2 in its KVM code, and then
> inside that start a nested L1 guest that thinks it has EL2 access,
> and then run an inner-nested L2 guest under that that can get
> to running userspace code. To do that you'll need some not-yet-upstream
> patches for both Linux and kvmtool:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/nv-6.8-nv2-only
> https://gitlab.arm.com/linux-arm/kvmtool/-/commits/nv-v6.6
>
> You'll also want to turn off SVE and SME emulation in QEMU
> (-cpu max,sve=off,sme=off) because at the moment the KVM patchset
> doesn't handle SVE and nested-virt together (the other option
> is to hack kvmtool to make it not ask for both at once, but this
> is easier).
>
> (kvmtool is needed here to run the L1 because QEMU itself as a VMM
> doesn't yet support asking KVM for an EL2 guest.)
>
> The first three patches in the series aren't strictly part of FEAT_NV:
> * patch 1 is already reviewed; I put it here to avoid having
> to deal with textual conflicts between it and this series
> * patch 2 sets CTR_EL0.{IDC,DIC} for '-cpu max', which is a good
> idea anyway and also works around what Marc Z and I think is
> a KVM bug that otherwise causes boot of the L2 kernel to hang
> * patch 3 is a GIC bug which is not FEAT_NV specific but for
> some reason only manifests when booting an L1 kernel under NV
>
I've successfully replicated this setup and reached inner-nested L2 guest
userspace.
FWIW, feel free to add
Tested-by: Miguel Luis <miguel.luis@oracle.com>
(I've been working on QEMU asking KVM for an EL2 guest on top of this series
although there's been yet some debugging to do.)
Thank you!
Miguel
> thanks
> -- PMM
>
> Peter Maydell (35):
> target/arm: Don't implement *32_EL2 registers when EL1 is AArch64 only
> target/arm: Set CTR_EL0.{IDC,DIC} for the 'max' CPU
> hw/intc/arm_gicv3_cpuif: handle LPIs in in the list registers
> target/arm: Handle HCR_EL2 accesses for bits introduced with FEAT_NV
> target/arm: Implement HCR_EL2.AT handling
> target/arm: Enable trapping of ERET for FEAT_NV
> target/arm: Always honour HCR_EL2.TSC when HCR_EL2.NV is set
> target/arm: Allow use of upper 32 bits of TBFLAG_A64
> target/arm: Record correct opcode fields in cpreg for E2H aliases
> target/arm: *_EL12 registers should UNDEF when HCR_EL2.E2H is 0
> target/arm: Make EL2 cpreg accessfns safe for FEAT_NV EL1 accesses
> target/arm: Move FPU/SVE/SME access checks up above
> ARM_CP_SPECIAL_MASK check
> target/arm: Trap sysreg accesses for FEAT_NV
> target/arm: Make NV reads of CurrentEL return EL2
> target/arm: Set SPSR_EL1.M correctly when nested virt is enabled
> target/arm: Trap registers when HCR_EL2.{NV,NV1} == {1,1}
> target/arm: Always use arm_pan_enabled() when checking if PAN is
> enabled
> target/arm: Don't honour PSTATE.PAN when HCR_EL2.{NV,NV1} == {1,1}
> target/arm: Treat LDTR* and STTR* as LDR/STR when NV,NV1 is 1,1
> target/arm: Handle FEAT_NV page table attribute changes
> target/arm: Add FEAT_NV to max, neoverse-n2, neoverse-v1 CPUs
> target/arm: Handle HCR_EL2 accesses for FEAT_NV2 bits
> target/arm: Implement VNCR_EL2 register
> target/arm: Handle FEAT_NV2 changes to when SPSR_EL1.M reports EL2
> target/arm: Handle FEAT_NV2 redirection of SPSR_EL2, ELR_EL2, ESR_EL2,
> FAR_EL2
> target/arm: Implement FEAT_NV2 redirection of sysregs to RAM
> target/arm: Report VNCR_EL2 based faults correctly
> target/arm: Mark up VNCR offsets (offsets 0x0..0xff)
> target/arm: Mark up VNCR offsets (offsets 0x100..0x160)
> target/arm: Mark up VNCR offsets (offsets 0x168..0x1f8)
> target/arm: Mark up VNCR offsets (offsets >= 0x200, except GIC)
> hw/intc/arm_gicv3_cpuif: Mark up VNCR offsets for GIC CPU registers
> target/arm: Report HCR_EL2.{NV,NV1,NV2} in cpu dumps
> target/arm: Enhance CPU_LOG_INT to show SPSR on AArch64
> exception-entry
> target/arm: Add FEAT_NV2 to max, neoverse-n2, neoverse-v1 CPUs
>
> docs/system/arm/emulation.rst | 2 +
> target/arm/cpregs.h | 54 ++++-
> target/arm/cpu-features.h | 10 +
> target/arm/cpu.h | 24 ++-
> target/arm/syndrome.h | 20 +-
> target/arm/tcg/translate.h | 16 +-
> hw/intc/arm_gicv3_cpuif.c | 28 ++-
> target/arm/cpu.c | 8 +-
> target/arm/debug_helper.c | 34 +++-
> target/arm/helper.c | 360 ++++++++++++++++++++++++++++-----
> target/arm/ptw.c | 21 ++
> target/arm/tcg/cpu64.c | 11 +
> target/arm/tcg/hflags.c | 30 ++-
> target/arm/tcg/op_helper.c | 16 +-
> target/arm/tcg/tlb_helper.c | 27 ++-
> target/arm/tcg/translate-a64.c | 162 +++++++++++++--
> 16 files changed, 725 insertions(+), 98 deletions(-)
>
> --
> 2.34.1
>
>