[RFC v2 0/5] ARM Nested Virt Support

Eric Auger posted 5 patches 2 weeks, 1 day ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20240209160039.677865-1-eric.auger@redhat.com
Maintainers: Peter Maydell <peter.maydell@linaro.org>, "Michael S. Tsirkin" <mst@redhat.com>, Cornelia Huck <cohuck@redhat.com>, Paolo Bonzini <pbonzini@redhat.com>
hw/arm/virt.c                      |  6 +++++-
hw/intc/arm_gicv3_common.c         |  1 +
hw/intc/arm_gicv3_kvm.c            | 21 +++++++++++++++++++++
include/hw/intc/arm_gicv3_common.h |  1 +
linux-headers/asm-arm64/kvm.h      |  1 +
linux-headers/linux/kvm.h          |  1 +
target/arm/kvm.c                   | 21 +++++++++++++++++++++
target/arm/kvm_arm.h               | 12 ++++++++++++
8 files changed, 63 insertions(+), 1 deletion(-)
[RFC v2 0/5] ARM Nested Virt Support
Posted by Eric Auger 2 weeks, 1 day ago
This series adds ARM Nested Virtualization support in KVM mode.
This is a respin of previous contributions from Miguel [1] and Haibo [2].

This was tested with Marc's v11 [3] on Ampere HW with fedora L1 guest and
L2 guests booted without EDK2. However it does not work yet with
EDK2 but it looks unrelated to this qemu integration (host hard lockups).

The host needs to be booted with "kvm-arm.mode=nested" option and
qemu needs to be invoked with :

-machine virt,virtualization=on

There is a known issue with hosts supporting SVE. Kernel does not support both
SVE and NV2 and the current qemu integration has an issue with the
scratch_host_vcpu startup because both are enabled if exposed by the kernel.
This is independent on whether sve is disabled on the command line. Unfortunately
I lost access to the HW that expose that issue so I couldn't fix it in this
version.

This series can be found at:
https://github.com/eauger/qemu/tree/v8.2-nv-rfcv2

Previous version from Miguel:
[1] https://lore.kernel.org/all/20230227163718.62003-1-miguel.luis@oracle.com/
Previous version from Haibo:
[2] https://lore.kernel.org/qemu-devel/cover.1617281290.git.haibo.xu@linaro.org/
[3] Marc's kernel v11 series:
    [PATCH v11 00/43] KVM: arm64: Nested Virtualization support (FEAT_NV2 only)
    https://lore.kernel.org/linux-arm-kernel/20231120131027.854038-1-maz@kernel.org/T/
    available at: https://github.com/eauger/linux/tree/nv-6.8-nv2-v11

Haibo Xu (5):
  [Placeholder] headers: Partial headers update for NV2 enablement
  hw/arm: Allow setting KVM vGIC maintenance IRQ
  target/arm/kvm: Add helper to detect EL2 when using KVM
  target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported
  hw/arm/virt: Allow virt extensions with KVM

 hw/arm/virt.c                      |  6 +++++-
 hw/intc/arm_gicv3_common.c         |  1 +
 hw/intc/arm_gicv3_kvm.c            | 21 +++++++++++++++++++++
 include/hw/intc/arm_gicv3_common.h |  1 +
 linux-headers/asm-arm64/kvm.h      |  1 +
 linux-headers/linux/kvm.h          |  1 +
 target/arm/kvm.c                   | 21 +++++++++++++++++++++
 target/arm/kvm_arm.h               | 12 ++++++++++++
 8 files changed, 63 insertions(+), 1 deletion(-)

-- 
2.41.0
Re: [RFC v2 0/5] ARM Nested Virt Support
Posted by Peter Maydell 2 weeks, 1 day ago
On Fri, 9 Feb 2024 at 16:00, Eric Auger <eric.auger@redhat.com> wrote:
>
> This series adds ARM Nested Virtualization support in KVM mode.
> This is a respin of previous contributions from Miguel [1] and Haibo [2].
>
> This was tested with Marc's v11 [3] on Ampere HW with fedora L1 guest and
> L2 guests booted without EDK2. However it does not work yet with
> EDK2 but it looks unrelated to this qemu integration (host hard lockups).
>
> The host needs to be booted with "kvm-arm.mode=nested" option and
> qemu needs to be invoked with :
>
> -machine virt,virtualization=on
>
> There is a known issue with hosts supporting SVE. Kernel does not support both
> SVE and NV2 and the current qemu integration has an issue with the
> scratch_host_vcpu startup because both are enabled if exposed by the kernel.
> This is independent on whether sve is disabled on the command line. Unfortunately
> I lost access to the HW that expose that issue so I couldn't fix it in this
> version.

You can probably repro that by running the whole setup under
QEMU's FEAT_NV emulation, which will be able to give you a CPU
with both FEAT_NV and SVE.

Personally I think that this is a kernel missing-feature that
should really be fixed as part of getting the kernel patches
upstreamed. There's no cause to force every userspace VMM to
develop extra complications for this.

thanks
-- PMM
Re: [RFC v2 0/5] ARM Nested Virt Support
Posted by Marc Zyngier 1 week, 5 days ago
On 2024-02-09 18:57, Peter Maydell wrote:
> On Fri, 9 Feb 2024 at 16:00, Eric Auger <eric.auger@redhat.com> wrote:
>> 
>> This series adds ARM Nested Virtualization support in KVM mode.
>> This is a respin of previous contributions from Miguel [1] and Haibo 
>> [2].
>> 
>> This was tested with Marc's v11 [3] on Ampere HW with fedora L1 guest 
>> and
>> L2 guests booted without EDK2. However it does not work yet with
>> EDK2 but it looks unrelated to this qemu integration (host hard 
>> lockups).
>> 
>> The host needs to be booted with "kvm-arm.mode=nested" option and
>> qemu needs to be invoked with :
>> 
>> -machine virt,virtualization=on
>> 
>> There is a known issue with hosts supporting SVE. Kernel does not 
>> support both
>> SVE and NV2 and the current qemu integration has an issue with the
>> scratch_host_vcpu startup because both are enabled if exposed by the 
>> kernel.
>> This is independent on whether sve is disabled on the command line. 
>> Unfortunately
>> I lost access to the HW that expose that issue so I couldn't fix it in 
>> this
>> version.
> 
> You can probably repro that by running the whole setup under
> QEMU's FEAT_NV emulation, which will be able to give you a CPU
> with both FEAT_NV and SVE.
> 
> Personally I think that this is a kernel missing-feature that
> should really be fixed as part of getting the kernel patches
> upstreamed. There's no cause to force every userspace VMM to
> develop extra complications for this.

I don't plan to make NV visible to userspace before this is fixed.
Which may delay KVM NV by another year or five, but I don't think
anyone is really waiting for it anyway.

         M.
-- 
Jazz is not dead. It just smells funny...
Re: [RFC v2 0/5] ARM Nested Virt Support
Posted by Eric Auger 1 week, 5 days ago
Hi Peter,

On 2/9/24 19:57, Peter Maydell wrote:
> On Fri, 9 Feb 2024 at 16:00, Eric Auger <eric.auger@redhat.com> wrote:
>> This series adds ARM Nested Virtualization support in KVM mode.
>> This is a respin of previous contributions from Miguel [1] and Haibo [2].
>>
>> This was tested with Marc's v11 [3] on Ampere HW with fedora L1 guest and
>> L2 guests booted without EDK2. However it does not work yet with
>> EDK2 but it looks unrelated to this qemu integration (host hard lockups).
>>
>> The host needs to be booted with "kvm-arm.mode=nested" option and
>> qemu needs to be invoked with :
>>
>> -machine virt,virtualization=on
>>
>> There is a known issue with hosts supporting SVE. Kernel does not support both
>> SVE and NV2 and the current qemu integration has an issue with the
>> scratch_host_vcpu startup because both are enabled if exposed by the kernel.
>> This is independent on whether sve is disabled on the command line. Unfortunately
>> I lost access to the HW that expose that issue so I couldn't fix it in this
>> version.
> You can probably repro that by running the whole setup under
> QEMU's FEAT_NV emulation, which will be able to give you a CPU
> with both FEAT_NV and SVE.

indeed, this should work now we have FEAT_NV.
>
> Personally I think that this is a kernel missing-feature that
> should really be fixed as part of getting the kernel patches
> upstreamed. There's no cause to force every userspace VMM to
> develop extra complications for this.
yes maybe this will be fixed later on.

Thanks!

Eric

>
> thanks
> -- PMM
>