This series add support for ARMv8.3/8.4 nested virtualization support
in KVM mode. It's based on Marc Zyngier's kernel KVM patches[1], and
has been tested on a FVP model to run a L2 guest with Qemu. Now the
feature can be enabled by "-M virt,accel=kvm,virtualization=on" when
starting a VM.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/nv-5.12-WIP

Haibo Xu (3):
  Update linux header with new arm64 NV macro.
  Enable support for setting KVM vGIC maintenance IRQ
  Enable nested virtualization support in arm64 KVM mode

 hw/arm/virt.c                      | 11 ++++++++---
 hw/intc/arm_gicv3_common.c         |  1 +
 hw/intc/arm_gicv3_kvm.c            | 16 ++++++++++++++++
 include/hw/intc/arm_gicv3_common.h |  1 +
 linux-headers/asm-arm64/kvm.h      |  2 ++
 linux-headers/linux/kvm.h          |  1 +
 target/arm/cpu.h                   |  8 ++++++++
 target/arm/kvm64.c                 | 14 ++++++++++++++
 target/arm/kvm_arm.h               | 28 ++++++++++++++++++++++++++++
 9 files changed, 79 insertions(+), 3 deletions(-)

-- 
2.17.1

On Mon, 2021-03-22 at 10:07 +0000, Haibo Xu wrote:
> This series add support for ARMv8.3/8.4 nested virtualization support
> in KVM mode. It's based on Marc Zyngier's kernel KVM patches[1], and
> has been tested on a FVP model to run a L2 guest with Qemu. Now the
> feature can be enabled by "-M virt,accel=kvm,virtualization=on" when
> starting a VM.

Why the need to enable this explicitly? AFAIK, that's not necessary
for any other architecture: on x86, you just need to make sure you're
using '-cpu host' and pass a parameter to the kernel module.

Even assuming this can't be enabled transparently, wouldn't its
availability it be controlled by a CPU feature flag, similar to what
already happens for SVE and PMU, rather than a machine type option?

That would also address the discoverability issue: unless I'm
mistaken (which I very well might be :), with the current
implementation there's no way to tell whether nested KVM will be
usable short of trying and seeing whether QEMU errors out.

-- 
Andrea Bolognani / Red Hat / Virtualization

On Mon, Mar 22, 2021 at 04:42:23PM +0100, Andrea Bolognani wrote:
> On Mon, 2021-03-22 at 10:07 +0000, Haibo Xu wrote:
> > This series add support for ARMv8.3/8.4 nested virtualization support
> > in KVM mode. It's based on Marc Zyngier's kernel KVM patches[1], and
> > has been tested on a FVP model to run a L2 guest with Qemu. Now the
> > feature can be enabled by "-M virt,accel=kvm,virtualization=on" when
> > starting a VM.
> 
> Why the need to enable this explicitly? AFAIK, that's not necessary
> for any other architecture: on x86, you just need to make sure you're
> using '-cpu host' and pass a parameter to the kernel module.
> 
> Even assuming this can't be enabled transparently, wouldn't its
> availability it be controlled by a CPU feature flag, similar to what
> already happens for SVE and PMU, rather than a machine type option?

I 100% agree. We should control this feature with a CPU feature property.
NV is a CPU feature, after all. Also, we should add it to the properties
that we can probe in cpu_model_advertised_features[].

Thanks,
drew

> 
> That would also address the discoverability issue: unless I'm
> mistaken (which I very well might be :), with the current
> implementation there's no way to tell whether nested KVM will be
> usable short of trying and seeing whether QEMU errors out.
> 
> -- 
> Andrea Bolognani / Red Hat / Virtualization
> 
> 

On Tue, 23 Mar 2021 at 00:32, Andrew Jones <drjones@redhat.com> wrote:
>
> On Mon, Mar 22, 2021 at 04:42:23PM +0100, Andrea Bolognani wrote:
> > On Mon, 2021-03-22 at 10:07 +0000, Haibo Xu wrote:
> > > This series add support for ARMv8.3/8.4 nested virtualization support
> > > in KVM mode. It's based on Marc Zyngier's kernel KVM patches[1], and
> > > has been tested on a FVP model to run a L2 guest with Qemu. Now the
> > > feature can be enabled by "-M virt,accel=kvm,virtualization=on" when
> > > starting a VM.
> >
> > Why the need to enable this explicitly? AFAIK, that's not necessary
> > for any other architecture: on x86, you just need to make sure you're
> > using '-cpu host' and pass a parameter to the kernel module.
> >
> > Even assuming this can't be enabled transparently, wouldn't its
> > availability it be controlled by a CPU feature flag, similar to what
> > already happens for SVE and PMU, rather than a machine type option?
>
> I 100% agree. We should control this feature with a CPU feature property.
> NV is a CPU feature, after all. Also, we should add it to the properties
> that we can probe in cpu_model_advertised_features[].
>

Thanks so much for the comments!

Yes, NV should be a vCPU feature, just as the kernel macro
KVM_ARM_VCPU_HAS_EL2 implies.
To implement it as a VM feature here just want to reuse the codes in
TCG mode which emulate a
guest CPU with virtualization extension support.

I will change the NV in KVM mode to a vCPU feature in  the next version.
@Peter Maydell and @Richard Henderson, shall we change that in TCG mode as well?

Thanks,
Haibo

> Thanks,
> drew
>
> >
> > That would also address the discoverability issue: unless I'm
> > mistaken (which I very well might be :), with the current
> > implementation there's no way to tell whether nested KVM will be
> > usable short of trying and seeing whether QEMU errors out.
> >
> > --
> > Andrea Bolognani / Red Hat / Virtualization
> >
> >
>