[PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM

Eric Auger posted 5 patches 5 months, 3 weeks ago
Maintainers: Peter Maydell <peter.maydell@linaro.org>, "Michael S. Tsirkin" <mst@redhat.com>, Cornelia Huck <cohuck@redhat.com>, Paolo Bonzini <pbonzini@redhat.com>
There is a newer version of this series
[PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM
Posted by Eric Auger 5 months, 3 weeks ago
From: Haibo Xu <haibo.xu@linaro.org>

Up to now virt support on guest has been only supported with TCG.
Now it becomes feasible to use it with KVM acceleration.

Also check only in-kernel GICv3 is used along with KVM EL2.

Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

---
v2 -> v3:
- check gic version/in-kernel implementation when kvm el2 is set (Peter)

v1 -> v2:
- fixed test ordering: virt && ((kvm && !kvm_el2) || hvf) [Richard]
- tweeked the commit title & message
---
 hw/arm/virt.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 675ddeec14..06a57380b8 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -795,6 +795,13 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem)
     default:
         g_assert_not_reached();
     }
+
+    if (kvm_enabled() && vms->virt &&
+        (revision != 3 || !kvm_irqchip_in_kernel())) {
+        error_report("KVM EL2 only is supported with in-kernel GICv3");
+        exit(1);
+    }
+
     vms->gic = qdev_new(gictype);
     qdev_prop_set_uint32(vms->gic, "revision", revision);
     qdev_prop_set_uint32(vms->gic, "num-cpu", smp_cpus);
@@ -2208,7 +2215,8 @@ static void machvirt_init(MachineState *machine)
         exit(1);
     }
 
-    if (vms->virt && (kvm_enabled() || hvf_enabled())) {
+    if (vms->virt &&
+        ((kvm_enabled() && !kvm_arm_el2_supported()) || hvf_enabled())) {
         error_report("mach-virt: %s does not support providing "
                      "Virtualization extensions to the guest CPU",
                      current_accel_name());
-- 
2.49.0
Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM
Posted by Alyssa Ross 5 months ago
Eric Auger <eric.auger@redhat.com> writes:

> From: Haibo Xu <haibo.xu@linaro.org>
>
> Up to now virt support on guest has been only supported with TCG.
> Now it becomes feasible to use it with KVM acceleration.
>
> Also check only in-kernel GICv3 is used along with KVM EL2.
>
> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

Hi!  From what I can tell, this will produce an error on hosts that
don't support nested virtualization when QEMU is invoked with -accel
kvm:tcg -machine virtualization=on, but I don't think that's the ideal
behaviour.  It would make more sense for it to fall back to the first
permitted accel option that does support running the machine as
configured, so if hardware nested virtualization is not supported, it
should fall back to TCG.

I maintain an OS development environment that includes scripts for
running images in QEMU, where running KVM on those images is a
requirement.  Currently, those scripts simply force TCG on aarch64.
With this change, to take advantage of KVM NV support, I'd have to try
to identify in the script whether NV would be supported.  QEMU would be
in a much better position to determine this and fall back to TCG if it's
unsupported, like how the -accel option with multiple values usually
works.
Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM
Posted by Eric Auger 5 months ago
Hi Alyssa,
On 6/17/25 4:17 PM, Alyssa Ross wrote:
> Eric Auger <eric.auger@redhat.com> writes:
>
>> From: Haibo Xu <haibo.xu@linaro.org>
>>
>> Up to now virt support on guest has been only supported with TCG.
>> Now it becomes feasible to use it with KVM acceleration.
>>
>> Also check only in-kernel GICv3 is used along with KVM EL2.
>>
>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> Hi!  From what I can tell, this will produce an error on hosts that
> don't support nested virtualization when QEMU is invoked with -accel
> kvm:tcg -machine virtualization=on, but I don't think that's the ideal
Thanks for reporting this issue. This is not an expected behavior. I
will respin hopefully this week so I will have give it a try.

What kind of error msg do you get?

Thanks

Eric
> behaviour.  It would make more sense for it to fall back to the first
> permitted accel option that does support running the machine as
> configured, so if hardware nested virtualization is not supported, it
> should fall back to TCG.
>
> I maintain an OS development environment that includes scripts for
> running images in QEMU, where running KVM on those images is a
> requirement.  Currently, those scripts simply force TCG on aarch64.
> With this change, to take advantage of KVM NV support, I'd have to try
> to identify in the script whether NV would be supported.  QEMU would be
> in a much better position to determine this and fall back to TCG if it's
> unsupported, like how the -accel option with multiple values usually
> works.
Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM
Posted by Eric Auger 5 months ago
Hi Alyssa,

On 6/17/25 4:17 PM, Alyssa Ross wrote:
> Eric Auger <eric.auger@redhat.com> writes:
>
>> From: Haibo Xu <haibo.xu@linaro.org>
>>
>> Up to now virt support on guest has been only supported with TCG.
>> Now it becomes feasible to use it with KVM acceleration.
>>
>> Also check only in-kernel GICv3 is used along with KVM EL2.
>>
>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> Hi!  From what I can tell, this will produce an error on hosts that
> don't support nested virtualization when QEMU is invoked with -accel
> kvm:tcg -machine virtualization=on, but I don't think that's the ideal
> behaviour.  It would make more sense for it to fall back to the first
> permitted accel option that does support running the machine as
> configured, so if hardware nested virtualization is not supported, it
> should fall back to TCG.

Hum I think I misundertood your point.
I guess you meant you get "qemu-system-aarch64: mach-virt: KVM does not
support providing Virtualization extensions to the guest CPU" which
sounds the usual behavior of qemu in case we attempt to run a feature on
a host that does not support it.


>
> I maintain an OS development environment that includes scripts for
> running images in QEMU, where running KVM on those images is a
> requirement.  Currently, those scripts simply force TCG on aarch64.
> With this change, to take advantage of KVM NV support, I'd have to try
> to identify in the script whether NV would be supported.  QEMU would be
> in a much better position to determine this and fall back to TCG if it's
> unsupported, like how the -accel option with multiple values usually
> works.
Switching to another accel mode is a policy that is not implemented by
qemu. There is no such situation where qemu would switch from kvm accel
mode to TCG. You need to implement this policy in your test env I think

Thanks

Eric
Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM
Posted by Miguel Luis 5 months ago
Hi Alyssa,

> On 17 Jun 2025, at 14:17, Alyssa Ross <hi@alyssa.is> wrote:
> 
> Eric Auger <eric.auger@redhat.com> writes:
> 
>> From: Haibo Xu <haibo.xu@linaro.org>
>> 
>> Up to now virt support on guest has been only supported with TCG.
>> Now it becomes feasible to use it with KVM acceleration.
>> 
>> Also check only in-kernel GICv3 is used along with KVM EL2.
>> 
>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> 
> Hi!  From what I can tell, this will produce an error on hosts that
> don't support nested virtualization when QEMU is invoked with -accel
> kvm:tcg

I didn’t know '-acell kvm:tcg’ could be used as a fallback mechanism between
acceleration modes. May I ask whether do you manage the ‘-cpu’ type for ‘-accel
kvm:tcg’ with cpu ‘max’ ?

But more importantly, is this what you’re referring to?

Although,

> -machine virtualization=on,

should work for both '-accel kvm’ and ‘-accel tcg’.

> but I don't think that's the ideal
> behaviour.  It would make more sense for it to fall back to the first
> permitted accel option that does support running the machine as
> configured, so if hardware nested virtualization is not supported, it
> should fall back to TCG.
> 
> I maintain an OS development environment that includes scripts for
> running images in QEMU, where running KVM on those images is a
> requirement.  Currently, those scripts simply force TCG on aarch64.
> With this change, to take advantage of KVM NV support, I'd have to try
> to identify in the script whether NV would be supported.  QEMU would be
> in a much better position to determine this and fall back to TCG if it's
> unsupported, like how the -accel option with multiple values usually
> works.

Thanks,
Miguel
Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM
Posted by Eric Auger 5 months ago

On 6/17/25 5:23 PM, Miguel Luis wrote:
> Hi Alyssa,
>
>> On 17 Jun 2025, at 14:17, Alyssa Ross <hi@alyssa.is> wrote:
>>
>> Eric Auger <eric.auger@redhat.com> writes:
>>
>>> From: Haibo Xu <haibo.xu@linaro.org>
>>>
>>> Up to now virt support on guest has been only supported with TCG.
>>> Now it becomes feasible to use it with KVM acceleration.
>>>
>>> Also check only in-kernel GICv3 is used along with KVM EL2.
>>>
>>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>> Hi!  From what I can tell, this will produce an error on hosts that
>> don't support nested virtualization when QEMU is invoked with -accel
>> kvm:tcg
> I didn’t know '-acell kvm:tcg’ could be used as a fallback mechanism between
> acceleration modes. May I ask whether do you manage the ‘-cpu’ type for ‘-accel
> kvm:tcg’ with cpu ‘max’ ?
Does it exist?
qemu-system-aarch64: -accel kvm:tcg: invalid accelerator kvm:tcg

Alyssa, didn't you mean -accel kvm or --accel tcg
>
> But more importantly, is this what you’re referring to?
>
> Although,
>
>> -machine virtualization=on,
> should work for both '-accel kvm’ and ‘-accel tcg’.
>
>> but I don't think that's the ideal
>> behaviour.  It would make more sense for it to fall back to the first
>> permitted accel option that does support running the machine as
>> configured, so if hardware nested virtualization is not supported, it
>> should fall back to TCG.
>>
>> I maintain an OS development environment that includes scripts for
>> running images in QEMU, where running KVM on those images is a
>> requirement.  Currently, those scripts simply force TCG on aarch64.
>> With this change, to take advantage of KVM NV support, I'd have to try
>> to identify in the script whether NV would be supported.  QEMU would be
>> in a much better position to determine this and fall back to TCG if it's
>> unsupported, like how the -accel option with multiple values usually
>> works.
> Thanks,
> Miguel


Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM
Posted by Miguel Luis 5 months ago
Hi Eric,

> On 17 Jun 2025, at 15:41, Eric Auger <eric.auger@redhat.com> wrote:
> 
> 
> 
> On 6/17/25 5:23 PM, Miguel Luis wrote:
>> Hi Alyssa,
>> 
>>> On 17 Jun 2025, at 14:17, Alyssa Ross <hi@alyssa.is> wrote:
>>> 
>>> Eric Auger <eric.auger@redhat.com> writes:
>>> 
>>>> From: Haibo Xu <haibo.xu@linaro.org>
>>>> 
>>>> Up to now virt support on guest has been only supported with TCG.
>>>> Now it becomes feasible to use it with KVM acceleration.
>>>> 
>>>> Also check only in-kernel GICv3 is used along with KVM EL2.
>>>> 
>>>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>>>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>>> Hi!  From what I can tell, this will produce an error on hosts that
>>> don't support nested virtualization when QEMU is invoked with -accel
>>> kvm:tcg
>> I didn’t know '-acell kvm:tcg’ could be used as a fallback mechanism between
>> acceleration modes. May I ask whether do you manage the ‘-cpu’ type for ‘-accel
>> kvm:tcg’ with cpu ‘max’ ?
> Does it exist?
> qemu-system-aarch64: -accel kvm:tcg: invalid accelerator kvm:tcg

Maybe Alyssa is referring to ‘-M virt,accel=kvm:tcg,virtualization=on,gic-version=3’ ?

The above didn’t triggered any error. Anyhow if the above does what Alyssa’s saying 
we would just be missing the check for || !tcg_enabled() in this patch, I believe.

Miguel

> 
> Alyssa, didn't you mean -accel kvm or --accel tcg
>> 
>> But more importantly, is this what you’re referring to?
>> 
>> Although,
>> 
>>> -machine virtualization=on,
>> should work for both '-accel kvm’ and ‘-accel tcg’.
>> 
>>> but I don't think that's the ideal
>>> behaviour.  It would make more sense for it to fall back to the first
>>> permitted accel option that does support running the machine as
>>> configured, so if hardware nested virtualization is not supported, it
>>> should fall back to TCG.
>>> 
>>> I maintain an OS development environment that includes scripts for
>>> running images in QEMU, where running KVM on those images is a
>>> requirement.  Currently, those scripts simply force TCG on aarch64.
>>> With this change, to take advantage of KVM NV support, I'd have to try
>>> to identify in the script whether NV would be supported.  QEMU would be
>>> in a much better position to determine this and fall back to TCG if it's
>>> unsupported, like how the -accel option with multiple values usually
>>> works.
>> Thanks,
>> Miguel


Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM
Posted by Eric Auger 4 months, 4 weeks ago
Hi,

On 6/17/25 5:50 PM, Miguel Luis wrote:
> Hi Eric,
>
>> On 17 Jun 2025, at 15:41, Eric Auger <eric.auger@redhat.com> wrote:
>>
>>
>>
>> On 6/17/25 5:23 PM, Miguel Luis wrote:
>>> Hi Alyssa,
>>>
>>>> On 17 Jun 2025, at 14:17, Alyssa Ross <hi@alyssa.is> wrote:
>>>>
>>>> Eric Auger <eric.auger@redhat.com> writes:
>>>>
>>>>> From: Haibo Xu <haibo.xu@linaro.org>
>>>>>
>>>>> Up to now virt support on guest has been only supported with TCG.
>>>>> Now it becomes feasible to use it with KVM acceleration.
>>>>>
>>>>> Also check only in-kernel GICv3 is used along with KVM EL2.
>>>>>
>>>>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>>>>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>>>> Hi!  From what I can tell, this will produce an error on hosts that
>>>> don't support nested virtualization when QEMU is invoked with -accel
>>>> kvm:tcg
>>> I didn’t know '-acell kvm:tcg’ could be used as a fallback mechanism between
>>> acceleration modes. May I ask whether do you manage the ‘-cpu’ type for ‘-accel
>>> kvm:tcg’ with cpu ‘max’ ?
>> Does it exist?
>> qemu-system-aarch64: -accel kvm:tcg: invalid accelerator kvm:tcg
> Maybe Alyssa is referring to ‘-M virt,accel=kvm:tcg,virtualization=on,gic-version=3’ ?
>
> The above didn’t triggered any error. Anyhow if the above does what Alyssa’s saying 
> we would just be missing the check for || !tcg_enabled() in this patch, I believe.

After discussion with Paolo, the lack of the CAP should be detected
earlier in kvm_init/kvm_arch_init to allow the fallback to TCG.
in target/arm/kvm.c kvm_arch_init() some generic caps are checked but
none of them are related to machine settings and this code is virt arm
machine agnostic.

I checked and adding

    if (object_dynamic_cast(OBJECT(ms), TYPE_VIRT_MACHINE)) {
        VirtMachineState *vms = VIRT_MACHINE(ms);

        if (vms->virt && !kvm_arm_el2_supported()) {
            error_report("KVM does not support nested virtualization");
            ret = -EINVAL;
        }
    }

at the end of the function would do the job. But as I said previously
this is not done for other virt arm machine options that are accel
specific or require special KVM caps (secure, mte for instance) so it
would be a change in the approach.

As pointed out before fallback spirit was rather: "KVM isn't enabled"
than for "KVM doesn't support a specific feature".

Thanks

Eric




it's purely the first accelerator that says it can work
1:03
<https://redhat-internal.slack.com/archives/C04KFKV2SE9/p1750331005242709>
pbonzini
Where "can work" is only based on the host.


>
> Miguel
>
>> Alyssa, didn't you mean -accel kvm or --accel tcg
>>> But more importantly, is this what you’re referring to?
>>>
>>> Although,
>>>
>>>> -machine virtualization=on,
>>> should work for both '-accel kvm’ and ‘-accel tcg’.
>>>
>>>> but I don't think that's the ideal
>>>> behaviour.  It would make more sense for it to fall back to the first
>>>> permitted accel option that does support running the machine as
>>>> configured, so if hardware nested virtualization is not supported, it
>>>> should fall back to TCG.
>>>>
>>>> I maintain an OS development environment that includes scripts for
>>>> running images in QEMU, where running KVM on those images is a
>>>> requirement.  Currently, those scripts simply force TCG on aarch64.
>>>> With this change, to take advantage of KVM NV support, I'd have to try
>>>> to identify in the script whether NV would be supported.  QEMU would be
>>>> in a much better position to determine this and fall back to TCG if it's
>>>> unsupported, like how the -accel option with multiple values usually
>>>> works.
>>> Thanks,
>>> Miguel
>


Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM
Posted by Miguel Luis 4 months, 4 weeks ago
Hi Eric,

> On 19 Jun 2025, at 13:29, Eric Auger <eric.auger@redhat.com> wrote:
> 
> Hi,
> 
> On 6/17/25 5:50 PM, Miguel Luis wrote:
>> Hi Eric,
>> 
>>> On 17 Jun 2025, at 15:41, Eric Auger <eric.auger@redhat.com> wrote:
>>> 
>>> 
>>> 
>>> On 6/17/25 5:23 PM, Miguel Luis wrote:
>>>> Hi Alyssa,
>>>> 
>>>>> On 17 Jun 2025, at 14:17, Alyssa Ross <hi@alyssa.is> wrote:
>>>>> 
>>>>> Eric Auger <eric.auger@redhat.com> writes:
>>>>> 
>>>>>> From: Haibo Xu <haibo.xu@linaro.org>
>>>>>> 
>>>>>> Up to now virt support on guest has been only supported with TCG.
>>>>>> Now it becomes feasible to use it with KVM acceleration.
>>>>>> 
>>>>>> Also check only in-kernel GICv3 is used along with KVM EL2.
>>>>>> 
>>>>>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>>>>>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>>>>> Hi!  From what I can tell, this will produce an error on hosts that
>>>>> don't support nested virtualization when QEMU is invoked with -accel
>>>>> kvm:tcg
>>>> I didn’t know '-acell kvm:tcg’ could be used as a fallback mechanism between
>>>> acceleration modes. May I ask whether do you manage the ‘-cpu’ type for ‘-accel
>>>> kvm:tcg’ with cpu ‘max’ ?
>>> Does it exist?
>>> qemu-system-aarch64: -accel kvm:tcg: invalid accelerator kvm:tcg
>> Maybe Alyssa is referring to ‘-M virt,accel=kvm:tcg,virtualization=on,gic-version=3’ ?
>> 
>> The above didn’t triggered any error. Anyhow if the above does what Alyssa’s saying 
>> we would just be missing the check for || !tcg_enabled() in this patch, I believe.
> 
> After discussion with Paolo, the lack of the CAP should be detected
> earlier in kvm_init/kvm_arch_init to allow the fallback to TCG.
> in target/arm/kvm.c kvm_arch_init() some generic caps are checked but
> none of them are related to machine settings and this code is virt arm
> machine agnostic.
> 
> I checked and adding
> 
>     if (object_dynamic_cast(OBJECT(ms), TYPE_VIRT_MACHINE)) {
>         VirtMachineState *vms = VIRT_MACHINE(ms);
> 
>         if (vms->virt && !kvm_arm_el2_supported()) {
>             error_report("KVM does not support nested virtualization");
>             ret = -EINVAL;
>         }
>     }
> 
> at the end of the function would do the job. But as I said previously
> this is not done for other virt arm machine options that are accel
> specific or require special KVM caps (secure, mte for instance) so it
> would be a change in the approach.
> 

Got it.

> As pointed out before fallback spirit was rather: "KVM isn't enabled"
> than for "KVM doesn't support a specific feature".
> 

Thank you for getting to the bottom of it.

Miguel

> Thanks
> 
> Eric
> 
> 
> 
> 
> it's purely the first accelerator that says it can work
> 1:03
> <https://redhat-internal.slack.com/archives/C04KFKV2SE9/p1750331005242709>
> pbonzini
> Where "can work" is only based on the host.
> 
> 
>> 
>> Miguel
>> 
>>> Alyssa, didn't you mean -accel kvm or --accel tcg
>>>> But more importantly, is this what you’re referring to?
>>>> 
>>>> Although,
>>>> 
>>>>> -machine virtualization=on,
>>>> should work for both '-accel kvm’ and ‘-accel tcg’.
>>>> 
>>>>> but I don't think that's the ideal
>>>>> behaviour.  It would make more sense for it to fall back to the first
>>>>> permitted accel option that does support running the machine as
>>>>> configured, so if hardware nested virtualization is not supported, it
>>>>> should fall back to TCG.
>>>>> 
>>>>> I maintain an OS development environment that includes scripts for
>>>>> running images in QEMU, where running KVM on those images is a
>>>>> requirement.  Currently, those scripts simply force TCG on aarch64.
>>>>> With this change, to take advantage of KVM NV support, I'd have to try
>>>>> to identify in the script whether NV would be supported.  QEMU would be
>>>>> in a much better position to determine this and fall back to TCG if it's
>>>>> unsupported, like how the -accel option with multiple values usually
>>>>> works.
>>>> Thanks,
>>>> Miguel


Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM
Posted by Alyssa Ross 4 months, 4 weeks ago
Eric Auger <eric.auger@redhat.com> writes:

> Hi,
>
> On 6/17/25 5:50 PM, Miguel Luis wrote:
>> Hi Eric,
>>
>>> On 17 Jun 2025, at 15:41, Eric Auger <eric.auger@redhat.com> wrote:
>>>
>>>
>>>
>>> On 6/17/25 5:23 PM, Miguel Luis wrote:
>>>> Hi Alyssa,
>>>>
>>>>> On 17 Jun 2025, at 14:17, Alyssa Ross <hi@alyssa.is> wrote:
>>>>>
>>>>> Eric Auger <eric.auger@redhat.com> writes:
>>>>>
>>>>>> From: Haibo Xu <haibo.xu@linaro.org>
>>>>>>
>>>>>> Up to now virt support on guest has been only supported with TCG.
>>>>>> Now it becomes feasible to use it with KVM acceleration.
>>>>>>
>>>>>> Also check only in-kernel GICv3 is used along with KVM EL2.
>>>>>>
>>>>>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>>>>>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>>>>> Hi!  From what I can tell, this will produce an error on hosts that
>>>>> don't support nested virtualization when QEMU is invoked with -accel
>>>>> kvm:tcg
>>>> I didn’t know '-acell kvm:tcg’ could be used as a fallback mechanism between
>>>> acceleration modes. May I ask whether do you manage the ‘-cpu’ type for ‘-accel
>>>> kvm:tcg’ with cpu ‘max’ ?
>>> Does it exist?
>>> qemu-system-aarch64: -accel kvm:tcg: invalid accelerator kvm:tcg
>> Maybe Alyssa is referring to ‘-M virt,accel=kvm:tcg,virtualization=on,gic-version=3’ ?
>>
>> The above didn’t triggered any error. Anyhow if the above does what Alyssa’s saying 
>> we would just be missing the check for || !tcg_enabled() in this patch, I believe.
>
> After discussion with Paolo, the lack of the CAP should be detected
> earlier in kvm_init/kvm_arch_init to allow the fallback to TCG.
> in target/arm/kvm.c kvm_arch_init() some generic caps are checked but
> none of them are related to machine settings and this code is virt arm
> machine agnostic.
>
> I checked and adding
>
>     if (object_dynamic_cast(OBJECT(ms), TYPE_VIRT_MACHINE)) {
>         VirtMachineState *vms = VIRT_MACHINE(ms);
>
>         if (vms->virt && !kvm_arm_el2_supported()) {
>             error_report("KVM does not support nested virtualization");
>             ret = -EINVAL;
>         }
>     }
>
> at the end of the function would do the job. But as I said previously
> this is not done for other virt arm machine options that are accel
> specific or require special KVM caps (secure, mte for instance) so it
> would be a change in the approach.

Right — if it's not done for other options it doesn't need to be done
here either; I wasn't aware of that.  It would be nice if QEMU worked
differently, but that's certainly not in scope of this series.
Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM
Posted by Eric Auger 4 months, 4 weeks ago
Hi Miguel,

On 6/17/25 5:50 PM, Miguel Luis wrote:
> Hi Eric,
>
>> On 17 Jun 2025, at 15:41, Eric Auger <eric.auger@redhat.com> wrote:
>>
>>
>>
>> On 6/17/25 5:23 PM, Miguel Luis wrote:
>>> Hi Alyssa,
>>>
>>>> On 17 Jun 2025, at 14:17, Alyssa Ross <hi@alyssa.is> wrote:
>>>>
>>>> Eric Auger <eric.auger@redhat.com> writes:
>>>>
>>>>> From: Haibo Xu <haibo.xu@linaro.org>
>>>>>
>>>>> Up to now virt support on guest has been only supported with TCG.
>>>>> Now it becomes feasible to use it with KVM acceleration.
>>>>>
>>>>> Also check only in-kernel GICv3 is used along with KVM EL2.
>>>>>
>>>>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>>>>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>>>> Hi!  From what I can tell, this will produce an error on hosts that
>>>> don't support nested virtualization when QEMU is invoked with -accel
>>>> kvm:tcg
>>> I didn’t know '-acell kvm:tcg’ could be used as a fallback mechanism between
>>> acceleration modes. May I ask whether do you manage the ‘-cpu’ type for ‘-accel
>>> kvm:tcg’ with cpu ‘max’ ?
>> Does it exist?
>> qemu-system-aarch64: -accel kvm:tcg: invalid accelerator kvm:tcg
> Maybe Alyssa is referring to ‘-M virt,accel=kvm:tcg,virtualization=on,gic-version=3’ ?
>
> The above didn’t triggered any error. Anyhow if the above does what Alyssa’s saying 
> we would just be missing the check for || !tcg_enabled() in this patch, I believe.
You meant you would report an error only if !tcg_enabled()?

In practice tcg_enabled() does not seem to be set when trying kvm first.

Refering to the ancestor of this feature - which I did not know I
acknowledge-,
https://patchwork.ozlabs.org/project/qemu-devel/patch/1275954730-8196-20-git-send-email-aliguori@us.ibm.com/
it is said:

kvm|tcg tries kvm and fallbacks on tcg if kvm support isn't available

It does not state that each option is evaluated with kvm_enabled() and in case it fails enabling one because the host does not support it, it falls back to TCG.

I checked for instance with already integrated MTE option and if the host does not support it, it does not offer the capability to fall back to TCG.

Adding Peter in TO to get his feeling on this.

Thanks

Eric

Besides

>
> Miguel
>
>> Alyssa, didn't you mean -accel kvm or --accel tcg
>>> But more importantly, is this what you’re referring to?
>>>
>>> Although,
>>>
>>>> -machine virtualization=on,
>>> should work for both '-accel kvm’ and ‘-accel tcg’.
>>>
>>>> but I don't think that's the ideal
>>>> behaviour.  It would make more sense for it to fall back to the first
>>>> permitted accel option that does support running the machine as
>>>> configured, so if hardware nested virtualization is not supported, it
>>>> should fall back to TCG.
>>>>
>>>> I maintain an OS development environment that includes scripts for
>>>> running images in QEMU, where running KVM on those images is a
>>>> requirement.  Currently, those scripts simply force TCG on aarch64.
>>>> With this change, to take advantage of KVM NV support, I'd have to try
>>>> to identify in the script whether NV would be supported.  QEMU would be
>>>> in a much better position to determine this and fall back to TCG if it's
>>>> unsupported, like how the -accel option with multiple values usually
>>>> works.
>>> Thanks,
>>> Miguel
>