include/hw/intc/arm_gicv3_common.h | 1 + include/standard-headers/linux/virtio_pci.h | 1 + linux-headers/asm-arm64/kvm.h | 9 +++++---- linux-headers/linux/kvm.h | 3 +++ linux-headers/linux/vhost.h | 4 ++-- target/arm/kvm_arm.h | 7 +++++++ hw/arm/virt.c | 13 ++++++++++++- hw/intc/arm_gicv3_common.c | 1 + hw/intc/arm_gicv3_kvm.c | 21 +++++++++++++++++++++ target/arm/kvm-stub.c | 5 +++++ target/arm/kvm.c | 21 +++++++++++++++++++++ 11 files changed, 79 insertions(+), 7 deletions(-)
Now that ARM nested virt has landed in kvm/next, let's turn the series into a PATCH series. The linux header update was made against kvm/next. For gaining virt functionality in KVM accelerated L1, The host needs to be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked with: -machine virt,virtualization=on. This series can be found at: https://github.com/eauger/qemu/tree/v10.0.0-nv-v5 Original version from Miguel: [1] https://lore.kernel.org/all/20230227163718.62003-1-miguel.luis@oracle.com/ version from Haibo: [2] https://lore.kernel.org/qemu-devel/cover.1617281290.git.haibo.xu@linaro.org/ History: v4 -> v5: - rebase on top of v10.0.0 v3 -> v4: - fix: only set maint_irq if vms->virt v2 -> v3: - KVM EL2 only is set if virtualization option is set - fixes regression with virtualization=off - Add checks against unsupported GIC configs until the kernel does Eric Auger (1): linux-headers: Update against kvm/next Haibo Xu (4): hw/arm: Allow setting KVM vGIC maintenance IRQ target/arm/kvm: Add helper to detect EL2 when using KVM target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported hw/arm/virt: Allow virt extensions with KVM include/hw/intc/arm_gicv3_common.h | 1 + include/standard-headers/linux/virtio_pci.h | 1 + linux-headers/asm-arm64/kvm.h | 9 +++++---- linux-headers/linux/kvm.h | 3 +++ linux-headers/linux/vhost.h | 4 ++-- target/arm/kvm_arm.h | 7 +++++++ hw/arm/virt.c | 13 ++++++++++++- hw/intc/arm_gicv3_common.c | 1 + hw/intc/arm_gicv3_kvm.c | 21 +++++++++++++++++++++ target/arm/kvm-stub.c | 5 +++++ target/arm/kvm.c | 21 +++++++++++++++++++++ 11 files changed, 79 insertions(+), 7 deletions(-) -- 2.49.0
Hi Eric, > On 27 May 2025, at 06:24, Eric Auger <eric.auger@redhat.com> wrote: > > Now that ARM nested virt has landed in kvm/next, let's turn the series > into a PATCH series. The linux header update was made against kvm/next. > > For gaining virt functionality in KVM accelerated L1, The host needs to > be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked > with: -machine virt,virtualization=on. For the current kvmarm/next the guest also needs “kvm-arm.mode=nested” I believe. > > This series can be found at: > https://github.com/eauger/qemu/tree/v10.0.0-nv-v5 > Thanks for the respin. Cheers Miguel > Original version from Miguel: > [1] https://lore.kernel.org/all/20230227163718.62003-1-miguel.luis@oracle.com/ > version from Haibo: > [2] https://lore.kernel.org/qemu-devel/cover.1617281290.git.haibo.xu@linaro.org/ > > History: > v4 -> v5: > - rebase on top of v10.0.0 > > v3 -> v4: > - fix: only set maint_irq if vms->virt > > v2 -> v3: > - KVM EL2 only is set if virtualization option is set > - fixes regression with virtualization=off > - Add checks against unsupported GIC configs until the kernel does > > Eric Auger (1): > linux-headers: Update against kvm/next > > Haibo Xu (4): > hw/arm: Allow setting KVM vGIC maintenance IRQ > target/arm/kvm: Add helper to detect EL2 when using KVM > target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported > hw/arm/virt: Allow virt extensions with KVM > > include/hw/intc/arm_gicv3_common.h | 1 + > include/standard-headers/linux/virtio_pci.h | 1 + > linux-headers/asm-arm64/kvm.h | 9 +++++---- > linux-headers/linux/kvm.h | 3 +++ > linux-headers/linux/vhost.h | 4 ++-- > target/arm/kvm_arm.h | 7 +++++++ > hw/arm/virt.c | 13 ++++++++++++- > hw/intc/arm_gicv3_common.c | 1 + > hw/intc/arm_gicv3_kvm.c | 21 +++++++++++++++++++++ > target/arm/kvm-stub.c | 5 +++++ > target/arm/kvm.c | 21 +++++++++++++++++++++ > 11 files changed, 79 insertions(+), 7 deletions(-) > > -- > 2.49.0 >
On Tue, 27 May 2025 12:33:23 +0100, Miguel Luis <miguel.luis@oracle.com> wrote: > > Hi Eric, > > > On 27 May 2025, at 06:24, Eric Auger <eric.auger@redhat.com> wrote: > > > > Now that ARM nested virt has landed in kvm/next, let's turn the series > > into a PATCH series. The linux header update was made against kvm/next. > > > > For gaining virt functionality in KVM accelerated L1, The host needs to > > be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked > > with: -machine virt,virtualization=on. > > For the current kvmarm/next the guest also needs > “kvm-arm.mode=nested” I believe. No, unless you want the guest itself to be NV-capable. M. -- Without deviation from the norm, progress is not possible.
> On 27 May 2025, at 12:01, Marc Zyngier <maz@kernel.org> wrote: > > On Tue, 27 May 2025 12:33:23 +0100, > Miguel Luis <miguel.luis@oracle.com> wrote: >> >> Hi Eric, >> >>> On 27 May 2025, at 06:24, Eric Auger <eric.auger@redhat.com> wrote: >>> >>> Now that ARM nested virt has landed in kvm/next, let's turn the series >>> into a PATCH series. The linux header update was made against kvm/next. >>> >>> For gaining virt functionality in KVM accelerated L1, The host needs to >>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked >>> with: -machine virt,virtualization=on. >> >> For the current kvmarm/next the guest also needs >> “kvm-arm.mode=nested” I believe. > > No, unless you want the guest itself to be NV-capable. > Correct, I got carried away with some mode combinations. Maybe we should depict here more broadly how NV might be used with different mode combinations. I'll think about this further ahead. As far this series go I couldn't found any issue booting a L1 guest with virtualization=on and a L2 guest with virtualization=off. Miguel > M. > > -- > Without deviation from the norm, progress is not possible.
Hi Miguel, Marc, On 5/27/25 2:54 PM, Miguel Luis wrote: > >> On 27 May 2025, at 12:01, Marc Zyngier <maz@kernel.org> wrote: >> >> On Tue, 27 May 2025 12:33:23 +0100, >> Miguel Luis <miguel.luis@oracle.com> wrote: >>> Hi Eric, >>> >>>> On 27 May 2025, at 06:24, Eric Auger <eric.auger@redhat.com> wrote: >>>> >>>> Now that ARM nested virt has landed in kvm/next, let's turn the series >>>> into a PATCH series. The linux header update was made against kvm/next. >>>> >>>> For gaining virt functionality in KVM accelerated L1, The host needs to >>>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked >>>> with: -machine virt,virtualization=on. >>> For the current kvmarm/next the guest also needs >>> “kvm-arm.mode=nested” I believe. >> No, unless you want the guest itself to be NV-capable. Effectively you don't need "kvm-arm.mode=nested" except if you want want multiple level of nesting. >> > Correct, I got carried away with some mode combinations. Maybe we should depict > here more broadly how NV might be used with different mode combinations. I'll > think about this further ahead. > > As far this series go I couldn't found any issue booting a L1 guest with > virtualization=on and a L2 guest with virtualization=off. on my end I tested with various untouched L2 guests (debian, fed, rhel) in 4kB/4kB/4KB page size mode (host, L1, L2). Those configs were successful. with 64kB/64kB/64kB configs I am less lucky atm. One one machine I cannot boot L1 with virtualization=on. On the other I can boot L1 but cannot boot L2. Trying my best to debug a little bit further with my setup. Anyway if somebody else can try 64kB configs, it would help to confirm whether there are pending issues. I don't think they are related to this qemu integration series though. Thanks Eric > > Miguel > >> M. >> >> -- >> Without deviation from the norm, progress is not possible. >
On Tue, 27 May 2025 14:11:41 +0100, Eric Auger <eric.auger@redhat.com> wrote: > > Hi Miguel, Marc, > > On 5/27/25 2:54 PM, Miguel Luis wrote: > > > >> On 27 May 2025, at 12:01, Marc Zyngier <maz@kernel.org> wrote: > >> > >> On Tue, 27 May 2025 12:33:23 +0100, > >> Miguel Luis <miguel.luis@oracle.com> wrote: > >>> Hi Eric, > >>> > >>>> On 27 May 2025, at 06:24, Eric Auger <eric.auger@redhat.com> wrote: > >>>> > >>>> Now that ARM nested virt has landed in kvm/next, let's turn the series > >>>> into a PATCH series. The linux header update was made against kvm/next. > >>>> > >>>> For gaining virt functionality in KVM accelerated L1, The host needs to > >>>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked > >>>> with: -machine virt,virtualization=on. > >>> For the current kvmarm/next the guest also needs > >>> “kvm-arm.mode=nested” I believe. > >> No, unless you want the guest itself to be NV-capable. > Effectively you don't need "kvm-arm.mode=nested" except if you want want > multiple level of nesting. You need kvm-arm.mode=nested on the command-line of a kernel that acts as a host exposing KVM_CAP_ARM_EL2. This is obviously a recursive property, but that's for the user to decide if they want it or not. > > Correct, I got carried away with some mode combinations. Maybe we should depict > > here more broadly how NV might be used with different mode combinations. I'll > > think about this further ahead. > > > > As far this series go I couldn't found any issue booting a L1 guest with > > virtualization=on and a L2 guest with virtualization=off. > > on my end I tested with various untouched L2 guests (debian, fed, rhel) > in 4kB/4kB/4KB page size mode (host, L1, L2). Those configs were successful. I'm doing 4/4/4 (x1e) and 16/16/4 (m2). I'm also doing 4/4/4/4/4, but that's not very reliable yet, > with 64kB/64kB/64kB configs I am less lucky atm. One one machine I > cannot boot L1 with virtualization=on. On the other I can boot L1 but > cannot boot L2. I think x1e supports 64k pages, but I'd need to try. Not exactly the highest priority on my list of things to debug. > Trying my best to debug a little bit further with my setup. Anyway if > somebody else can try 64kB configs, it would help to confirm whether > there are pending issues. I don't think they are related to this qemu > integration series though. Yeah, definitely a kernel issue IMO. Thanks, M. -- Without deviation from the norm, progress is not possible.
Hi Eric, On Tue, 27 May 2025 07:24:32 +0100, Eric Auger <eric.auger@redhat.com> wrote: > > Now that ARM nested virt has landed in kvm/next, let's turn the series > into a PATCH series. The linux header update was made against kvm/next. > > For gaining virt functionality in KVM accelerated L1, The host needs to > be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked > with: -machine virt,virtualization=on. Thanks for respinning this series. Do you have any plan to support the non-VHE version of the NV support (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser hypervisors (such as *cough* Xen *cough*), which completely rely on HCR_EL2.E2H being 0? Thanks, M. -- Without deviation from the norm, progress is not possible.
Hi Marc, Miguel, On 5/27/25 9:39 AM, Marc Zyngier wrote: > Hi Eric, > > On Tue, 27 May 2025 07:24:32 +0100, > Eric Auger <eric.auger@redhat.com> wrote: >> Now that ARM nested virt has landed in kvm/next, let's turn the series >> into a PATCH series. The linux header update was made against kvm/next. >> >> For gaining virt functionality in KVM accelerated L1, The host needs to >> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked >> with: -machine virt,virtualization=on. > Thanks for respinning this series. > > Do you have any plan to support the non-VHE version of the NV support > (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser > hypervisors (such as *cough* Xen *cough*), which completely rely on > HCR_EL2.E2H being 0? While respinning I'd rather not support KVM_ARM_VCPU_HAS_EL2_E2H0 in next respin. The code is mostly reviewed and I would rather add support for this in a separate series. It will most probably rely on a change of the virtualization property type to support a "restricted" enum value. Thanks Eric > > > Thanks, > > M. >
________________________________ De: Eric Auger <eric.auger@redhat.com> Enviado: quinta-feira, junho 19, 2025 8:19:47 da manhã Para: Marc Zyngier <maz@kernel.org> Cc: eric.auger.pro@gmail.com <eric.auger.pro@gmail.com>; qemu-devel@nongnu.org <qemu-devel@nongnu.org>; qemu-arm@nongnu.org <qemu-arm@nongnu.org>; Miguel Luis <miguel.luis@oracle.com>; peter.maydell@linaro.org <peter.maydell@linaro.org>; richard.henderson@linaro.org <richard.henderson@linaro.org>; gkulkarni@amperecomputing.com <gkulkarni@amperecomputing.com>; gankulkarni@os.amperecomputing.com <gankulkarni@os.amperecomputing.com> Assunto: Re: [PATCH v5 0/5] ARM Nested Virt Support Hi Marc, Miguel, On 5/27/25 9:39 AM, Marc Zyngier wrote: > Hi Eric, > > On Tue, 27 May 2025 07:24:32 +0100, > Eric Auger <eric.auger@redhat.com> wrote: >> Now that ARM nested virt has landed in kvm/next, let's turn the series >> into a PATCH series. The linux header update was made against kvm/next. >> >> For gaining virt functionality in KVM accelerated L1, The host needs to >> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked >> with: -machine virt,virtualization=on. > Thanks for respinning this series. > > Do you have any plan to support the non-VHE version of the NV support > (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser > hypervisors (such as *cough* Xen *cough*), which completely rely on > HCR_EL2.E2H being 0? While respinning I'd rather not support KVM_ARM_VCPU_HAS_EL2_E2H0 in next respin. The code is mostly reviewed and I would rather add support for this in a separate series. It will most probably rely on a change of the virtualization property type to support a "restricted" enum value. Ack! Thank you. Miguel Thanks Eric > > > Thanks, > > M. >
Hi Marc, > On 27 May 2025, at 07:39, Marc Zyngier <maz@kernel.org> wrote: > > Hi Eric, > > On Tue, 27 May 2025 07:24:32 +0100, > Eric Auger <eric.auger@redhat.com> wrote: >> >> Now that ARM nested virt has landed in kvm/next, let's turn the series >> into a PATCH series. The linux header update was made against kvm/next. >> >> For gaining virt functionality in KVM accelerated L1, The host needs to >> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked >> with: -machine virt,virtualization=on. > > Thanks for respinning this series. > > Do you have any plan to support the non-VHE version of the NV support > (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser > hypervisors (such as *cough* Xen *cough*), which completely rely on > HCR_EL2.E2H being 0? > Something that pops up is early_kvm_mode_cfg trying to handle nested mode while KVM_ARM_VCPU_HAS_EL2_E2H0 is set. Miguel > > Thanks, > > M. > > -- > Without deviation from the norm, progress is not possible.
On Tue, 27 May 2025 12:40:35 +0100, Miguel Luis <miguel.luis@oracle.com> wrote: > > Hi Marc, > > > On 27 May 2025, at 07:39, Marc Zyngier <maz@kernel.org> wrote: > > > > Hi Eric, > > > > On Tue, 27 May 2025 07:24:32 +0100, > > Eric Auger <eric.auger@redhat.com> wrote: > >> > >> Now that ARM nested virt has landed in kvm/next, let's turn the series > >> into a PATCH series. The linux header update was made against kvm/next. > >> > >> For gaining virt functionality in KVM accelerated L1, The host needs to > >> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked > >> with: -machine virt,virtualization=on. > > > > Thanks for respinning this series. > > > > Do you have any plan to support the non-VHE version of the NV support > > (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser > > hypervisors (such as *cough* Xen *cough*), which completely rely on > > HCR_EL2.E2H being 0? > > > > Something that pops up is early_kvm_mode_cfg trying to handle nested mode > while KVM_ARM_VCPU_HAS_EL2_E2H0 is set. Care to elaborate? M. -- Without deviation from the norm, progress is not possible.
> On 27 May 2025, at 12:02, Marc Zyngier <maz@kernel.org> wrote: > > On Tue, 27 May 2025 12:40:35 +0100, > Miguel Luis <miguel.luis@oracle.com> wrote: >> >> Hi Marc, >> >>> On 27 May 2025, at 07:39, Marc Zyngier <maz@kernel.org> wrote: >>> >>> Hi Eric, >>> >>> On Tue, 27 May 2025 07:24:32 +0100, >>> Eric Auger <eric.auger@redhat.com> wrote: >>>> >>>> Now that ARM nested virt has landed in kvm/next, let's turn the series >>>> into a PATCH series. The linux header update was made against kvm/next. >>>> >>>> For gaining virt functionality in KVM accelerated L1, The host needs to >>>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked >>>> with: -machine virt,virtualization=on. >>> >>> Thanks for respinning this series. >>> >>> Do you have any plan to support the non-VHE version of the NV support >>> (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser >>> hypervisors (such as *cough* Xen *cough*), which completely rely on >>> HCR_EL2.E2H being 0? >>> >> >> Something that pops up is early_kvm_mode_cfg trying to handle nested mode >> while KVM_ARM_VCPU_HAS_EL2_E2H0 is set. > > Care to elaborate? > Say host is booted in nested mode (kvm-arm.mode=nested) and host's KVM supports both KVM_CAP_ARM_EL2 and KVM_CAP_ARM_E2H0. A L1 guest boots setting both KVM_ARM_VCPU_HAS_EL2 and KVM_ARM_VCPU_HAS_EL2_E2H0 and guest kernel's command line state kvm-arm.mode=nested. This splats the kernel from early_kvm_mode_cfg along a malformed early option message. This is not related to this qemu integration though, should you take it somewhere else, please let me know? Thanks Miguel > M. > > -- > Without deviation from the norm, progress is not possible.
On Tue, 27 May 2025 14:24:31 +0100, Miguel Luis <miguel.luis@oracle.com> wrote: > > > > > On 27 May 2025, at 12:02, Marc Zyngier <maz@kernel.org> wrote: > > > > On Tue, 27 May 2025 12:40:35 +0100, > > Miguel Luis <miguel.luis@oracle.com> wrote: > >> > >> Hi Marc, > >> > >>> On 27 May 2025, at 07:39, Marc Zyngier <maz@kernel.org> wrote: > >>> > >>> Hi Eric, > >>> > >>> On Tue, 27 May 2025 07:24:32 +0100, > >>> Eric Auger <eric.auger@redhat.com> wrote: > >>>> > >>>> Now that ARM nested virt has landed in kvm/next, let's turn the series > >>>> into a PATCH series. The linux header update was made against kvm/next. > >>>> > >>>> For gaining virt functionality in KVM accelerated L1, The host needs to > >>>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked > >>>> with: -machine virt,virtualization=on. > >>> > >>> Thanks for respinning this series. > >>> > >>> Do you have any plan to support the non-VHE version of the NV support > >>> (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser > >>> hypervisors (such as *cough* Xen *cough*), which completely rely on > >>> HCR_EL2.E2H being 0? > >>> > >> > >> Something that pops up is early_kvm_mode_cfg trying to handle nested mode > >> while KVM_ARM_VCPU_HAS_EL2_E2H0 is set. > > > > Care to elaborate? > > > > Say host is booted in nested mode (kvm-arm.mode=nested) and host's KVM supports > both KVM_CAP_ARM_EL2 and KVM_CAP_ARM_E2H0. > > A L1 guest boots setting both KVM_ARM_VCPU_HAS_EL2 and > KVM_ARM_VCPU_HAS_EL2_E2H0 and guest kernel's command line state > kvm-arm.mode=nested. > > This splats the kernel from early_kvm_mode_cfg along a malformed early option > message. BEBKAC. You are asking for nested on a (virtual) machine that doesn't support it, and the kernel tells you so with a warning. Try the same thing on a physical machine that doesn't have NV, and observe the result. M. -- Without deviation from the norm, progress is not possible.
Hi Marc, > On 27 May 2025, at 13:46, Marc Zyngier <maz@kernel.org> wrote: > > On Tue, 27 May 2025 14:24:31 +0100, > Miguel Luis <miguel.luis@oracle.com> wrote: >> >> >> >>> On 27 May 2025, at 12:02, Marc Zyngier <maz@kernel.org> wrote: >>> >>> On Tue, 27 May 2025 12:40:35 +0100, >>> Miguel Luis <miguel.luis@oracle.com> wrote: >>>> >>>> Hi Marc, >>>> >>>>> On 27 May 2025, at 07:39, Marc Zyngier <maz@kernel.org> wrote: >>>>> >>>>> Hi Eric, >>>>> >>>>> On Tue, 27 May 2025 07:24:32 +0100, >>>>> Eric Auger <eric.auger@redhat.com> wrote: >>>>>> >>>>>> Now that ARM nested virt has landed in kvm/next, let's turn the series >>>>>> into a PATCH series. The linux header update was made against kvm/next. >>>>>> >>>>>> For gaining virt functionality in KVM accelerated L1, The host needs to >>>>>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked >>>>>> with: -machine virt,virtualization=on. >>>>> >>>>> Thanks for respinning this series. >>>>> >>>>> Do you have any plan to support the non-VHE version of the NV support >>>>> (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser >>>>> hypervisors (such as *cough* Xen *cough*), which completely rely on >>>>> HCR_EL2.E2H being 0? >>>>> >>>> >>>> Something that pops up is early_kvm_mode_cfg trying to handle nested mode >>>> while KVM_ARM_VCPU_HAS_EL2_E2H0 is set. >>> >>> Care to elaborate? >>> >> >> Say host is booted in nested mode (kvm-arm.mode=nested) and host's KVM supports >> both KVM_CAP_ARM_EL2 and KVM_CAP_ARM_E2H0. >> >> A L1 guest boots setting both KVM_ARM_VCPU_HAS_EL2 and >> KVM_ARM_VCPU_HAS_EL2_E2H0 and guest kernel's command line state >> kvm-arm.mode=nested. >> >> This splats the kernel from early_kvm_mode_cfg along a malformed early option >> message. > > BEBKAC. You are asking for nested on a (virtual) machine that doesn't > support it, and the kernel tells you so with a warning. Try the same > thing on a physical machine that doesn't have NV, and observe the > result. > Ack. I find trying them a great way to improve resilience. I’ve tried the scenarios below which have similar results on the guest: 1. Host: kvm-arm.mode=nested L1 Guest: kvm-arm.mode=nvhe setting both KVM_ARM_VCPU_HAS_EL2 and KVM_ARM_VCPU_HAS_EL2_E2H0 Result on the guest: No early_kvm_mode_cfg splat, boot proceeds, ends up in a hard lockup splat. 2. Host: kvm-arm.mode=nested L1 Guest: kvm-arm.mode=nested setting both KVM_ARM_VCPU_HAS_EL2 and KVM_ARM_VCPU_HAS_EL2_E2H0 Result on the guest: Splat at early_kvm_mode_cfg, boot proceeds, ends up in hard lockup splat. Does this means there’s a default fallback mode in which nv gets on when kvm-arm.mode fed to the guest kernel cmdline differs from the expected? Miguel > M. > > -- > Without deviation from the norm, progress is not possible.
On Tue, 27 May 2025 16:55:32 +0100, Miguel Luis <miguel.luis@oracle.com> wrote: > > Hi Marc, > > > On 27 May 2025, at 13:46, Marc Zyngier <maz@kernel.org> wrote: > > > > On Tue, 27 May 2025 14:24:31 +0100, > > Miguel Luis <miguel.luis@oracle.com> wrote: > >> > >> > >> > >>> On 27 May 2025, at 12:02, Marc Zyngier <maz@kernel.org> wrote: > >>> > >>> On Tue, 27 May 2025 12:40:35 +0100, > >>> Miguel Luis <miguel.luis@oracle.com> wrote: > >>>> > >>>> Hi Marc, > >>>> > >>>>> On 27 May 2025, at 07:39, Marc Zyngier <maz@kernel.org> wrote: > >>>>> > >>>>> Hi Eric, > >>>>> > >>>>> On Tue, 27 May 2025 07:24:32 +0100, > >>>>> Eric Auger <eric.auger@redhat.com> wrote: > >>>>>> > >>>>>> Now that ARM nested virt has landed in kvm/next, let's turn the series > >>>>>> into a PATCH series. The linux header update was made against kvm/next. > >>>>>> > >>>>>> For gaining virt functionality in KVM accelerated L1, The host needs to > >>>>>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked > >>>>>> with: -machine virt,virtualization=on. > >>>>> > >>>>> Thanks for respinning this series. > >>>>> > >>>>> Do you have any plan to support the non-VHE version of the NV support > >>>>> (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser > >>>>> hypervisors (such as *cough* Xen *cough*), which completely rely on > >>>>> HCR_EL2.E2H being 0? > >>>>> > >>>> > >>>> Something that pops up is early_kvm_mode_cfg trying to handle nested mode > >>>> while KVM_ARM_VCPU_HAS_EL2_E2H0 is set. > >>> > >>> Care to elaborate? > >>> > >> > >> Say host is booted in nested mode (kvm-arm.mode=nested) and host's KVM supports > >> both KVM_CAP_ARM_EL2 and KVM_CAP_ARM_E2H0. > >> > >> A L1 guest boots setting both KVM_ARM_VCPU_HAS_EL2 and > >> KVM_ARM_VCPU_HAS_EL2_E2H0 and guest kernel's command line state > >> kvm-arm.mode=nested. > >> > >> This splats the kernel from early_kvm_mode_cfg along a malformed early option > >> message. > > > > BEBKAC. You are asking for nested on a (virtual) machine that doesn't > > support it, and the kernel tells you so with a warning. Try the same > > thing on a physical machine that doesn't have NV, and observe the > > result. > > > > Ack. > > I find trying them a great way to improve resilience. > I’ve tried the scenarios below which have similar results on the guest: > > 1. > Host: kvm-arm.mode=nested > > L1 Guest: kvm-arm.mode=nvhe setting both > KVM_ARM_VCPU_HAS_EL2 and KVM_ARM_VCPU_HAS_EL2_E2H0 > > Result on the guest: No early_kvm_mode_cfg splat, boot proceeds, ends up in a hard lockup splat. Setting kvm-arm.mode=nvhe when KVM_ARM_VCPU_HAS_EL2_E2H0 is set is a tautology. The very definition of nVHE is that HCR_EL2.E2H=0. > > 2. > Host: kvm-arm.mode=nested > > L1 Guest: kvm-arm.mode=nested setting both > KVM_ARM_VCPU_HAS_EL2 and KVM_ARM_VCPU_HAS_EL2_E2H0 > > Result on the guest: Splat at early_kvm_mode_cfg, boot proceeds, ends up in hard lockup splat. I don't see any of these lockups with kvmtool. See this: https://pastebin.com/uyYzsBHc for an example of a boot with both capabilities set and the nonsense "nested" on the command-line (your #2). > Does this means there’s a default fallback mode in which nv gets on when > kvm-arm.mode fed to the guest kernel cmdline differs from the expected? I don't understand your question. We have two modes of operation: - HAS_EL2 enables NV on the host, and additionally enables recursive NV. As a consequence, HCR_EL2.E2H is RES1. This is how NV will be supported long term. - HAS_EL2_E2H0 restricts the above by not exposing NV to the guest, and enforcing HCR_EL2.E2H to be RES0. I expect this to gradually be removed from implementations, and eventually disappear. As you can see, there is no "fallback mode". You pick the mode you want based on the guest you want to run and the capabilities of the hardware. M. -- Without deviation from the norm, progress is not possible.
> On 27 May 2025, at 16:52, Marc Zyngier <maz@kernel.org> wrote: > > On Tue, 27 May 2025 16:55:32 +0100, > Miguel Luis <miguel.luis@oracle.com> wrote: >> >> Hi Marc, >> >>> On 27 May 2025, at 13:46, Marc Zyngier <maz@kernel.org> wrote: >>> >>> On Tue, 27 May 2025 14:24:31 +0100, >>> Miguel Luis <miguel.luis@oracle.com> wrote: >>>> >>>> >>>> >>>>> On 27 May 2025, at 12:02, Marc Zyngier <maz@kernel.org> wrote: >>>>> >>>>> On Tue, 27 May 2025 12:40:35 +0100, >>>>> Miguel Luis <miguel.luis@oracle.com> wrote: >>>>>> >>>>>> Hi Marc, >>>>>> >>>>>>> On 27 May 2025, at 07:39, Marc Zyngier <maz@kernel.org> wrote: >>>>>>> >>>>>>> Hi Eric, >>>>>>> >>>>>>> On Tue, 27 May 2025 07:24:32 +0100, >>>>>>> Eric Auger <eric.auger@redhat.com> wrote: >>>>>>>> >>>>>>>> Now that ARM nested virt has landed in kvm/next, let's turn the series >>>>>>>> into a PATCH series. The linux header update was made against kvm/next. >>>>>>>> >>>>>>>> For gaining virt functionality in KVM accelerated L1, The host needs to >>>>>>>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked >>>>>>>> with: -machine virt,virtualization=on. >>>>>>> >>>>>>> Thanks for respinning this series. >>>>>>> >>>>>>> Do you have any plan to support the non-VHE version of the NV support >>>>>>> (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser >>>>>>> hypervisors (such as *cough* Xen *cough*), which completely rely on >>>>>>> HCR_EL2.E2H being 0? >>>>>>> >>>>>> >>>>>> Something that pops up is early_kvm_mode_cfg trying to handle nested mode >>>>>> while KVM_ARM_VCPU_HAS_EL2_E2H0 is set. >>>>> >>>>> Care to elaborate? >>>>> >>>> >>>> Say host is booted in nested mode (kvm-arm.mode=nested) and host's KVM supports >>>> both KVM_CAP_ARM_EL2 and KVM_CAP_ARM_E2H0. >>>> >>>> A L1 guest boots setting both KVM_ARM_VCPU_HAS_EL2 and >>>> KVM_ARM_VCPU_HAS_EL2_E2H0 and guest kernel's command line state >>>> kvm-arm.mode=nested. >>>> >>>> This splats the kernel from early_kvm_mode_cfg along a malformed early option >>>> message. >>> >>> BEBKAC. You are asking for nested on a (virtual) machine that doesn't >>> support it, and the kernel tells you so with a warning. Try the same >>> thing on a physical machine that doesn't have NV, and observe the >>> result. >>> >> >> Ack. >> >> I find trying them a great way to improve resilience. >> I’ve tried the scenarios below which have similar results on the guest: >> >> 1. >> Host: kvm-arm.mode=nested >> >> L1 Guest: kvm-arm.mode=nvhe setting both >> KVM_ARM_VCPU_HAS_EL2 and KVM_ARM_VCPU_HAS_EL2_E2H0 >> >> Result on the guest: No early_kvm_mode_cfg splat, boot proceeds, ends up in a hard lockup splat. > > Setting kvm-arm.mode=nvhe when KVM_ARM_VCPU_HAS_EL2_E2H0 is set is a > tautology. The very definition of nVHE is that HCR_EL2.E2H=0. > >> >> 2. >> Host: kvm-arm.mode=nested >> >> L1 Guest: kvm-arm.mode=nested setting both >> KVM_ARM_VCPU_HAS_EL2 and KVM_ARM_VCPU_HAS_EL2_E2H0 >> >> Result on the guest: Splat at early_kvm_mode_cfg, boot proceeds, ends up in hard lockup splat. > > I don't see any of these lockups with kvmtool. See this: > > https://pastebin.com/uyYzsBHc Could you try bigger values for -c and check whether you can reproduce the issue? > > for an example of a boot with both capabilities set and the nonsense > "nested" on the command-line (your #2). > >> Does this means there’s a default fallback mode in which nv gets on when >> kvm-arm.mode fed to the guest kernel cmdline differs from the expected? > > I don't understand your question. We have two modes of operation: > > - HAS_EL2 enables NV on the host, and additionally enables recursive > NV. As a consequence, HCR_EL2.E2H is RES1. This is how NV will be > supported long term. > > - HAS_EL2_E2H0 restricts the above by not exposing NV to the guest, > and enforcing HCR_EL2.E2H to be RES0. I expect this to gradually be > removed from implementations, and eventually disappear. > > As you can see, there is no "fallback mode". You pick the mode you > want based on the guest you want to run and the capabilities of the > hardware. > I’m now suspecting the lockups might have a different reason than guest’s mode. Thanks Miguel > M. > > -- > Without deviation from the norm, progress is not possible.
On Wed, 28 May 2025 00:52:40 +0100, Miguel Luis <miguel.luis@oracle.com> wrote: > > > > > On 27 May 2025, at 16:52, Marc Zyngier <maz@kernel.org> wrote: > > > > On Tue, 27 May 2025 16:55:32 +0100, > > Miguel Luis <miguel.luis@oracle.com> wrote: > >> >> Result on the guest: Splat at early_kvm_mode_cfg, boot proceeds, ends up in hard lockup splat. > > > > I don't see any of these lockups with kvmtool. See this: > > > > https://pastebin.com/uyYzsBHc > > Could you try bigger values for -c and check whether you can reproduce the issue? How about 512, which is the maximum we support, and works for me with an oversubscribing ratio of 42:1? More importantly, how about *you* analyse the issue and report something that is actually actionable? > I’m now suspecting the lockups might have a different reason than guest’s mode. I'm shocked! :) M. -- Without deviation from the norm, progress is not possible.
Hi Marc, On 5/27/25 9:39 AM, Marc Zyngier wrote: > Hi Eric, > > On Tue, 27 May 2025 07:24:32 +0100, > Eric Auger <eric.auger@redhat.com> wrote: >> Now that ARM nested virt has landed in kvm/next, let's turn the series >> into a PATCH series. The linux header update was made against kvm/next. >> >> For gaining virt functionality in KVM accelerated L1, The host needs to >> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked >> with: -machine virt,virtualization=on. > Thanks for respinning this series. > > Do you have any plan to support the non-VHE version of the NV support > (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser > hypervisors (such as *cough* Xen *cough*), which completely rely on > HCR_EL2.E2H being 0? yes I can definitively extend the series to support KVM_ARM_VCPU_HAS_EL2_E2H0. Eric > > > Thanks, > > M. >
© 2016 - 2025 Red Hat, Inc.