[PATCH v14 00/19] HVF: Add support for platform vGIC and nested virtualisation

Mohamed Mediouni posted 19 patches 1 month, 1 week ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20260306083113.74177-1-mohamed@unpredictable.fr
Maintainers: Cameron Esfahani <dirty@apple.com>, Roman Bolshakov <rbolshakov@ddn.com>, Phil Dennis-Jordan <phil@philjordan.eu>, Peter Maydell <peter.maydell@linaro.org>, Paolo Bonzini <pbonzini@redhat.com>, Alexander Graf <agraf@csgraf.de>
There is a newer version of this series
accel/hvf/hvf-all.c                |  52 +-
accel/stubs/hvf-stub.c             |   2 +
hw/arm/virt.c                      |  38 +-
hw/intc/arm_gicv3_common.c         |   4 +
hw/intc/arm_gicv3_hvf.c            | 822 +++++++++++++++++++++++++++++
hw/intc/arm_gicv3_hvf_stub.c       |  25 +
hw/intc/meson.build                |   2 +
include/hw/intc/arm_gicv3_common.h |   4 +
include/system/hvf.h               |   8 +
system/vl.c                        |   2 +
target/arm/cpu64.c                 |   2 +
target/arm/hvf/hvf.c               | 468 +++++++++++++++-
target/arm/hvf/hvf_sme_stubs.h     | 158 ++++++
target/arm/hvf/sysreg.c.inc        |  52 ++
target/arm/hvf_arm.h               |  46 ++
target/arm/machine.c               |   2 +-
16 files changed, 1659 insertions(+), 28 deletions(-)
create mode 100644 hw/intc/arm_gicv3_hvf.c
create mode 100644 hw/intc/arm_gicv3_hvf_stub.c
create mode 100644 target/arm/hvf/hvf_sme_stubs.h
[PATCH v14 00/19] HVF: Add support for platform vGIC and nested virtualisation
Posted by Mohamed Mediouni 1 month, 1 week ago
Link to branch: https://github.com/mediouni-m/qemu hvf-irqchip-and-nested
(tag for this submission: hvf-irqchip-and-nested-v13)

This series adds supports for nested virtualisation when using HVF on arm64 Macs.

Reviews would be very welcome :-) It's maybe a bit unlikely that this will make
it to QEMU 11.0 I might as well try.

Patches pending review:

1, 2, 3, 15, 17, 18

It has two parts:
- Apple vGICv3 support and necessary infrastructure changes for it
- Nested virtualisation support. Note that the nested virtualisation implementation
shipping as of macOS 26.3 is nVHE only (but it _does_ use VNCR as shipped).

This series doesn't contain EL2 physical timer emulation, which is
needed if not leveraging the Apple vGIC. As such, nested virt requires kernel-irqchip=on.

Known issues:
- when nested virt is enabled, no UI response within EDK2
and a permanent wait. Workaround: -boot menu=on,splash-time=0.

Apple Feedback Assistant item: FB21649319

When the VM is running at EL2 at the very moment the virtual timer fires:
HV_EXIT_REASON_VTIMER_ACTIVATED doesn’t fire (when not using the provided vGIC)
- using a GICv2 doesn’t require having transition notifiers which Hypervisor.framework doesn’t have…)

and when using the provided vGIC, the interrupt never gets delivered back to the guest

Linux as a guest OS is fine with this… but the reference ArmVirtQemu edk2 build always uses the virtual timer even when running EFI at EL2, so it gets broken unless this patch is applied to edk2:

diff of ArmVirtPkg/ArmVirt.dsc.inc:

-  ArmGenericTimerCounterLib|ArmPkg/Library/ArmGenericTimerVirtCounterLib/ArmGenericTimerVirtCounterLib.inf
+  ArmGenericTimerCounterLib|ArmPkg/Library/ArmGenericTimerPhyCounterLib/ArmGenericTimerPhyCounterLib.inf^M

Changelog:

v1->v2:
Oops. I did a mistake when preparing my patches.

- Add hvf_arm_el2_enable(_) call to virt_set_virt
- Fix nested virt support check to add HVF

v2->v3:
- LORC_EL1 patch was merged separately, remove from this series.
- fix LPIs when kernel-irqchip disabled and using TCG
- remove spurious if case in vGIC supported version detection (inapplicable now)
- Add hvf_enabled() check in combination with hvf kernel-irqchip checks
- cleanly fail on attempt to use the platform vGIC together with ITS

v3->v4:
- GIC state save improvements, including saving the opaque Apple-specific state
- Saving HVF system register state when using the vGIC and/or EL2

v5:
- oops, fixed up save/restore to be functional
- misc changes otherwise

v6:
- Addressing review comments

v7:
- Address review comments, adapt around Qemu changes and bugfixes.

v8:
- Rebase, and misc fixes

v9:
- Misc fixes from WHPX review comments
- Split enablement commit for EL2 into its own separate commit

v10:

- A bit of changes across the board
- move HVF irqchip save/restore into irqchip code
- block nested virt support when kernel-irqchip=off with a GICv2 as
we don't implement an emulated physical timer yet.

v11:
- fixups for the save/restore Apple GIC state patch
- block the same registers as the revert in 28b0ed32b32c7e5094cf2f1ec9c0645c65fad2aa
for "hvf: sync registers used at EL2"
- drop patch that warns instead of aborts on mismatch

v12:
- Have the SME2 series at the start of this.
- Adaptation for the SME2 series.

v13:
- include "fix SME capabilities reporting" patch instead of it being standalone
- include "remove hvf_arm.h include" patch at the beginning of the series

v14:
- Turns out that the HVF SME2 patches broke the build for pretty much anything
non-macOS, and didn't address that properly in v13
- Addressed some checkpatch nits

Manos Pitsidianakis (2):
  hvf/arm: handle FEAT_SME2 migration
  hvf/arm: expose FEAT_SME2 to guest if available

Mohamed Mediouni (17):
  hw/arm: virt: remove hvf_arm.h include
  target/arm: cpu64: only include hvf_arm.h when HVF is enabled
  hvf: hvf-all: stop including hvf_arm.h
  hw/intc: Add hvf vGIC interrupt controller support
  hw/intc: arm_gicv3_hvf: save/restore Apple GIC state
  accel, hw/arm, include/system/hvf: infrastructure changes for HVF vGIC
  target/arm: hvf: instantiate GIC early
  hw/arm, target/arm: nested virtualisation on HVF
  hvf: only call hvf_sync_vtimer() when running without the platform
    vGIC
  hvf: gate ARM_FEATURE_PMU register emulation behind not being at EL2
  hvf: arm: allow exposing minimal PMU when running with nested virt on
  target/arm: hvf: add asserts for code paths not leveraged when using
    the vGIC
  hvf: sync registers used at EL2
  target/arm: hvf: pass through CNTHCTL_EL2 and MDCCINT_EL1
  hvf: arm: SME2 availability fixes
  hvf: arm: fix SME capabilities reporting
  hvf: enable nested virtualisation support

 accel/hvf/hvf-all.c                |  52 +-
 accel/stubs/hvf-stub.c             |   2 +
 hw/arm/virt.c                      |  38 +-
 hw/intc/arm_gicv3_common.c         |   4 +
 hw/intc/arm_gicv3_hvf.c            | 822 +++++++++++++++++++++++++++++
 hw/intc/arm_gicv3_hvf_stub.c       |  25 +
 hw/intc/meson.build                |   2 +
 include/hw/intc/arm_gicv3_common.h |   4 +
 include/system/hvf.h               |   8 +
 system/vl.c                        |   2 +
 target/arm/cpu64.c                 |   2 +
 target/arm/hvf/hvf.c               | 468 +++++++++++++++-
 target/arm/hvf/hvf_sme_stubs.h     | 158 ++++++
 target/arm/hvf/sysreg.c.inc        |  52 ++
 target/arm/hvf_arm.h               |  46 ++
 target/arm/machine.c               |   2 +-
 16 files changed, 1659 insertions(+), 28 deletions(-)
 create mode 100644 hw/intc/arm_gicv3_hvf.c
 create mode 100644 hw/intc/arm_gicv3_hvf_stub.c
 create mode 100644 target/arm/hvf/hvf_sme_stubs.h

-- 
2.50.1 (Apple Git-155)


Re: [PATCH v14 00/19] HVF: Add support for platform vGIC and nested virtualisation
Posted by Mohamed Mediouni 1 month, 1 week ago

> On 6. Mar 2026, at 09:30, Mohamed Mediouni <mohamed@unpredictable.fr> wrote:
> 
> Link to branch: https://github.com/mediouni-m/qemu hvf-irqchip-and-nested
> (tag for this submission: hvf-irqchip-and-nested-v13)
> 
Hello,

Oops, hvf-irqchip-and-nested-v14

> This series adds supports for nested virtualisation when using HVF on arm64 Macs.
> 
> Reviews would be very welcome :-) It's maybe a bit unlikely that this will make
> it to QEMU 11.0 I might as well try.
> 
> Patches pending review:
> 
> 1, 2, 3, 15, 17, 18
Please have patches 1 to 3 applied before the HVF SME2 patches… otherwise the Linux build breaks…
And patch 18 is also not nested virt related as such but merely making HVF SME2 support feature reporting functional.

Note: I didn’t test x86 macOS builds but I believe this should be enough.

Sorry for sending a bit too many revisions in too short of a time but this _should_ be the last one...

> 
> It has two parts:
> - Apple vGICv3 support and necessary infrastructure changes for it
> - Nested virtualisation support. Note that the nested virtualisation implementation
> shipping as of macOS 26.3 is nVHE only (but it _does_ use VNCR as shipped).
> 
> This series doesn't contain EL2 physical timer emulation, which is
> needed if not leveraging the Apple vGIC. As such, nested virt requires kernel-irqchip=on.
> 
> Known issues:
> - when nested virt is enabled, no UI response within EDK2
> and a permanent wait. Workaround: -boot menu=on,splash-time=0.
> 
> Apple Feedback Assistant item: FB21649319
> 
Apple replied to that and said:

Investigation complete - works as currently designed

Which is a bit fair given how timer virt works on NV2 without relying on ECV…

> When the VM is running at EL2 at the very moment the virtual timer fires:
> HV_EXIT_REASON_VTIMER_ACTIVATED doesn’t fire (when not using the provided vGIC)
> - using a GICv2 doesn’t require having transition notifiers which Hypervisor.framework doesn’t have…)
> 
> and when using the provided vGIC, the interrupt never gets delivered back to the guest
> 
> Linux as a guest OS is fine with this… but the reference ArmVirtQemu edk2 build always uses the virtual timer even when running EFI at EL2, so it gets broken unless this patch is applied to edk2:
> 
> diff of ArmVirtPkg/ArmVirt.dsc.inc:
> 
> -  ArmGenericTimerCounterLib|ArmPkg/Library/ArmGenericTimerVirtCounterLib/ArmGenericTimerVirtCounterLib.inf
> +  ArmGenericTimerCounterLib|ArmPkg/Library/ArmGenericTimerPhyCounterLib/ArmGenericTimerPhyCounterLib.inf^M
> 
> Changelog:
> 
> v1->v2:
> Oops. I did a mistake when preparing my patches.
> 
> - Add hvf_arm_el2_enable(_) call to virt_set_virt
> - Fix nested virt support check to add HVF
> 
> v2->v3:
> - LORC_EL1 patch was merged separately, remove from this series.
> - fix LPIs when kernel-irqchip disabled and using TCG
> - remove spurious if case in vGIC supported version detection (inapplicable now)
> - Add hvf_enabled() check in combination with hvf kernel-irqchip checks
> - cleanly fail on attempt to use the platform vGIC together with ITS
> 
> v3->v4:
> - GIC state save improvements, including saving the opaque Apple-specific state
> - Saving HVF system register state when using the vGIC and/or EL2
> 
> v5:
> - oops, fixed up save/restore to be functional
> - misc changes otherwise
> 
> v6:
> - Addressing review comments
> 
> v7:
> - Address review comments, adapt around Qemu changes and bugfixes.
> 
> v8:
> - Rebase, and misc fixes
> 
> v9:
> - Misc fixes from WHPX review comments
> - Split enablement commit for EL2 into its own separate commit
> 
> v10:
> 
> - A bit of changes across the board
> - move HVF irqchip save/restore into irqchip code
> - block nested virt support when kernel-irqchip=off with a GICv2 as
> we don't implement an emulated physical timer yet.
> 
> v11:
> - fixups for the save/restore Apple GIC state patch
> - block the same registers as the revert in 28b0ed32b32c7e5094cf2f1ec9c0645c65fad2aa
> for "hvf: sync registers used at EL2"
> - drop patch that warns instead of aborts on mismatch
> 
> v12:
> - Have the SME2 series at the start of this.
> - Adaptation for the SME2 series.
> 
> v13:
> - include "fix SME capabilities reporting" patch instead of it being standalone
> - include "remove hvf_arm.h include" patch at the beginning of the series
> 
> v14:
> - Turns out that the HVF SME2 patches broke the build for pretty much anything
> non-macOS, and didn't address that properly in v13
> - Addressed some checkpatch nits
> 
> Manos Pitsidianakis (2):
>  hvf/arm: handle FEAT_SME2 migration
>  hvf/arm: expose FEAT_SME2 to guest if available
> 
> Mohamed Mediouni (17):
>  hw/arm: virt: remove hvf_arm.h include
>  target/arm: cpu64: only include hvf_arm.h when HVF is enabled
>  hvf: hvf-all: stop including hvf_arm.h
>  hw/intc: Add hvf vGIC interrupt controller support
>  hw/intc: arm_gicv3_hvf: save/restore Apple GIC state
>  accel, hw/arm, include/system/hvf: infrastructure changes for HVF vGIC
>  target/arm: hvf: instantiate GIC early
>  hw/arm, target/arm: nested virtualisation on HVF
>  hvf: only call hvf_sync_vtimer() when running without the platform
>    vGIC
>  hvf: gate ARM_FEATURE_PMU register emulation behind not being at EL2
>  hvf: arm: allow exposing minimal PMU when running with nested virt on
>  target/arm: hvf: add asserts for code paths not leveraged when using
>    the vGIC
>  hvf: sync registers used at EL2
>  target/arm: hvf: pass through CNTHCTL_EL2 and MDCCINT_EL1
>  hvf: arm: SME2 availability fixes
>  hvf: arm: fix SME capabilities reporting
>  hvf: enable nested virtualisation support
> 
> accel/hvf/hvf-all.c                |  52 +-
> accel/stubs/hvf-stub.c             |   2 +
> hw/arm/virt.c                      |  38 +-
> hw/intc/arm_gicv3_common.c         |   4 +
> hw/intc/arm_gicv3_hvf.c            | 822 +++++++++++++++++++++++++++++
> hw/intc/arm_gicv3_hvf_stub.c       |  25 +
> hw/intc/meson.build                |   2 +
> include/hw/intc/arm_gicv3_common.h |   4 +
> include/system/hvf.h               |   8 +
> system/vl.c                        |   2 +
> target/arm/cpu64.c                 |   2 +
> target/arm/hvf/hvf.c               | 468 +++++++++++++++-
> target/arm/hvf/hvf_sme_stubs.h     | 158 ++++++
> target/arm/hvf/sysreg.c.inc        |  52 ++
> target/arm/hvf_arm.h               |  46 ++
> target/arm/machine.c               |   2 +-
> 16 files changed, 1659 insertions(+), 28 deletions(-)
> create mode 100644 hw/intc/arm_gicv3_hvf.c
> create mode 100644 hw/intc/arm_gicv3_hvf_stub.c
> create mode 100644 target/arm/hvf/hvf_sme_stubs.h
> 
> -- 
> 2.50.1 (Apple Git-155)
>