[RFC PATCH 00/34] Running Qualcomm's Gunyah Guests via KVM in EL1

Karim Manaouil posted 34 patches 7 months, 3 weeks ago
.../bindings/firmware/gunyah-hypervisor.yaml  |   82 +
Documentation/virt/gunyah/index.rst           |  135 ++
Documentation/virt/gunyah/message-queue.rst   |   68 +
Documentation/virt/index.rst                  |    1 +
arch/arm64/Kbuild                             |    1 +
arch/arm64/gunyah/Makefile                    |    3 +
arch/arm64/gunyah/gunyah_hypercall.c          |  279 +++
arch/arm64/include/asm/gunyah.h               |   57 +
arch/arm64/include/asm/kvm_host.h             |   10 +-
arch/arm64/include/asm/virt.h                 |    7 +
arch/arm64/kernel/cpufeature.c                |    4 +
arch/arm64/kernel/image-vars.h                |    2 +-
arch/arm64/kvm/Kconfig                        |   22 +-
arch/arm64/kvm/Makefile                       |   14 +-
arch/arm64/kvm/gunyah.c                       | 2085 +++++++++++++++++
drivers/virt/Kconfig                          |    2 +
drivers/virt/Makefile                         |    1 +
drivers/virt/gunyah/Kconfig                   |   29 +
drivers/virt/gunyah/Makefile                  |    7 +
drivers/virt/gunyah/gunyah.c                  |   55 +
drivers/virt/gunyah/gunyah_platform_hooks.c   |  117 +
drivers/virt/gunyah/gunyah_qcom.c             |  220 ++
drivers/virt/gunyah/rsc_mgr.c                 |  792 +++++++
drivers/virt/gunyah/rsc_mgr_rpc.c             |  572 +++++
include/kvm/arm_pmu.h                         |    2 +-
include/linux/gunyah.h                        |  425 ++++
include/linux/gunyah_rsc_mgr.h                |  182 ++
include/linux/irqchip/arm-vgic-info.h         |    2 +-
include/linux/kvm_host.h                      |    3 +
include/linux/kvm_irqfd.h                     |    5 +
include/linux/perf/arm_pmu.h                  |    2 +-
include/uapi/linux/kvm.h                      |   14 +-
virt/kvm/eventfd.c                            |   56 +-
virt/kvm/kvm_main.c                           |   24 +-
34 files changed, 5258 insertions(+), 22 deletions(-)
create mode 100644 Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
create mode 100644 Documentation/virt/gunyah/index.rst
create mode 100644 Documentation/virt/gunyah/message-queue.rst
create mode 100644 arch/arm64/gunyah/Makefile
create mode 100644 arch/arm64/gunyah/gunyah_hypercall.c
create mode 100644 arch/arm64/include/asm/gunyah.h
create mode 100644 arch/arm64/kvm/gunyah.c
create mode 100644 drivers/virt/gunyah/Kconfig
create mode 100644 drivers/virt/gunyah/Makefile
create mode 100644 drivers/virt/gunyah/gunyah.c
create mode 100644 drivers/virt/gunyah/gunyah_platform_hooks.c
create mode 100644 drivers/virt/gunyah/gunyah_qcom.c
create mode 100644 drivers/virt/gunyah/rsc_mgr.c
create mode 100644 drivers/virt/gunyah/rsc_mgr_rpc.c
create mode 100644 include/linux/gunyah.h
create mode 100644 include/linux/gunyah_rsc_mgr.h
[RFC PATCH 00/34] Running Qualcomm's Gunyah Guests via KVM in EL1
Posted by Karim Manaouil 7 months, 3 weeks ago
This series introduces the capability of running Gunyah guests via KVM on
Qualcomm SoCs shipped with Gunyah hypervisor [1] (e.g. RB3 Gen2).

The goal of this work is to port the existing Gunyah hypervisor support from a
standalone driver interface [2] to KVM, with the aim of leveraging as much of the
existing KVM infrastructure as possible to reduce duplication of effort around
memory management (e.g. guest_memfd), irqfd, and other core components.

In short, Gunyah is a Type-1 hypervisor, meaning that it runs independently of any
high-level OS kernel such as Linux and runs in a higher CPU privilege level than VMs.
Gunyah is shipped as firmware and guests typically talk with Gunyah via hypercalls.
KVM is designed to run as Type-2 hypervisor. This port allows KVM to run in EL1 and
serve as the interface for VM lifecycle management,while offloading virtualization
to Gunyah.

This series is heavily based on previous work from Elliot Berman and others,
available at:
https://lore.kernel.org/lkml/20240222-gunyah-v17-0-1e9da6763d38@quicinc.com/

Many commits in this series are identical to or derived from previous work. To
preserve authorship and attribution, original `Author:` and `Signed-off-by:` tags
have been retained where appropriate, and occasionally `Reviewed-by:` lines were
kept as well.

While this series builds on much of the original Gunyah implementation, it drops
certain parts in favor of existing upstream features:
  - `gunyah_memfd` is dropped (currently using pinned anonymous pages — see below).
  - `gunyah_irqfd` is dropped in favor of KVM's irqfd.
  - Resource management, vCPU creation, and VM lifecycle are integrated directly
    into KVM’s architecture hooks.

At this stage, the port is functional but still **work in progress**. Notably:
  - Memory for guests is currently backed by **pinned anonymous pages**. This is a
    temporary solution: we intend to migrate to `guest_memfd`.
  - Memory compaction or swap must be avoided for now, as pages donated to Gunyah
    are no longer accessible from the host once mapped into the guest.
  - SMP boot is not available at the moment because Gunyah does not yet
    forward PSCI hypercalls to the host.
  - Virtio is not supported yet.

There is a lot of room for performance improvment. For example, ATM.
there is a lot of privilege level switching between EL1, EL2 (and
possible EL3). As well as context swicthing between guest kernel space
and guest userspace. For those interested, I have a compiled some slides
here to summarise the issues:
https://docs.google.com/presentation/d/1fL1TM1oxBrWFSL8KKw4jQtMDh7NOvyqbWWVu8iQxvoI/edit?usp=sharing

To test the port, a modified version of Qemu is needed. A version can be
found here:
https://github.com/karim-manaouil/qemu-for-gunyah

The series is based on linux-next as of today. A git tree is also
available here (gunyah-kvm branch):
https://github.com/karim-manaouil/linux-next

To test the guest, run (with the modified qemu):
./qemu-system-aarch64 \
	-enable-kvm \
	-machine virt,highmem=off \
	-cpu host \
	-smp 1 \
	-m 1G \
	-nographic \
	-kernel Image \
	-initrd initrd.img \
	-append "console=ttyAMA0 earlycon rdinit=/bin/sh"

Feedback Welcome!

[1] https://www.qualcomm.com/developer/blog/2024/08/learn-about-gunyah--qualcomm-s-open-source--lightweight-hypervis
[2] https://lore.kernel.org/lkml/20240222-gunyah-v17-0-1e9da6763d38@quicinc.com/

Elliot Berman (20):
  docs: gunyah: Introduce Gunyah Hypervisor
  dt-bindings: Add binding for gunyah hypervisor
  gunyah: Common types and error codes for Gunyah hypercalls
  gunyah: Add hypercalls to identify Gunyah
  gunyah: Add hypervisor driver
  gunyah: Add hypercalls to send and receive messages
  gunyah: Add resource manager RPC core
  gunyah: Add VM lifecycle RPC
  gunyah: Translate gh_rm_hyp_resource into gunyah_resource
  gunyah: Add resource tickets
  gunyah: Add hypercalls for running a vCPU
  gunyah: Add hypercalls for demand paging
  gunyah: Add memory parcel RPC
  gunyah: Add interfaces to map memory into guest address space
  gunyah: Add platform ops on mem_lend/mem_reclaim
  gunyah: Add Qualcomm Gunyah platform ops
  gunyah: Share guest VM dtb configuration to Gunyah
  gunyah: Add RPC to enable demand paging
  gunyah: Add RPC to set VM boot context
  gunyah: Add hypercalls for sending doorbell

Karim Manaouil (14):
  KVM: Allow arch-specific vCPU allocation and freeing
  KVM: irqfd: Add architecture hooks for irqfd allocation and
    initialization
  KVM: irqfd: Allow KVM backends to override IRQ injection via set_irq
    callback
  KVM: Add weak stubs for irqchip-related functions for Gunyah builds
  KVM: Add KVM_SET_DTB_ADDRESS ioctl to pass guest DTB address from
    userspace
  KVM: gunyah: Add initial Gunyah backend support
  KVM: gunyah: Pin guest memory
  gunyah: Add basic VM lifecycle management
  gunyah: add proxy-scheduled vCPUs
  gunyah: Share memory parcels
  gunyah: Enable demand paging
  gunyah: allow userspace to set boot cpu context
  KVM: gunyah: Implement irqfd interface
  KVM: gunyah: enable KVM for Gunyah

 .../bindings/firmware/gunyah-hypervisor.yaml  |   82 +
 Documentation/virt/gunyah/index.rst           |  135 ++
 Documentation/virt/gunyah/message-queue.rst   |   68 +
 Documentation/virt/index.rst                  |    1 +
 arch/arm64/Kbuild                             |    1 +
 arch/arm64/gunyah/Makefile                    |    3 +
 arch/arm64/gunyah/gunyah_hypercall.c          |  279 +++
 arch/arm64/include/asm/gunyah.h               |   57 +
 arch/arm64/include/asm/kvm_host.h             |   10 +-
 arch/arm64/include/asm/virt.h                 |    7 +
 arch/arm64/kernel/cpufeature.c                |    4 +
 arch/arm64/kernel/image-vars.h                |    2 +-
 arch/arm64/kvm/Kconfig                        |   22 +-
 arch/arm64/kvm/Makefile                       |   14 +-
 arch/arm64/kvm/gunyah.c                       | 2085 +++++++++++++++++
 drivers/virt/Kconfig                          |    2 +
 drivers/virt/Makefile                         |    1 +
 drivers/virt/gunyah/Kconfig                   |   29 +
 drivers/virt/gunyah/Makefile                  |    7 +
 drivers/virt/gunyah/gunyah.c                  |   55 +
 drivers/virt/gunyah/gunyah_platform_hooks.c   |  117 +
 drivers/virt/gunyah/gunyah_qcom.c             |  220 ++
 drivers/virt/gunyah/rsc_mgr.c                 |  792 +++++++
 drivers/virt/gunyah/rsc_mgr_rpc.c             |  572 +++++
 include/kvm/arm_pmu.h                         |    2 +-
 include/linux/gunyah.h                        |  425 ++++
 include/linux/gunyah_rsc_mgr.h                |  182 ++
 include/linux/irqchip/arm-vgic-info.h         |    2 +-
 include/linux/kvm_host.h                      |    3 +
 include/linux/kvm_irqfd.h                     |    5 +
 include/linux/perf/arm_pmu.h                  |    2 +-
 include/uapi/linux/kvm.h                      |   14 +-
 virt/kvm/eventfd.c                            |   56 +-
 virt/kvm/kvm_main.c                           |   24 +-
 34 files changed, 5258 insertions(+), 22 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
 create mode 100644 Documentation/virt/gunyah/index.rst
 create mode 100644 Documentation/virt/gunyah/message-queue.rst
 create mode 100644 arch/arm64/gunyah/Makefile
 create mode 100644 arch/arm64/gunyah/gunyah_hypercall.c
 create mode 100644 arch/arm64/include/asm/gunyah.h
 create mode 100644 arch/arm64/kvm/gunyah.c
 create mode 100644 drivers/virt/gunyah/Kconfig
 create mode 100644 drivers/virt/gunyah/Makefile
 create mode 100644 drivers/virt/gunyah/gunyah.c
 create mode 100644 drivers/virt/gunyah/gunyah_platform_hooks.c
 create mode 100644 drivers/virt/gunyah/gunyah_qcom.c
 create mode 100644 drivers/virt/gunyah/rsc_mgr.c
 create mode 100644 drivers/virt/gunyah/rsc_mgr_rpc.c
 create mode 100644 include/linux/gunyah.h
 create mode 100644 include/linux/gunyah_rsc_mgr.h

-- 
2.39.5

Re: [RFC PATCH 00/34] Running Qualcomm's Gunyah Guests via KVM in EL1
Posted by Oliver Upton 7 months, 3 weeks ago
On Thu, Apr 24, 2025 at 03:13:07PM +0100, Karim Manaouil wrote:
> This series introduces the capability of running Gunyah guests via KVM on
> Qualcomm SoCs shipped with Gunyah hypervisor [1] (e.g. RB3 Gen2).
> 
> The goal of this work is to port the existing Gunyah hypervisor support from a
> standalone driver interface [2] to KVM, with the aim of leveraging as much of the
> existing KVM infrastructure as possible to reduce duplication of effort around
> memory management (e.g. guest_memfd), irqfd, and other core components.
> 
> In short, Gunyah is a Type-1 hypervisor, meaning that it runs independently of any
> high-level OS kernel such as Linux and runs in a higher CPU privilege level than VMs.
> Gunyah is shipped as firmware and guests typically talk with Gunyah via hypercalls.
> KVM is designed to run as Type-2 hypervisor. This port allows KVM to run in EL1 and
> serve as the interface for VM lifecycle management,while offloading virtualization
> to Gunyah.

If you're keen on running your own hypervisor then I'm sorry, you get to
deal with it soup to nuts. Other hypervisors (e.g. mshv) have their own
kernel drivers for managing the host / UAPI parts of driving VMs.

The KVM arch interface is *internal* to KVM, not something to be
(ab)used for cramming in a non-KVM hypervisor. KVM and other hypervisors
can still share other bits of truly common infrastructure, like
guest_memfd.

I understand the value in what you're trying to do, but if you want it
to smell like KVM you may as well just let the user run it at EL2.

Thanks,
Oliver
Re: [RFC PATCH 00/34] Running Qualcomm's Gunyah Guests via KVM in EL1
Posted by Marc Zyngier 7 months, 3 weeks ago
On Thu, 24 Apr 2025 16:34:50 +0100,
Oliver Upton <oliver.upton@linux.dev> wrote:
> 
> On Thu, Apr 24, 2025 at 03:13:07PM +0100, Karim Manaouil wrote:
> > This series introduces the capability of running Gunyah guests via KVM on
> > Qualcomm SoCs shipped with Gunyah hypervisor [1] (e.g. RB3 Gen2).
> > 
> > The goal of this work is to port the existing Gunyah hypervisor support from a
> > standalone driver interface [2] to KVM, with the aim of leveraging as much of the
> > existing KVM infrastructure as possible to reduce duplication of effort around
> > memory management (e.g. guest_memfd), irqfd, and other core components.
> > 
> > In short, Gunyah is a Type-1 hypervisor, meaning that it runs independently of any
> > high-level OS kernel such as Linux and runs in a higher CPU privilege level than VMs.
> > Gunyah is shipped as firmware and guests typically talk with Gunyah via hypercalls.
> > KVM is designed to run as Type-2 hypervisor. This port allows KVM to run in EL1 and
> > serve as the interface for VM lifecycle management,while offloading virtualization
> > to Gunyah.
> 
> If you're keen on running your own hypervisor then I'm sorry, you get to
> deal with it soup to nuts. Other hypervisors (e.g. mshv) have their own
> kernel drivers for managing the host / UAPI parts of driving VMs.
> 
> The KVM arch interface is *internal* to KVM, not something to be
> (ab)used for cramming in a non-KVM hypervisor. KVM and other hypervisors
> can still share other bits of truly common infrastructure, like
> guest_memfd.
> 
> I understand the value in what you're trying to do, but if you want it
> to smell like KVM you may as well just let the user run it at EL2.

+1. KVM is not a generic interface for random third party hypervisors.

If you want to run KVM on your Qualcomm HW, boot at EL2, and enjoy the
real thing -- it is worth it. If Gunyah is what you want, then there
is enough code out there to use it with crosvm.

But mixing the two is not happening, sorry.

	M.

-- 
Without deviation from the norm, progress is not possible.
Re: [RFC PATCH 00/34] Running Qualcomm's Gunyah Guests via KVM in EL1
Posted by David Woodhouse 6 months ago
On Thu, 2025-04-24 at 17:57 +0100, Marc Zyngier wrote:
> On Thu, 24 Apr 2025 16:34:50 +0100,
> Oliver Upton <oliver.upton@linux.dev> wrote:
> > 
> > On Thu, Apr 24, 2025 at 03:13:07PM +0100, Karim Manaouil wrote:
> > > This series introduces the capability of running Gunyah guests via KVM on
> > > Qualcomm SoCs shipped with Gunyah hypervisor [1] (e.g. RB3 Gen2).
> > > 
> > > The goal of this work is to port the existing Gunyah hypervisor support from a
> > > standalone driver interface [2] to KVM, with the aim of leveraging as much of the
> > > existing KVM infrastructure as possible to reduce duplication of effort around
> > > memory management (e.g. guest_memfd), irqfd, and other core components.
> > > 
> > > In short, Gunyah is a Type-1 hypervisor, meaning that it runs independently of any
> > > high-level OS kernel such as Linux and runs in a higher CPU privilege level than VMs.
> > > Gunyah is shipped as firmware and guests typically talk with Gunyah via hypercalls.
> > > KVM is designed to run as Type-2 hypervisor. This port allows KVM to run in EL1 and
> > > serve as the interface for VM lifecycle management,while offloading virtualization
> > > to Gunyah.
> > 
> > If you're keen on running your own hypervisor then I'm sorry, you get to
> > deal with it soup to nuts. Other hypervisors (e.g. mshv) have their own
> > kernel drivers for managing the host / UAPI parts of driving VMs.
> > 
> > The KVM arch interface is *internal* to KVM, not something to be
> > (ab)used for cramming in a non-KVM hypervisor. KVM and other hypervisors
> > can still share other bits of truly common infrastructure, like
> > guest_memfd.
> > 
> > I understand the value in what you're trying to do, but if you want it
> > to smell like KVM you may as well just let the user run it at EL2.
> 
> +1. KVM is not a generic interface for random third party hypervisors.

I don't think that should be true in the general case. At least, it
depends on whether you mean the literal implementation in
arch/arm64/kvm/ vs. the userspace API and set of ioctls on /dev/kvm.

The kernel exists to provide a coherent userspace API for all kinds of
hardware. That's what it's *for*. It provides users with a consistent
interface to all kinds of network cards, serial ports, etc. — and that
includes firmware/platform features too. 

There's no reason that shouldn't be the same for virtualisation. If the
kernel cannot provide an API which supports *all* kinds of
virtualization, then it seems like we've done something wrong.

On x86 we have /dev/kvm backed by different vendor-specific support for
Intel vs. AMD. And in recent years we've retrofitted confidential
compute to it too, with SEV-SNP, TDX, etc.

We haven't resorted to saying "no, sorry, KVM doesn't support that".

We shouldn't say that for Arm either.

Re: [RFC PATCH 00/34] Running Qualcomm's Gunyah Guests via KVM in EL1
Posted by Karim Manaouil 7 months, 3 weeks ago
On Thu, Apr 24, 2025 at 08:34:50AM -0700, Oliver Upton wrote:
> On Thu, Apr 24, 2025 at 03:13:07PM +0100, Karim Manaouil wrote:
> > This series introduces the capability of running Gunyah guests via KVM on
> > Qualcomm SoCs shipped with Gunyah hypervisor [1] (e.g. RB3 Gen2).
> > 
> > The goal of this work is to port the existing Gunyah hypervisor support from a
> > standalone driver interface [2] to KVM, with the aim of leveraging as much of the
> > existing KVM infrastructure as possible to reduce duplication of effort around
> > memory management (e.g. guest_memfd), irqfd, and other core components.
> > 
> > In short, Gunyah is a Type-1 hypervisor, meaning that it runs independently of any
> > high-level OS kernel such as Linux and runs in a higher CPU privilege level than VMs.
> > Gunyah is shipped as firmware and guests typically talk with Gunyah via hypercalls.
> > KVM is designed to run as Type-2 hypervisor. This port allows KVM to run in EL1 and
> > serve as the interface for VM lifecycle management,while offloading virtualization
> > to Gunyah.
> 
> If you're keen on running your own hypervisor then I'm sorry, you get to
> deal with it soup to nuts. Other hypervisors (e.g. mshv) have their own
> kernel drivers for managing the host / UAPI parts of driving VMs.
> 
> The KVM arch interface is *internal* to KVM, not something to be
> (ab)used for cramming in a non-KVM hypervisor. KVM and other hypervisors
> can still share other bits of truly common infrastructure, like
> guest_memfd.
> 
> I understand the value in what you're trying to do, but if you want it
> to smell like KVM you may as well just let the user run it at EL2.

Hi Oliver,

Thanks for your reply! I very much expected this take and my discussion
with Qauclomm engineers also more or less had the same conclusion. There
was a previous effort to go into this direction, but it never got
implemented, so this was my (no bullshit) attempt at doing that!

I believe that Qcom are more interested in having their own driver,
instead (as Trilok pointed out in another thread). But this port could
still be interesting for those who wanted to see KVM in EL1.

On the positive side, you get all the userspace tools (Qemu, Crosvm,
kvmtool, cloud-hypervisor, firecracker) working with minimal effort.

Cheers
Karim
Re: [RFC PATCH 00/34] Running Qualcomm's Gunyah Guests via KVM in EL1
Posted by Trilok Soni 7 months, 3 weeks ago
On 4/24/2025 8:34 AM, Oliver Upton wrote:
> On Thu, Apr 24, 2025 at 03:13:07PM +0100, Karim Manaouil wrote:
>> This series introduces the capability of running Gunyah guests via KVM on
>> Qualcomm SoCs shipped with Gunyah hypervisor [1] (e.g. RB3 Gen2).
>>
>> The goal of this work is to port the existing Gunyah hypervisor support from a
>> standalone driver interface [2] to KVM, with the aim of leveraging as much of the
>> existing KVM infrastructure as possible to reduce duplication of effort around
>> memory management (e.g. guest_memfd), irqfd, and other core components.
>>
>> In short, Gunyah is a Type-1 hypervisor, meaning that it runs independently of any
>> high-level OS kernel such as Linux and runs in a higher CPU privilege level than VMs.
>> Gunyah is shipped as firmware and guests typically talk with Gunyah via hypercalls.
>> KVM is designed to run as Type-2 hypervisor. This port allows KVM to run in EL1 and
>> serve as the interface for VM lifecycle management,while offloading virtualization
>> to Gunyah.
> 
> If you're keen on running your own hypervisor then I'm sorry, you get to
> deal with it soup to nuts. Other hypervisors (e.g. mshv) have their own
> kernel drivers for managing the host / UAPI parts of driving VMs.
> 
> The KVM arch interface is *internal* to KVM, not something to be
> (ab)used for cramming in a non-KVM hypervisor. KVM and other hypervisors
> can still share other bits of truly common infrastructure, like
> guest_memfd.
> 
> I understand the value in what you're trying to do, but if you want it
> to smell like KVM you may as well just let the user run it at EL2.

I agree, this is not the approach Qualcomm would like to use. Our approach
will be similar to Elliot Berman's patches (v17). We will revive
that series once Faud's patches are accepted on guest_memfd. 

-- 
---Trilok Soni