[PATCH v8 00/43] arm64: Support for Arm CCA in KVM

Steven Price posted 43 patches 8 months ago
There is a newer version of this series
Documentation/virt/kvm/api.rst       |   91 +-
arch/arm64/include/asm/kvm_emulate.h |   40 +
arch/arm64/include/asm/kvm_host.h    |   17 +-
arch/arm64/include/asm/kvm_rme.h     |  137 +++
arch/arm64/include/asm/rmi_cmds.h    |  508 ++++++++
arch/arm64/include/asm/rmi_smc.h     |  259 ++++
arch/arm64/include/asm/virt.h        |    1 +
arch/arm64/include/uapi/asm/kvm.h    |   49 +
arch/arm64/kvm/Kconfig               |    1 +
arch/arm64/kvm/Makefile              |    3 +-
arch/arm64/kvm/arch_timer.c          |   48 +-
arch/arm64/kvm/arm.c                 |  160 ++-
arch/arm64/kvm/guest.c               |  104 +-
arch/arm64/kvm/hypercalls.c          |    4 +-
arch/arm64/kvm/inject_fault.c        |    5 +-
arch/arm64/kvm/mmio.c                |   16 +-
arch/arm64/kvm/mmu.c                 |  201 ++-
arch/arm64/kvm/pmu-emul.c            |    6 +
arch/arm64/kvm/psci.c                |   30 +
arch/arm64/kvm/reset.c               |   23 +-
arch/arm64/kvm/rme-exit.c            |  199 +++
arch/arm64/kvm/rme.c                 | 1708 ++++++++++++++++++++++++++
arch/arm64/kvm/sys_regs.c            |   49 +-
arch/arm64/kvm/vgic/vgic-init.c      |    2 +-
arch/arm64/kvm/vgic/vgic-v3.c        |    6 +-
arch/arm64/kvm/vgic/vgic.c           |   54 +-
arch/arm64/mm/fault.c                |   31 +-
drivers/perf/arm_pmu.c               |   15 +
include/kvm/arm_arch_timer.h         |    2 +
include/kvm/arm_pmu.h                |    4 +
include/kvm/arm_psci.h               |    2 +
include/linux/perf/arm_pmu.h         |    5 +
include/uapi/linux/kvm.h             |   29 +-
33 files changed, 3709 insertions(+), 100 deletions(-)
create mode 100644 arch/arm64/include/asm/kvm_rme.h
create mode 100644 arch/arm64/include/asm/rmi_cmds.h
create mode 100644 arch/arm64/include/asm/rmi_smc.h
create mode 100644 arch/arm64/kvm/rme-exit.c
create mode 100644 arch/arm64/kvm/rme.c
[PATCH v8 00/43] arm64: Support for Arm CCA in KVM
Posted by Steven Price 8 months ago
This series adds support for running protected VMs using KVM under the
Arm Confidential Compute Architecture (CCA).

The related guest support was merged for v6.14-rc1 so you no longer need
that separately.

There are a few changes since v7, many thanks for the review
comments. The highlights are below, and individual patches have a changelog.

 * More documentation - the new ioctls and capabilties are now all
   documented.

 * Initial patch adding "only_private"/"only_shared" to struct
   kvm_gfn_range replaced with already upstream "attr_filter".

 * Improvement in variable naming and error codes, and some improved/new
   comments. All following valuable review feedback (thanks!).

 * Drop the final WIP patch for enabling large PAGE_SIZE support. It's
   not ready for merging and I want to focus on landing the 4k support.

 * Rebased onto v6.15-rc1.

Things to note:

 * The magic numbers for capabilities and ioctls have been updated. So
   you'll need to update your VMM. See below for update kvmtool branch.

 * Patch 42 increases KVM_VCPU_MAX_FEATURES to expose the new feature.
   This also exposes the NV features (as they are currently numbered
   lower). This will resolve when Marc's NV series has landed, see [2].

 * There are some conflicts with v6.15-rc2, mostly documentation, but
   also commit 26fbdf369227 ("KVM: arm64: Don't translate FAR if
   invalid/unsafe") 'hijacks' HPFAR_EL2_NS as a valid bit. This will
   require corresponding changes to the CCA code.

The ABI to the RMM (the RMI) is based on RMM v1.0-rel0 specification[1].

This series is based on v6.15-rc1. It is also available as a git
repository:

https://gitlab.arm.com/linux-arm/linux-cca cca-host/v8

Work in progress changes for kvmtool are available from the git
repository below:

https://gitlab.arm.com/linux-arm/kvmtool-cca cca/v6

[1] https://developer.arm.com/documentation/den0137/1-0rel0/
[2] https://lore.kernel.org/r/20250408105225.4002637-17-maz%40kernel.org

Jean-Philippe Brucker (7):
  arm64: RME: Propagate number of breakpoints and watchpoints to
    userspace
  arm64: RME: Set breakpoint parameters through SET_ONE_REG
  arm64: RME: Initialize PMCR.N with number counter supported by RMM
  arm64: RME: Propagate max SVE vector length from RMM
  arm64: RME: Configure max SVE vector length for a Realm
  arm64: RME: Provide register list for unfinalized RME RECs
  arm64: RME: Provide accurate register list

Joey Gouly (2):
  arm64: RME: allow userspace to inject aborts
  arm64: RME: support RSI_HOST_CALL

Steven Price (31):
  arm64: RME: Handle Granule Protection Faults (GPFs)
  arm64: RME: Add SMC definitions for calling the RMM
  arm64: RME: Add wrappers for RMI calls
  arm64: RME: Check for RME support at KVM init
  arm64: RME: Define the user ABI
  arm64: RME: ioctls to create and configure realms
  KVM: arm64: Allow passing machine type in KVM creation
  arm64: RME: RTT tear down
  arm64: RME: Allocate/free RECs to match vCPUs
  KVM: arm64: vgic: Provide helper for number of list registers
  arm64: RME: Support for the VGIC in realms
  KVM: arm64: Support timers in realm RECs
  arm64: RME: Allow VMM to set RIPAS
  arm64: RME: Handle realm enter/exit
  arm64: RME: Handle RMI_EXIT_RIPAS_CHANGE
  KVM: arm64: Handle realm MMIO emulation
  arm64: RME: Allow populating initial contents
  arm64: RME: Runtime faulting of memory
  KVM: arm64: Handle realm VCPU load
  KVM: arm64: Validate register access for a Realm VM
  KVM: arm64: Handle Realm PSCI requests
  KVM: arm64: WARN on injected undef exceptions
  arm64: Don't expose stolen time for realm guests
  arm64: RME: Always use 4k pages for realms
  arm64: RME: Prevent Device mappings for Realms
  arm_pmu: Provide a mechanism for disabling the physical IRQ
  arm64: RME: Enable PMU support with a realm guest
  arm64: RME: Hide KVM_CAP_READONLY_MEM for realm guests
  KVM: arm64: Expose support for private memory
  KVM: arm64: Expose KVM_ARM_VCPU_REC to user space
  KVM: arm64: Allow activating realms

Suzuki K Poulose (3):
  kvm: arm64: Include kvm_emulate.h in kvm/arm_psci.h
  kvm: arm64: Don't expose debug capabilities for realm guests
  arm64: RME: Allow checking SVE on VM instance

 Documentation/virt/kvm/api.rst       |   91 +-
 arch/arm64/include/asm/kvm_emulate.h |   40 +
 arch/arm64/include/asm/kvm_host.h    |   17 +-
 arch/arm64/include/asm/kvm_rme.h     |  137 +++
 arch/arm64/include/asm/rmi_cmds.h    |  508 ++++++++
 arch/arm64/include/asm/rmi_smc.h     |  259 ++++
 arch/arm64/include/asm/virt.h        |    1 +
 arch/arm64/include/uapi/asm/kvm.h    |   49 +
 arch/arm64/kvm/Kconfig               |    1 +
 arch/arm64/kvm/Makefile              |    3 +-
 arch/arm64/kvm/arch_timer.c          |   48 +-
 arch/arm64/kvm/arm.c                 |  160 ++-
 arch/arm64/kvm/guest.c               |  104 +-
 arch/arm64/kvm/hypercalls.c          |    4 +-
 arch/arm64/kvm/inject_fault.c        |    5 +-
 arch/arm64/kvm/mmio.c                |   16 +-
 arch/arm64/kvm/mmu.c                 |  201 ++-
 arch/arm64/kvm/pmu-emul.c            |    6 +
 arch/arm64/kvm/psci.c                |   30 +
 arch/arm64/kvm/reset.c               |   23 +-
 arch/arm64/kvm/rme-exit.c            |  199 +++
 arch/arm64/kvm/rme.c                 | 1708 ++++++++++++++++++++++++++
 arch/arm64/kvm/sys_regs.c            |   49 +-
 arch/arm64/kvm/vgic/vgic-init.c      |    2 +-
 arch/arm64/kvm/vgic/vgic-v3.c        |    6 +-
 arch/arm64/kvm/vgic/vgic.c           |   54 +-
 arch/arm64/mm/fault.c                |   31 +-
 drivers/perf/arm_pmu.c               |   15 +
 include/kvm/arm_arch_timer.h         |    2 +
 include/kvm/arm_pmu.h                |    4 +
 include/kvm/arm_psci.h               |    2 +
 include/linux/perf/arm_pmu.h         |    5 +
 include/uapi/linux/kvm.h             |   29 +-
 33 files changed, 3709 insertions(+), 100 deletions(-)
 create mode 100644 arch/arm64/include/asm/kvm_rme.h
 create mode 100644 arch/arm64/include/asm/rmi_cmds.h
 create mode 100644 arch/arm64/include/asm/rmi_smc.h
 create mode 100644 arch/arm64/kvm/rme-exit.c
 create mode 100644 arch/arm64/kvm/rme.c

-- 
2.43.0
Re: [PATCH v8 00/43] arm64: Support for Arm CCA in KVM
Posted by Gavin Shan 7 months, 2 weeks ago
On 4/16/25 11:41 PM, Steven Price wrote:

[...]

> 
> The ABI to the RMM (the RMI) is based on RMM v1.0-rel0 specification[1].
> 
> This series is based on v6.15-rc1. It is also available as a git
> repository:
> 
> https://gitlab.arm.com/linux-arm/linux-cca cca-host/v8
> 
> Work in progress changes for kvmtool are available from the git
> repository below:
> 
> https://gitlab.arm.com/linux-arm/kvmtool-cca cca/v6
> 
> [1] https://developer.arm.com/documentation/den0137/1-0rel0/
> [2] https://lore.kernel.org/r/20250408105225.4002637-17-maz%40kernel.org
> 

I got a chance to try the following combination, the guest can boot using
qemu/kvmtool except a RCU stall is observed (more details provided below)

host.tf-rmm      https://git.codelinaro.org/linaro/dcap/rmm.git                      (cca/v8)
host.edk2        git@github.com:tianocore/edk2.git                                   (edk2-stable202411)
host.tf-a        https://git.codelinaro.org/linaro/dcap/tf-a/trusted-firmware-a.git  (cca/v4)
host.qemu        https://git.qemu.org/git/qemu.git                                   (stable-9.2)
host.linux       https://git.gitlab.arm.com/linux-arm/linux-cca.git                  (cca-host/v8)
host.buildroot   https://github.com/buildroot/buildroot                              (master)
guest.qemu       https://git.codelinaro.org/linaro/dcap/qemu.git                     (cca/latest)
guest.kvmtool    https://gitlab.arm.com/linux-arm/kvmtool-cca                        (cca/latest)
guest.buildroot  https://github.com/buildroot/buildroot                              (master)

RCU stall report
----------------

[ 7816.381336] rcu: INFO: rcu_preempt self-detected stall on CPU
[ 7816.382816] rcu:     6-....: (5249 ticks this GP) idle=3a4c/1/0x4000000000000000 softirq=6001/6003 fqs=2624
[ 7816.384399] rcu:     (t=5250 jiffies g=29821 q=47 ncpus=8)
[ 7816.386059] CPU: 6 UID: 0 PID: 203 Comm: qemu-system-aar Not tainted 6.15.0-rc1-gavin-g78b23c56de79 #34 PREEMPT
[ 7816.387133] Hardware name: QEMU QEMU Virtual Machine, BIOS unknown 02/02/2022
[ 7816.387926] pstate: 61402009 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[ 7816.388678] pc : realm_unmap_private_range+0x19c/0x2c0
[ 7816.389878] lr : realm_unmap_private_range+0x94/0x2c0
[ 7816.390388] sp : ffff80008095bb60
[ 7816.390765] x29: ffff80008095bb60 x28: 000000007caef000 x27: ffffba3322764000
[ 7816.392321] x26: 00007fffffffffff x25: 000000007caf0000 x24: 0001000000000000
[ 7816.393168] x23: 00000000c4000155 x22: ffff8000801b5e98 x21: 0000000106481000
[ 7816.393999] x20: 000000007caef000 x19: 0000000000000000 x18: 0000000000000000
[ 7816.394833] x17: 0000000000000000 x16: 0000000000000000 x15: 0000ffff8e997058
[ 7816.395668] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 7816.396548] x11: 0000000038e38e39 x10: 0000000000000004 x9 : ffffba33218a2564
[ 7816.397419] x8 : ffff8000801b5e98 x7 : 000000003fffffff x6 : 0000000000000001
[ 7816.398243] x5 : 000000011e795000 x4 : 0000000000000002 x3 : 0000000000000000
[ 7816.399062] x2 : 000000007caf0000 x1 : 000000011e796000 x0 : 0000000000000000
[ 7816.400021] Call trace:
[ 7816.400604]  realm_unmap_private_range+0x19c/0x2c0 (P)
[ 7816.401347]  kvm_realm_unmap_range+0x94/0xb0
[ 7816.401894]  __unmap_stage2_range+0x70/0xa0
[ 7816.402421]  kvm_arch_post_set_memory_attributes+0x68/0xa8
[ 7816.403011]  kvm_vm_ioctl+0x6bc/0x1b58
[ 7816.403509]  __arm64_sys_ioctl+0xa4/0xe8
[ 7816.404020]  invoke_syscall+0x50/0x120
[ 7816.404562]  el0_svc_common.constprop.0+0x48/0xf0
[ 7816.405113]  do_el0_svc+0x24/0x38
[ 7816.405584]  el0_svc+0x34/0xf0
[ 7816.406045]  el0t_64_sync_handler+0x10c/0x138
[ 7816.406571]  el0t_64_sync+0x1ac/0x1b0

Thanks,
Gavin
Re: [PATCH v8 00/43] arm64: Support for Arm CCA in KVM
Posted by Steven Price 7 months ago
Hi Gavin,

On 02/05/2025 01:46, Gavin Shan wrote:
> On 4/16/25 11:41 PM, Steven Price wrote:
> 
> [...]
> 
>>
>> The ABI to the RMM (the RMI) is based on RMM v1.0-rel0 specification[1].
>>
>> This series is based on v6.15-rc1. It is also available as a git
>> repository:
>>
>> https://gitlab.arm.com/linux-arm/linux-cca cca-host/v8
>>
>> Work in progress changes for kvmtool are available from the git
>> repository below:
>>
>> https://gitlab.arm.com/linux-arm/kvmtool-cca cca/v6
>>
>> [1] https://developer.arm.com/documentation/den0137/1-0rel0/
>> [2] https://lore.kernel.org/r/20250408105225.4002637-17-maz%40kernel.org
>>
> 
> I got a chance to try the following combination, the guest can boot using
> qemu/kvmtool except a RCU stall is observed (more details provided below)
> 
> host.tf-rmm      https://git.codelinaro.org/linaro/dcap/
> rmm.git                      (cca/v8)
> host.edk2        git@github.com:tianocore/
> edk2.git                                   (edk2-stable202411)
> host.tf-a        https://git.codelinaro.org/linaro/dcap/tf-a/trusted-
> firmware-a.git  (cca/v4)
> host.qemu        https://git.qemu.org/git/
> qemu.git                                   (stable-9.2)
> host.linux       https://git.gitlab.arm.com/linux-arm/linux-
> cca.git                  (cca-host/v8)
> host.buildroot   https://github.com/buildroot/
> buildroot                              (master)
> guest.qemu       https://git.codelinaro.org/linaro/dcap/
> qemu.git                     (cca/latest)
> guest.kvmtool    https://gitlab.arm.com/linux-arm/kvmtool-
> cca                        (cca/latest)
> guest.buildroot  https://github.com/buildroot/
> buildroot                              (master)
> 
> RCU stall report
> ----------------
> 
> [ 7816.381336] rcu: INFO: rcu_preempt self-detected stall on CPU
> [ 7816.382816] rcu:     6-....: (5249 ticks this GP)
> idle=3a4c/1/0x4000000000000000 softirq=6001/6003 fqs=2624
> [ 7816.384399] rcu:     (t=5250 jiffies g=29821 q=47 ncpus=8)
> [ 7816.386059] CPU: 6 UID: 0 PID: 203 Comm: qemu-system-aar Not tainted
> 6.15.0-rc1-gavin-g78b23c56de79 #34 PREEMPT
> [ 7816.387133] Hardware name: QEMU QEMU Virtual Machine, BIOS unknown
> 02/02/2022
> [ 7816.387926] pstate: 61402009 (nZCv daif +PAN -UAO -TCO +DIT -SSBS
> BTYPE=--)
> [ 7816.388678] pc : realm_unmap_private_range+0x19c/0x2c0
> [ 7816.389878] lr : realm_unmap_private_range+0x94/0x2c0
> [ 7816.390388] sp : ffff80008095bb60
> [ 7816.390765] x29: ffff80008095bb60 x28: 000000007caef000 x27:
> ffffba3322764000
> [ 7816.392321] x26: 00007fffffffffff x25: 000000007caf0000 x24:
> 0001000000000000
> [ 7816.393168] x23: 00000000c4000155 x22: ffff8000801b5e98 x21:
> 0000000106481000
> [ 7816.393999] x20: 000000007caef000 x19: 0000000000000000 x18:
> 0000000000000000
> [ 7816.394833] x17: 0000000000000000 x16: 0000000000000000 x15:
> 0000ffff8e997058
> [ 7816.395668] x14: 0000000000000000 x13: 0000000000000000 x12:
> 0000000000000000
> [ 7816.396548] x11: 0000000038e38e39 x10: 0000000000000004 x9 :
> ffffba33218a2564
> [ 7816.397419] x8 : ffff8000801b5e98 x7 : 000000003fffffff x6 :
> 0000000000000001
> [ 7816.398243] x5 : 000000011e795000 x4 : 0000000000000002 x3 :
> 0000000000000000
> [ 7816.399062] x2 : 000000007caf0000 x1 : 000000011e796000 x0 :
> 0000000000000000
> [ 7816.400021] Call trace:
> [ 7816.400604]  realm_unmap_private_range+0x19c/0x2c0 (P)

So I suspect this is because I didn't have a cond_resched_rwlock_write()
in here - which was (as you pointed out) because I hadn't propagated
"may_block" down. Finger's crossed this will be fixed with that change.

Thanks,
Steve

> [ 7816.401347]  kvm_realm_unmap_range+0x94/0xb0
> [ 7816.401894]  __unmap_stage2_range+0x70/0xa0
> [ 7816.402421]  kvm_arch_post_set_memory_attributes+0x68/0xa8
> [ 7816.403011]  kvm_vm_ioctl+0x6bc/0x1b58
> [ 7816.403509]  __arm64_sys_ioctl+0xa4/0xe8
> [ 7816.404020]  invoke_syscall+0x50/0x120
> [ 7816.404562]  el0_svc_common.constprop.0+0x48/0xf0
> [ 7816.405113]  do_el0_svc+0x24/0x38
> [ 7816.405584]  el0_svc+0x34/0xf0
> [ 7816.406045]  el0t_64_sync_handler+0x10c/0x138
> [ 7816.406571]  el0t_64_sync+0x1ac/0x1b0
> 
> Thanks,
> Gavin
>