Documentation/virt/kvm/api.rst | 70 +- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/kvm_asm.h | 2 + arch/arm64/include/asm/kvm_emulate.h | 37 + arch/arm64/include/asm/kvm_host.h | 12 +- arch/arm64/include/asm/kvm_pgtable.h | 6 +- arch/arm64/include/asm/kvm_pkvm.h | 2 +- arch/arm64/include/asm/kvm_rmi.h | 139 +++ arch/arm64/include/asm/rmi_cmds.h | 465 ++++++++ arch/arm64/include/asm/virt.h | 1 + arch/arm64/kernel/cpufeature.c | 1 + arch/arm64/kvm/Kconfig | 2 + arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/arch_timer.c | 34 +- arch/arm64/kvm/arm.c | 139 ++- arch/arm64/kvm/guest.c | 93 +- arch/arm64/kvm/handle_exit.c | 14 + arch/arm64/kvm/hyp/pgtable.c | 1 + arch/arm64/kvm/hypercalls.c | 4 +- arch/arm64/kvm/inject_fault.c | 5 +- arch/arm64/kvm/mmio.c | 16 +- arch/arm64/kvm/mmu.c | 144 ++- arch/arm64/kvm/reset.c | 13 +- arch/arm64/kvm/rmi-exit.c | 178 +++ arch/arm64/kvm/rmi.c | 1561 ++++++++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 47 +- arch/arm64/kvm/vgic/vgic-init.c | 2 +- arch/arm64/mm/fault.c | 28 +- drivers/firmware/Kconfig | 1 + drivers/firmware/Makefile | 1 + drivers/firmware/arm_rmm/Kconfig | 26 + drivers/firmware/arm_rmm/Makefile | 2 + drivers/firmware/arm_rmm/rmi.c | 776 +++++++++++++ include/kvm/arm_psci.h | 2 + include/linux/arm-rmi-cmds.h | 201 ++++ include/linux/arm-smccc-rmi.h | 493 ++++++++ include/uapi/linux/kvm.h | 20 +- 37 files changed, 4446 insertions(+), 95 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_rmi.h create mode 100644 arch/arm64/include/asm/rmi_cmds.h create mode 100644 arch/arm64/kvm/rmi-exit.c create mode 100644 arch/arm64/kvm/rmi.c create mode 100644 drivers/firmware/arm_rmm/Kconfig create mode 100644 drivers/firmware/arm_rmm/Makefile create mode 100644 drivers/firmware/arm_rmm/rmi.c create mode 100644 include/linux/arm-rmi-cmds.h create mode 100644 include/linux/arm-smccc-rmi.h
This series adds support for running protected VMs using KVM under the
Arm Confidential Compute Architecture (CCA), including the firmware
support needed to communicate with the Realm Management Monitor (RMM).
For v15 the series was split into a 6-patch generic firmware/RMM series
and a 37-patch KVM series. The two parts are combined again for v16, as
requested to allow the complete stack to be tested by Sashiko as a
single series.
The first six patches add the generic firmware layer for talking to the
RMM, as specified by version 2.0-bet2 of the RMM specification[0]. They
provide the RMI definitions and wrappers, discover and configure the
RMM, implement Stateful RMI Operations (SROs), and ensure that the RMM
has GPT entries for host memory. The remaining patches add the KVM
support needed to create, populate and run Realm VMs.
Note that RMM v2.0 Beta 3 specification should be published soon. The
changes introduced in that are minor and I expect this series to be
largely compatible with the new spec.
The RMM v2.0 specification introduces SROs, which allow the RMM to
complete an operation over several SMC calls while requesting or
returning memory to the host. This allows interrupts to be handled in
the middle of an operation and lets the RMM dynamically allocate memory
for internal tracking purposes. For example, RMI_REC_CREATE no longer
needs auxiliary granules to be provided up front, and can instead
request memory during the operation.
The main changes since v15 are:
* Recombine the firmware/RMM and KVM portions into one series. The
Realm-specific RMI wrappers remain separate from the generic
firmware wrappers, in a new KVM patch.
* Rewrite Realm entry and exit handling to fit into the generic KVM
run loop. Work which cannot be performed in the entry path is now
completed through KVM requests, and MMIO, PSCI, RIPAS changes and
host calls are adapted to the new flow.
* Rewrite Realm timer support to use KVM's IRQ-ops infrastructure and
software resampling.
* Improve SRO cancellation and error handling. Ensure a cancelled
operation isn't treated as successful, wrapper return values can carry
negative Linux errors as well as RMI return values, and those errors
are propagated through the KVM users.
* Check that the RMM supports the host page size before configuring it,
and exclude firmware-reserved NOMAP memory when creating GPT entries.
* Expand the KVM_ARM_RMI_POPULATE documentation to make clear that Arm
CCA cannot preserve memory contents during an in-place
shared-to-private conversion.
* Allocate Realm parameters only while creating the Realm descriptor.
* Document the encoding of the arm64 VM types and require
ICH_HCR_EL2.TDIR before advertising or creating Realm VMs.
This series is based on the guest_memfd in-place conversion v9 tree[1],
which is itself based on kvm-x86/next. It is also available as a git
repository:
https://gitlab.arm.com/linux-arm/linux-cca cca-host/v16
Work in progress changes for kvmtool are available from the git
repository below:
https://gitlab.arm.com/linux-arm/kvmtool-cca cca/v13
The TF-RMM branch used for testing this series is available here:
https://git.trustedfirmware.org/TF-RMM/tf-rmm.git topics/rmm-v2.0-poc_3
There is a kvm-unit-test branch updated to support the attestation used
in RMM v2.0 available here:
https://gitlab.arm.com/linux-arm/kvm-unit-tests-cca cca/v4
[0] https://developer.arm.com/documentation/den0137/2-0bet2/
One bet2 change, which moves metadata out of the individual address
range descriptors, has intentionally not been implemented because that
part of the specification is expected to be reverted.
[1] https://github.com/googleprodkernel/linux-cc/commits/guest_memfd-inplace-conversion-v9
Jean-Philippe Brucker (6):
KVM: arm64: CCA: Propagate breakpoint and watchpoint counts to
userspace
KVM: arm64: CCA: Set breakpoint parameters through SET_ONE_REG
KVM: arm64: CCA: Propagate max SVE vector length from the RMM
KVM: arm64: CCA: Configure max SVE vector length for a Realm
KVM: arm64: CCA: Provide register list for unfinalized RECs
KVM: arm64: CCA: Provide an accurate register list
Joey Gouly (2):
KVM: arm64: CCA: Allow userspace to inject aborts
KVM: arm64: CCA: Support RSI_HOST_CALL
Steven Price (34):
firmware: arm_rmm: Add SMC definitions for calling the RMM
firmware: arm_rmm: Add wrappers for direct RMI calls
firmware: arm_rmm: Check for RMI support at init
firmware: arm_rmm: Configure the RMM with the host's page size
firmware: arm_rmm: Add support for SRO
firmware: arm_rmm: Ensure the RMM has GPT entries for memory
arm64: mm: Handle Granule Protection Faults (GPFs)
KVM: arm64: Avoid including linux/kvm_host.h in kvm_pgtable.h
KVM: arm64: CCA: Add wrappers for realm related RMIs
KVM: arm64: CCA: Check for RMI support at KVM init
KVM: arm64: CCA: Check for LPA2 support
KVM: arm64: CCA: Define the user ABI
KVM: arm64: CCA: Add basic infrastructure for creating a realm
KVM: arm64: CCA: Allow passing the machine type in KVM creation
KVM: arm64: CCA: Tear down RTTs
KVM: arm64: CCA: Allocate and free RECs to match vCPUs
KVM: arm64: CCA: Support the VGIC in realms
KVM: arm64: CCA: Support timers in realm RECs
KVM: arm64: CCA: Handle realm enter/exit
KVM: arm64: CCA: Handle RMI_EXIT_RIPAS_CHANGE
KVM: arm64: CCA: Handle realm MMIO emulation
KVM: arm64: Expose support for private memory
KVM: arm64: CCA: Create the realm descriptor
KVM: arm64: CCA: Activate realms on first vCPU run
KVM: arm64: CCA: Allow populating initial contents
KVM: arm64: CCA: Set RIPAS of initial memslots
KVM: arm64: CCA: Support runtime faulting of memory
KVM: arm64: CCA: Handle realm vCPU load
KVM: arm64: CCA: Validate register access for Realm VMs
KVM: arm64: CCA: Handle Realm PSCI requests
KVM: arm64: WARN on injected undef exceptions
KVM: arm64: CCA: Prevent Device mappings for realms
KVM: arm64: CCA: Require ICH_HCR_EL2.TDIR for realms
KVM: arm64: CCA: Enable realms to be created
Suzuki K Poulose (3):
KVM: arm64: Include kvm_emulate.h in kvm/arm_psci.h
KVM: arm64: CCA: Don't expose unsupported capabilities for realm
guests
KVM: arm64: CCA: Allow checking SVE on VM instance
Documentation/virt/kvm/api.rst | 70 +-
arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/kvm_asm.h | 2 +
arch/arm64/include/asm/kvm_emulate.h | 37 +
arch/arm64/include/asm/kvm_host.h | 12 +-
arch/arm64/include/asm/kvm_pgtable.h | 6 +-
arch/arm64/include/asm/kvm_pkvm.h | 2 +-
arch/arm64/include/asm/kvm_rmi.h | 139 +++
arch/arm64/include/asm/rmi_cmds.h | 465 ++++++++
arch/arm64/include/asm/virt.h | 1 +
arch/arm64/kernel/cpufeature.c | 1 +
arch/arm64/kvm/Kconfig | 2 +
arch/arm64/kvm/Makefile | 2 +-
arch/arm64/kvm/arch_timer.c | 34 +-
arch/arm64/kvm/arm.c | 139 ++-
arch/arm64/kvm/guest.c | 93 +-
arch/arm64/kvm/handle_exit.c | 14 +
arch/arm64/kvm/hyp/pgtable.c | 1 +
arch/arm64/kvm/hypercalls.c | 4 +-
arch/arm64/kvm/inject_fault.c | 5 +-
arch/arm64/kvm/mmio.c | 16 +-
arch/arm64/kvm/mmu.c | 144 ++-
arch/arm64/kvm/reset.c | 13 +-
arch/arm64/kvm/rmi-exit.c | 178 +++
arch/arm64/kvm/rmi.c | 1561 ++++++++++++++++++++++++++
arch/arm64/kvm/sys_regs.c | 47 +-
arch/arm64/kvm/vgic/vgic-init.c | 2 +-
arch/arm64/mm/fault.c | 28 +-
drivers/firmware/Kconfig | 1 +
drivers/firmware/Makefile | 1 +
drivers/firmware/arm_rmm/Kconfig | 26 +
drivers/firmware/arm_rmm/Makefile | 2 +
drivers/firmware/arm_rmm/rmi.c | 776 +++++++++++++
include/kvm/arm_psci.h | 2 +
include/linux/arm-rmi-cmds.h | 201 ++++
include/linux/arm-smccc-rmi.h | 493 ++++++++
include/uapi/linux/kvm.h | 20 +-
37 files changed, 4446 insertions(+), 95 deletions(-)
create mode 100644 arch/arm64/include/asm/kvm_rmi.h
create mode 100644 arch/arm64/include/asm/rmi_cmds.h
create mode 100644 arch/arm64/kvm/rmi-exit.c
create mode 100644 arch/arm64/kvm/rmi.c
create mode 100644 drivers/firmware/arm_rmm/Kconfig
create mode 100644 drivers/firmware/arm_rmm/Makefile
create mode 100644 drivers/firmware/arm_rmm/rmi.c
create mode 100644 include/linux/arm-rmi-cmds.h
create mode 100644 include/linux/arm-smccc-rmi.h
--
2.43.0
Hi Steven, On Mon, 3 Aug 2026 at 14:44, Steven Price <steven.price@arm.com> wrote: ... > * Rewrite Realm entry and exit handling to fit into the generic KVM > run loop. Work which cannot be performed in the entry path is now > completed through KVM requests, and MMIO, PSCI, RIPAS changes and > host calls are adapted to the new flow. Reading v16 with pKVM in mind: this rework helps, but the series still adds a few kvm_is_realm()/vcpu_is_rec() checks to the core arm64 code, and several sit next to, or inside the same condition as, the existing pKVM checks (the nommu path in kvm_arch_vcpu_load(), the timer offsets, the NISV abort injection). Both checks answer the same question: is the guest's state owned by something other than KVM. It might be worth a common predicate covering both. Cheers, /fuad > > * Rewrite Realm timer support to use KVM's IRQ-ops infrastructure and > software resampling. > > * Improve SRO cancellation and error handling. Ensure a cancelled > operation isn't treated as successful, wrapper return values can carry > negative Linux errors as well as RMI return values, and those errors > are propagated through the KVM users. > > * Check that the RMM supports the host page size before configuring it, > and exclude firmware-reserved NOMAP memory when creating GPT entries. > > * Expand the KVM_ARM_RMI_POPULATE documentation to make clear that Arm > CCA cannot preserve memory contents during an in-place > shared-to-private conversion. > > * Allocate Realm parameters only while creating the Realm descriptor. > > * Document the encoding of the arm64 VM types and require > ICH_HCR_EL2.TDIR before advertising or creating Realm VMs. > > This series is based on the guest_memfd in-place conversion v9 tree[1], > which is itself based on kvm-x86/next. It is also available as a git > repository: > > https://gitlab.arm.com/linux-arm/linux-cca cca-host/v16 > > Work in progress changes for kvmtool are available from the git > repository below: > > https://gitlab.arm.com/linux-arm/kvmtool-cca cca/v13 > > The TF-RMM branch used for testing this series is available here: > > https://git.trustedfirmware.org/TF-RMM/tf-rmm.git topics/rmm-v2.0-poc_3 > > There is a kvm-unit-test branch updated to support the attestation used > in RMM v2.0 available here: > > https://gitlab.arm.com/linux-arm/kvm-unit-tests-cca cca/v4 > > [0] https://developer.arm.com/documentation/den0137/2-0bet2/ > One bet2 change, which moves metadata out of the individual address > range descriptors, has intentionally not been implemented because that > part of the specification is expected to be reverted. > > [1] https://github.com/googleprodkernel/linux-cc/commits/guest_memfd-inplace-conversion-v9 > > Jean-Philippe Brucker (6): > KVM: arm64: CCA: Propagate breakpoint and watchpoint counts to > userspace > KVM: arm64: CCA: Set breakpoint parameters through SET_ONE_REG > KVM: arm64: CCA: Propagate max SVE vector length from the RMM > KVM: arm64: CCA: Configure max SVE vector length for a Realm > KVM: arm64: CCA: Provide register list for unfinalized RECs > KVM: arm64: CCA: Provide an accurate register list > > Joey Gouly (2): > KVM: arm64: CCA: Allow userspace to inject aborts > KVM: arm64: CCA: Support RSI_HOST_CALL > > Steven Price (34): > firmware: arm_rmm: Add SMC definitions for calling the RMM > firmware: arm_rmm: Add wrappers for direct RMI calls > firmware: arm_rmm: Check for RMI support at init > firmware: arm_rmm: Configure the RMM with the host's page size > firmware: arm_rmm: Add support for SRO > firmware: arm_rmm: Ensure the RMM has GPT entries for memory > arm64: mm: Handle Granule Protection Faults (GPFs) > KVM: arm64: Avoid including linux/kvm_host.h in kvm_pgtable.h > KVM: arm64: CCA: Add wrappers for realm related RMIs > KVM: arm64: CCA: Check for RMI support at KVM init > KVM: arm64: CCA: Check for LPA2 support > KVM: arm64: CCA: Define the user ABI > KVM: arm64: CCA: Add basic infrastructure for creating a realm > KVM: arm64: CCA: Allow passing the machine type in KVM creation > KVM: arm64: CCA: Tear down RTTs > KVM: arm64: CCA: Allocate and free RECs to match vCPUs > KVM: arm64: CCA: Support the VGIC in realms > KVM: arm64: CCA: Support timers in realm RECs > KVM: arm64: CCA: Handle realm enter/exit > KVM: arm64: CCA: Handle RMI_EXIT_RIPAS_CHANGE > KVM: arm64: CCA: Handle realm MMIO emulation > KVM: arm64: Expose support for private memory > KVM: arm64: CCA: Create the realm descriptor > KVM: arm64: CCA: Activate realms on first vCPU run > KVM: arm64: CCA: Allow populating initial contents > KVM: arm64: CCA: Set RIPAS of initial memslots > KVM: arm64: CCA: Support runtime faulting of memory > KVM: arm64: CCA: Handle realm vCPU load > KVM: arm64: CCA: Validate register access for Realm VMs > KVM: arm64: CCA: Handle Realm PSCI requests > KVM: arm64: WARN on injected undef exceptions > KVM: arm64: CCA: Prevent Device mappings for realms > KVM: arm64: CCA: Require ICH_HCR_EL2.TDIR for realms > KVM: arm64: CCA: Enable realms to be created > > Suzuki K Poulose (3): > KVM: arm64: Include kvm_emulate.h in kvm/arm_psci.h > KVM: arm64: CCA: Don't expose unsupported capabilities for realm > guests > KVM: arm64: CCA: Allow checking SVE on VM instance > > Documentation/virt/kvm/api.rst | 70 +- > arch/arm64/Kconfig | 1 + > arch/arm64/include/asm/kvm_asm.h | 2 + > arch/arm64/include/asm/kvm_emulate.h | 37 + > arch/arm64/include/asm/kvm_host.h | 12 +- > arch/arm64/include/asm/kvm_pgtable.h | 6 +- > arch/arm64/include/asm/kvm_pkvm.h | 2 +- > arch/arm64/include/asm/kvm_rmi.h | 139 +++ > arch/arm64/include/asm/rmi_cmds.h | 465 ++++++++ > arch/arm64/include/asm/virt.h | 1 + > arch/arm64/kernel/cpufeature.c | 1 + > arch/arm64/kvm/Kconfig | 2 + > arch/arm64/kvm/Makefile | 2 +- > arch/arm64/kvm/arch_timer.c | 34 +- > arch/arm64/kvm/arm.c | 139 ++- > arch/arm64/kvm/guest.c | 93 +- > arch/arm64/kvm/handle_exit.c | 14 + > arch/arm64/kvm/hyp/pgtable.c | 1 + > arch/arm64/kvm/hypercalls.c | 4 +- > arch/arm64/kvm/inject_fault.c | 5 +- > arch/arm64/kvm/mmio.c | 16 +- > arch/arm64/kvm/mmu.c | 144 ++- > arch/arm64/kvm/reset.c | 13 +- > arch/arm64/kvm/rmi-exit.c | 178 +++ > arch/arm64/kvm/rmi.c | 1561 ++++++++++++++++++++++++++ > arch/arm64/kvm/sys_regs.c | 47 +- > arch/arm64/kvm/vgic/vgic-init.c | 2 +- > arch/arm64/mm/fault.c | 28 +- > drivers/firmware/Kconfig | 1 + > drivers/firmware/Makefile | 1 + > drivers/firmware/arm_rmm/Kconfig | 26 + > drivers/firmware/arm_rmm/Makefile | 2 + > drivers/firmware/arm_rmm/rmi.c | 776 +++++++++++++ > include/kvm/arm_psci.h | 2 + > include/linux/arm-rmi-cmds.h | 201 ++++ > include/linux/arm-smccc-rmi.h | 493 ++++++++ > include/uapi/linux/kvm.h | 20 +- > 37 files changed, 4446 insertions(+), 95 deletions(-) > create mode 100644 arch/arm64/include/asm/kvm_rmi.h > create mode 100644 arch/arm64/include/asm/rmi_cmds.h > create mode 100644 arch/arm64/kvm/rmi-exit.c > create mode 100644 arch/arm64/kvm/rmi.c > create mode 100644 drivers/firmware/arm_rmm/Kconfig > create mode 100644 drivers/firmware/arm_rmm/Makefile > create mode 100644 drivers/firmware/arm_rmm/rmi.c > create mode 100644 include/linux/arm-rmi-cmds.h > create mode 100644 include/linux/arm-smccc-rmi.h > > -- > 2.43.0 >
On Mon, 03 Aug 2026 14:43:16 +0100, Steven Price <steven.price@arm.com> wrote: > > This series is based on the guest_memfd in-place conversion v9 tree[1], > which is itself based on kvm-x86/next. It is also available as a git > repository: Well done. Series on top of another random series, based on a random tree. Unusable. Have you realised there are reasons why we (KVM/arm64) ask people to base their series on a *released* tag from Linus' tree? This means that Sashiko cannot, once again, provide any feedback on this series, as it fails to apply. Just to avoid any ambiguity: Sashiko reviews, as annoying as they are, are not optional. They are very much mandatory, and I wish I could turn back the clocks to run it on my own code... Running it on your own is not enough, because *I* want to see what the tool finds. Not to mention that non-Gemini runs of sashiko are much less interesting (unsurprisingly). M. -- Without deviation from the norm, progress is not possible.
On 03/08/2026 16:01, Marc Zyngier wrote: > On Mon, 03 Aug 2026 14:43:16 +0100, > Steven Price <steven.price@arm.com> wrote: >> >> This series is based on the guest_memfd in-place conversion v9 tree[1], >> which is itself based on kvm-x86/next. It is also available as a git >> repository: > > Well done. Series on top of another random series, based on a random > tree. Unusable. Have you realised there are reasons why we (KVM/arm64) > ask people to base their series on a *released* tag from Linus' tree? > This means that Sashiko cannot, once again, provide any feedback on > this series, as it fails to apply. > > Just to avoid any ambiguity: Sashiko reviews, as annoying as they are, > are not optional. They are very much mandatory, and I wish I could > turn back the clocks to run it on my own code... > > Running it on your own is not enough, because *I* want to see what the > tool finds. Not to mention that non-Gemini runs of sashiko are much > less interesting (unsurprisingly). Sorry, but I'm not sure what you expect me to do - clearly I want to base my changes on top of the guest_memfd series which is being worked on and not just ignore that work. So I've based my changes on top of the tree that Ackerley published. I'm certainly not trying to avoid the Sashiko reviews - it's a rather unfortunate limitation of the public tool that it doesn't seem to be able to deal with these dependencies. Or is there some magic that I can do to fix this? Thanks, Steve
On Mon, 03 Aug 2026 16:06:24 +0100, Steven Price <steven.price@arm.com> wrote: > > On 03/08/2026 16:01, Marc Zyngier wrote: > > On Mon, 03 Aug 2026 14:43:16 +0100, > > Steven Price <steven.price@arm.com> wrote: > >> > >> This series is based on the guest_memfd in-place conversion v9 tree[1], > >> which is itself based on kvm-x86/next. It is also available as a git > >> repository: > > > > Well done. Series on top of another random series, based on a random > > tree. Unusable. Have you realised there are reasons why we (KVM/arm64) > > ask people to base their series on a *released* tag from Linus' tree? > > This means that Sashiko cannot, once again, provide any feedback on > > this series, as it fails to apply. > > > > Just to avoid any ambiguity: Sashiko reviews, as annoying as they are, > > are not optional. They are very much mandatory, and I wish I could > > turn back the clocks to run it on my own code... > > > > Running it on your own is not enough, because *I* want to see what the > > tool finds. Not to mention that non-Gemini runs of sashiko are much > > less interesting (unsurprisingly). > > Sorry, but I'm not sure what you expect me to do - clearly I want to > base my changes on top of the guest_memfd series which is being worked > on and not just ignore that work. So I've based my changes on top of the > tree that Ackerley published. But that completely ignores the way the KVM/arm64 tree is managed. If you base a series on a bunch of unmerged patches, how do you expect people to do a sensible job at reviewing it? Now I need to ingest another pile of random patches just to review this set? Not happening, sorry. M. -- Without deviation from the norm, progress is not possible.
On 03/08/2026 16:20, Marc Zyngier wrote: > On Mon, 03 Aug 2026 16:06:24 +0100, > Steven Price <steven.price@arm.com> wrote: >> >> On 03/08/2026 16:01, Marc Zyngier wrote: >>> On Mon, 03 Aug 2026 14:43:16 +0100, >>> Steven Price <steven.price@arm.com> wrote: >>>> >>>> This series is based on the guest_memfd in-place conversion v9 tree[1], >>>> which is itself based on kvm-x86/next. It is also available as a git >>>> repository: >>> >>> Well done. Series on top of another random series, based on a random >>> tree. Unusable. Have you realised there are reasons why we (KVM/arm64) >>> ask people to base their series on a *released* tag from Linus' tree? >>> This means that Sashiko cannot, once again, provide any feedback on >>> this series, as it fails to apply. >>> >>> Just to avoid any ambiguity: Sashiko reviews, as annoying as they are, >>> are not optional. They are very much mandatory, and I wish I could >>> turn back the clocks to run it on my own code... >>> >>> Running it on your own is not enough, because *I* want to see what the >>> tool finds. Not to mention that non-Gemini runs of sashiko are much >>> less interesting (unsurprisingly). >> >> Sorry, but I'm not sure what you expect me to do - clearly I want to >> base my changes on top of the guest_memfd series which is being worked >> on and not just ignore that work. So I've based my changes on top of the >> tree that Ackerley published. > > But that completely ignores the way the KVM/arm64 tree is managed. If > you base a series on a bunch of unmerged patches, how do you expect > people to do a sensible job at reviewing it? Now I need to ingest > another pile of random patches just to review this set? It's based on the same 'base' as v15 was - it depends on the unmerged guest_memfd changes. I've just updated to the latest revision from Ackerley. You've been reviewing the changes, and I've attempted to update the code based on that valuable feedback. Clearly I'm not expecting this series to be merged yet - we're waiting for the guest_memfd changes. If you have a suggestion on how I should tackle this in the future then please let me know. I could bombard you with an even bigger series with Ackerley's series on the front, but I think you'd shout at me even more if I did that. > Not happening, sorry. Well I'm off on 'holiday' or more accurately to move house. The new house's (brand new) hot water cylinder is bust and the plumber is currently saying we can't even have cold water, never mind hot, so I'm going to spend the next week worrying about things other than this series. If you provide review comments then I'll look at them when I get back. Thanks, Steve
On 03/08/2026 14:43, Steven Price wrote: > This series adds support for running protected VMs using KVM under the > Arm Confidential Compute Architecture (CCA), including the firmware > support needed to communicate with the Realm Management Monitor (RMM). > ... > This series is based on the guest_memfd in-place conversion v9 tree[1], > which is itself based on kvm-x86/next. It is also available as a git > repository: > > https://gitlab.arm.com/linux-arm/linux-cca cca-host/v16 > > Work in progress changes for kvmtool are available from the git > repository below: Here is a cleaned up version, rebased on to Will's kvmtool master branch: https://gitlab.arm.com/linux-arm/kvmtool-cca cca/kvm-v16 (This works with the v15 of the KVM series too) Build on top of Fuad's Guest memfd support patches. Cheers Suzuki > > https://gitlab.arm.com/linux-arm/kvmtool-cca cca/v13
On 8/7/26 8:05 AM, Suzuki K Poulose wrote:
[...]
>
> Here is a cleaned up version, rebased on to Will's kvmtool master
> branch:
>
> https://gitlab.arm.com/linux-arm/kvmtool-cca cca/kvm-v16
>
> (This works with the v15 of the KVM series too)
> Build on top of Fuad's Guest memfd support patches.
>
With the following combination, I'm able boot up the realm guest.
tf-rmm: https://git.trustedfirmware.org/TF-RMM/tf-rmm.git (branch: topics/rmm-v2.0-poc_2)
tf-a: https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git (branch: master)
host: https://git.gitlab.arm.com/linux-arm/linux-cca.git (branch: cca-host/v16)
kvmtool: https://gitlab.arm.com/linux-arm/kvmtool-cca (branch: cca/kvm-v16)
However, the guest can't boot up and become stuck in SMC_RSI_IPA_STATE_SET request,
which can't completed by the host.
tf-rmm: https://git.trustedfirmware.org/TF-RMM/tf-rmm.git (branch: topics/rmm-v2.0-poc_3)
(1) Login the emulated host
machine$ ssh -o StrictHostKeyChecking=no root@10.26.1.240
(2) Start realm guest using kvmtool
root@host:~# lkvm run --realm -c 1 -m 256 \
-k /mnt/linux/arch/arm64/boot/Image \
-i /mnt/buildroot/output/images/rootfs.cpio.xz \
-p earlycon=uart,mmio,0x101000000
:
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
[ 0.000000] Linux version 7.2.0-rc5-gavin-gf5098b6bae76 (gshan@nvidia-grace-hopper-01.khw.eng.bos2.dc.redhat.com) (gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), GNU ld version 2.41-65.el10) #47 SMP PREEMPT Mon Jul 27 02:29:03 EDT 2026
[ 0.000000] KASLR enabled
[ 0.000000] Machine model: linux,dummy-virt
[ 0.000000] earlycon: uart0 at MMIO 0x0000000101000000 (options '')
[ 0.000000] printk: legacy bootconsole [uart0] enabled
[ 0.000000] efi: UEFI not found.
[ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT
[ 0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x000000008fffffff]
[ 0.000000] NODE_DATA(0) allocated [mem 0x8ff76dc0-0x8ff7ac7f]
[ 0.000000] psci: probing for conduit method from DT.
[ 0.000000] psci: PSCIv1.1 detected in firmware.
[ 0.000000] psci: Using standard PSCI v0.2 function IDs
[ 0.000000] psci: MIGRATE_INFO_TYPE not supported.
[ 0.000000] psci: SMC Calling Convention v1.2
[ 0.000000] RME: Using RSI version 1.0
<... no more output from the guest ...>
(3) The output from host's serial console
SMC_RMI_REALM_CREATE 12a4c7000 12a4c6000 > RMI_INCOMPLETE 0 24 0 0
SMC_RMI_OP_MEM_DONATE 0 1002c7098 9 > RMI_INCOMPLETE 9 0 0 0
SMC_RMI_OP_CONTINUE 0 0 > RMI_SUCCESS 0 0
SMC_RMI_RTT_UNPROT_UNMAP 12a4c7000 18f85c000 18fe00000 0 0 > RMI_ERROR_RTT 2 0 0 0 0
SMC_RMI_RTT_UNPROT_UNMAP 12a4c7000 18fe00000 18fe10000 0 0 > RMI_ERROR_RTT 2 0 0 0 0
SMC_RMI_RTT_UNPROT_UNMAP 12a4c7000 18fe00000 18fe10000 0 0 > RMI_ERROR_RTT 2 0 0 0 0
SMC_RMI_REC_CREATE 12a4c7000 12a392000 12a391000 > RMI_INCOMPLETE 0 40 0 0
SMC_RMI_OP_MEM_DONATE 0 106406098 10 > RMI_INCOMPLETE 10 0 0 0
SMC_RMI_OP_CONTINUE 0 0 > RMI_SUCCESS 0 0
SMC_RMI_REALM_ACTIVATE 12a4c7000 > RMI_SUCCESS
Unhandled write S2_0_C0_C2_2
SMC_RMI_RTT_DATA_MAP 12a4c7000 8ffb4000 8ffb5000 1 129f1d004 > RMI_INCOMPLETE 0 0 0 1
SMC_RMI_OP_CONTINUE 0 0 > RMI_SUCCESS 8ffb5000 0
PSCI_84000000 0 0 0 0 0 0 0 > 10001 0 0 0
PSCI_84000006 0 0 0 0 0 0 0 > ffffffffffffffff 0 0 0
PSCI_8400000a 80000000 0 0 0 0 0 0 > 0 0 0 0
SMC_80000000 0 0 0 0 0 0 0 > 10002 0 0 0
SMC_84000050 1 0 0 ffffacd67d1e8000 ffffacd67d124000 0 0 > ffffffffffffffff 0 0 0
SMC_80000001 80000002 ffff 0 ffffacd67d1e8000 ffffacd67d124000 ffffacd67cd5d000 ffffacd67cd5dd38 > ffffffffffffffff 0 0 0
PSCI_8400000a c4000001 0 0 0 0 0 0 > 0 0 0 0
PSCI_8400000a c4000012 0 0 0 0 0 0 > ffffffffffffffff 0 0 0
PSCI_8400000a c4000015 0 0 0 0 0 0 > ffffffffffffffff 0 0 0
SMC_8600ff01 ffffacd67cfc0740 10002 0 0 0 0 ffffacd67cfb3cc8 > ffffffffffffffff 0 0 0
SMC_RSI_VERSION 10000 > RSI_SUCCESS 10000 10001
SMC_RSI_REALM_CONFIG 81385000 > RSI_SUCCESS
SMC_RSI_IPA_STATE_SET 80000000 90000000 1 0
SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > RMI_ERROR_RTT 3
SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > RMI_ERROR_RTT 3
SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > RMI_ERROR_RTT 3
SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > RMI_ERROR_RTT 3
SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > RMI_ERROR_RTT 3
SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > RMI_ERROR_RTT 3
<... the last message repeats ...>
Thanks,
Gavin
On 11/08/2026 05:44, Gavin Shan wrote: > On 8/7/26 8:05 AM, Suzuki K Poulose wrote: > > [...] > >> >> Here is a cleaned up version, rebased on to Will's kvmtool master >> branch: >> >> https://gitlab.arm.com/linux-arm/kvmtool-cca cca/kvm-v16 >> >> (This works with the v15 of the KVM series too) >> Build on top of Fuad's Guest memfd support patches. >> > > With the following combination, I'm able boot up the realm guest. > > tf-rmm: https://git.trustedfirmware.org/TF-RMM/tf- > rmm.git (branch: topics/rmm-v2.0-poc_2) Please be aware that CCA KVM v15 onwards, the above branch is not compatible. You should be using rmm-v2.0-poc_3. Please could you confirm if this is still an issue ? Cheers Suzuki > tf-a: https://git.trustedfirmware.org/TF-A/trusted-firmware- > a.git (branch: master) > host: https://git.gitlab.arm.com/linux-arm/linux- > cca.git (branch: cca-host/v16) > kvmtool: https://gitlab.arm.com/linux-arm/kvmtool- > cca (branch: cca/kvm-v16) > > However, the guest can't boot up and become stuck in > SMC_RSI_IPA_STATE_SET request, > which can't completed by the host. > > tf-rmm: https://git.trustedfirmware.org/TF-RMM/tf- > rmm.git (branch: topics/rmm-v2.0-poc_3) > > (1) Login the emulated host > > machine$ ssh -o StrictHostKeyChecking=no root@10.26.1.240 > > (2) Start realm guest using kvmtool > > root@host:~# lkvm run --realm -c 1 -m 256 \ > -k /mnt/linux/arch/arm64/boot/Image \ > -i /mnt/buildroot/output/images/rootfs.cpio.xz \ > -p earlycon=uart,mmio,0x101000000 > : > [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510] > [ 0.000000] Linux version 7.2.0-rc5-gavin-gf5098b6bae76 > (gshan@nvidia-grace-hopper-01.khw.eng.bos2.dc.redhat.com) (gcc (GCC) > 14.3.1 20251022 (Red Hat 14.3.1-4), GNU ld version 2.41-65.el10) #47 SMP > PREEMPT Mon Jul 27 02:29:03 EDT 2026 > [ 0.000000] KASLR enabled > [ 0.000000] Machine model: linux,dummy-virt > [ 0.000000] earlycon: uart0 at MMIO 0x0000000101000000 (options '') > [ 0.000000] printk: legacy bootconsole [uart0] enabled > [ 0.000000] efi: UEFI not found. > [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory > node in the DT > [ 0.000000] NUMA: Faking a node at [mem > 0x0000000080000000-0x000000008fffffff] > [ 0.000000] NODE_DATA(0) allocated [mem 0x8ff76dc0-0x8ff7ac7f] > [ 0.000000] psci: probing for conduit method from DT. > [ 0.000000] psci: PSCIv1.1 detected in firmware. > [ 0.000000] psci: Using standard PSCI v0.2 function IDs > [ 0.000000] psci: MIGRATE_INFO_TYPE not supported. > [ 0.000000] psci: SMC Calling Convention v1.2 > [ 0.000000] RME: Using RSI version 1.0 > <... no more output from the guest ...> > > > (3) The output from host's serial console > > SMC_RMI_REALM_CREATE 12a4c7000 12a4c6000 > RMI_INCOMPLETE 0 > 24 0 0 > SMC_RMI_OP_MEM_DONATE 0 1002c7098 9 > RMI_INCOMPLETE 9 0 0 0 > SMC_RMI_OP_CONTINUE 0 0 > RMI_SUCCESS 0 0 > SMC_RMI_RTT_UNPROT_UNMAP 12a4c7000 18f85c000 18fe00000 0 0 > > RMI_ERROR_RTT 2 0 0 0 0 > SMC_RMI_RTT_UNPROT_UNMAP 12a4c7000 18fe00000 18fe10000 0 0 > > RMI_ERROR_RTT 2 0 0 0 0 > SMC_RMI_RTT_UNPROT_UNMAP 12a4c7000 18fe00000 18fe10000 0 0 > > RMI_ERROR_RTT 2 0 0 0 0 > SMC_RMI_REC_CREATE 12a4c7000 12a392000 12a391000 > > RMI_INCOMPLETE 0 40 0 0 > SMC_RMI_OP_MEM_DONATE 0 106406098 10 > RMI_INCOMPLETE 10 0 0 0 > SMC_RMI_OP_CONTINUE 0 0 > RMI_SUCCESS 0 0 > SMC_RMI_REALM_ACTIVATE 12a4c7000 > RMI_SUCCESS > Unhandled write S2_0_C0_C2_2 > SMC_RMI_RTT_DATA_MAP 12a4c7000 8ffb4000 8ffb5000 1 > 129f1d004 > RMI_INCOMPLETE 0 0 0 1 > SMC_RMI_OP_CONTINUE 0 0 > RMI_SUCCESS 8ffb5000 0 > PSCI_84000000 0 0 0 0 0 0 0 > 10001 0 0 0 > PSCI_84000006 0 0 0 0 0 0 0 > ffffffffffffffff 0 0 0 > PSCI_8400000a 80000000 0 0 0 0 0 0 > 0 0 0 0 > SMC_80000000 0 0 0 0 0 0 0 > 10002 0 0 0 > SMC_84000050 1 0 0 ffffacd67d1e8000 > ffffacd67d124000 0 0 > ffffffffffffffff 0 0 0 > SMC_80000001 80000002 ffff 0 ffffacd67d1e8000 > ffffacd67d124000 ffffacd67cd5d000 ffffacd67cd5dd38 > ffffffffffffffff 0 0 0 > PSCI_8400000a c4000001 0 0 0 0 0 0 > 0 0 0 0 > PSCI_8400000a c4000012 0 0 0 0 0 0 > > ffffffffffffffff 0 0 0 > PSCI_8400000a c4000015 0 0 0 0 0 0 > > ffffffffffffffff 0 0 0 > SMC_8600ff01 ffffacd67cfc0740 10002 0 0 0 0 > ffffacd67cfb3cc8 > ffffffffffffffff 0 0 0 > SMC_RSI_VERSION 10000 > RSI_SUCCESS 10000 10001 > SMC_RSI_REALM_CONFIG 81385000 > RSI_SUCCESS > SMC_RSI_IPA_STATE_SET 80000000 90000000 1 0 > SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > > RMI_ERROR_RTT 3 > SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > > RMI_ERROR_RTT 3 > SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > > RMI_ERROR_RTT 3 > SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > > RMI_ERROR_RTT 3 > SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > > RMI_ERROR_RTT 3 > SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > > RMI_ERROR_RTT 3 > <... the last message repeats ...> > > Thanks, > Gavin > >
On 8/11/26 9:12 PM, Suzuki K Poulose wrote:
> On 11/08/2026 05:44, Gavin Shan wrote:
>> On 8/7/26 8:05 AM, Suzuki K Poulose wrote:
>>
>> [...]
>>
>>>
>>> Here is a cleaned up version, rebased on to Will's kvmtool master
>>> branch:
>>>
>>> https://gitlab.arm.com/linux-arm/kvmtool-cca cca/kvm-v16
>>>
>>> (This works with the v15 of the KVM series too)
>>> Build on top of Fuad's Guest memfd support patches.
>>>
>>
>> With the following combination, I'm able boot up the realm guest.
>>
>> tf-rmm: https://git.trustedfirmware.org/TF-RMM/tf- rmm.git (branch: topics/rmm-v2.0-poc_2)
>
>
> Please be aware that CCA KVM v15 onwards, the above branch is not compatible. You should be using rmm-v2.0-poc_3. Please could you
> confirm if this is still an issue ?
>
With tf-rmm/topics/rmm-v2.0-poc_2 + cca/host-v16 + kvmtool/cca/v16, there is no issue
and the realm guest can boot up successfully.
When tf-rmm/topics/rmm-v2.0-poc_3 is used, the realm guest boot gets stuck as I reported
earlier. Note the host is emulated by QEMU (TCG mode).
As the following calltrace indicates, -EAGAIN is returned from tf-rmm::update_ripas()
because true is returned from s2tte_drain_pending() for the S2TTE corresponding to
IPA 0x80000000. Linux host received error (RMI_ERROR_RTT, level=3) in ripas_change().
Upon this specific error and the IPA range [0x80000000 0x90000000], find_map_level()
returns level of 2, and realm_create_rtt_levels() returns 0 without populating any
RTTs. After that, rmi_rtt_set_ripas() is re-executed and the above loop starts over
again.
Linux host
==========
kvm_arch_vcpu_ioctl_run // cca/host-v16
check_vcpu_requests
kvm_check_request
kvm_rec_handle_request
kvm_complete_ripas_change
realm_set_ipa_state
ripas_change
rmi_rtt_set_ripas
SMC_RMI_RTT_SET_RIPAS
TF-RMM
======
SMC_RMI_RTT_SET_RIPAS // tf-rmm/topics/rmm-v2.0-poc_3
smc_rtt_set_ripas
s2tt_walk_lock_unlock
rtt_set_ripas_range
update_ripas
s2tte_drain_pending // true, returns -EAGAIN
The problem is the pending-bit for RTE corresponding IPA address 0x80000000 isn't cleared
when SMC_RMI_RTT_SET_RIPAS is invoked. I didn't figure out how this bit is set and why
it's not cleared in time.
Thanks,
Gavin
>
> Cheers
> Suzuki
>
>
>> tf-a: https://git.trustedfirmware.org/TF-A/trusted-firmware- a.git (branch: master)
>> host: https://git.gitlab.arm.com/linux-arm/linux- cca.git (branch: cca-host/v16)
>> kvmtool: https://gitlab.arm.com/linux-arm/kvmtool- cca (branch: cca/kvm-v16)
>>
>> However, the guest can't boot up and become stuck in SMC_RSI_IPA_STATE_SET request,
>> which can't completed by the host.
>>
>> tf-rmm: https://git.trustedfirmware.org/TF-RMM/tf- rmm.git (branch: topics/rmm-v2.0-poc_3)
>>
>> (1) Login the emulated host
>>
>> machine$ ssh -o StrictHostKeyChecking=no root@10.26.1.240
>>
>> (2) Start realm guest using kvmtool
>>
>> root@host:~# lkvm run --realm -c 1 -m 256 \
>> -k /mnt/linux/arch/arm64/boot/Image \
>> -i /mnt/buildroot/output/images/rootfs.cpio.xz \
>> -p earlycon=uart,mmio,0x101000000
>> :
>> [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
>> [ 0.000000] Linux version 7.2.0-rc5-gavin-gf5098b6bae76 (gshan@nvidia-grace-hopper-01.khw.eng.bos2.dc.redhat.com) (gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), GNU ld version 2.41-65.el10) #47 SMP PREEMPT Mon Jul 27 02:29:03 EDT 2026
>> [ 0.000000] KASLR enabled
>> [ 0.000000] Machine model: linux,dummy-virt
>> [ 0.000000] earlycon: uart0 at MMIO 0x0000000101000000 (options '')
>> [ 0.000000] printk: legacy bootconsole [uart0] enabled
>> [ 0.000000] efi: UEFI not found.
>> [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT
>> [ 0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x000000008fffffff]
>> [ 0.000000] NODE_DATA(0) allocated [mem 0x8ff76dc0-0x8ff7ac7f]
>> [ 0.000000] psci: probing for conduit method from DT.
>> [ 0.000000] psci: PSCIv1.1 detected in firmware.
>> [ 0.000000] psci: Using standard PSCI v0.2 function IDs
>> [ 0.000000] psci: MIGRATE_INFO_TYPE not supported.
>> [ 0.000000] psci: SMC Calling Convention v1.2
>> [ 0.000000] RME: Using RSI version 1.0
>> <... no more output from the guest ...>
>>
>>
>> (3) The output from host's serial console
>>
>> SMC_RMI_REALM_CREATE 12a4c7000 12a4c6000 > RMI_INCOMPLETE 0 24 0 0
>> SMC_RMI_OP_MEM_DONATE 0 1002c7098 9 > RMI_INCOMPLETE 9 0 0 0
>> SMC_RMI_OP_CONTINUE 0 0 > RMI_SUCCESS 0 0
>> SMC_RMI_RTT_UNPROT_UNMAP 12a4c7000 18f85c000 18fe00000 0 0 > RMI_ERROR_RTT 2 0 0 0 0
>> SMC_RMI_RTT_UNPROT_UNMAP 12a4c7000 18fe00000 18fe10000 0 0 > RMI_ERROR_RTT 2 0 0 0 0
>> SMC_RMI_RTT_UNPROT_UNMAP 12a4c7000 18fe00000 18fe10000 0 0 > RMI_ERROR_RTT 2 0 0 0 0
>> SMC_RMI_REC_CREATE 12a4c7000 12a392000 12a391000 > RMI_INCOMPLETE 0 40 0 0
>> SMC_RMI_OP_MEM_DONATE 0 106406098 10 > RMI_INCOMPLETE 10 0 0 0
>> SMC_RMI_OP_CONTINUE 0 0 > RMI_SUCCESS 0 0
>> SMC_RMI_REALM_ACTIVATE 12a4c7000 > RMI_SUCCESS
>> Unhandled write S2_0_C0_C2_2
>> SMC_RMI_RTT_DATA_MAP 12a4c7000 8ffb4000 8ffb5000 1 129f1d004 > RMI_INCOMPLETE 0 0 0 1
>> SMC_RMI_OP_CONTINUE 0 0 > RMI_SUCCESS 8ffb5000 0
>> PSCI_84000000 0 0 0 0 0 0 0 > 10001 0 0 0
>> PSCI_84000006 0 0 0 0 0 0 0 > ffffffffffffffff 0 0 0
>> PSCI_8400000a 80000000 0 0 0 0 0 0 > 0 0 0 0
>> SMC_80000000 0 0 0 0 0 0 0 > 10002 0 0 0
>> SMC_84000050 1 0 0 ffffacd67d1e8000 ffffacd67d124000 0 0 > ffffffffffffffff 0 0 0
>> SMC_80000001 80000002 ffff 0 ffffacd67d1e8000 ffffacd67d124000 ffffacd67cd5d000 ffffacd67cd5dd38 > ffffffffffffffff 0 0 0
>> PSCI_8400000a c4000001 0 0 0 0 0 0 > 0 0 0 0
>> PSCI_8400000a c4000012 0 0 0 0 0 0 > ffffffffffffffff 0 0 0
>> PSCI_8400000a c4000015 0 0 0 0 0 0 > ffffffffffffffff 0 0 0
>> SMC_8600ff01 ffffacd67cfc0740 10002 0 0 0 0 ffffacd67cfb3cc8 > ffffffffffffffff 0 0 0
>> SMC_RSI_VERSION 10000 > RSI_SUCCESS 10000 10001
>> SMC_RSI_REALM_CONFIG 81385000 > RSI_SUCCESS
>> SMC_RSI_IPA_STATE_SET 80000000 90000000 1 0
>> SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > RMI_ERROR_RTT 3
>> SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > RMI_ERROR_RTT 3
>> SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > RMI_ERROR_RTT 3
>> SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > RMI_ERROR_RTT 3
>> SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > RMI_ERROR_RTT 3
>> SMC_RMI_RTT_SET_RIPAS 12a4c7000 12a392000 80000000 90000000 > RMI_ERROR_RTT 3
>> <... the last message repeats ...>
>>
>> Thanks,
>> Gavin
>>
>>
>
On Tue, Aug 11, 2026 at 8:08 PM Gavin Shan <gshan@redhat.com> wrote:
> As the following calltrace indicates, -EAGAIN is returned from tf-rmm::update_ripas()
> because true is returned from s2tte_drain_pending() for the S2TTE corresponding to
> IPA 0x80000000. Linux host received error (RMI_ERROR_RTT, level=3) in ripas_change().
> Upon this specific error and the IPA range [0x80000000 0x90000000], find_map_level()
> returns level of 2, and realm_create_rtt_levels() returns 0 without populating any
> RTTs. After that, rmi_rtt_set_ripas() is re-executed and the above loop starts over
> again.
>
> Linux host
> ==========
> kvm_arch_vcpu_ioctl_run // cca/host-v16
> check_vcpu_requests
> kvm_check_request
> kvm_rec_handle_request
> kvm_complete_ripas_change
> realm_set_ipa_state
> ripas_change
> rmi_rtt_set_ripas
> SMC_RMI_RTT_SET_RIPAS
>
> TF-RMM
> ======
> SMC_RMI_RTT_SET_RIPAS // tf-rmm/topics/rmm-v2.0-poc_3
> smc_rtt_set_ripas
> s2tt_walk_lock_unlock
> rtt_set_ripas_range
> update_ripas
> s2tte_drain_pending // true, returns -EAGAIN
>
> The problem is the pending-bit for RTE corresponding IPA address 0x80000000 isn't cleared
> when SMC_RMI_RTT_SET_RIPAS is invoked. I didn't figure out how this bit is set and why
> it's not cleared in time.
>
Hi Gavin, Suzuki,
I think I ran into a similar issue on rmm-v2.0-poc_3 last week.
This looks like a potential RMM bug: could bit 32 be part of the physical
Address (if PA >= 4 GiB)?
It seems s2tte_drain_pending() in lib/s2tt/src/s2tt.c checks bit 32 without
checking whether the descriptor is valid or invalid.
In my testing, guarding the drain checks with a check for S2TTE_INVALID seemed
to resolve the boot hang:
--- a/lib/s2tt/src/s2tt.c
+++ b/lib/s2tt/src/s2tt.c
@@ -1701,6 +1701,10 @@ unsigned long
s2tte_clear_drain_pending(unsigned long s2tte)
bool s2tte_drain_pending(unsigned long s2tte)
{
+ if ((s2tte & S2TT_DESC_VALID_MASK) != S2TTE_INVALID) {
+ return false;
+ }
+
return (s2tte & S2TTE_SW_DRAIN_PENDING_BIT) != 0UL;
}
@@ -1730,11 +1734,19 @@ unsigned long
s2tte_clear_tlbi_pending(unsigned long s2tte)
bool s2tte_tlbi_pending(unsigned long s2tte)
{
+ if ((s2tte & S2TT_DESC_VALID_MASK) != S2TTE_INVALID) {
+ return false;
+ }
+
return (s2tte & S2TTE_SW_TLBI_PENDING_BIT) != 0UL;
}
unsigned int s2tte_drain_handle(unsigned long s2tte)
{
+ if ((s2tte & S2TT_DESC_VALID_MASK) != S2TTE_INVALID) {
+ return 0U;
+ }
+
return (unsigned int)EXTRACT(S2TTE_SW_HANDLE, s2tte);
}
Sharing in case it helps.
Thanks,
Alper
Hi Alper, Gavin
On 12/08/2026 04:25, Alper Gun wrote:
> On Tue, Aug 11, 2026 at 8:08 PM Gavin Shan <gshan@redhat.com> wrote:
>> As the following calltrace indicates, -EAGAIN is returned from tf-rmm::update_ripas()
>> because true is returned from s2tte_drain_pending() for the S2TTE corresponding to
>> IPA 0x80000000. Linux host received error (RMI_ERROR_RTT, level=3) in ripas_change().
>> Upon this specific error and the IPA range [0x80000000 0x90000000], find_map_level()
>> returns level of 2, and realm_create_rtt_levels() returns 0 without populating any
>> RTTs. After that, rmi_rtt_set_ripas() is re-executed and the above loop starts over
>> again.
>>
>> Linux host
>> ==========
>> kvm_arch_vcpu_ioctl_run // cca/host-v16
>> check_vcpu_requests
>> kvm_check_request
>> kvm_rec_handle_request
>> kvm_complete_ripas_change
>> realm_set_ipa_state
>> ripas_change
>> rmi_rtt_set_ripas
>> SMC_RMI_RTT_SET_RIPAS
>>
>> TF-RMM
>> ======
>> SMC_RMI_RTT_SET_RIPAS // tf-rmm/topics/rmm-v2.0-poc_3
>> smc_rtt_set_ripas
>> s2tt_walk_lock_unlock
>> rtt_set_ripas_range
>> update_ripas
>> s2tte_drain_pending // true, returns -EAGAIN
>>
>> The problem is the pending-bit for RTE corresponding IPA address 0x80000000 isn't cleared
>> when SMC_RMI_RTT_SET_RIPAS is invoked. I didn't figure out how this bit is set and why
>> it's not cleared in time.
Thanks for the details.
>>
>
> Hi Gavin, Suzuki,
>
> I think I ran into a similar issue on rmm-v2.0-poc_3 last week.
> This looks like a potential RMM bug: could bit 32 be part of the physical
> Address (if PA >= 4 GiB)?
>
> It seems s2tte_drain_pending() in lib/s2tt/src/s2tt.c checks bit 32 without
> checking whether the descriptor is valid or invalid.
>
> In my testing, guarding the drain checks with a check for S2TTE_INVALID seemed
> to resolve the boot hang:
> --- a/lib/s2tt/src/s2tt.c
> +++ b/lib/s2tt/src/s2tt.c
> @@ -1701,6 +1701,10 @@ unsigned long
> s2tte_clear_drain_pending(unsigned long s2tte)
>
> bool s2tte_drain_pending(unsigned long s2tte)
> {
> + if ((s2tte & S2TT_DESC_VALID_MASK) != S2TTE_INVALID) {
> + return false;
We should use also consider cases where the entry is INVALID, but
has HIPAS=ASSIGNED/ASSIGNED_DEV to make it tighter. So, I think
it is better to use :
s2tte_is_unassigned() or in the library stick to :
if (!s2tte_has_hipas(s2tte, S2TTE_INVALID_HIPAS_UNASSIGNED))
return false;
May be we should assert this and make the caller responsible for
checking the bit. I will leave it to the tf-RMM team to fix.
But for now, please use the above fix.
Cheers
Suzuki
> + }
> +
> return (s2tte & S2TTE_SW_DRAIN_PENDING_BIT) != 0UL;
> }
>
> @@ -1730,11 +1734,19 @@ unsigned long
> s2tte_clear_tlbi_pending(unsigned long s2tte)
>
> bool s2tte_tlbi_pending(unsigned long s2tte)
> {
> + if ((s2tte & S2TT_DESC_VALID_MASK) != S2TTE_INVALID) {
> + return false;
> + }
> +
> return (s2tte & S2TTE_SW_TLBI_PENDING_BIT) != 0UL;
> }
>
> unsigned int s2tte_drain_handle(unsigned long s2tte)
> {
> + if ((s2tte & S2TT_DESC_VALID_MASK) != S2TTE_INVALID) {
> + return 0U;
> + }
> +
> return (unsigned int)EXTRACT(S2TTE_SW_HANDLE, s2tte);
> }
>
> Sharing in case it helps.
> Thanks,
> Alper
Hi Alper and Suzuki,
On 8/12/26 4:04 PM, Suzuki K Poulose wrote:
> Hi Alper, Gavin
>
> On 12/08/2026 04:25, Alper Gun wrote:
>> On Tue, Aug 11, 2026 at 8:08 PM Gavin Shan <gshan@redhat.com> wrote:
>>> As the following calltrace indicates, -EAGAIN is returned from tf-rmm::update_ripas()
>>> because true is returned from s2tte_drain_pending() for the S2TTE corresponding to
>>> IPA 0x80000000. Linux host received error (RMI_ERROR_RTT, level=3) in ripas_change().
>>> Upon this specific error and the IPA range [0x80000000 0x90000000], find_map_level()
>>> returns level of 2, and realm_create_rtt_levels() returns 0 without populating any
>>> RTTs. After that, rmi_rtt_set_ripas() is re-executed and the above loop starts over
>>> again.
>>>
>>> Linux host
>>> ==========
>>> kvm_arch_vcpu_ioctl_run // cca/host-v16
>>> check_vcpu_requests
>>> kvm_check_request
>>> kvm_rec_handle_request
>>> kvm_complete_ripas_change
>>> realm_set_ipa_state
>>> ripas_change
>>> rmi_rtt_set_ripas
>>> SMC_RMI_RTT_SET_RIPAS
>>>
>>> TF-RMM
>>> ======
>>> SMC_RMI_RTT_SET_RIPAS // tf-rmm/topics/rmm-v2.0-poc_3
>>> smc_rtt_set_ripas
>>> s2tt_walk_lock_unlock
>>> rtt_set_ripas_range
>>> update_ripas
>>> s2tte_drain_pending // true, returns -EAGAIN
>>>
>>> The problem is the pending-bit for RTE corresponding IPA address 0x80000000 isn't cleared
>>> when SMC_RMI_RTT_SET_RIPAS is invoked. I didn't figure out how this bit is set and why
>>> it's not cleared in time.
>
> Thanks for the details.
>
>>>
>>
>> Hi Gavin, Suzuki,
>>
>> I think I ran into a similar issue on rmm-v2.0-poc_3 last week.
>> This looks like a potential RMM bug: could bit 32 be part of the physical
>> Address (if PA >= 4 GiB)?
>>
>> It seems s2tte_drain_pending() in lib/s2tt/src/s2tt.c checks bit 32 without
>> checking whether the descriptor is valid or invalid.
>>
>> In my testing, guarding the drain checks with a check for S2TTE_INVALID seemed
>> to resolve the boot hang:
>> --- a/lib/s2tt/src/s2tt.c
>> +++ b/lib/s2tt/src/s2tt.c
>> @@ -1701,6 +1701,10 @@ unsigned long
>> s2tte_clear_drain_pending(unsigned long s2tte)
>>
>> bool s2tte_drain_pending(unsigned long s2tte)
>> {
>> + if ((s2tte & S2TT_DESC_VALID_MASK) != S2TTE_INVALID) {
>> + return false;
>
> We should use also consider cases where the entry is INVALID, but
> has HIPAS=ASSIGNED/ASSIGNED_DEV to make it tighter. So, I think
> it is better to use :
>
> s2tte_is_unassigned() or in the library stick to :
>
> if (!s2tte_has_hipas(s2tte, S2TTE_INVALID_HIPAS_UNASSIGNED))
> return false;
>
> May be we should assert this and make the caller responsible for
> checking the bit. I will leave it to the tf-RMM team to fix.
>
> But for now, please use the above fix.
>
Both worked for me. With the extra check in place, the realm guest can boot
up successfully.
FYI, The below additional checks in s2tte_tlbi_pending() and s2tte_drain_handle()
aren't needed because they're always guarded by s2tte_drain_pending() in all
calling sites.
Thanks,
Gavin
> Cheers
> Suzuki
>
>
>> + }
>> +
>> return (s2tte & S2TTE_SW_DRAIN_PENDING_BIT) != 0UL;
>> }
>>
>> @@ -1730,11 +1734,19 @@ unsigned long
>> s2tte_clear_tlbi_pending(unsigned long s2tte)
>>
>> bool s2tte_tlbi_pending(unsigned long s2tte)
>> {
>> + if ((s2tte & S2TT_DESC_VALID_MASK) != S2TTE_INVALID) {
>> + return false;
>> + }
>> +
>> return (s2tte & S2TTE_SW_TLBI_PENDING_BIT) != 0UL;
>> }
>>
>> unsigned int s2tte_drain_handle(unsigned long s2tte)
>> {
>> + if ((s2tte & S2TT_DESC_VALID_MASK) != S2TTE_INVALID) {
>> + return 0U;
>> + }
>> +
>> return (unsigned int)EXTRACT(S2TTE_SW_HANDLE, s2tte);
>> }
>>
>
>
>
>
>> Sharing in case it helps.
>> Thanks,
>> Alper
>
On 12/08/2026 11:35, Gavin Shan wrote:
> Hi Alper and Suzuki,
>
> On 8/12/26 4:04 PM, Suzuki K Poulose wrote:
>> Hi Alper, Gavin
>>
>> On 12/08/2026 04:25, Alper Gun wrote:
>>> On Tue, Aug 11, 2026 at 8:08 PM Gavin Shan <gshan@redhat.com> wrote:
>>>> As the following calltrace indicates, -EAGAIN is returned from tf-
>>>> rmm::update_ripas()
>>>> because true is returned from s2tte_drain_pending() for the S2TTE
>>>> corresponding to
>>>> IPA 0x80000000. Linux host received error (RMI_ERROR_RTT, level=3)
>>>> in ripas_change().
>>>> Upon this specific error and the IPA range [0x80000000 0x90000000],
>>>> find_map_level()
>>>> returns level of 2, and realm_create_rtt_levels() returns 0 without
>>>> populating any
>>>> RTTs. After that, rmi_rtt_set_ripas() is re-executed and the above
>>>> loop starts over
>>>> again.
>>>>
>>>> Linux host
>>>> ==========
>>>> kvm_arch_vcpu_ioctl_run // cca/host-v16
>>>> check_vcpu_requests
>>>> kvm_check_request
>>>> kvm_rec_handle_request
>>>> kvm_complete_ripas_change
>>>> realm_set_ipa_state
>>>> ripas_change
>>>> rmi_rtt_set_ripas
>>>> SMC_RMI_RTT_SET_RIPAS
>>>>
>>>> TF-RMM
>>>> ======
>>>> SMC_RMI_RTT_SET_RIPAS // tf-rmm/topics/rmm-
>>>> v2.0-poc_3
>>>> smc_rtt_set_ripas
>>>> s2tt_walk_lock_unlock
>>>> rtt_set_ripas_range
>>>> update_ripas
>>>> s2tte_drain_pending // true, returns -EAGAIN
>>>>
>>>> The problem is the pending-bit for RTE corresponding IPA address
>>>> 0x80000000 isn't cleared
>>>> when SMC_RMI_RTT_SET_RIPAS is invoked. I didn't figure out how this
>>>> bit is set and why
>>>> it's not cleared in time.
>>
>> Thanks for the details.
>>
>>>>
>>>
>>> Hi Gavin, Suzuki,
>>>
>>> I think I ran into a similar issue on rmm-v2.0-poc_3 last week.
>>> This looks like a potential RMM bug: could bit 32 be part of the
>>> physical
>>> Address (if PA >= 4 GiB)?
>>>
>>> It seems s2tte_drain_pending() in lib/s2tt/src/s2tt.c checks bit 32
>>> without
>>> checking whether the descriptor is valid or invalid.
>>>
>>> In my testing, guarding the drain checks with a check for
>>> S2TTE_INVALID seemed
>>> to resolve the boot hang:
>>> --- a/lib/s2tt/src/s2tt.c
>>> +++ b/lib/s2tt/src/s2tt.c
>>> @@ -1701,6 +1701,10 @@ unsigned long
>>> s2tte_clear_drain_pending(unsigned long s2tte)
>>>
>>> bool s2tte_drain_pending(unsigned long s2tte)
>>> {
>>> + if ((s2tte & S2TT_DESC_VALID_MASK) != S2TTE_INVALID) {
>>> + return false;
>>
>> We should use also consider cases where the entry is INVALID, but
>> has HIPAS=ASSIGNED/ASSIGNED_DEV to make it tighter. So, I think
>> it is better to use :
>>
>> s2tte_is_unassigned() or in the library stick to :
>>
>> if (!s2tte_has_hipas(s2tte, S2TTE_INVALID_HIPAS_UNASSIGNED))
>> return false;
>>
>> May be we should assert this and make the caller responsible for
>> checking the bit. I will leave it to the tf-RMM team to fix.
>>
>> But for now, please use the above fix.
>>
>
> Both worked for me. With the extra check in place, the realm guest can boot
> up successfully.
>
> FYI, The below additional checks in s2tte_tlbi_pending() and
> s2tte_drain_handle()
> aren't needed because they're always guarded by s2tte_drain_pending() in
> all
> calling sites.
fyi, the tf-RMM patch is out for review here : (Thanks Javier)
https://review.trustedfirmware.org/c/TF-RMM/tf-rmm/+/53531
Please feel free to cherry-pick that one
Cheers
Suzuki
© 2016 - 2026 Red Hat, Inc.