This series implements the KVM stage 2 page table pre-faulting feature for
arm64.

== Foundations ==

The series begins by establishing required foundations:

1. Updating kvm_s2_fault_desc to independently store the exception syndrome
   register (ESR) value, and updating all code paths to use this value
   exclusively.

This is needed so we can later generate a synthetic fault to perform the
pre-faulting - we need to be sure the code doesn't grab an incorrect ESR
from elsewhere.

2. Updating kvm_s2_fault_desc to independently store the kvm_s2_mmu and
   updating all code paths to use this value exclusively.

Similarly this is needed so we can generate a synthetic fault against the
canonical stage 2 MMU (it would make no sense for it to touch nested shadow
page tables) - we need to be sure that the code doesn't grab an incorrect
MMU from elsewhere.

3. Updating the abort paths which consume kvm_s2_fault_desc to also return
   a kvm_s2_fault_result data structure.

To perform pre-faulting the code must know the granule size of what was
just walked. So the abort paths have to tell us what that was.

4. Pass walk flags to kvm_pgtable_get_leaf() to permit walking page tables
   under the MMU read lock.

This is Jack's patch, verbatim, which allows the use of the
KVM_PGTABLE_WALK_SHARED flag to walk page tables under the MMU read lock.

Pre-faulting requires it to be able to work in parallel as specified by the
API. The read lock precludes page tables being torn down behind our back.

== Implementation ==

Pre-faulting is implemented in kvm_arch_vcpu_pre_fault_memory() whose job
is to pre-fault the stage 2 page tables which map a specific GPA (which,
for arm64, is the guest's IPA).

This function is called by kvm_vcpu_pre_fault_memory() for each GPA in the
range, which itself is ultimately invoked by userland via the
KVM_PRE_FAULT_MEMORY ioctl.

The implementation is simple - try to walk to the stage 2 page table
mapping the GPA - if unmapped, fault it in through a synthetic page fault.

pKVM is not supported regardless of whether the VM is protected or
not.

This is because pKVM instantiates vCPUs upon run, but pre-faulting is
typically performed before a vCPU is run. It would be confusing and
inconsistent to error out on non-running vCPUs but to pre-fault running
ones.

== Credits ==

This series is based, with gratitude, on Jack Thomson's series and their
respins (links provided below) as well as the feedback he received.

The series includes Jack's v5 "KVM: arm64: Pass walk flags to
kvm_pgtable_get_leaf()" patch verbatim and two of his test patches (with
minor fixups), plus a nested pre-fault test based on his.

Link: https://patch.msgid.link/20260612162354.73378-1-jackabt.amazon@gmail.com/
Link: https://patch.msgid.link/20260113152643.18858-1-jackabt.amazon@gmail.com/
Link: https://patch.msgid.link/20251119154910.97716-1-jackabt.amazon@gmail.com/
Link: https://patch.msgid.link/20251013151502.6679-1-jackabt.amazon@gmail.com/
Link: https://patch.msgid.link/20250911134648.58945-1-jackabt.amazon@gmail.com/

== Reviewer Notes ==

I synced with maintainers on this who asked me to take a look, as there hadn't
been progress on the series for some time.

Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
---
v2:
* Split series - Added 1/13 for the esr.h helpers, 2/13 for converting
  callers to use them, and move what was 1/8 to 3/13 to propagate
  s2fd->esr, as per Marc.
* Made esr helpers __always_inline as per Marc.
* Renamed esr_trap_get_class() -> esr_get_ec() + dropped churn as per Marc.
* Updated 5/13 (was 3/8) to drop the kvm_s2_fault_result.mapped flag.
* Also updated 5/13 so abort handlers return -EAGAIN instead of swallowing, as
  per Marc.
* Split implementation patch further -> topup_mmu_memcache() change to 6/13,
  EHWPOISON change -> 7/13 and docs change -> 10/13, as per Marc.
* No longer mkyoung existing walked ranges, updated docs to reflect, as per
  Marc.
* Eliminated s2fd->pre_fault as per Marc.
* Eliminated the redundant page table level in pre_fault_s2() as per Marc.
* Put the commit message for 9/13 on a diet as per Marc.
* Fixed up commit message for 11/13 (was 6/8) to correct reasoning about
  exit-reason assert, as per Fuad.
* Simplified: nested test 13/13 (was 8/8), dropping the empty nested-S2
  setup and using test_supports_el2() to handle NV=0 opt-out, as per
  Wei-Lin and Fuad.
* Re-authored patch 13/13 to me as the changes are substantial enough to
  require it. Updated commit message to give Jack credit for original.

v1:
https://lore.kernel.org/r/20260825-kvm-arm-prefault-v1-0-befe8947702e@kernel.org

---
Jack Thomson (3):
      KVM: arm64: Pass walk flags to kvm_pgtable_get_leaf()
      KVM: selftests: Enable pre_fault_memory_test for arm64
      KVM: selftests: Add option for different backing in pre-fault tests

Lorenzo Stoakes (ARM) (10):
      arm64: Add ESR fault helpers
      KVM: arm64: Use ESR helpers in guest abort handling
      KVM: arm64: Propagate and use esr in s2fd when handling guest aborts
      KVM: arm64: Propagate and use mmu in s2fd when handling guest aborts
      KVM: arm64: Propagate and use kvm_s2_fault_result on S2 fault
      KVM: arm64: Size the stage-2 memcache from the fault MMU
      KVM: arm64: Propagate EHWPOISON in kvm_s2_fault_pin_pfn()
      KVM: arm64: Implement KVM_PRE_FAULT_MEMORY
      Documentation: KVM: document arm64 KVM_PRE_FAULT_MEMORY
      KVM: selftests: Add nested pre-fault test for arm64

 Documentation/virt/kvm/api.rst                     |  18 +-
 arch/arm64/include/asm/esr.h                       |  44 +++
 arch/arm64/include/asm/kvm_emulate.h               |  52 ++--
 arch/arm64/include/asm/kvm_pgtable.h               |   5 +-
 arch/arm64/include/asm/kvm_pkvm.h                  |   2 +-
 arch/arm64/kvm/Kconfig                             |   1 +
 arch/arm64/kvm/arm.c                               |   1 +
 arch/arm64/kvm/hyp/nvhe/mem_protect.c              |  10 +-
 arch/arm64/kvm/hyp/pgtable.c                       |   5 +-
 arch/arm64/kvm/mmu.c                               | 300 +++++++++++++++++----
 arch/arm64/kvm/nested.c                            |   2 +-
 tools/testing/selftests/kvm/Makefile.kvm           |   2 +
 .../selftests/kvm/arm64/nv_pre_fault_memory_test.c | 158 +++++++++++
 .../testing/selftests/kvm/pre_fault_memory_test.c  | 152 +++++++++--
 14 files changed, 615 insertions(+), 137 deletions(-)
---
base-commit: aa8e5dc6a7a2a1141ab40706a51010adcd0e57d2
change-id: 20260815-kvm-arm-prefault-9bb411b6897d

Best regards,
-- 
Lorenzo Stoakes (ARM) <ljs@kernel.org>

"Lorenzo Stoakes (ARM)" <ljs@kernel.org> writes:

> This series implements the KVM stage 2 page table pre-faulting feature for
> arm64.
>

> == Foundations ==
>
> The series begins by establishing the required foundations:
>
> 1. Update kvm_s2_fault_desc to store the exception syndrome register (ESR)
>    value independently, and update all code paths to use this value
>    exclusively.
>
> This is required to generate a synthetic fault for pre-faulting without
> inadvertently obtaining an incorrect ESR from elsewhere.
>
> 2. Update kvm_s2_fault_desc to store the kvm_s2_mmu independently, and
>    update all code paths to use this value exclusively.
>
> This is similarly required to generate a synthetic fault against the
> canonical stage-2 MMU. It would make no sense for pre-faulting to modify
> nested shadow page tables, so the code must not obtain an incorrect MMU
> from elsewhere.
>
> 3. Update the abort paths that consume kvm_s2_fault_desc to also return a
>    kvm_s2_fault_result.
>
> Pre-faulting needs to know the granule size handled by the fault, so the
> abort paths must return that information.
>
> 4. Pass walk flags to kvm_pgtable_get_leaf() to allow page-table walks
>    under the MMU read lock.
>
> This is Jack's patch verbatim. It allows KVM_PGTABLE_WALK_SHARED to be used
> when walking page tables under the MMU read lock.
>
> The API permits pre-faulting to run in parallel, and the read lock prevents
> the page tables from being torn down during the walk.
>
> == Implementation ==
>
> Pre-faulting is implemented in kvm_arch_vcpu_pre_fault_memory(), which
> pre-faults the stage-2 page tables for a specific GPA (the guest IPA on
> arm64).
>
> kvm_vcpu_pre_fault_memory() calls this function for each GPA in the range
> requested by userspace through the KVM_PRE_FAULT_MEMORY ioctl.
>
> The implementation is straightforward: walk the stage-2 page tables for
> the GPA and, if it is unmapped, populate the mapping by handling a
> synthetic page fault.

 [ ... 47 lines skipped ... ] 

>
> pKVM is not supported regardless of whether the VM is protected or
> not.
>
> This is because pKVM instantiates vCPUs upon run,
>

Can pKVM instantiate the hyp vCPU during pre-faulting ?


> but pre-faulting is  typically performed before a vCPU is run.  It would be confusing and
> inconsistent to error out on non-running vCPUs but to pre-fault running
> ones.


I use KVM pre-faulting when transitioning pages from shared to private
with CoCo guest. This ensures that a trusted device can DMA to private
memory before the guest accesses it.

-aneesh

On Tue, Sep 22, 2026 at 03:28:33PM +0530, Aneesh Kumar K.V wrote:
> > This is because pKVM instantiates vCPUs upon run,
> >
>
> Can pKVM instantiate the hyp vCPU during pre-faulting ?

It would be an unusual and unexpected thing to do - suddenly a pre-fault
operation is initialising a vCPU explicitly for pKVM.

A caller is not going to reasonably expect this and might treat a failure to
pre-alloc as fine to carry on whereas in fact it was a failure to initailised a
pKVM vCPU.

It'd also require significant changes to how pKVM is set up, right now it's
hardcoded to be done unconditionally at run via kvm_arch_vcpu_run_pid_change()
-> pkvm_create_hyp_vcpu(), so all that would have to change and be checked and
tested and... that'd be really out of scope I think :)

And pre-faulting really makes most sense BEFORE you run a VM. It doesn't make so
much sense mid-run.

But more fundamentally, the stage 2 page tables, as I understand it, are owned
by pKVM and so aren't really available to be pre-faulted.

Maybe unprotected-under-pKVM VMs but then it's questionable as to how useful
that would be given that it would be confusing to users vs. how it works for
other VMs.

So in general, no I don't think it's a good idea.

And even if we wanted to pursue some version of this, it's _definitely_ out
of scope for the initial pre-faulting bring-up series.

>
>
> > but pre-faulting is  typically performed before a vCPU is run.  It would be confusing and
> > inconsistent to error out on non-running vCPUs but to pre-fault running
> > ones.
>
>
> I use KVM pre-faulting when transitioning pages from shared to private

You mean you'd prefer to use? Or you are using it on another arch?

> with CoCo guest. This ensures that a trusted device can DMA to private
> memory before the guest accesses it.

Hm what do you mean by private memory?

I see:

#ifndef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
static inline bool kvm_arch_has_private_mem(struct kvm *kvm)
{
	return false;
}
#endif

And only x86 selects KVM_GENERIC_MEMORY_ATTRIBUTES?

Do you mean something else?

Or are you saying we should match x86? I don't think we have the same
semantics as them, I don't think there's an equivalent for us, and of
course no private memory in the sense of the predicate above.

In any case I think anything wanting to add additional functionality to
pre-faulting would have to be a follow-up anyway.

>
> -aneesh

--
Cheers, Lorenzo

"Lorenzo Stoakes (ARM)" <ljs@kernel.org> writes:

> On Tue, Sep 22, 2026 at 03:28:33PM +0530, Aneesh Kumar K.V wrote:
>> > This is because pKVM instantiates vCPUs upon run,
>> >
>>
>> Can pKVM instantiate the hyp vCPU during pre-faulting ?
>
> It would be an unusual and unexpected thing to do - suddenly a pre-fault
> operation is initialising a vCPU explicitly for pKVM.
>
> A caller is not going to reasonably expect this and might treat a failure to
> pre-alloc as fine to carry on whereas in fact it was a failure to initailised a
> pKVM vCPU.
>
> It'd also require significant changes to how pKVM is set up, right now it's
> hardcoded to be done unconditionally at run via kvm_arch_vcpu_run_pid_change()
> -> pkvm_create_hyp_vcpu(), so all that would have to change and be checked and
> tested and... that'd be really out of scope I think :)
>
> And pre-faulting really makes most sense BEFORE you run a VM. It doesn't make so
> much sense mid-run.
>
> But more fundamentally, the stage 2 page tables, as I understand it, are owned
> by pKVM and so aren't really available to be pre-faulted.
>
> Maybe unprotected-under-pKVM VMs but then it's questionable as to how useful
> that would be given that it would be confusing to users vs. how it works for
> other VMs.
>
> So in general, no I don't think it's a good idea.
>
> And even if we wanted to pursue some version of this, it's _definitely_ out
> of scope for the initial pre-faulting bring-up series.
>
>>
>>
>> > but pre-faulting is  typically performed before a vCPU is run.  It would be confusing and
>> > inconsistent to error out on non-running vCPUs but to pre-fault running
>> > ones.
>>
>>
>> I use KVM pre-faulting when transitioning pages from shared to private
>
> You mean you'd prefer to use? Or you are using it on another arch?
>
>> with CoCo guest. This ensures that a trusted device can DMA to private
>> memory before the guest accesses it.
>
> Hm what do you mean by private memory?
>
> I see:
>
> #ifndef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
> static inline bool kvm_arch_has_private_mem(struct kvm *kvm)
> {
> 	return false;
> }
> #endif
>
> And only x86 selects KVM_GENERIC_MEMORY_ATTRIBUTES?
>
> Do you mean something else?
>

I am using this with ARM CCA-DA, based on the patch series from Jack Thomson <jackabt.amazon@gmail.com>.

https://gitlab.arm.com/linux-arm/kvmtool-cca/-/commit/80e7aad61c5639de2f0cb4a5525dad0c96156428

We do this while the VM is running.

-aneesh

On Tue, Sep 22, 2026 at 06:37:04PM +0530, Aneesh Kumar K.V wrote:
> "Lorenzo Stoakes (ARM)" <ljs@kernel.org> writes:
>
> > On Tue, Sep 22, 2026 at 03:28:33PM +0530, Aneesh Kumar K.V wrote:
> >> > This is because pKVM instantiates vCPUs upon run,
> >> >
> >>
> >> Can pKVM instantiate the hyp vCPU during pre-faulting ?
> >
> > It would be an unusual and unexpected thing to do - suddenly a pre-fault
> > operation is initialising a vCPU explicitly for pKVM.
> >
> > A caller is not going to reasonably expect this and might treat a failure to
> > pre-alloc as fine to carry on whereas in fact it was a failure to initailised a
> > pKVM vCPU.
> >
> > It'd also require significant changes to how pKVM is set up, right now it's
> > hardcoded to be done unconditionally at run via kvm_arch_vcpu_run_pid_change()
> > -> pkvm_create_hyp_vcpu(), so all that would have to change and be checked and
> > tested and... that'd be really out of scope I think :)
> >
> > And pre-faulting really makes most sense BEFORE you run a VM. It doesn't make so
> > much sense mid-run.
> >
> > But more fundamentally, the stage 2 page tables, as I understand it, are owned
> > by pKVM and so aren't really available to be pre-faulted.
> >
> > Maybe unprotected-under-pKVM VMs but then it's questionable as to how useful
> > that would be given that it would be confusing to users vs. how it works for
> > other VMs.
> >
> > So in general, no I don't think it's a good idea.
> >
> > And even if we wanted to pursue some version of this, it's _definitely_ out
> > of scope for the initial pre-faulting bring-up series.
> >
> >>
> >>
> >> > but pre-faulting is  typically performed before a vCPU is run.  It would be confusing and
> >> > inconsistent to error out on non-running vCPUs but to pre-fault running
> >> > ones.
> >>
> >>
> >> I use KVM pre-faulting when transitioning pages from shared to private
> >
> > You mean you'd prefer to use? Or you are using it on another arch?
> >
> >> with CoCo guest. This ensures that a trusted device can DMA to private
> >> memory before the guest accesses it.
> >
> > Hm what do you mean by private memory?
> >
> > I see:
> >
> > #ifndef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
> > static inline bool kvm_arch_has_private_mem(struct kvm *kvm)
> > {
> > 	return false;
> > }
> > #endif
> >
> > And only x86 selects KVM_GENERIC_MEMORY_ATTRIBUTES?
> >
> > Do you mean something else?
> >
>
> I am using this with ARM CCA-DA, based on the patch series from Jack Thomson <jackabt.amazon@gmail.com>.
>
> https://gitlab.arm.com/linux-arm/kvmtool-cca/-/commit/80e7aad61c5639de2f0cb4a5525dad0c96156428
>
> We do this while the VM is running.

Right, that's a non-mainline kernel I guess? Which presumably implements private memory.

Jack himself experienced a panic with his pKVM code, so the code you're using is
not upstreamable, unfortunately. And he'd already shelved pKVM support AFAICT.

And reviewers pointed out actually implementing the pKVM stuff properly would be
quite involved, even if you wanted to do that (hence follow-up).

Also you end up stuck with the same problems as I mentioned above - you can't
sanely bring the vCPU pre-run, so now you have extremely weird behaviour - only
pre-faults if vCPU initialised, running, and unprotected pKVM.

Protected pKVM support is a whole other layer of complexity and it's not obvious
that you're really achieving what pre-fault is supposed to.

In any case Oliver literally just asked me to _simplify_ weird edge cases for
this series :) so I am not sure something like that is going to be accepted.

Are you sure you're actually running in pKVM mode btw? CCA doesn't AFAICT? In
which case this series _should_ work fine for you.

Anyway, if we really do need to add something for pKVM it needs to be a follow
up. Let's get the basics working first :)

(Note that kvmtool will need to be updated to retry pre-fault on -EAGAIN, -EINTR
 as this series can, albeit unlikely, return -EAGAIN.)

>
> -aneesh

--
Cheers, Lorenzo

"Lorenzo Stoakes (ARM)" <ljs@kernel.org> writes:

> On Tue, Sep 22, 2026 at 06:37:04PM +0530, Aneesh Kumar K.V wrote:
>> "Lorenzo Stoakes (ARM)" <ljs@kernel.org> writes:
>>
>> > On Tue, Sep 22, 2026 at 03:28:33PM +0530, Aneesh Kumar K.V wrote:
>> >> > This is because pKVM instantiates vCPUs upon run,
>> >> >
>> >>
>> >> Can pKVM instantiate the hyp vCPU during pre-faulting ?
>> >
>> > It would be an unusual and unexpected thing to do - suddenly a pre-fault
>> > operation is initialising a vCPU explicitly for pKVM.
>> >
>> > A caller is not going to reasonably expect this and might treat a failure to
>> > pre-alloc as fine to carry on whereas in fact it was a failure to initailised a
>> > pKVM vCPU.
>> >
>> > It'd also require significant changes to how pKVM is set up, right now it's
>> > hardcoded to be done unconditionally at run via kvm_arch_vcpu_run_pid_change()
>> > -> pkvm_create_hyp_vcpu(), so all that would have to change and be checked and
>> > tested and... that'd be really out of scope I think :)
>> >
>> > And pre-faulting really makes most sense BEFORE you run a VM. It doesn't make so
>> > much sense mid-run.
>> >
>> > But more fundamentally, the stage 2 page tables, as I understand it, are owned
>> > by pKVM and so aren't really available to be pre-faulted.
>> >
>> > Maybe unprotected-under-pKVM VMs but then it's questionable as to how useful
>> > that would be given that it would be confusing to users vs. how it works for
>> > other VMs.
>> >
>> > So in general, no I don't think it's a good idea.
>> >
>> > And even if we wanted to pursue some version of this, it's _definitely_ out
>> > of scope for the initial pre-faulting bring-up series.
>> >
>> >>
>> >>
>> >> > but pre-faulting is  typically performed before a vCPU is run.  It would be confusing and
>> >> > inconsistent to error out on non-running vCPUs but to pre-fault running
>> >> > ones.
>> >>
>> >>
>> >> I use KVM pre-faulting when transitioning pages from shared to private
>> >
>> > You mean you'd prefer to use? Or you are using it on another arch?
>> >
>> >> with CoCo guest. This ensures that a trusted device can DMA to private
>> >> memory before the guest accesses it.
>> >
>> > Hm what do you mean by private memory?
>> >
>> > I see:
>> >
>> > #ifndef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
>> > static inline bool kvm_arch_has_private_mem(struct kvm *kvm)
>> > {
>> > 	return false;
>> > }
>> > #endif
>> >
>> > And only x86 selects KVM_GENERIC_MEMORY_ATTRIBUTES?
>> >
>> > Do you mean something else?
>> >
>>
>> I am using this with ARM CCA-DA, based on the patch series from Jack Thomson <jackabt.amazon@gmail.com>.
>>
>> https://gitlab.arm.com/linux-arm/kvmtool-cca/-/commit/80e7aad61c5639de2f0cb4a5525dad0c96156428
>>
>> We do this while the VM is running.
>
> Right, that's a non-mainline kernel I guess? Which presumably implements private memory.
>
> Jack himself experienced a panic with his pKVM code, so the code you're using is
> not upstreamable, unfortunately. And he'd already shelved pKVM support AFAICT.
>
> And reviewers pointed out actually implementing the pKVM stuff properly would be
> quite involved, even if you wanted to do that (hence follow-up).
>
> Also you end up stuck with the same problems as I mentioned above - you can't
> sanely bring the vCPU pre-run, so now you have extremely weird behaviour - only
> pre-faults if vCPU initialised, running, and unprotected pKVM.
>

IIUC, pkvm_pgtable_stage2_map() only uses the hyp vCPU's pKVM memcache
(&vcpu->vcpu.arch.pkvm_memcache). I agree that this does not need to be
addressed in this series. However, there is also a desire to keep the CCA
and pKVM code paths similar by using helpers such as kvm_vm_is_protected().
Since an RMM can create a stage-2 mapping without a REC (vCPU), rejecting
pre-faulting for all protected VM configurations may be incorrect. I can
handle this in my series though.

I also want to point out that there are use cases for pre-faulting while
the VM is running.

>
> Protected pKVM support is a whole other layer of complexity and it's not obvious
> that you're really achieving what pre-fault is supposed to.
>
> In any case Oliver literally just asked me to _simplify_ weird edge cases for
> this series :) so I am not sure something like that is going to be accepted.
>
> Are you sure you're actually running in pKVM mode btw? CCA doesn't AFAICT? In
> which case this series _should_ work fine for you.
>


It is not pKVM; it runs in Realm mode.

>
> Anyway, if we really do need to add something for pKVM it needs to be a follow
> up. Let's get the basics working first :)
>

sure.

> (Note that kvmtool will need to be updated to retry pre-fault on -EAGAIN, -EINTR
>  as this series can, albeit unlikely, return -EAGAIN.)
>

I will check this when I rebase my kernel onto this series.

-aneesh

On Tue, Sep 22, 2026 at 07:52:25PM +0530, Aneesh Kumar K.V wrote:
> "Lorenzo Stoakes (ARM)" <ljs@kernel.org> writes:
>
> > On Tue, Sep 22, 2026 at 06:37:04PM +0530, Aneesh Kumar K.V wrote:
> >> "Lorenzo Stoakes (ARM)" <ljs@kernel.org> writes:
> >>
> >> > On Tue, Sep 22, 2026 at 03:28:33PM +0530, Aneesh Kumar K.V wrote:
> >> >> > This is because pKVM instantiates vCPUs upon run,
> >> >> >
> >> >>
> >> >> Can pKVM instantiate the hyp vCPU during pre-faulting ?
> >> >
> >> > It would be an unusual and unexpected thing to do - suddenly a pre-fault
> >> > operation is initialising a vCPU explicitly for pKVM.
> >> >
> >> > A caller is not going to reasonably expect this and might treat a failure to
> >> > pre-alloc as fine to carry on whereas in fact it was a failure to initailised a
> >> > pKVM vCPU.
> >> >
> >> > It'd also require significant changes to how pKVM is set up, right now it's
> >> > hardcoded to be done unconditionally at run via kvm_arch_vcpu_run_pid_change()
> >> > -> pkvm_create_hyp_vcpu(), so all that would have to change and be checked and
> >> > tested and... that'd be really out of scope I think :)
> >> >
> >> > And pre-faulting really makes most sense BEFORE you run a VM. It doesn't make so
> >> > much sense mid-run.
> >> >
> >> > But more fundamentally, the stage 2 page tables, as I understand it, are owned
> >> > by pKVM and so aren't really available to be pre-faulted.
> >> >
> >> > Maybe unprotected-under-pKVM VMs but then it's questionable as to how useful
> >> > that would be given that it would be confusing to users vs. how it works for
> >> > other VMs.
> >> >
> >> > So in general, no I don't think it's a good idea.
> >> >
> >> > And even if we wanted to pursue some version of this, it's _definitely_ out
> >> > of scope for the initial pre-faulting bring-up series.
> >> >
> >> >>
> >> >>
> >> >> > but pre-faulting is  typically performed before a vCPU is run.  It would be confusing and
> >> >> > inconsistent to error out on non-running vCPUs but to pre-fault running
> >> >> > ones.
> >> >>
> >> >>
> >> >> I use KVM pre-faulting when transitioning pages from shared to private
> >> >
> >> > You mean you'd prefer to use? Or you are using it on another arch?
> >> >
> >> >> with CoCo guest. This ensures that a trusted device can DMA to private
> >> >> memory before the guest accesses it.
> >> >
> >> > Hm what do you mean by private memory?
> >> >
> >> > I see:
> >> >
> >> > #ifndef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
> >> > static inline bool kvm_arch_has_private_mem(struct kvm *kvm)
> >> > {
> >> > 	return false;
> >> > }
> >> > #endif
> >> >
> >> > And only x86 selects KVM_GENERIC_MEMORY_ATTRIBUTES?
> >> >
> >> > Do you mean something else?
> >> >
> >>
> >> I am using this with ARM CCA-DA, based on the patch series from Jack Thomson <jackabt.amazon@gmail.com>.
> >>
> >> https://gitlab.arm.com/linux-arm/kvmtool-cca/-/commit/80e7aad61c5639de2f0cb4a5525dad0c96156428
> >>
> >> We do this while the VM is running.
> >
> > Right, that's a non-mainline kernel I guess? Which presumably implements private memory.
> >
> > Jack himself experienced a panic with his pKVM code, so the code you're using is
> > not upstreamable, unfortunately. And he'd already shelved pKVM support AFAICT.
> >
> > And reviewers pointed out actually implementing the pKVM stuff properly would be
> > quite involved, even if you wanted to do that (hence follow-up).
> >
> > Also you end up stuck with the same problems as I mentioned above - you can't
> > sanely bring the vCPU pre-run, so now you have extremely weird behaviour - only
> > pre-faults if vCPU initialised, running, and unprotected pKVM.
> >
>
> IIUC, pkvm_pgtable_stage2_map() only uses the hyp vCPU's pKVM memcache
> (&vcpu->vcpu.arch.pkvm_memcache). I agree that this does not need to be
> addressed in this series. However, there is also a desire to keep the CCA
> and pKVM code paths similar by using helpers such as kvm_vm_is_protected().
> Since an RMM can create a stage-2 mapping without a REC (vCPU), rejecting
> pre-faulting for all protected VM configurations may be incorrect. I can
> handle this in my series though.

Ack, understood. Feel free to cc- me on that when you send it out!

>
> I also want to point out that there are use cases for pre-faulting while
> the VM is running.

Ack yeah understood! And it does support that fine :)

I added further reasoning about pKVM in the cover/commit msg in the respin
([0]) so hopefully that helps clarify from that sid eof things.


>
> >
> > Protected pKVM support is a whole other layer of complexity and it's not obvious
> > that you're really achieving what pre-fault is supposed to.
> >
> > In any case Oliver literally just asked me to _simplify_ weird edge cases for
> > this series :) so I am not sure something like that is going to be accepted.
> >
> > Are you sure you're actually running in pKVM mode btw? CCA doesn't AFAICT? In
> > which case this series _should_ work fine for you.
> >
>
>
> It is not pKVM; it runs in Realm mode.

Ack yeah, then the series should hopefully work for you.

If you could test it and report any issues I'd be grateful! Thanks :)

>
> >
> > Anyway, if we really do need to add something for pKVM it needs to be a follow
> > up. Let's get the basics working first :)
> >
>
> sure.

Thanks!

>
> > (Note that kvmtool will need to be updated to retry pre-fault on -EAGAIN, -EINTR
> >  as this series can, albeit unlikely, return -EAGAIN.)
> >
>
> I will check this when I rebase my kernel onto this series.

OK cool, let me know if you hit any issues.

>
> -aneesh

--
Cheers, Lorenzo

[0]: https://lore.kernel.org/kvmarm/20260922-kvm-arm-prefault-v3-0-787bd3bc7e3f@kernel.org/T/#t