From nobody Sat Jul 25 19:28:41 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D1D62385D73 for ; Tue, 14 Jul 2026 12:00:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784030404; cv=none; b=Znzb/490bD1TB+WU3HkYEOa2myC/WtUW4zhwEjoHPNNQaDGp4AI8BHk7QA50Aft+KDe4BFQfUDyZ8JPMsE27/aAU2ZP3n8XcdX1nzofADaq8ea3dKUfHaMpkPaRfxYpCf61PvCiqSKjPGD1jy0lfzY9mLPNDku5S4gcpqbzfNqM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784030404; c=relaxed/simple; bh=RPfike1o9nuXZBOiQAjbQMhBFKE/+EBRVPOQI5nvn6I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uDB+DnOwZkLjz62mv2NUsf5lAHThtcSZ1B88/NOUvxyYwHrQI/UJH2Emmxv8aQzbZpD5TlYcITgdE4LzlNfAzwtHIThNGnHPWKmImIWdg6QUWUwO1mzaqCcc7tvzgS0rOKVohQVSv8A4U5WyjUsB7qyYP0VodflZP/CnLDcdnrI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=kzhJlb0n; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="kzhJlb0n" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EE7061476; Tue, 14 Jul 2026 04:59:57 -0700 (PDT) Received: from workstation-e142269.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id BD06E3F93E; Tue, 14 Jul 2026 04:59:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1784030402; bh=RPfike1o9nuXZBOiQAjbQMhBFKE/+EBRVPOQI5nvn6I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kzhJlb0nAI9smDy/4lMOJ0AcSAkvKnNWn2soWpbbvZtDl2ruR8Gb/+ENw6cqhjTIS 4bjzQe121JTL2/WPP0EtjvbCpOAkGafnLZVOMFpJ/LCgCDk1ONEC+x4fJh+4k0aM/Q whm6MV7IF6jry9SjgmYOM6Ofc1FLFo6eBxRBlmh0= From: Wei-Lin Chang To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org Cc: Marc Zyngier , Oliver Upton , Fuad Tabba , Joey Gouly , Steffen Eiden , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Itaru Kitayama , Sebastian Ene , Wei-Lin Chang Subject: [PATCH v4 1/6] KVM: arm64: Use a variable for the canonical GPA in kvm_s2_fault_map() Date: Tue, 14 Jul 2026 12:59:20 +0100 Message-ID: <20260714115926.2044757-2-weilin.chang@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260714115926.2044757-1-weilin.chang@arm.com> References: <20260714115926.2044757-1-weilin.chang@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Create a variable to store the canonical GPA, instead of calculating it when needed. This will be useful when we need to use the canonical GPA for the nested reverse map later. Signed-off-by: Wei-Lin Chang --- arch/arm64/kvm/mmu.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 8811ad60cf72..54395a608fd3 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -2021,6 +2021,7 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault= _desc *s2fd, long mapping_size; kvm_pfn_t pfn; gfn_t gfn; + phys_addr_t canonical_gpa; int ret; =20 kvm_fault_lock(kvm); @@ -2034,6 +2035,7 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault= _desc *s2fd, mapping_size =3D s2vi->vma_pagesize; pfn =3D s2vi->pfn; gfn =3D s2vi->gfn; + canonical_gpa =3D gfn_to_gpa(get_canonical_gfn(s2fd, s2vi)); =20 /* * If we are not forced to use page mapping, check if we are @@ -2052,6 +2054,7 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault= _desc *s2fd, goto out_unlock; } } + canonical_gpa =3D ALIGN_DOWN(canonical_gpa, mapping_size); } =20 if (!perm_fault_granule && !s2vi->map_non_cacheable && kvm_has_mte(kvm)) @@ -2085,11 +2088,9 @@ static int kvm_s2_fault_map(const struct kvm_s2_faul= t_desc *s2fd, * making sure we adjust the canonical IPA if the mapping size has * been updated (via a THP upgrade, for example). */ - if (writable && !ret) { - phys_addr_t ipa =3D gfn_to_gpa(get_canonical_gfn(s2fd, s2vi)); - ipa &=3D ~(mapping_size - 1); - mark_page_dirty_in_slot(kvm, s2fd->memslot, gpa_to_gfn(ipa)); - } + if (writable && !ret) + mark_page_dirty_in_slot(kvm, s2fd->memslot, + gpa_to_gfn(canonical_gpa)); =20 if (ret !=3D -EAGAIN) return ret; --=20 2.43.0 From nobody Sat Jul 25 19:28:41 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C660938C2AE for ; Tue, 14 Jul 2026 12:00:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784030407; cv=none; b=GrAtfq7/ScvIh/c2UmsJPsIPjm/xBuJIhfMTpKYWR8OG5PhyqkzSbgLW8AIixNGii3VPRat6KDBZSOXKRSxRV5nCwHWzSU9rym28jaJb+BP7pPUkQyN70I9EdERZ+4AuUE9cd7wf28KlOcsfphiiojUDUaDCEaDZM51GTaISepU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784030407; c=relaxed/simple; bh=GPIPzN0jFwepejcD52PL/IcNSdWmNA3pYoGfSzQoRms=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dJlPT3YLc/i6cWtbfNvGzzVwlmh+8Gwy3V5/SbMTsh7lU+LOP/Tpl0aAXOB9T9K4OMRoeKVDkgzihu4NG7nz/wsvo1bj+ke3HTOgf0NMtBvBKdPtI0pH34yqzblKvx4IgV5KoOl17gcrnLS7XpjYoLF6m6PJ+fCOD5I5AFI1QXw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=Ak0Zk7fM; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="Ak0Zk7fM" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EEBBC497; Tue, 14 Jul 2026 05:00:00 -0700 (PDT) Received: from workstation-e142269.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9934C3F93E; Tue, 14 Jul 2026 05:00:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1784030405; bh=GPIPzN0jFwepejcD52PL/IcNSdWmNA3pYoGfSzQoRms=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ak0Zk7fM1suQ5CCrf7e2o3trFUtCJSqAv1yhepnVYZdve4RxnfKMDhW8fGfs1iWl5 D70VIKVDGh+oKTwAcdgQ1W0E5JS3VpaTWZIIIqaSwzYkvgyCmfMdirnkAw6A9dCAJ2 9fLbkwbyyt1HLDogCelkTrg1aHJfxcaEa7aHizhU= From: Wei-Lin Chang To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org Cc: Marc Zyngier , Oliver Upton , Fuad Tabba , Joey Gouly , Steffen Eiden , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Itaru Kitayama , Sebastian Ene , Wei-Lin Chang Subject: [PATCH v4 2/6] KVM: arm64: nv: Avoid full shadow s2 unmap Date: Tue, 14 Jul 2026 12:59:21 +0100 Message-ID: <20260714115926.2044757-3-weilin.chang@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260714115926.2044757-1-weilin.chang@arm.com> References: <20260714115926.2044757-1-weilin.chang@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently we are forced to fully unmap all shadow stage-2 for a VM when unmapping a page from the canonical stage-2, for example during an MMU notifier call. This is because we are not tracking which canonical IPAs are mapped in the shadow stage-2 page tables hence there is no way to know what to unmap. Create a per kvm_s2_mmu maple tree to track canonical IPA range -> nested IPA range, so that it is possible to partially unmap shadow stage-2 when a canonical IPA range is unmapped. The algorithm is simple and conservative: At each shadow stage-2 map, insert the nested IPA range into the maple tree, with the canonical IPA range as the key. If the canonical IPA range doesn't overlap with existing ranges in the tree, insert as is, and a reverse mapping for this range is established. But if the canonical IPA range overlaps with any existing ranges in the tree, create a new range that spans all the overlapping ranges including the input range and replace those existing ranges. In the meantime, mark this new spanning canonical IPA range with an "UNKNOWN_IPA" bit, indicating we give up tracking the nested IPA ranges that map to this canonical IPA range. The maple tree's 64-bit entry is enough to store the nested IPA and the UNKNOWN_IPA status, therefore besides maple tree's internal operation, memory allocation is avoided. Example: |||| means existing range, ---- means empty range input: $$$$$$$$$$$$$$$$$$$$$$$$$$ tree: --||||-----|||||||---------||||||||||----------- insert spanning range and replace overlapping ones: --||||-----||||||||||||||||||||||||||----------- ^^^^marked UNKNOWN_IPA^^^^ With the reverse map created, when a canonical IPA range gets unmapped, look into each s2 mmu's maple tree and look for canonical IPA ranges affected, and based on their UNKNOWN_IPA status: UNKNOWN_IPA -> fall back and fully unmap the current shadow stage-2, also clear the tree not UNKNOWN_IPA -> unmap the nested IPA range, and remove the reverse map entry Suggested-by: Marc Zyngier Signed-off-by: Wei-Lin Chang --- arch/arm64/include/asm/kvm_host.h | 4 + arch/arm64/include/asm/kvm_nested.h | 4 + arch/arm64/kvm/mmu.c | 21 +++- arch/arm64/kvm/nested.c | 167 +++++++++++++++++++++++++++- 4 files changed, 190 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index 2faa60df847d..42ab121a04f8 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -227,6 +227,10 @@ struct kvm_s2_mmu { */ bool pending_unmap; =20 + bool nested_revmap_broken; + /* canonical IPA to nested IPA range lookup */ + struct maple_tree nested_revmap_mt; + /* * 0: Nobody is currently using this, check vttbr for validity * >0: Somebody is actively using this. diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/k= vm_nested.h index cbdaaa2a2903..79de224f3438 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -77,6 +77,8 @@ extern void kvm_s2_mmu_iterate_by_vmid(struct kvm *kvm, u= 16 vmid, const union tlbi_info *info, void (*)(struct kvm_s2_mmu *, const union tlbi_info *)); +extern void kvm_record_nested_revmap(gpa_t canonical_ipa, struct kvm_s2_mm= u *mmu, + gpa_t nested_ipa, size_t map_size); extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu); extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu); =20 @@ -165,6 +167,8 @@ extern int kvm_s2_handle_perm_fault(struct kvm_vcpu *vc= pu, struct kvm_s2_trans *trans); extern int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2); extern void kvm_nested_s2_wp(struct kvm *kvm); +extern void kvm_unmap_cipa_range_nested(struct kvm *kvm, gpa_t canonical_i= pa, size_t size, + bool may_block); extern void kvm_nested_s2_unmap(struct kvm *kvm, bool may_block); extern void kvm_nested_s2_flush(struct kvm *kvm); =20 diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 54395a608fd3..73629aa1ade6 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -5,6 +5,7 @@ */ =20 #include +#include #include #include #include @@ -1126,6 +1127,7 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu) { struct kvm *kvm =3D kvm_s2_mmu_to_kvm(mmu); struct kvm_pgtable *pgt =3D NULL; + struct maple_tree *revmap_mt =3D &mmu->nested_revmap_mt; =20 write_lock(&kvm->mmu_lock); pgt =3D mmu->pgt; @@ -1135,8 +1137,11 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu) free_percpu(mmu->last_vcpu_ran); } =20 - if (kvm_is_nested_s2_mmu(kvm, mmu)) + if (kvm_is_nested_s2_mmu(kvm, mmu)) { + if (!mtree_empty(revmap_mt)) + mtree_destroy(revmap_mt); kvm_init_nested_s2_mmu(mmu); + } =20 write_unlock(&kvm->mmu_lock); =20 @@ -1670,6 +1675,9 @@ static int gmem_abort(const struct kvm_s2_fault_desc = *s2fd) ret =3D KVM_PGT_FN(kvm_pgtable_stage2_relax_perms)(pgt, s2fd->fault_ipa, prot, flags); } else { + if (s2fd->nested) + kvm_record_nested_revmap(gfn << PAGE_SHIFT, pgt->mmu, + s2fd->fault_ipa, PAGE_SIZE); ret =3D KVM_PGT_FN(kvm_pgtable_stage2_map)(pgt, s2fd->fault_ipa, PAGE_SI= ZE, __pfn_to_phys(pfn), prot, memcache, flags); @@ -2074,6 +2082,10 @@ static int kvm_s2_fault_map(const struct kvm_s2_faul= t_desc *s2fd, ret =3D KVM_PGT_FN(kvm_pgtable_stage2_relax_perms)(pgt, gfn_to_gpa(gfn), prot, flags); } else { + if (s2fd->nested) + kvm_record_nested_revmap(canonical_gpa, pgt->mmu, + gfn_to_gpa(gfn), mapping_size); + ret =3D KVM_PGT_FN(kvm_pgtable_stage2_map)(pgt, gfn_to_gpa(gfn), mapping= _size, __pfn_to_phys(pfn), prot, memcache, flags); @@ -2435,8 +2447,9 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_= gfn_range *range) __unmap_stage2_range(&kvm->arch.mmu, range->start << PAGE_SHIFT, (range->end - range->start) << PAGE_SHIFT, range->may_block); - - kvm_nested_s2_unmap(kvm, range->may_block); + kvm_unmap_cipa_range_nested(kvm, range->start << PAGE_SHIFT, + (range->end - range->start) << PAGE_SHIFT, + range->may_block); return false; } =20 @@ -2714,7 +2727,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, =20 write_lock(&kvm->mmu_lock); kvm_stage2_unmap_range(&kvm->arch.mmu, gpa, size, true); - kvm_nested_s2_unmap(kvm, true); + kvm_unmap_cipa_range_nested(kvm, gpa, size, true); write_unlock(&kvm->mmu_lock); } =20 diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 94df26de6990..065823e415ce 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -7,6 +7,7 @@ #include #include #include +#include =20 #include #include @@ -44,6 +45,32 @@ struct vncr_tlb { */ #define S2_MMU_PER_VCPU 2 =20 +/* + * Per shadow S2 reverse map (canonical IPA -> nested IPA range) maple tree + * payload layout: + * + * bit 62: valid, prevents the case where the nested IPA is 0 and tur= ning + * the whole value to 0 (NULL for maple tree) + * bits 55-12: nested IPA bits 55-12 + * bit 0: UNKNOWN_IPA bit, 1 indicates we give up on tracking what n= ested + * IPA maps to this canonical IPA in the shadow stage-2 + */ +#define VALID_ENTRY BIT(62) +#define ADDR_MASK GENMASK_ULL(55, 12) +#define UNKNOWN_IPA BIT(0) + +static bool valid_entry(unsigned long entry) +{ + WARN_ON(entry & VALID_ENTRY && entry & UNKNOWN_IPA); + return entry & VALID_ENTRY; +} + +static bool unknown_ipa_entry(unsigned long entry) +{ + WARN_ON(entry & VALID_ENTRY && entry & UNKNOWN_IPA); + return entry & UNKNOWN_IPA; +} + void kvm_init_nested(struct kvm *kvm) { kvm->arch.nested_mmus =3D NULL; @@ -844,12 +871,65 @@ static struct kvm_s2_mmu *get_s2_mmu_nested(struct kv= m_vcpu *vcpu) return s2_mmu; } =20 +void kvm_record_nested_revmap(gpa_t canonical_ipa, struct kvm_s2_mmu *mmu, + gpa_t nested_ipa, size_t map_size) +{ + struct maple_tree *revmap_mt =3D &mmu->nested_revmap_mt; + gpa_t canonical_ipa_end; + u64 entry, new_entry =3D 0; + + lockdep_assert_held_read(&kvm_s2_mmu_to_kvm(mmu)->mmu_lock); + + if (mmu->nested_revmap_broken) + return; + + if (WARN_ON(!IS_ALIGNED(canonical_ipa, map_size))) + canonical_ipa =3D ALIGN_DOWN(canonical_ipa, map_size); + + canonical_ipa_end =3D canonical_ipa + map_size - 1; + MA_STATE(mas_rmap, revmap_mt, canonical_ipa, canonical_ipa_end); + + mtree_lock(revmap_mt); + entry =3D xa_to_value(mas_find(&mas_rmap, canonical_ipa_end)); + + if (entry) { + /* parallel faults can be adding the same mapping */ + if (valid_entry(entry) && + mas_rmap.index =3D=3D canonical_ipa && + mas_rmap.last =3D=3D canonical_ipa_end && + nested_ipa =3D=3D (entry & ADDR_MASK)) + goto unlock; + /* + * Create a "UNKNOWN_IPA" range that spans all the overlapping + * ranges and store it. + */ + while (entry && mas_rmap.index <=3D canonical_ipa_end) { + canonical_ipa =3D min(mas_rmap.index, canonical_ipa); + canonical_ipa_end =3D max(mas_rmap.last, canonical_ipa_end); + entry =3D xa_to_value(mas_find(&mas_rmap, canonical_ipa_end)); + } + new_entry |=3D UNKNOWN_IPA; + } else { + new_entry |=3D nested_ipa; + new_entry |=3D VALID_ENTRY; + } + + mas_set_range(&mas_rmap, canonical_ipa, canonical_ipa_end); + if (mas_store_gfp(&mas_rmap, xa_mk_value(new_entry), + GFP_NOWAIT | __GFP_ACCOUNT)) + mmu->nested_revmap_broken =3D true; +unlock: + mtree_unlock(revmap_mt); +} + void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu) { /* CnP being set denotes an invalid entry */ mmu->tlb_vttbr =3D VTTBR_CNP_BIT; mmu->nested_stage2_enabled =3D false; atomic_set(&mmu->refcnt, 0); + mt_init(&mmu->nested_revmap_mt); + mmu->nested_revmap_broken =3D false; } =20 void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu) @@ -1225,6 +1305,89 @@ void kvm_nested_s2_wp(struct kvm *kvm) kvm_invalidate_vncr_ipa(kvm, 0, BIT(kvm->arch.mmu.pgt->ia_bits)); } =20 +static void reset_revmap_and_unmap(struct kvm_s2_mmu *mmu, bool may_block) +{ + mtree_destroy(&mmu->nested_revmap_mt); + mmu->nested_revmap_broken =3D false; + kvm_stage2_unmap_range(mmu, 0, kvm_phys_size(mmu), may_block); +} + +static void unmap_mmu_cipa_range(struct kvm_s2_mmu *mmu, gpa_t canonical_i= pa, + size_t unmap_size, bool may_block) +{ + struct maple_tree *revmap_mt =3D &mmu->nested_revmap_mt; + gpa_t canonical_ipa_end =3D canonical_ipa + unmap_size - 1; + size_t entry_size; + gpa_t next_addr; + u64 entry; + MA_STATE(mas_rmap, revmap_mt, canonical_ipa, canonical_ipa_end); + + lockdep_assert_held_write(&kvm_s2_mmu_to_kvm(mmu)->mmu_lock); + + if (mmu->nested_revmap_broken) { + reset_revmap_and_unmap(mmu, may_block); + return; + } + + if (!mmu->nested_stage2_enabled) { + kvm_stage2_unmap_range(mmu, canonical_ipa, unmap_size, may_block); + return; + } + + mtree_lock(revmap_mt); + entry =3D xa_to_value(mas_find(&mas_rmap, canonical_ipa_end)); + + while (entry && mas_rmap.index <=3D canonical_ipa_end) { + entry_size =3D mas_rmap.last - mas_rmap.index + 1; + next_addr =3D mas_rmap.index + entry_size; + /* + * Give up and invalidate this s2 mmu if the unmap range + * touches any UNKNOWN_IPA range. + */ + if (unknown_ipa_entry(entry)) { + mtree_unlock(revmap_mt); + reset_revmap_and_unmap(mmu, may_block); + return; + } + + /* + * Ignore result, it is okay if a reverse mapping erase + * fails. + */ + mas_store_gfp(&mas_rmap, NULL, GFP_NOWAIT | __GFP_ACCOUNT); + + mtree_unlock(revmap_mt); + kvm_stage2_unmap_range(mmu, entry & ADDR_MASK, entry_size, + may_block); + mtree_lock(revmap_mt); + /* + * Other maple tree operations during preemption could render + * this ma_state invalid, so reset it. + */ + mas_set_range(&mas_rmap, next_addr, canonical_ipa_end); + entry =3D xa_to_value(mas_find(&mas_rmap, canonical_ipa_end)); + } + mtree_unlock(revmap_mt); +} + +void kvm_unmap_cipa_range_nested(struct kvm *kvm, gpa_t canonical_ipa, siz= e_t size, + bool may_block) +{ + int i; + + if (!kvm->arch.nested_mmus_size) + return; + + for (i =3D 0; i < kvm->arch.nested_mmus_size; i++) { + struct kvm_s2_mmu *mmu =3D &kvm->arch.nested_mmus[i]; + + if (kvm_s2_mmu_valid(mmu)) + unmap_mmu_cipa_range(mmu, canonical_ipa, size, may_block); + } + + kvm_invalidate_vncr_ipa(kvm, canonical_ipa, canonical_ipa + size); +} + void kvm_nested_s2_unmap(struct kvm *kvm, bool may_block) { int i; @@ -1238,7 +1401,7 @@ void kvm_nested_s2_unmap(struct kvm *kvm, bool may_bl= ock) struct kvm_s2_mmu *mmu =3D &kvm->arch.nested_mmus[i]; =20 if (kvm_s2_mmu_valid(mmu)) - kvm_stage2_unmap_range(mmu, 0, kvm_phys_size(mmu), may_block); + reset_revmap_and_unmap(mmu, may_block); } =20 kvm_invalidate_vncr_ipa(kvm, 0, BIT(kvm->arch.mmu.pgt->ia_bits)); @@ -1924,7 +2087,7 @@ void check_nested_vcpu_requests(struct kvm_vcpu *vcpu) =20 write_lock(&vcpu->kvm->mmu_lock); if (mmu->pending_unmap) { - kvm_stage2_unmap_range(mmu, 0, kvm_phys_size(mmu), true); + reset_revmap_and_unmap(mmu, true); mmu->pending_unmap =3D false; } write_unlock(&vcpu->kvm->mmu_lock); --=20 2.43.0 From nobody Sat Jul 25 19:28:41 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B554A38D403 for ; Tue, 14 Jul 2026 12:00:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784030410; cv=none; b=I30GhshQcSRRqasQ0YJ0qRf5j/HyExjdb2ZHORQ88Po9yGtHh5iEL1NsYsp/LjXRMXRT8hmOLEilVKF9XzDrdX8z7wjQW3QyZOgK/u8Xq57l8Rz78WBVqjEkiJCmuctVgNjRWzRU48++0sPiAsy2jyo/4wBhcHkSQdgRy8xLx2Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784030410; c=relaxed/simple; bh=0sHCsiFR68L+YiQqudbh7dXb+D2OosNbQD6DJoCp7Fk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qeCsNgCY1iuG2oogqee8DRt48TNechcCXy+hrru6gog0ZklutkThDqW7vw/CRaEg3XOjwHoF1kr2wwiyW81ltbz/g3hjxOq1lYS+H+1Z7D1cDutJFCG50iV82ZO1D1hM0fNZcaxHa/IWyzFuzpj3IjpVJe59AIBFHcrfe5KuKXw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=YjU4N3bd; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="YjU4N3bd" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CCEDF1477; Tue, 14 Jul 2026 05:00:03 -0700 (PDT) Received: from workstation-e142269.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 99ACC3F93E; Tue, 14 Jul 2026 05:00:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1784030408; bh=0sHCsiFR68L+YiQqudbh7dXb+D2OosNbQD6DJoCp7Fk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YjU4N3bdOSPXXEn2diDSlkjIBmOqSHXjvjYKDQy7IeV2AUTdV2OGKfEkPD6GjhDR9 /fzMnKugMlyxjJaUaPQn2Qhgz28K/JqqRc9O4z6+niQRI05C8pkW5nSN+QyswWwrHI IrXqcKGSeh38bjN9L5YgE5OiTQoVx3tK7uopgpVI= From: Wei-Lin Chang To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org Cc: Marc Zyngier , Oliver Upton , Fuad Tabba , Joey Gouly , Steffen Eiden , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Itaru Kitayama , Sebastian Ene , Wei-Lin Chang Subject: [PATCH v4 3/6] KVM: arm64: nv: Add nested revmap broken tracepoint Date: Tue, 14 Jul 2026 12:59:22 +0100 Message-ID: <20260714115926.2044757-4-weilin.chang@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260714115926.2044757-1-weilin.chang@arm.com> References: <20260714115926.2044757-1-weilin.chang@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a tracepoint to track when a nested reverse map becomes broken. This makes it easier to detect NV performance degradation caused by full shadow stage-2 unmaps. Signed-off-by: Wei-Lin Chang --- arch/arm64/kvm/nested.c | 7 ++++++- arch/arm64/kvm/trace_arm.h | 27 +++++++++++++++++++++++++++ 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 065823e415ce..22f8a1daca93 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -17,6 +17,7 @@ #include =20 #include "sys_regs.h" +#include "trace.h" =20 struct vncr_tlb { /* The guest's VNCR_EL2 */ @@ -916,8 +917,12 @@ void kvm_record_nested_revmap(gpa_t canonical_ipa, str= uct kvm_s2_mmu *mmu, =20 mas_set_range(&mas_rmap, canonical_ipa, canonical_ipa_end); if (mas_store_gfp(&mas_rmap, xa_mk_value(new_entry), - GFP_NOWAIT | __GFP_ACCOUNT)) + GFP_NOWAIT | __GFP_ACCOUNT)) { + trace_kvm_nested_revmap_broken(mmu - mmu->arch->nested_mmus, + canonical_ipa, canonical_ipa_end, + nested_ipa, new_entry); mmu->nested_revmap_broken =3D true; + } unlock: mtree_unlock(revmap_mt); } diff --git a/arch/arm64/kvm/trace_arm.h b/arch/arm64/kvm/trace_arm.h index 9c60f6465c78..0db90068ff63 100644 --- a/arch/arm64/kvm/trace_arm.h +++ b/arch/arm64/kvm/trace_arm.h @@ -415,6 +415,33 @@ TRACE_EVENT(kvm_forward_sysreg_trap, sys_reg_Op2(__entry->sysreg)) ); =20 +TRACE_EVENT(kvm_nested_revmap_broken, + TP_PROTO(int mmu_idx, unsigned long canonical_ipa, + unsigned long canonical_ipa_end, unsigned long nested_ipa, + unsigned long revmap_entry), + TP_ARGS(mmu_idx, canonical_ipa, canonical_ipa_end, nested_ipa, revmap= _entry), + + TP_STRUCT__entry( + __field(int, mmu_idx) + __field(unsigned long, canonical_ipa) + __field(unsigned long, nested_ipa) + __field(size_t, size) + __field(unsigned long, revmap_entry) + ), + + TP_fast_assign( + __entry->mmu_idx =3D mmu_idx; + __entry->canonical_ipa =3D canonical_ipa; + __entry->nested_ipa =3D nested_ipa; + __entry->size =3D canonical_ipa_end - canonical_ipa + 1; + __entry->revmap_entry =3D revmap_entry; + ), + + TP_printk("mmu idx: %d, cipa: 0x%016lx, nipa: 0x%016lx, size: 0x%016l= x, entry: %016lx", + __entry->mmu_idx, __entry->canonical_ipa, __entry->nested_ipa, + __entry->size, __entry->revmap_entry) +); + #endif /* _TRACE_ARM_ARM64_KVM_H */ =20 #undef TRACE_INCLUDE_PATH --=20 2.43.0 From nobody Sat Jul 25 19:28:41 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 747D738E8A5 for ; Tue, 14 Jul 2026 12:00:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784030412; cv=none; b=faDVm+d+4hrbcC7HuchJfDwZI9Ohk9ZRhgq1FZ8x3tbi55BWD3+q+PSIIdCklQFXSC61fmF9IANV+lBeSrRtgwtJyG9Fjpw2sdZAx3Tkd8CFgJWoJ/dzBX4h8xjxr+XJMbfik8EGQQDGjSS4I7sluRzeh0xeD6oi9Fprh243iNo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784030412; c=relaxed/simple; bh=dEOA4qysSRG7n58hsk4J7X5+FYVGdQ3ln2cyFcm1ZyE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Mk0qTh/n9Fx1+kyIoQALyw3Vv5LUrlhc2nv6cL/yhz0xZwlqXxFD6/pDj0G9/CXkCGnxJjj8uHLgmvK4HpQ69pD66JwJhBHSUxpfqYLnhT1a29Ruo6P5HVdbaMSpLPDb/vhVuWM8E3aiHGJVzY9TL5gfb/HuDlURjBNrVwZQ4Wg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=srF8MC90; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="srF8MC90" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id ABABB497; Tue, 14 Jul 2026 05:00:06 -0700 (PDT) Received: from workstation-e142269.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 74CC93FAF5; Tue, 14 Jul 2026 05:00:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1784030410; bh=dEOA4qysSRG7n58hsk4J7X5+FYVGdQ3ln2cyFcm1ZyE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=srF8MC90nVSBrVEL7QmF7JLmmo8Sv8R3E3KsdFRWfZU1J878GxNO9qhYIiAQb5Q8N FXIx+lnr/rY/WmSScZds/P1s0xqA8awh9vzBebK8k+08Z0xUjctiUIoH4mpazdURc1 /O4vbxzfPregZE9Qw4ba+7MZZaQm3ROtT/68SYbs= From: Wei-Lin Chang To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org Cc: Marc Zyngier , Oliver Upton , Fuad Tabba , Joey Gouly , Steffen Eiden , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Itaru Kitayama , Sebastian Ene , Wei-Lin Chang Subject: [PATCH v4 4/6] KVM: arm64: Refactor kvm_unmap_gfn_range() with common variables Date: Tue, 14 Jul 2026 12:59:23 +0100 Message-ID: <20260714115926.2044757-5-weilin.chang@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260714115926.2044757-1-weilin.chang@arm.com> References: <20260714115926.2044757-1-weilin.chang@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" __unmap_stage2_range() and kvm_unmap_cipa_range_nested() are using the same arguments, clean this up by using local variables. Signed-off-by: Wei-Lin Chang --- arch/arm64/kvm/mmu.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 73629aa1ade6..6bfc1fbc5387 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -2441,15 +2441,16 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) =20 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) { + gpa_t gpa =3D range->start << PAGE_SHIFT; + size_t size =3D (range->end - range->start) << PAGE_SHIFT; + bool may_block =3D range->may_block; + if (!kvm->arch.mmu.pgt || kvm_vm_is_protected(kvm)) return false; =20 - __unmap_stage2_range(&kvm->arch.mmu, range->start << PAGE_SHIFT, - (range->end - range->start) << PAGE_SHIFT, - range->may_block); - kvm_unmap_cipa_range_nested(kvm, range->start << PAGE_SHIFT, - (range->end - range->start) << PAGE_SHIFT, - range->may_block); + __unmap_stage2_range(&kvm->arch.mmu, gpa, size, may_block); + kvm_unmap_cipa_range_nested(kvm, gpa, size, may_block); + return false; } =20 --=20 2.43.0 From nobody Sat Jul 25 19:28:41 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2428D39060B for ; Tue, 14 Jul 2026 12:00:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784030415; cv=none; b=Zu3b77wj+oV5pW4srS227YwD8tJoBbJd3Z0sQLvLPwaq799Q/ja2+5lNttT+hJVRrDruRzTuK4W6lc7823Y/J5W2ArT1qaQjB4vcjYmFkdg1SkWcDxpXLkxPfbgu2Fp3ZnWLfu0DDpck8nv9MLD4C93+RBE4j2ictsdB/gDtfgw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784030415; c=relaxed/simple; bh=T73ydasjvqIW4dG81U6rIei8nMSD0vZur1SW676PBRs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Fs7lzzMyIw9z192Lu40FCPfx63Ozo8MFAuIfrmlllacngR+1j8wqOVXSyFQXS3zMjZp3xy/Igqqvx5LMee+2Qb9gTTzemn2ky3SalJ74Z2wUq/XaO7HLpPWs+NGbCc/rlf8+4iaaUjKVA36zWGk7/eQ0J2CM4oa2w+EBUOIhlwk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=l3fAi3fu; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="l3fAi3fu" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8329E1476; Tue, 14 Jul 2026 05:00:09 -0700 (PDT) Received: from workstation-e142269.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 50AB93F93E; Tue, 14 Jul 2026 05:00:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1784030413; bh=T73ydasjvqIW4dG81U6rIei8nMSD0vZur1SW676PBRs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=l3fAi3fud6DlOGS2BXfXbzkGb3wNJukwl+N9BrPZurZnpyrPc7Yjhga4JFiBT+ryS rWLbv2cFQxk2EHLFNezB4P+FgusgOjkIA61kzTWS0864J5nQpYyTGyWM2Hy7LxzTSO Elr1NAng6U6Zt3CaX3BDQKX/AVxlm23MqqQFm7uo= From: Wei-Lin Chang To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org Cc: Marc Zyngier , Oliver Upton , Fuad Tabba , Joey Gouly , Steffen Eiden , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Itaru Kitayama , Sebastian Ene , Wei-Lin Chang Subject: [PATCH v4 5/6] KVM: arm64: nv: Remove reverse map entries during TLBI handling Date: Tue, 14 Jul 2026 12:59:24 +0100 Message-ID: <20260714115926.2044757-6-weilin.chang@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260714115926.2044757-1-weilin.chang@arm.com> References: <20260714115926.2044757-1-weilin.chang@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When a guest hypervisor issues a TLBI for a specific IPA range, KVM unmaps that range from all the affected shadow stage-2s. During this we get the opportunity to remove the reverse map, and lower the probability of creating UNKNOWN_IPA reverse map ranges at subsequent stage-2 faults. However, the TLBI ranges are specified in nested IPA, so in order to locate the affected ranges in the reverse map maple tree, which is a mapping from canonical IPA to nested IPA, we can only iterate through the entire tree and check each entry. Suggested-by: Marc Zyngier Signed-off-by: Wei-Lin Chang --- arch/arm64/include/asm/kvm_nested.h | 2 ++ arch/arm64/kvm/nested.c | 37 +++++++++++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 3 +++ 3 files changed, 42 insertions(+) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/k= vm_nested.h index 79de224f3438..664d789d17b1 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -77,6 +77,8 @@ extern void kvm_s2_mmu_iterate_by_vmid(struct kvm *kvm, u= 16 vmid, const union tlbi_info *info, void (*)(struct kvm_s2_mmu *, const union tlbi_info *)); +extern void kvm_remove_nested_revmap(struct kvm_s2_mmu *mmu, u64 nested_ip= a, + size_t size); extern void kvm_record_nested_revmap(gpa_t canonical_ipa, struct kvm_s2_mm= u *mmu, gpa_t nested_ipa, size_t map_size); extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 22f8a1daca93..cd93793fe89d 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -872,6 +872,43 @@ static struct kvm_s2_mmu *get_s2_mmu_nested(struct kvm= _vcpu *vcpu) return s2_mmu; } =20 +void kvm_remove_nested_revmap(struct kvm_s2_mmu *mmu, u64 nested_ipa, size= _t size) +{ + /* + * Iterate through the mt of this mmu, remove all canonical ipa ranges + * with !UNKNOWN_IPA that maps to ranges that are strictly within + * [addr, addr + size). + */ + struct maple_tree *revmap_mt =3D &mmu->nested_revmap_mt; + u64 entry_val, nested_ipa_end =3D nested_ipa + size; + u64 this_nested_ipa, this_nested_ipa_end; + size_t revmap_size; + void *entry; + + MA_STATE(mas_rmap, revmap_mt, 0, ULONG_MAX); + + mtree_lock(revmap_mt); + mas_for_each(&mas_rmap, entry, ULONG_MAX) { + entry_val =3D xa_to_value(entry); + if (unknown_ipa_entry(entry_val)) + continue; + + revmap_size =3D mas_rmap.last - mas_rmap.index + 1; + this_nested_ipa =3D entry_val & ADDR_MASK; + this_nested_ipa_end =3D this_nested_ipa + revmap_size; + + if (this_nested_ipa >=3D nested_ipa && + this_nested_ipa_end <=3D nested_ipa_end) { + /* + * Ignore result, failure to remove reverse mapping will + * only cause extra unmaps. + */ + mas_store_gfp(&mas_rmap, NULL, GFP_NOWAIT | __GFP_ACCOUNT); + } + } + mtree_unlock(revmap_mt); +} + void kvm_record_nested_revmap(gpa_t canonical_ipa, struct kvm_s2_mmu *mmu, gpa_t nested_ipa, size_t map_size) { diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 33c921df19b5..e4ce70508af6 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -4006,6 +4006,7 @@ union tlbi_info { static void s2_mmu_unmap_range(struct kvm_s2_mmu *mmu, const union tlbi_info *info) { + kvm_remove_nested_revmap(mmu, info->range.start, info->range.size); /* * The unmap operation is allowed to drop the MMU lock and block, which * means that @mmu could be used for a different context than the one @@ -4104,6 +4105,8 @@ static void s2_mmu_unmap_ipa(struct kvm_s2_mmu *mmu, max_size =3D compute_tlb_inval_range(mmu, info->ipa.addr); base_addr &=3D ~(max_size - 1); =20 + kvm_remove_nested_revmap(mmu, base_addr, max_size); + /* * See comment in s2_mmu_unmap_range() for why this is allowed to * reschedule. --=20 2.43.0 From nobody Sat Jul 25 19:28:41 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 59537390CB8 for ; Tue, 14 Jul 2026 12:00:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784030419; cv=none; b=a6828PE/kCdon+GaWJ+T0gnqD2VJhEYrq6+bx9fqZIDHVuXMmOi/Mw1ArayV6mXkllc+u+Z2KQ+rhpA1ceg8Rwf+ddwBr6B6d7b80fkEJbbZL9bAUzeki5rqFNnf5cRvaZcOq3Jt2ezFJ2ygLHO445mAR0o78wsY+xozGiYi9L4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784030419; c=relaxed/simple; bh=IQR1Dv1T87h3rluP4EOjqIIc0OVDqRoRbfBoLL9DAnQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tvAzVV+5s0lHJl9sN+6kMzfgayWqDFKlHNOccU76mIjpMCuLqP8xmhGpXgmFiN+8Xx9ztIji/js/wBI8YIZtzSskZgQmi0jpYOGXDfOzOK/3xVJyKEAGddDM2+qUm+gDXKXBSsyZC9UJQRRwohfXpNLXV7mmMA2GREylAM0dJQM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=GfqLBJum; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="GfqLBJum" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 61BAC497; Tue, 14 Jul 2026 05:00:12 -0700 (PDT) Received: from workstation-e142269.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2F2503F93E; Tue, 14 Jul 2026 05:00:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1784030416; bh=IQR1Dv1T87h3rluP4EOjqIIc0OVDqRoRbfBoLL9DAnQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GfqLBJumVoE8E1s01F+ZhXkKTS4V1PXEk5c8kEgM0BWil1b2Y6ulWkZt+avz3Aoer n69TDCqmjtZz6IEU/kTWlCQhjdJV/2XGaeTGul1aiL3a7SMg/wRpwhWSOQ+AOaidrC bTca6vNTj68SKF+ShWE0X2q3VFRA386mtdwoaHRE= From: Wei-Lin Chang To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org Cc: Marc Zyngier , Oliver Upton , Fuad Tabba , Joey Gouly , Steffen Eiden , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Itaru Kitayama , Sebastian Ene , Wei-Lin Chang Subject: [PATCH v4 6/6] KVM: arm64: nv: Create nested IPA direct map to speed up reverse map removal Date: Tue, 14 Jul 2026 12:59:25 +0100 Message-ID: <20260714115926.2044757-7-weilin.chang@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260714115926.2044757-1-weilin.chang@arm.com> References: <20260714115926.2044757-1-weilin.chang@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Iterating through the whole reverse map to find which entries to remove when handling guest hypervisor TLBIs is not efficient. Create a direct map that goes from nested IPA to canonical IPA so that the canonical IPA range affected by the TLBI can be quickly determined, then remove the entries in the reverse map accordingly. Since insertion and deletion to the maple tree aren't guaranteed to succeed, the direct map is implemented best-effort, and we always check the direct mapping and the reverse mapping are symmetric (they store the same mapping in reverse) before removing the reverse mapping. Suggested-by: Marc Zyngier Signed-off-by: Wei-Lin Chang --- arch/arm64/include/asm/kvm_host.h | 5 ++ arch/arm64/kvm/mmu.c | 9 +- arch/arm64/kvm/nested.c | 131 ++++++++++++++++++++++-------- 3 files changed, 108 insertions(+), 37 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index 42ab121a04f8..9b385897c352 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -230,6 +230,11 @@ struct kvm_s2_mmu { bool nested_revmap_broken; /* canonical IPA to nested IPA range lookup */ struct maple_tree nested_revmap_mt; + /* + * Nested IPA to canonical IPA range lookup, essentially a cache of + * the guest's stage-2. + */ + struct maple_tree nested_direct_mt; =20 /* * 0: Nobody is currently using this, check vttbr for validity diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 6bfc1fbc5387..b0cca48040c4 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1128,6 +1128,7 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu) struct kvm *kvm =3D kvm_s2_mmu_to_kvm(mmu); struct kvm_pgtable *pgt =3D NULL; struct maple_tree *revmap_mt =3D &mmu->nested_revmap_mt; + struct maple_tree *direct_mt =3D &mmu->nested_direct_mt; =20 write_lock(&kvm->mmu_lock); pgt =3D mmu->pgt; @@ -1138,8 +1139,12 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu) } =20 if (kvm_is_nested_s2_mmu(kvm, mmu)) { - if (!mtree_empty(revmap_mt)) - mtree_destroy(revmap_mt); + if (!mtree_empty(revmap_mt) || !mtree_empty(direct_mt)) { + mtree_lock(revmap_mt); + __mt_destroy(revmap_mt); + __mt_destroy(direct_mt); + mtree_unlock(revmap_mt); + } kvm_init_nested_s2_mmu(mmu); } =20 diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index cd93793fe89d..975a1b38d545 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -47,14 +47,14 @@ struct vncr_tlb { #define S2_MMU_PER_VCPU 2 =20 /* - * Per shadow S2 reverse map (canonical IPA -> nested IPA range) maple tree - * payload layout: + * Per shadow S2 reverse & direct map maple tree payload layout: * - * bit 62: valid, prevents the case where the nested IPA is 0 and tur= ning + * bit 62: valid, prevents the case where the address is 0 and turning * the whole value to 0 (NULL for maple tree) - * bits 55-12: nested IPA bits 55-12 + * bits 55-12: {nested, canonical} IPA bits 55-12 * bit 0: UNKNOWN_IPA bit, 1 indicates we give up on tracking what n= ested - * IPA maps to this canonical IPA in the shadow stage-2 + * IPA maps to this canonical IPA in the shadow stage-2, only= used + * in reverse map */ #define VALID_ENTRY BIT(62) #define ADDR_MASK GENMASK_ULL(55, 12) @@ -127,12 +127,18 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) swap(kvm->arch.nested_mmus, tmp); =20 /* - * If we went through a realocation, adjust the MMU back-pointers in - * the previously initialised kvm_pgtable structures. + * If we went through a reallocation, adjust pointers in the + * previously initialised MMUs. */ - if (kvm->arch.nested_mmus !=3D tmp) - for (int i =3D 0; i < kvm->arch.nested_mmus_size; i++) - kvm->arch.nested_mmus[i].pgt->mmu =3D &kvm->arch.nested_mmus[i]; + if (kvm->arch.nested_mmus !=3D tmp) { + for (int i =3D 0; i < kvm->arch.nested_mmus_size; i++) { + struct kvm_s2_mmu *mmu =3D &kvm->arch.nested_mmus[i]; + + mmu->pgt->mmu =3D &kvm->arch.nested_mmus[i]; + mt_set_external_lock(&mmu->nested_direct_mt, + &mmu->nested_revmap_mt.ma_lock); + } + } =20 for (int i =3D kvm->arch.nested_mmus_size; !ret && i < num_mmus; i++) ret =3D init_nested_s2_mmu(kvm, &kvm->arch.nested_mmus[i]); @@ -875,36 +881,61 @@ static struct kvm_s2_mmu *get_s2_mmu_nested(struct kv= m_vcpu *vcpu) void kvm_remove_nested_revmap(struct kvm_s2_mmu *mmu, u64 nested_ipa, size= _t size) { /* - * Iterate through the mt of this mmu, remove all canonical ipa ranges - * with !UNKNOWN_IPA that maps to ranges that are strictly within - * [addr, addr + size). + * Remove the reverse map and its corresponding direct map together, + * when these conditions are met: + * + * 1. The reverse map is valid (not UNKNOWN_IPA). + * 2. The reverse map is completely covered by the TLBI range. + * 3. The reverse map and the direct map are symmetric i.e. they map to + * each other, with the same size. + * + * The symmetry check ensures we don't remove wrong reverse mappings. */ struct maple_tree *revmap_mt =3D &mmu->nested_revmap_mt; - u64 entry_val, nested_ipa_end =3D nested_ipa + size; - u64 this_nested_ipa, this_nested_ipa_end; - size_t revmap_size; - void *entry; - - MA_STATE(mas_rmap, revmap_mt, 0, ULONG_MAX); - + struct maple_tree *direct_mt =3D &mmu->nested_direct_mt; + gpa_t nested_ipa_end =3D nested_ipa + size - 1; + u64 entry_dmap; + struct mapping { + u64 from; + u64 to; + size_t size; + }; + + MA_STATE(mas_dmap, direct_mt, nested_ipa, nested_ipa_end); mtree_lock(revmap_mt); - mas_for_each(&mas_rmap, entry, ULONG_MAX) { - entry_val =3D xa_to_value(entry); - if (unknown_ipa_entry(entry_val)) - continue; + entry_dmap =3D xa_to_value(mas_find(&mas_dmap, nested_ipa_end)); =20 - revmap_size =3D mas_rmap.last - mas_rmap.index + 1; - this_nested_ipa =3D entry_val & ADDR_MASK; - this_nested_ipa_end =3D this_nested_ipa + revmap_size; + while (entry_dmap && mas_dmap.index <=3D nested_ipa_end) { + struct mapping dmap, rmap; + u64 entry_rmap; =20 - if (this_nested_ipa >=3D nested_ipa && - this_nested_ipa_end <=3D nested_ipa_end) { - /* - * Ignore result, failure to remove reverse mapping will - * only cause extra unmaps. - */ + dmap.from =3D mas_dmap.index; + dmap.to =3D entry_dmap & ADDR_MASK; + dmap.size =3D mas_dmap.last - mas_dmap.index + 1; + + /* + * Use the canonical ipa range to find the corresponding entry + * in revmap. + */ + MA_STATE(mas_rmap, revmap_mt, dmap.to, dmap.to + dmap.size - 1); + entry_rmap =3D xa_to_value(mas_find(&mas_rmap, + dmap.to + dmap.size - 1)); + + rmap.from =3D mas_rmap.index; + rmap.to =3D entry_rmap & ADDR_MASK; + rmap.size =3D mas_rmap.last - mas_rmap.index + 1; + + /* The three conditions outlined above. */ + if (entry_rmap && valid_entry(entry_rmap) && + dmap.from >=3D nested_ipa && + dmap.from + dmap.size - 1 <=3D nested_ipa_end && + dmap.from =3D=3D rmap.to && + rmap.from =3D=3D dmap.to && + dmap.size =3D=3D rmap.size) { + mas_store_gfp(&mas_dmap, NULL, GFP_NOWAIT | __GFP_ACCOUNT); mas_store_gfp(&mas_rmap, NULL, GFP_NOWAIT | __GFP_ACCOUNT); } + entry_dmap =3D xa_to_value(mas_find(&mas_dmap, nested_ipa_end)); } mtree_unlock(revmap_mt); } @@ -913,7 +944,8 @@ void kvm_record_nested_revmap(gpa_t canonical_ipa, stru= ct kvm_s2_mmu *mmu, gpa_t nested_ipa, size_t map_size) { struct maple_tree *revmap_mt =3D &mmu->nested_revmap_mt; - gpa_t canonical_ipa_end; + struct maple_tree *direct_mt =3D &mmu->nested_direct_mt; + gpa_t canonical_ipa_end, nested_ipa_end; u64 entry, new_entry =3D 0; =20 lockdep_assert_held_read(&kvm_s2_mmu_to_kvm(mmu)->mmu_lock); @@ -925,7 +957,9 @@ void kvm_record_nested_revmap(gpa_t canonical_ipa, stru= ct kvm_s2_mmu *mmu, canonical_ipa =3D ALIGN_DOWN(canonical_ipa, map_size); =20 canonical_ipa_end =3D canonical_ipa + map_size - 1; + nested_ipa_end =3D nested_ipa + map_size - 1; MA_STATE(mas_rmap, revmap_mt, canonical_ipa, canonical_ipa_end); + MA_STATE(mas_dmap, direct_mt, nested_ipa, nested_ipa_end); =20 mtree_lock(revmap_mt); entry =3D xa_to_value(mas_find(&mas_rmap, canonical_ipa_end)); @@ -960,6 +994,15 @@ void kvm_record_nested_revmap(gpa_t canonical_ipa, str= uct kvm_s2_mmu *mmu, nested_ipa, new_entry); mmu->nested_revmap_broken =3D true; } + + /* + * Add direct map but ignore the result, missing a direct map does not + * affect correctness. + */ + if (valid_entry(new_entry) && !mmu->nested_revmap_broken) + mas_store_gfp(&mas_dmap, xa_mk_value(canonical_ipa | VALID_ENTRY), + GFP_NOWAIT | __GFP_ACCOUNT); + unlock: mtree_unlock(revmap_mt); } @@ -971,6 +1014,8 @@ void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu) mmu->nested_stage2_enabled =3D false; atomic_set(&mmu->refcnt, 0); mt_init(&mmu->nested_revmap_mt); + mt_init_flags(&mmu->nested_direct_mt, MT_FLAGS_LOCK_EXTERN); + mt_set_external_lock(&mmu->nested_direct_mt, &mmu->nested_revmap_mt.ma_lo= ck); mmu->nested_revmap_broken =3D false; } =20 @@ -1349,7 +1394,10 @@ void kvm_nested_s2_wp(struct kvm *kvm) =20 static void reset_revmap_and_unmap(struct kvm_s2_mmu *mmu, bool may_block) { - mtree_destroy(&mmu->nested_revmap_mt); + mtree_lock(&mmu->nested_revmap_mt); + __mt_destroy(&mmu->nested_revmap_mt); + __mt_destroy(&mmu->nested_direct_mt); + mtree_unlock(&mmu->nested_revmap_mt); mmu->nested_revmap_broken =3D false; kvm_stage2_unmap_range(mmu, 0, kvm_phys_size(mmu), may_block); } @@ -1357,12 +1405,15 @@ static void reset_revmap_and_unmap(struct kvm_s2_mm= u *mmu, bool may_block) static void unmap_mmu_cipa_range(struct kvm_s2_mmu *mmu, gpa_t canonical_i= pa, size_t unmap_size, bool may_block) { + struct maple_tree *direct_mt =3D &mmu->nested_direct_mt; struct maple_tree *revmap_mt =3D &mmu->nested_revmap_mt; gpa_t canonical_ipa_end =3D canonical_ipa + unmap_size - 1; + gpa_t nested_ipa, nested_ipa_end; size_t entry_size; gpa_t next_addr; u64 entry; MA_STATE(mas_rmap, revmap_mt, canonical_ipa, canonical_ipa_end); + MA_STATE(mas_dmap, direct_mt, 0, ULONG_MAX); =20 lockdep_assert_held_write(&kvm_s2_mmu_to_kvm(mmu)->mmu_lock); =20 @@ -1398,6 +1449,16 @@ static void unmap_mmu_cipa_range(struct kvm_s2_mmu *= mmu, gpa_t canonical_ipa, */ mas_store_gfp(&mas_rmap, NULL, GFP_NOWAIT | __GFP_ACCOUNT); =20 + /* + * Try to also remove the direct map, it is okay if this fails, + * as we check for direct map consistency in + * kvm_remove_nested_revmap(). + */ + nested_ipa =3D entry & ADDR_MASK; + nested_ipa_end =3D nested_ipa + entry_size - 1; + mas_set_range(&mas_dmap, nested_ipa, nested_ipa_end); + mas_store_gfp(&mas_dmap, NULL, GFP_NOWAIT | __GFP_ACCOUNT); + mtree_unlock(revmap_mt); kvm_stage2_unmap_range(mmu, entry & ADDR_MASK, entry_size, may_block); --=20 2.43.0