From nobody Sun Feb 8 05:56:43 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED23DC71153 for ; Mon, 4 Sep 2023 07:29:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242180AbjIDH3z (ORCPT ); Mon, 4 Sep 2023 03:29:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56274 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352431AbjIDH3w (ORCPT ); Mon, 4 Sep 2023 03:29:52 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 92855139 for ; Mon, 4 Sep 2023 00:28:58 -0700 (PDT) Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-21-Xm6JU2DSMD-NIYWwlG_p3A-1; Mon, 04 Sep 2023 03:28:52 -0400 X-MC-Unique: Xm6JU2DSMD-NIYWwlG_p3A-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 547561817903; Mon, 4 Sep 2023 07:28:51 +0000 (UTC) Received: from gshan.redhat.com (unknown [10.64.136.25]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 45B071121318; Mon, 4 Sep 2023 07:28:46 +0000 (UTC) From: Gavin Shan To: kvmarm@lists.linux.dev Cc: linux-kernel@vger.kernel.org, maz@kernel.org, oliver.upton@linux.dev, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, qperret@google.com, gshan@redhat.com, ricarkol@google.com, tabba@google.com, bgardon@google.com, zhenyzha@redhat.com, yihyu@redhat.com, shan.gavin@gmail.com Subject: [PATCH] KVM: arm64: Fix soft-lockup on relaxing PTE permission Date: Mon, 4 Sep 2023 17:28:26 +1000 Message-ID: <20230904072826.1468907-1-gshan@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We observed soft-lockup on the host in a specific scenario where the host on Ampere's Altra Max CPU has 64KB base page size and the guest has 4KB base page size, 64 vCPUs and 13GB memory. The guest's memory is backed by 512MB huge pages via hugetlbfs. All the 64 vCPUs are simultaneously trapped into the host due to permission page faults, to request adding the execution permission to the corresponding PMD entry, before the soft-lockup is raised on the host. On handling the parallel requests, the instruction cache for the 512MB huge page is invalidated by mm_ops->icache_inval_pou() in stage2_attr_walker() on 64 hardware CPUs. Unfortunately, the instruction cache invalidation on one CPU interfere with that on another CPU in the hardware level. It takes 37 seconds for mm_ops->icache_inval_pou() to finish in the worst case. So we can't scale out to handle the permission faults at will. They need to be serialized to some extent with the help of a interval tree, to track IPA ranges, currently under service. For the incoming permission faults, the vCPU is asked to bail for a retry if its IPA range is being served since the vCPU can't proceed its execution. Fixes: 1577cb5823ce ("KVM: arm64: Handle stage-2 faults in parallel") Cc: stable@vger.kernel.org # v6.2+ Reported-by: Yihuang Yu Reported-by: Zhenyu Zhang Signed-off-by: Gavin Shan --- arch/arm64/include/asm/kvm_host.h | 4 ++ arch/arm64/include/asm/kvm_pgtable.h | 3 +- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 4 +- arch/arm64/kvm/hyp/pgtable.c | 25 +++++++++--- arch/arm64/kvm/mmu.c | 55 ++++++++++++++++++++++++++- 5 files changed, 83 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index d3dd05bbfe23..a457720b5caf 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -175,6 +175,10 @@ struct kvm_s2_mmu { struct kvm_mmu_memory_cache split_page_cache; uint64_t split_page_chunk_size; =20 + /* Page fault ranges */ + struct mutex fault_ranges_mutex; + struct rb_root_cached fault_ranges; + struct kvm_arch *arch; }; =20 diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/= kvm_pgtable.h index 929d355eae0a..dca0bf81616f 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -149,7 +149,8 @@ struct kvm_pgtable_mm_ops { void* (*phys_to_virt)(phys_addr_t phys); phys_addr_t (*virt_to_phys)(void *addr); void (*dcache_clean_inval_poc)(void *addr, size_t size); - void (*icache_inval_pou)(void *addr, size_t size); + int (*icache_inval_pou)(struct kvm_s2_mmu *mmu, + void *addr, u64 ipa, size_t size); }; =20 /** diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvh= e/mem_protect.c index 9d703441278b..9bbe7c641770 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -223,10 +223,12 @@ static void clean_dcache_guest_page(void *va, size_t = size) hyp_fixmap_unmap(); } =20 -static void invalidate_icache_guest_page(void *va, size_t size) +static int invalidate_icache_guest_page(struct kvm_s2_mmu *mmu, + void *va, u64 ipa, size_t size) { __invalidate_icache_guest_page(hyp_fixmap_map(__hyp_pa(va)), size); hyp_fixmap_unmap(); + return 0; } =20 int kvm_guest_prepare_stage2(struct pkvm_hyp_vm *vm, void *pgd) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index f7a93ef29250..fabfdb4d1e00 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -875,6 +875,7 @@ static int stage2_map_walker_try_leaf(const struct kvm_= pgtable_visit_ctx *ctx, u64 granule =3D kvm_granule_size(ctx->level); struct kvm_pgtable *pgt =3D data->mmu->pgt; struct kvm_pgtable_mm_ops *mm_ops =3D ctx->mm_ops; + int ret; =20 if (!stage2_leaf_mapping_allowed(ctx, data)) return -E2BIG; @@ -903,8 +904,14 @@ static int stage2_map_walker_try_leaf(const struct kvm= _pgtable_visit_ctx *ctx, granule); =20 if (!kvm_pgtable_walk_skip_cmo(ctx) && mm_ops->icache_inval_pou && - stage2_pte_executable(new)) - mm_ops->icache_inval_pou(kvm_pte_follow(new, mm_ops), granule); + stage2_pte_executable(new)) { + ret =3D mm_ops->icache_inval_pou(data->mmu, + kvm_pte_follow(new, mm_ops), + ALIGN_DOWN(ctx->addr, granule), + granule); + if (ret) + return ret; + } =20 stage2_make_pte(ctx, new); =20 @@ -1101,6 +1108,7 @@ int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt,= u64 addr, u64 size) } =20 struct stage2_attr_data { + struct kvm_s2_mmu *mmu; kvm_pte_t attr_set; kvm_pte_t attr_clr; kvm_pte_t pte; @@ -1113,6 +1121,8 @@ static int stage2_attr_walker(const struct kvm_pgtabl= e_visit_ctx *ctx, kvm_pte_t pte =3D ctx->old; struct stage2_attr_data *data =3D ctx->arg; struct kvm_pgtable_mm_ops *mm_ops =3D ctx->mm_ops; + u64 granule =3D kvm_granule_size(ctx->level); + int ret; =20 if (!kvm_pte_valid(ctx->old)) return -EAGAIN; @@ -1133,9 +1143,13 @@ static int stage2_attr_walker(const struct kvm_pgtab= le_visit_ctx *ctx, * stage-2 PTE if we are going to add executable permission. */ if (mm_ops->icache_inval_pou && - stage2_pte_executable(pte) && !stage2_pte_executable(ctx->old)) - mm_ops->icache_inval_pou(kvm_pte_follow(pte, mm_ops), - kvm_granule_size(ctx->level)); + stage2_pte_executable(pte) && !stage2_pte_executable(ctx->old)) { + ret =3D mm_ops->icache_inval_pou(data->mmu, + kvm_pte_follow(pte, mm_ops), + ALIGN_DOWN(ctx->addr, granule), granule); + if (ret) + return ret; + } =20 if (!stage2_try_set_pte(ctx, pte)) return -EAGAIN; @@ -1152,6 +1166,7 @@ static int stage2_update_leaf_attrs(struct kvm_pgtabl= e *pgt, u64 addr, int ret; kvm_pte_t attr_mask =3D KVM_PTE_LEAF_ATTR_LO | KVM_PTE_LEAF_ATTR_HI; struct stage2_attr_data data =3D { + .mmu =3D pgt->mmu, .attr_set =3D attr_set & attr_mask, .attr_clr =3D attr_clr & attr_mask, }; diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index d3b4feed460c..a778f48beb56 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -267,9 +267,58 @@ static void clean_dcache_guest_page(void *va, size_t s= ize) __clean_dcache_guest_page(va, size); } =20 -static void invalidate_icache_guest_page(void *va, size_t size) +static struct interval_tree_node *add_fault_range(struct kvm_s2_mmu *mmu, + u64 ipa, size_t size) { + struct interval_tree_node *node; + unsigned long start =3D ipa, end =3D start + size - 1; /* inclusive */ + + mutex_lock(&mmu->fault_ranges_mutex); + + node =3D interval_tree_iter_first(&mmu->fault_ranges, start, end); + if (node) { + node =3D NULL; + goto unlock; + } + + node =3D kzalloc(sizeof(*node), GFP_KERNEL_ACCOUNT); + if (!node) + goto unlock; + + node->start =3D start; + node->last =3D end; + interval_tree_insert(node, &mmu->fault_ranges); + +unlock: + mutex_unlock(&mmu->fault_ranges_mutex); + return node; +} + +static void remove_fault_range(struct kvm_s2_mmu *mmu, + struct interval_tree_node *node) +{ + mutex_lock(&mmu->fault_ranges_mutex); + + interval_tree_remove(node, &mmu->fault_ranges); + kfree(node); + + mutex_unlock(&mmu->fault_ranges_mutex); +} + + +static int invalidate_icache_guest_page(struct kvm_s2_mmu *mmu, + void *va, u64 ipa, size_t size) +{ + struct interval_tree_node *node; + + node =3D add_fault_range(mmu, ipa, size); + if (!node) + return -EAGAIN; + __invalidate_icache_guest_page(va, size); + remove_fault_range(mmu, node); + + return 0; } =20 /* @@ -859,6 +908,10 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2= _mmu *mmu, unsigned long t mmu->split_page_chunk_size =3D KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT; mmu->split_page_cache.gfp_zero =3D __GFP_ZERO; =20 + /* Initialize the page fault ranges */ + mutex_init(&mmu->fault_ranges_mutex); + mmu->fault_ranges =3D RB_ROOT_CACHED; + mmu->pgt =3D pgt; mmu->pgd_phys =3D __pa(pgt->pgd); return 0; --=20 2.41.0