From nobody Tue Jun 16 16:52:56 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 816D63FD152; Thu, 30 Apr 2026 11:15:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547726; cv=none; b=gF15Z1Hda1KD6h7l9kCnqowlSZB5+CN9QphVZdVK9D0xmvU4/jOUeJRehl4GfGrKBGhkwDzDPu7Z0TjCLRxB1B3bkWeG2/GvyPaQE6CJJlCwn8xyH1tfGViH6KxrsjgANOwAfANILrVlUk1BEAfSKzpqumCB5LWiyZYxhKOciwc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547726; c=relaxed/simple; bh=BxOAhxaI3ep0pCLZHvFXFWhljUjYRpulIZeDdhTGMx0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=M5D5MUSMSlXfTZMoqYfYWGiANyxAJsHLzR0me2Q0kiT70KDnLN4BGNAKg/CL5y93DjZ6K2K8l+Wf3DsvVWC5I6d7G5NUEpPcmDU0tyzE8pNTsLJT0XDZeRUZIo0I7w316s45FSre4u9WPQDldrvmY5M8epr4eAZL1/Yfo2JCPsI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=tW6C+0X8; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="tW6C+0X8" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 32AF13311; Thu, 30 Apr 2026 04:15:18 -0700 (PDT) Received: from devkitleo.cambridge.arm.com (devkitleo.cambridge.arm.com [10.1.196.90]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 37C233F763; Thu, 30 Apr 2026 04:15:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1777547723; bh=BxOAhxaI3ep0pCLZHvFXFWhljUjYRpulIZeDdhTGMx0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tW6C+0X8vWRpU3tueo+qCaEfYKtSSWq3qISrRQm4ow8VE9Xc/V868i7jDfltdB2Wa W4LAqThuXJkL6IDVuGj4WbH6zdbCygpLtMzu87JLzao4jj2Vh3q+EO/fhzFl/y7ncK LYWyKsSrgT7UthhHo3POxnaq9EpFRZiy1RBSCdHs= From: Leonardo Bras To: Catalin Marinas , Will Deacon , Leonardo Bras , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Zenghui Yu , "Rafael J. Wysocki" , Len Brown , Saket Dumbre , Paolo Bonzini , Chengwen Feng , Jonathan Cameron , Kees Cook , =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , Ryan Roberts , Yang Shi , Thomas Huth , mrigendrachaubey , Yeoreum Yun , Mark Brown , Kevin Brodsky , James Clark , Ard Biesheuvel , Fuad Tabba , Raghavendra Rao Ananta , Nathan Chancellor , Vincent Donnefort , Lorenzo Pieralisi , Sascha Bischoff , Anshuman Khandual , Tian Zheng , Wei-Lin Chang Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-acpi@vger.kernel.org, acpica-devel@lists.linux.dev, kvm@vger.kernel.org Subject: [PATCH v1 01/12] KVM: arm64: Enable eager hugepage splitting if HDBSS is available Date: Thu, 30 Apr 2026 12:14:05 +0100 Message-ID: <20260430111424.3479613-3-leo.bras@arm.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260430111424.3479613-2-leo.bras@arm.com> References: <20260430111424.3479613-2-leo.bras@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" FEAT_HDBSS speeds up guest memory dirty tracking by avoiding a page fault and saving the entry in a tracking structure. That may be a problem when we have guest memory backed by hugepages or transparent huge pages, as it's not possible to do on-demand hugepage splitting, relying only on eager hugepage splitting. So, at stage2 initialization, enable eager hugepage splitting with chunk =3D PAGE_SIZE if the system supports HDBSS. Signed-off-by: Leonardo Bras --- arch/arm64/kvm/mmu.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index d089c107d9b7..42c734423253 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -993,22 +993,26 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s= 2_mmu *mmu, unsigned long t =20 mmu->last_vcpu_ran =3D alloc_percpu(typeof(*mmu->last_vcpu_ran)); if (!mmu->last_vcpu_ran) { err =3D -ENOMEM; goto out_destroy_pgtable; } =20 for_each_possible_cpu(cpu) *per_cpu_ptr(mmu->last_vcpu_ran, cpu) =3D -1; =20 - /* The eager page splitting is disabled by default */ - mmu->split_page_chunk_size =3D KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT; + /* The eager page splitting is disabled by default if system has no HDBS= S */ + if (system_supports_hacdbs()) + mmu->split_page_chunk_size =3D PAGE_SIZE; + else + mmu->split_page_chunk_size =3D KVM_ARM_EAGER_SPLIT_CHUNK_SIZE_DEFAULT; + mmu->split_page_cache.gfp_zero =3D __GFP_ZERO; =20 mmu->pgd_phys =3D __pa(pgt->pgd); =20 if (kvm_is_nested_s2_mmu(kvm, mmu)) kvm_init_nested_s2_mmu(mmu); =20 return 0; =20 out_destroy_pgtable: --=20 2.54.0 From nobody Tue Jun 16 16:52:56 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 02333401A07; Thu, 30 Apr 2026 11:15:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547731; cv=none; b=mbH7KviLGX6Z/P+S/m+UPuZ5KaFP7yN1xsmMiJZRPH0GbOQ9HmfCPfhJynFjC30H3YkVRYzEMpgq/e+AGO2rQXA0aQqJzb/b7OlwtpPy3r62DwjfEwlQJQ/16JkESuki6JtQDZoO6thuXXLiFAAY9teGo85HOh8Ck2ixBKPaBhM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547731; c=relaxed/simple; bh=IJnhFRETYA3/XOyItIN8jRNbzWUD8pBmuScgZbJBJ+s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VYTYUK1dyEeGg5Q4aK/raOh9nbApRi7yFYGEpiol6fhLrX4vHb+WGZ9dVMndHKA/tCteOiiGQzBxEaQqfrKq8JKFzJZMKYsOMHHCQ2VtDJ3sw+mTXwyOO35u6UbUvsAYYgCoccFpKe1sLel6Tv3SQkq9YxHTY5lZBa2qX4JFmOQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=hwbU60+M; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="hwbU60+M" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 002CC34FB; Thu, 30 Apr 2026 04:15:23 -0700 (PDT) Received: from devkitleo.cambridge.arm.com (devkitleo.cambridge.arm.com [10.1.196.90]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E42A73F763; Thu, 30 Apr 2026 04:15:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1777547728; bh=IJnhFRETYA3/XOyItIN8jRNbzWUD8pBmuScgZbJBJ+s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hwbU60+MRUATS4TmL0gah19/ogqn/VzSzNRDIGcQQPQY4Nb3B0m3bE6Io/YJKs3DP N9whYfyK9mf867Bz+RuahLPusbfBp8QpEUqfl0glyBQCBOG2Z+qeSzWR0qPmqMy1WB eq0rs04VCgCXe7HdhltnqOgnEyUlJgQyMCVtW9zo= From: Leonardo Bras To: Catalin Marinas , Will Deacon , Leonardo Bras , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Zenghui Yu , "Rafael J. Wysocki" , Len Brown , Saket Dumbre , Paolo Bonzini , Chengwen Feng , Jonathan Cameron , Kees Cook , =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , Ryan Roberts , Yang Shi , Thomas Huth , mrigendrachaubey , Yeoreum Yun , Mark Brown , Kevin Brodsky , James Clark , Ard Biesheuvel , Fuad Tabba , Raghavendra Rao Ananta , Nathan Chancellor , Vincent Donnefort , Lorenzo Pieralisi , Sascha Bischoff , Anshuman Khandual , Tian Zheng , Wei-Lin Chang Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-acpi@vger.kernel.org, acpica-devel@lists.linux.dev, kvm@vger.kernel.org Subject: [PATCH v1 02/12] KVM: arm64: HDBSS bits Date: Thu, 30 Apr 2026 12:14:06 +0100 Message-ID: <20260430111424.3479613-4-leo.bras@arm.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260430111424.3479613-2-leo.bras@arm.com> References: <20260430111424.3479613-2-leo.bras@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" All those bits should come from a future version of HDBSS patchset: https://lore.kernel.org/lkml/20260225040421.2683931-1-zhengtian10@huawei.com I added them here in order to fulfill the dependencies and be able to easily build this patchset, but this particular patch should not be merged upstream. Signed-off-by: Leonardo Bras --- arch/arm64/include/asm/cpufeature.h | 5 +++++ arch/arm64/include/asm/kvm_dirty_bit.h | 12 ++++++++++++ arch/arm64/include/asm/kvm_pgtable.h | 3 +++ arch/arm64/kernel/cpufeature.c | 12 ++++++++++++ arch/arm64/kvm/dirty_bit.c | 23 +++++++++++++++++++++++ arch/arm64/kvm/hyp/pgtable.c | 15 +++++++++++++-- arch/arm64/kvm/Makefile | 2 +- arch/arm64/tools/cpucaps | 1 + 8 files changed, 70 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_dirty_bit.h create mode 100644 arch/arm64/kvm/dirty_bit.c diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/c= pufeature.h index 4de51f8d92cb..dcc2e2cad5ad 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -849,20 +849,25 @@ static inline bool system_supports_poe(void) static inline bool system_supports_gcs(void) { return alternative_has_cap_unlikely(ARM64_HAS_GCS); } =20 static inline bool system_supports_haft(void) { return cpus_have_final_cap(ARM64_HAFT); } =20 +static inline bool system_supports_hdbss(void) +{ + return cpus_have_final_cap(ARM64_HAS_HDBSS); +} + static __always_inline bool system_supports_mpam(void) { return alternative_has_cap_unlikely(ARM64_MPAM); } =20 static __always_inline bool system_supports_mpam_hcr(void) { return alternative_has_cap_unlikely(ARM64_MPAM_HCR); } =20 diff --git a/arch/arm64/include/asm/kvm_dirty_bit.h b/arch/arm64/include/as= m/kvm_dirty_bit.h new file mode 100644 index 000000000000..dd16438f0651 --- /dev/null +++ b/arch/arm64/include/asm/kvm_dirty_bit.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2026 ARM Ltd. + * Author: Leonardo Bras + */ + +#ifndef __ARM64_KVM_DIRTY_BIT_H__ +#define __ARM64_KVM_DIRTY_BIT_H__ + +#include + +#endif /* __ARM64_KVM_DIRTY_BIT_H__ */ diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/= kvm_pgtable.h index 41a8687938eb..646ff88e0258 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -86,20 +86,22 @@ typedef u64 kvm_pte_t; #define KVM_PTE_LEAF_ATTR_HI GENMASK(63, 50) =20 #define KVM_PTE_LEAF_ATTR_HI_SW GENMASK(58, 55) =20 #define KVM_PTE_LEAF_ATTR_HI_S1_XN BIT(54) #define KVM_PTE_LEAF_ATTR_HI_S1_UXN BIT(54) #define KVM_PTE_LEAF_ATTR_HI_S1_PXN BIT(53) =20 #define KVM_PTE_LEAF_ATTR_HI_S2_XN GENMASK(54, 53) =20 +#define KVM_PTE_LEAF_ATTR_HI_S2_DBM BIT(51) + #define KVM_PTE_LEAF_ATTR_HI_S1_GP BIT(50) =20 #define KVM_PTE_LEAF_ATTR_S2_PERMS (KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R | \ KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W | \ KVM_PTE_LEAF_ATTR_HI_S2_XN) =20 /* pKVM invalid pte encodings */ #define KVM_INVALID_PTE_TYPE_MASK GENMASK(63, 60) #define KVM_INVALID_PTE_ANNOT_MASK ~(KVM_PTE_VALID | \ KVM_INVALID_PTE_TYPE_MASK) @@ -246,20 +248,21 @@ struct kvm_pgtable_mm_ops { }; =20 /** * enum kvm_pgtable_stage2_flags - Stage-2 page-table flags. * @KVM_PGTABLE_S2_IDMAP: Only use identity mappings. * @KVM_PGTABLE_S2_AS_S1: Final memory attributes are that of Stage-1. */ enum kvm_pgtable_stage2_flags { KVM_PGTABLE_S2_IDMAP =3D BIT(0), KVM_PGTABLE_S2_AS_S1 =3D BIT(1), + KVM_PGTABLE_S2_DBM =3D BIT(2), }; =20 /** * enum kvm_pgtable_prot - Page-table permissions and attributes. * @KVM_PGTABLE_PROT_UX: Unprivileged execute permission. * @KVM_PGTABLE_PROT_PX: Privileged execute permission. * @KVM_PGTABLE_PROT_X: Privileged and unprivileged execute permission. * @KVM_PGTABLE_PROT_W: Write permission. * @KVM_PGTABLE_PROT_R: Read permission. * @KVM_PGTABLE_PROT_DEVICE: Device attributes. diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 6d53bb15cf7b..eac7784463e3 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2112,20 +2112,25 @@ static bool has_nested_virt_support(const struct ar= m64_cpu_capabilities *cap, =20 return true; } =20 static bool hvhe_possible(const struct arm64_cpu_capabilities *entry, int __unused) { return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_HVHE); } =20 +static bool has_vhe_hdbss(const struct arm64_cpu_capabilities *entry, int = cope) +{ + return is_kernel_in_hyp_mode() && has_cpuid_feature(entry, cope); +} + bool cpu_supports_bbml2_noabort(void) { /* * We want to allow usage of BBML2 in as wide a range of kernel contexts * as possible. This list is therefore an allow-list of known-good * implementations that both support BBML2 and additionally, fulfill the * extra constraint of never generating TLB conflict aborts when using * the relaxed BBML2 semantics (such aborts make use of BBML2 in certain * kernel contexts difficult to prove safe against recursive aborts). * @@ -2762,20 +2767,27 @@ static const struct arm64_cpu_capabilities arm64_fe= atures[] =3D { * cannot be emulated in software (no access fault will occur). * Therefore this should be used only if it's supported system * wide. */ .type =3D ARM64_CPUCAP_SYSTEM_FEATURE, .capability =3D ARM64_HAFT, .matches =3D has_cpuid_feature, ARM64_CPUID_FIELDS(ID_AA64MMFR1_EL1, HAFDBS, HAFT) }, #endif + { + .desc =3D "Hardware Dirty state tracking structure (HDBSS)", + .type =3D ARM64_CPUCAP_SYSTEM_FEATURE, + .capability =3D ARM64_HAS_HDBSS, + .matches =3D has_vhe_hdbss, + ARM64_CPUID_FIELDS(ID_AA64MMFR1_EL1, HAFDBS, HDBSS) + }, { .desc =3D "CRC32 instructions", .capability =3D ARM64_HAS_CRC32, .type =3D ARM64_CPUCAP_SYSTEM_FEATURE, .matches =3D has_cpuid_feature, ARM64_CPUID_FIELDS(ID_AA64ISAR0_EL1, CRC32, IMP) }, { .desc =3D "Speculative Store Bypassing Safe (SSBS)", .capability =3D ARM64_SSBS, diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c new file mode 100644 index 000000000000..31ef35f3b72f --- /dev/null +++ b/arch/arm64/kvm/dirty_bit.c @@ -0,0 +1,23 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2026 ARM Ltd. + * Author: Leonardo Bras + */ + +#include + +/* HDBSS entry field definitions */ +#define HDBSS_ENTRY_VALID BIT(0) +#define HDBSS_ENTRY_TTWL_SHIFT (1) +#define HDBSS_ENTRY_TTWL_MASK (GENMASK(3, 1)) +#define HDBSS_ENTRY_TTWL(x) \ + (((x) << HDBSS_ENTRY_TTWL_SHIFT) & HDBSS_ENTRY_TTWL_MASK) +#define HDBSS_ENTRY_TTWL_RESV HDBSS_ENTRY_TTWL(-4) +#define HDBSS_ENTRY_IPA GENMASK_ULL(55, 12) + +inline u64 hdbss_get_ttwl(u64 chunk_size) +{ + u64 hw_lvl =3D ARM64_HW_PGTABLE_LEVELS(ilog2(chunk_size)); + + return HDBSS_ENTRY_TTWL(3 - hw_lvl); +} diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 0c1defa5fb0f..68ade2623748 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -724,23 +724,27 @@ static int stage2_set_prot_attr(struct kvm_pgtable *p= gt, enum kvm_pgtable_prot p attr =3D KVM_S2_MEMATTR(pgt, NORMAL); } =20 r =3D stage2_set_xn_attr(prot, &attr); if (r) return r; =20 if (prot & KVM_PGTABLE_PROT_R) attr |=3D KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R; =20 - if (prot & KVM_PGTABLE_PROT_W) + if (prot & KVM_PGTABLE_PROT_W) { attr |=3D KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W; =20 + if (pgt->flags & KVM_PGTABLE_S2_DBM) + attr |=3D KVM_PTE_LEAF_ATTR_HI_S2_DBM; + } + if (!kvm_lpa2_is_enabled()) attr |=3D FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S2_SH, sh); =20 attr |=3D KVM_PTE_LEAF_ATTR_LO_S2_AF; attr |=3D prot & KVM_PTE_LEAF_ATTR_HI_SW; *ptep =3D attr; =20 return 0; } =20 @@ -1358,23 +1362,27 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtab= le *pgt, u64 addr, kvm_pte_t xn =3D 0, set =3D 0, clr =3D 0; s8 level; int ret; =20 if (prot & KVM_PTE_LEAF_ATTR_HI_SW) return -EINVAL; =20 if (prot & KVM_PGTABLE_PROT_R) set |=3D KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R; =20 - if (prot & KVM_PGTABLE_PROT_W) + if (prot & KVM_PGTABLE_PROT_W) { set |=3D KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W; =20 + if (pgt->flags & KVM_PGTABLE_S2_DBM) + set |=3D KVM_PTE_LEAF_ATTR_HI_S2_DBM; + } + ret =3D stage2_set_xn_attr(prot, &xn); if (ret) return ret; =20 set |=3D xn & KVM_PTE_LEAF_ATTR_HI_S2_XN; clr |=3D ~xn & KVM_PTE_LEAF_ATTR_HI_S2_XN; =20 ret =3D stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level, fl= ags); if (!ret || ret =3D=3D -EAGAIN) kvm_call_hyp(__kvm_tlb_flush_vmid_ipa_nsh, pgt->mmu, addr, level); @@ -1576,20 +1584,23 @@ int __kvm_pgtable_stage2_init(struct kvm_pgtable *p= gt, struct kvm_s2_mmu *mmu, u64 vtcr =3D mmu->vtcr; u32 ia_bits =3D VTCR_EL2_IPA(vtcr); u32 sl0 =3D FIELD_GET(VTCR_EL2_SL0_MASK, vtcr); s8 start_level =3D VTCR_EL2_TGRAN_SL0_BASE - sl0; =20 pgd_sz =3D kvm_pgd_pages(ia_bits, start_level) * PAGE_SIZE; pgt->pgd =3D (kvm_pteref_t)mm_ops->zalloc_pages_exact(pgd_sz); if (!pgt->pgd) return -ENOMEM; =20 + if (system_supports_hdbss()) + flags |=3D KVM_PGTABLE_S2_DBM; + pgt->ia_bits =3D ia_bits; pgt->start_level =3D start_level; pgt->mm_ops =3D mm_ops; pgt->mmu =3D mmu; pgt->flags =3D flags; pgt->force_pte_cb =3D force_pte_cb; =20 /* Ensure zeroed PGD pages are visible to the hardware walker */ dsb(ishst); return 0; diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 59612d2f277c..6faacd857346 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -17,21 +17,21 @@ kvm-y +=3D arm.o mmu.o mmio.o psci.o hypercalls.o pvtim= e.o \ inject_fault.o va_layout.o handle_exit.o config.o \ guest.o debug.o reset.o sys_regs.o stacktrace.o \ vgic-sys-reg-v3.o fpsimd.o pkvm.o \ arch_timer.o trng.o vmid.o emulate-nested.o nested.o at.o \ vgic/vgic.o vgic/vgic-init.o \ vgic/vgic-irqfd.o vgic/vgic-v2.o \ vgic/vgic-v3.o vgic/vgic-v4.o \ vgic/vgic-mmio.o vgic/vgic-mmio-v2.o \ vgic/vgic-mmio-v3.o vgic/vgic-kvm-device.o \ vgic/vgic-its.o vgic/vgic-debug.o vgic/vgic-v3-nested.o \ - vgic/vgic-v5.o + vgic/vgic-v5.o dirty_bit.o =20 kvm-$(CONFIG_HW_PERF_EVENTS) +=3D pmu-emul.o pmu.o kvm-$(CONFIG_ARM64_PTR_AUTH) +=3D pauth.o kvm-$(CONFIG_PTDUMP_STAGE2_DEBUGFS) +=3D ptdump.o =20 kvm-$(CONFIG_NVHE_EL2_TRACING) +=3D hyp_trace.o =20 always-y :=3D hyp_constants.h hyp-constants.s =20 define rule_gen_hyp_constants diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index 811c2479e82d..6bd563e0bc62 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -62,20 +62,21 @@ HAS_RASV1P1_EXTN HAS_RNG HAS_SB HAS_STAGE2_FWB HAS_TCR2 HAS_TIDCP1 HAS_TLB_RANGE HAS_VA52 HAS_VIRT_HOST_EXTN HAS_WFXT HAS_XNX +HAS_HDBSS HAFT HW_DBM KVM_HVHE KVM_PROTECTED_MODE MISMATCHED_CACHE_TYPE MPAM MPAM_HCR MTE MTE_ASYMM MTE_FAR --=20 2.54.0 From nobody Tue Jun 16 16:52:56 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id BC4DD406264; Thu, 30 Apr 2026 11:15:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547735; cv=none; b=sbxP36EJc+gklqzcIByZSWn0aqnyklxZzLW64DSqIRMRnYmKQXdv2sFrHhEHbUQyaK/i2nbAtXsERTg6/yXTvGfJ2mTOzeT9Za1m4a9gyPiYKl98ekJEmnhs6/p1SyE2M1HHDylCXLQVOh3mlHKo0aRZBpuZUj78IJcYDod/X/A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547735; c=relaxed/simple; bh=UhKkxZ44K9ifGGbHJk8Ib05WN/5cyjeFhz1eO97Wcm0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kTaSmE82UTKktZH3P4wj3VyQxi4KoJIW8T3fLbhz/sJ9UbXYki727XTG4GUHOOiVaO3+Il9zjP37tTFHKNjNlNsLd7npVSaGMHSw8jgwLErC+DaUrveCs5yqGMJ6El7qaYyFYTN7okGjmmvnZwkrqIEJfLFUAqh3DlLAq6cuFrU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=CLeHBWER; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="CLeHBWER" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AF1263500; Thu, 30 Apr 2026 04:15:27 -0700 (PDT) Received: from devkitleo.cambridge.arm.com (devkitleo.cambridge.arm.com [10.1.196.90]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B5B583F763; Thu, 30 Apr 2026 04:15:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1777547733; bh=UhKkxZ44K9ifGGbHJk8Ib05WN/5cyjeFhz1eO97Wcm0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CLeHBWERpoD73pW7riRaqhkZrRB8pPBVJX4Mstz4h3vM1/eMOEngiMdSbZBPAOzWT P2KYxxHNXGuRtPIH9FnSNsyrgbxr9BmgvKOtBuda6a2OoUgO60N7vLf68q8vFaWNNU Ulnxeny0ebrtMiBPqasx+DTUs5UShke4yzl27t9Q= From: Leonardo Bras To: Catalin Marinas , Will Deacon , Leonardo Bras , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Zenghui Yu , "Rafael J. Wysocki" , Len Brown , Saket Dumbre , Paolo Bonzini , Chengwen Feng , Jonathan Cameron , Kees Cook , =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , Ryan Roberts , Yang Shi , Thomas Huth , mrigendrachaubey , Yeoreum Yun , Mark Brown , Kevin Brodsky , James Clark , Ard Biesheuvel , Fuad Tabba , Raghavendra Rao Ananta , Nathan Chancellor , Vincent Donnefort , Lorenzo Pieralisi , Sascha Bischoff , Anshuman Khandual , Tian Zheng , Wei-Lin Chang Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-acpi@vger.kernel.org, acpica-devel@lists.linux.dev, kvm@vger.kernel.org Subject: [PATCH v1 03/12] arm64/cpufeature: Add system-wide FEAT_HACDBS detection Date: Thu, 30 Apr 2026 12:14:07 +0100 Message-ID: <20260430111424.3479613-5-leo.bras@arm.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260430111424.3479613-2-leo.bras@arm.com> References: <20260430111424.3479613-2-leo.bras@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" FEAT_HACDBS will only be used for dirty-bit cleaning if it is detected in all running cpus. Signed-off-by: Leonardo Bras --- arch/arm64/include/asm/cpufeature.h | 5 +++++ arch/arm64/kernel/cpufeature.c | 8 ++++++++ arch/arm64/tools/cpucaps | 1 + 3 files changed, 14 insertions(+) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/c= pufeature.h index dcc2e2cad5ad..5d438a00e0ff 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -854,20 +854,25 @@ static inline bool system_supports_gcs(void) static inline bool system_supports_haft(void) { return cpus_have_final_cap(ARM64_HAFT); } =20 static inline bool system_supports_hdbss(void) { return cpus_have_final_cap(ARM64_HAS_HDBSS); } =20 +static inline bool system_supports_hacdbs(void) +{ + return cpus_have_final_cap(ARM64_HACDBS); +} + static __always_inline bool system_supports_mpam(void) { return alternative_has_cap_unlikely(ARM64_MPAM); } =20 static __always_inline bool system_supports_mpam_hcr(void) { return alternative_has_cap_unlikely(ARM64_MPAM_HCR); } =20 diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index eac7784463e3..95b92060cae1 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -512,20 +512,21 @@ static const struct arm64_ftr_bits ftr_id_aa64mmfr3[]= =3D { FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64MMFR3_EL1_S1POE_SHIFT, 4, 0= ), ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64MMFR3_EL= 1_S1PIE_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR3_EL1_S= CTLRX_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64MMFR3_EL= 1_TCRX_SHIFT, 4, 0), ARM64_FTR_END, }; =20 static const struct arm64_ftr_bits ftr_id_aa64mmfr4[] =3D { S_ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR4_EL1= _E2H0_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR4_EL1_N= V_frac_SHIFT, 4, 0), + ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR4_EL1_H= ACDBS_SHIFT, 4, 0), ARM64_FTR_END, }; =20 static const struct arm64_ftr_bits ftr_ctr[] =3D { ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_EXACT, 31, 1, 1), /* RES1 */ ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, CTR_EL0_DIC_SHIFT= , 1, 1), ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, CTR_EL0_IDC_SHIFT= , 1, 1), ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_HIGHER_OR_ZERO_SAFE, CTR_EL0_= CWG_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_HIGHER_OR_ZERO_SAFE, CTR_EL0_= ERG_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, CTR_EL0_DminLine_= SHIFT, 4, 1), @@ -2752,20 +2753,27 @@ static const struct arm64_cpu_capabilities arm64_fe= atures[] =3D { { .desc =3D "Hardware dirty bit management", .type =3D ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE, .capability =3D ARM64_HW_DBM, .matches =3D has_hw_dbm, .cpu_enable =3D cpu_enable_hw_dbm, .cpus =3D &dbm_cpus, ARM64_CPUID_FIELDS(ID_AA64MMFR1_EL1, HAFDBS, DBM) }, #endif + { + .desc =3D "Hardware dirty bit Cleaning", + .type =3D ARM64_CPUCAP_SYSTEM_FEATURE, + .capability =3D ARM64_HACDBS, + .matches =3D has_cpuid_feature, + ARM64_CPUID_FIELDS(ID_AA64MMFR4_EL1, HACDBS, IMP) + }, #ifdef CONFIG_ARM64_HAFT { .desc =3D "Hardware managed Access Flag for Table Descriptors", /* * Contrary to the page/block access flag, the table access flag * cannot be emulated in software (no access fault will occur). * Therefore this should be used only if it's supported system * wide. */ .type =3D ARM64_CPUCAP_SYSTEM_FEATURE, diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index 6bd563e0bc62..cd53cec79d34 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -65,20 +65,21 @@ HAS_STAGE2_FWB HAS_TCR2 HAS_TIDCP1 HAS_TLB_RANGE HAS_VA52 HAS_VIRT_HOST_EXTN HAS_WFXT HAS_XNX HAS_HDBSS HAFT HW_DBM +HACDBS KVM_HVHE KVM_PROTECTED_MODE MISMATCHED_CACHE_TYPE MPAM MPAM_HCR MTE MTE_ASYMM MTE_FAR MTE_STORE_ONLY SME --=20 2.54.0 From nobody Tue Jun 16 16:52:56 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 557BA407561; Thu, 30 Apr 2026 11:15:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547739; cv=none; b=WVynXd0jRE62/0LH+3otK7b3583eHGZX36JM9XwBaiklria6ZwdEsJRAbpJfQas8cOs1+PfGrC6KOF+CuM/+qpT5gk9cUWtAcaeZbfcbVxkdUWthmz1NElwEeWC9Y4tQSRuO27lBi3QvCxJnFhcAoPysrMPV4IA50Pf+sHFrD2E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547739; c=relaxed/simple; bh=gfrKgtXGA9jbSPGpNbN2YuT5jDSPo32v7BItwOrMJ6I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eeihLc7LQP/521FpXvmKIJ+7axO1DPcelrows1mPJXD6TGwTrXuP7Pg0Mnu5DnWb16sfDduA/tCrzcmhOCgK/9vEe7gR1vZMRkZbl7tDqBcugenRxVlI0dxWudAPfogrZzbLGSy7OAyLfwH5JpSl5f2tMYJxbCCFEkDnZD/ltVo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=WzcC5e8a; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="WzcC5e8a" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 66D743547; Thu, 30 Apr 2026 04:15:32 -0700 (PDT) Received: from devkitleo.cambridge.arm.com (devkitleo.cambridge.arm.com [10.1.196.90]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6D7723F763; Thu, 30 Apr 2026 04:15:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1777547737; bh=gfrKgtXGA9jbSPGpNbN2YuT5jDSPo32v7BItwOrMJ6I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WzcC5e8aL5JvmjSDzEtK6LU52kY92jTkYQUdQus06jIGlfM2QzMb1ZF50zwPdtHdL txKodUqFM+OA+K3tdIslPl74Yrqp/NKOHH7S93Z40+fND3y8Bh4n3qFD5GMF/+yccX eXXg/NptcueB0TnV/K3ziiwsW/6AGDXQtzK4+KYw= From: Leonardo Bras To: Catalin Marinas , Will Deacon , Leonardo Bras , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Zenghui Yu , "Rafael J. Wysocki" , Len Brown , Saket Dumbre , Paolo Bonzini , Chengwen Feng , Jonathan Cameron , Kees Cook , =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , Ryan Roberts , Yang Shi , Thomas Huth , mrigendrachaubey , Yeoreum Yun , Mark Brown , Kevin Brodsky , James Clark , Ard Biesheuvel , Fuad Tabba , Raghavendra Rao Ananta , Nathan Chancellor , Vincent Donnefort , Lorenzo Pieralisi , Sascha Bischoff , Anshuman Khandual , Tian Zheng , Wei-Lin Chang Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-acpi@vger.kernel.org, acpica-devel@lists.linux.dev, kvm@vger.kernel.org Subject: [PATCH v1 04/12] arm64/sysreg: Add HACDBS consumer and base registers Date: Thu, 30 Apr 2026 12:14:08 +0100 Message-ID: <20260430111424.3479613-6-leo.bras@arm.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260430111424.3479613-2-leo.bras@arm.com> References: <20260430111424.3479613-2-leo.bras@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" They will be used on a later commit to make use of the FEAT_HACDBS mechanism if available. Signed-off-by: Leonardo Bras Reviewed-by: Mark Brown --- arch/arm64/tools/sysreg | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg index 6c3ff14e561e..7ad392a5c436 100644 --- a/arch/arm64/tools/sysreg +++ b/arch/arm64/tools/sysreg @@ -4582,20 +4582,50 @@ EndEnum Enum 9:8 IRGN0 0b00 NC 0b01 WBWA 0b10 WT 0b11 WBnWA EndEnum Field 7:6 SL0 Field 5:0 T0SZ EndSysreg =20 +Sysreg HACDBSBR_EL2 3 4 2 3 4 +Res0 63:56 +Field 55:12 BADDR +Field 11 EN +Res0 10:4 +UnsignedEnum 3:0 SZ + 0b0000 4K + 0b0001 8K + 0b0010 16K + 0b0011 32K + 0b0100 64K + 0b0101 128K + 0b0110 256K + 0b0111 512K + 0b1000 1M + 0b1001 2M +EndEnum +EndSysreg + +Sysreg HACDBSCONS_EL2 3 4 2 3 5 +UnsignedEnum 63:62 ERR_REASON + 0b00 NOF + 0b01 STRUCTF + 0b10 IPAF + 0b11 IPAHACF +EndEnum +Res0 61:19 +Field 18:0 INDEX +EndSysreg + Sysreg GCSCR_EL2 3 4 2 5 0 Fields GCSCR_ELx EndSysreg =20 Sysreg GCSPR_EL2 3 4 2 5 1 Fields GCSPR_ELx EndSysreg =20 Sysreg DACR32_EL2 3 4 3 0 0 Res0 63:32 --=20 2.54.0 From nobody Tue Jun 16 16:52:56 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id EBBE340758E; Thu, 30 Apr 2026 11:15:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547744; cv=none; b=RS/cTPVuRl3uFTR/qayBTugywQPfnpmwzX4CLbxaScazAQwCWF7kZg9OjvHF6hIu/KB35okJVqWmR9mgGdTSBNpC9DEZKl39bwUZ3mOk7AMQsw0G62178PXdIKhKDSl3QAx9t9GLfKoi4WB3cW2oV4clihI4gog9D7xrqkGvoo0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547744; c=relaxed/simple; bh=RnNmUi25WQATOqzbYmUHJ+Iu1/LzjMsiRxtUGsjdP1w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Tpp9GhYHI5GUietv0vxHBMlEDTvzSoEFD7kbIVlAwofFPzSV+YeHdkntYUbEHzFKXhLsbPKDJxBotWt4UpARxRUYj5RBHRqyasP3RJ51zPMFLWiKdGrbeSKtG9PdHX2A08CkXg/9Ir0q8Nuun20FzqEEBVc8B+g9Et9GVirm9FA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=Q2fZJxgK; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="Q2fZJxgK" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 21BC63548; Thu, 30 Apr 2026 04:15:37 -0700 (PDT) Received: from devkitleo.cambridge.arm.com (devkitleo.cambridge.arm.com [10.1.196.90]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2586A3F763; Thu, 30 Apr 2026 04:15:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1777547742; bh=RnNmUi25WQATOqzbYmUHJ+Iu1/LzjMsiRxtUGsjdP1w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Q2fZJxgKC25GnHizrIMLhZj3kwBhkBrD8tX9MUtFbGt/IWt6VDDhoK4ndFnTDnnr4 zl2ZLEnjJCPesxZXgJkfHsSutOVGwe+r5zfdHTJUvOyryLjhUM9zi3pAQeYvFUbkUp ZpVe3mGYnnCVhr9y413ng7uqhrYXCsxTLlTUFxbs= From: Leonardo Bras To: Catalin Marinas , Will Deacon , Leonardo Bras , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Zenghui Yu , "Rafael J. Wysocki" , Len Brown , Saket Dumbre , Paolo Bonzini , Chengwen Feng , Jonathan Cameron , Kees Cook , =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , Ryan Roberts , Yang Shi , Thomas Huth , mrigendrachaubey , Yeoreum Yun , Mark Brown , Kevin Brodsky , James Clark , Ard Biesheuvel , Fuad Tabba , Raghavendra Rao Ananta , Nathan Chancellor , Vincent Donnefort , Lorenzo Pieralisi , Sascha Bischoff , Anshuman Khandual , Tian Zheng , Wei-Lin Chang Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-acpi@vger.kernel.org, acpica-devel@lists.linux.dev, kvm@vger.kernel.org Subject: [PATCH v1 05/12] KVM: arm64: Detect (via ACPI) and initialize HACDBSIRQ Date: Thu, 30 Apr 2026 12:14:09 +0100 Message-ID: <20260430111424.3479613-7-leo.bras@arm.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260430111424.3479613-2-leo.bras@arm.com> References: <20260430111424.3479613-2-leo.bras@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Find via ACPI [1] the Id for HACDBSIRQ, initialize it as a per-cpu IRQ and make sure any cpu able to run virtualization has it active. Introduce a per-cpu structure used by the HACDBSIRQ handler to keep track of entries size and the status of HACDBS. Size is used to detect end of processing in case the number of entries being processed is different of the supported entries size. Status may look easily replaceable by checking HACDBS registers now, but will make the OFF/IDLE detection easier in next patches. Signed-off-by: Leonardo Bras [1] https://github.com/tianocore/edk2/issues/12409 --- arch/arm64/include/asm/acpi.h | 3 + arch/arm64/include/asm/kvm_dirty_bit.h | 18 +++++ include/acpi/actbl2.h | 1 + arch/arm64/kvm/arm.c | 5 ++ arch/arm64/kvm/dirty_bit.c | 97 ++++++++++++++++++++++++++ 5 files changed, 124 insertions(+) diff --git a/arch/arm64/include/asm/acpi.h b/arch/arm64/include/asm/acpi.h index 8a54ca6ba602..883315e9d79d 100644 --- a/arch/arm64/include/asm/acpi.h +++ b/arch/arm64/include/asm/acpi.h @@ -38,20 +38,23 @@ =20 #define BAD_MADT_GICC_ENTRY(entry, end) \ (!(entry) || (entry)->header.length < ACPI_MADT_GICC_MIN_LENGTH || \ (unsigned long)(entry) + (entry)->header.length > (end)) =20 #define ACPI_MADT_GICC_SPE (offsetof(struct acpi_madt_generic_interrupt, \ spe_interrupt) + sizeof(u16)) =20 #define ACPI_MADT_GICC_TRBE (offsetof(struct acpi_madt_generic_interrupt,= \ trbe_interrupt) + sizeof(u16)) + +#define ACPI_MADT_GICC_HACDBSIRQ (offsetof(struct acpi_madt_generic_inter= rupt, \ + hacdbsirq_gsi) + sizeof(u32)) /* * Arm=C2=AE Functional Fixed Hardware Specification Version 1.2. * Table 2: Arm Architecture context loss flags */ #define CPUIDLE_CORE_CTXT BIT(0) /* Core context Lost */ =20 static inline unsigned int arch_get_idle_state_flags(u32 arch_flags) { if (arch_flags & CPUIDLE_CORE_CTXT) return CPUIDLE_FLAG_TIMER_STOP; diff --git a/arch/arm64/include/asm/kvm_dirty_bit.h b/arch/arm64/include/as= m/kvm_dirty_bit.h index dd16438f0651..904e59f95b7e 100644 --- a/arch/arm64/include/asm/kvm_dirty_bit.h +++ b/arch/arm64/include/asm/kvm_dirty_bit.h @@ -2,11 +2,29 @@ /* * Copyright (C) 2026 ARM Ltd. * Author: Leonardo Bras */ =20 #ifndef __ARM64_KVM_DIRTY_BIT_H__ #define __ARM64_KVM_DIRTY_BIT_H__ =20 #include =20 +enum hacdbs_status { + HACDBS_OFF, + HACDBS_IDLE, + HACDBS_RUNNING, + HACDBS_ERROR +}; + +struct hacdbs { + enum hacdbs_status status; + int size; +}; + +DECLARE_PER_CPU(struct hacdbs, hacdbs_pcp); + +void __init kvm_hacdbs_init(void); +void kvm_hacdbs_cpu_up(void); +void kvm_hacdbs_cpu_down(void); + #endif /* __ARM64_KVM_DIRTY_BIT_H__ */ diff --git a/include/acpi/actbl2.h b/include/acpi/actbl2.h index 5c0b55e7b3e4..96b664dde9c2 100644 --- a/include/acpi/actbl2.h +++ b/include/acpi/actbl2.h @@ -1442,20 +1442,21 @@ struct acpi_madt_generic_interrupt { u64 gich_base_address; u32 vgic_interrupt; u64 gicr_base_address; u64 arm_mpidr; u8 efficiency_class; u8 reserved2[1]; u16 spe_interrupt; /* ACPI 6.3 */ u16 trbe_interrupt; /* ACPI 6.5 */ u16 iaffid; /* ACPI 6.7 */ u32 irs_id; + u32 hacdbsirq_gsi; /* ACPI 6.X */ }; =20 /* Masks for Flags field above */ =20 /* ACPI_MADT_ENABLED (1) Processor is usable if se= t */ #define ACPI_MADT_PERFORMANCE_IRQ_MODE (1<<1) /* 01: Performance Interrup= t Mode */ #define ACPI_MADT_VGIC_IRQ_MODE (1<<2) /* 02: VGIC Maintenance Int= errupt mode */ #define ACPI_MADT_GICC_ONLINE_CAPABLE (1<<3) /* 03: Processor is online = capable */ #define ACPI_MADT_GICC_NON_COHERENT (1<<4) /* 04: GIC redistributor is= not coherent */ =20 diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 176cbe8baad3..aae230c33c41 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -34,20 +34,21 @@ #include #include #include #include #include #include #include #include #include #include +#include #include #include =20 #include #include #include #include =20 #include =20 @@ -2278,28 +2279,30 @@ int kvm_arch_enable_virtualization_cpu(void) * disabled, but not with preemption disabled. The former is * enough to ensure correctness, but most of the helpers * expect the later and will throw a tantrum otherwise. */ preempt_disable(); =20 cpu_hyp_init(NULL); =20 kvm_vgic_cpu_up(); kvm_timer_cpu_up(); + kvm_hacdbs_cpu_up(); =20 preempt_enable(); =20 return 0; } =20 void kvm_arch_disable_virtualization_cpu(void) { + kvm_hacdbs_cpu_down(); kvm_timer_cpu_down(); kvm_vgic_cpu_down(); =20 if (!is_protected_kvm_enabled()) cpu_hyp_uninit(NULL); } =20 #ifdef CONFIG_CPU_PM static int hyp_init_cpu_pm_notifier(struct notifier_block *self, unsigned long cmd, @@ -2450,20 +2453,22 @@ static int __init init_subsystems(void) goto out; } =20 /* * Init HYP architected timer support */ err =3D kvm_timer_hyp_init(vgic_present); if (err) goto out; =20 + kvm_hacdbs_init(); + kvm_register_perf_callbacks(); =20 err =3D kvm_hyp_trace_init(); if (err) kvm_err("Failed to initialize Hyp tracing\n"); =20 out: if (err) hyp_cpu_pm_exit(); =20 diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c index 31ef35f3b72f..765ef609ff70 100644 --- a/arch/arm64/kvm/dirty_bit.c +++ b/arch/arm64/kvm/dirty_bit.c @@ -1,23 +1,120 @@ // SPDX-License-Identifier: GPL-2.0-only /* * Copyright (C) 2026 ARM Ltd. * Author: Leonardo Bras */ =20 #include +#include +#include + +DEFINE_PER_CPU(struct hacdbs, hacdbs_pcp) =3D { + .status =3D HACDBS_OFF, + .size =3D 0, +}; =20 /* HDBSS entry field definitions */ #define HDBSS_ENTRY_VALID BIT(0) #define HDBSS_ENTRY_TTWL_SHIFT (1) #define HDBSS_ENTRY_TTWL_MASK (GENMASK(3, 1)) #define HDBSS_ENTRY_TTWL(x) \ (((x) << HDBSS_ENTRY_TTWL_SHIFT) & HDBSS_ENTRY_TTWL_MASK) #define HDBSS_ENTRY_TTWL_RESV HDBSS_ENTRY_TTWL(-4) #define HDBSS_ENTRY_IPA GENMASK_ULL(55, 12) =20 inline u64 hdbss_get_ttwl(u64 chunk_size) { u64 hw_lvl =3D ARM64_HW_PGTABLE_LEVELS(ilog2(chunk_size)); =20 return HDBSS_ENTRY_TTWL(3 - hw_lvl); } + +static __ro_after_init int hacdbsirq =3D -1; + +static irqreturn_t hacdbsirq_handler(int irq, void *pcpu) +{ + u64 cons =3D read_sysreg_s(SYS_HACDBSCONS_EL2); + unsigned long err =3D FIELD_GET(HACDBSCONS_EL2_ERR_REASON, cons); + + switch (err) { + case HACDBSCONS_EL2_ERR_REASON_NOF: + this_cpu_write(hacdbs_pcp.status, HACDBS_IDLE); + break; + case HACDBSCONS_EL2_ERR_REASON_IPAHACF: + /* When size not a power of two >=3D 4k, exit with reserved TTLW */ + int index =3D FIELD_GET(HACDBSCONS_EL2_INDEX, cons); + + if (index >=3D this_cpu_read(hacdbs_pcp.size)) { + this_cpu_write(hacdbs_pcp.status, HACDBS_IDLE); + break; + } + fallthrough; + case HACDBSCONS_EL2_ERR_REASON_STRUCTF: + case HACDBSCONS_EL2_ERR_REASON_IPAF: + this_cpu_write(hacdbs_pcp.status, HACDBS_ERROR); + break; + } + + return IRQ_HANDLED; +} + +void kvm_hacdbs_cpu_up(void) +{ + if (hacdbsirq < 0) + return; + + enable_percpu_irq(hacdbsirq, IRQ_TYPE_LEVEL_HIGH); + this_cpu_write(hacdbs_pcp.status, HACDBS_IDLE); +} + +void kvm_hacdbs_cpu_down(void) +{ + if (hacdbsirq < 0) + return; + + disable_percpu_irq(hacdbsirq); + this_cpu_write(hacdbs_pcp.status, HACDBS_OFF); +} + +#ifdef CONFIG_ACPI +static int __init hacdbs_acpi_get_irq(void) +{ + struct acpi_madt_generic_interrupt *gicc; + u32 gsi; + int irq; + + gicc =3D acpi_cpu_get_madt_gicc(smp_processor_id()); + if (gicc->header.length < ACPI_MADT_GICC_HACDBSIRQ) + return -ENXIO; + + gsi =3D gicc->hacdbsirq_gsi; + + irq =3D acpi_register_gsi(NULL, gsi, ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_HI= GH); + if (irq < 0) { + pr_warn("ACPI: Unable to register HACDBS interrupt: %d\n", gsi); + return -ENXIO; + } + + return irq; +} +#else +#define hacdbs_acpi_get_irq() (-ENXIO) +#endif + +void __init kvm_hacdbs_init(void) +{ + int irq; + + /* FEAT_HACDBS is only supported if Linux runs in EL2 (VHE) */ + if (!system_supports_hacdbs() || !is_kernel_in_hyp_mode()) + return; + + irq =3D hacdbs_acpi_get_irq(); + if (irq < 0) + return; + + if (request_percpu_irq(irq, hacdbsirq_handler, "HACDBSIRQ", &hacdbs_pcp) = < 0) + return; + + hacdbsirq =3D irq; +} --=20 2.54.0 From nobody Tue Jun 16 16:52:56 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B39E840B6E4; Thu, 30 Apr 2026 11:15:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547749; cv=none; b=OXGF9Nsim5MZ5dVmptGLqRO439zsimZSJ5B8r/PpLrK1sHamy5oC70SDhtVBpcXtJUYiMjATY5rxOsVmHEn86MscpcnslUgR3zWLNlom/V7BATOicQw1835M4KMMH+WfZkCWuzQpl5Pwa76m+zx2ghlhwyArpjdgDTv7vEtmbUc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547749; c=relaxed/simple; bh=wZ+B2i5T+YTs9KjTiOk5jbT7BSpcv7bgMbQ7JYPWchg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pOjlG085rvwzF+OBaaPIuOHIs5ZxNfURVXfy6VvlWPaUK+uhbJg++E25F4zK9UqSj25LungZbZKrQaVV3frLBcTHtHf8TGr3Xe5q3ovO4hm6nlBfkXkumciy2sMp36ISreUahI4mbOPVqJBSw3kdXPcpW3q2oMsIBVG63XCJ2zo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=jjmBGAm/; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="jjmBGAm/" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C9EEF354E; Thu, 30 Apr 2026 04:15:41 -0700 (PDT) Received: from devkitleo.cambridge.arm.com (devkitleo.cambridge.arm.com [10.1.196.90]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D2D6D3F763; Thu, 30 Apr 2026 04:15:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1777547747; bh=wZ+B2i5T+YTs9KjTiOk5jbT7BSpcv7bgMbQ7JYPWchg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jjmBGAm/d8papjvaMBr0lo1UQvwyvDA/gKC4gwQknxiPPsQn2EBUvTJPXBWAI32bM XtnLw5KnXMhUvswEeFrI9/QT4uAVUsh+3jiUnpbhE9VQnwpNs4JG6GBaGbKQmbozQx M77mFI3b1VeX3w2ftIGOnqnDj+1UnWQrtgz5Tl5A= From: Leonardo Bras To: Catalin Marinas , Will Deacon , Leonardo Bras , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Zenghui Yu , "Rafael J. Wysocki" , Len Brown , Saket Dumbre , Paolo Bonzini , Chengwen Feng , Jonathan Cameron , Kees Cook , =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , Ryan Roberts , Yang Shi , Thomas Huth , mrigendrachaubey , Yeoreum Yun , Mark Brown , Kevin Brodsky , James Clark , Ard Biesheuvel , Fuad Tabba , Raghavendra Rao Ananta , Nathan Chancellor , Vincent Donnefort , Lorenzo Pieralisi , Sascha Bischoff , Anshuman Khandual , Tian Zheng , Wei-Lin Chang Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-acpi@vger.kernel.org, acpica-devel@lists.linux.dev, kvm@vger.kernel.org Subject: [PATCH v1 06/12] KVM: arm64: dirty_bit: Add base FEAT_HACDBS cleaning routine Date: Thu, 30 Apr 2026 12:14:10 +0100 Message-ID: <20260430111424.3479613-8-leo.bras@arm.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260430111424.3479613-2-leo.bras@arm.com> References: <20260430111424.3479613-2-leo.bras@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce the basic cleaning routine that is going to be used for both dirty-bitmap and dirty-ring routines. It sets the required registers with the input buffer, and wait for HACDBSIRQ to happen, which means either the task is done, or there was some error during processing. It is ran with preemption disabled, as a task being scheduled in could change the translation registers used by HACDBS and end up corrupting the current dirty-bit tracking and the sched-in task's S2 pagetables. Signed-off-by: Leonardo Bras --- arch/arm64/kvm/dirty_bit.c | 86 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c index 765ef609ff70..22e3ed07256a 100644 --- a/arch/arm64/kvm/dirty_bit.c +++ b/arch/arm64/kvm/dirty_bit.c @@ -1,17 +1,18 @@ // SPDX-License-Identifier: GPL-2.0-only /* * Copyright (C) 2026 ARM Ltd. * Author: Leonardo Bras */ =20 #include +#include #include #include =20 DEFINE_PER_CPU(struct hacdbs, hacdbs_pcp) =3D { .status =3D HACDBS_OFF, .size =3D 0, }; =20 /* HDBSS entry field definitions */ #define HDBSS_ENTRY_VALID BIT(0) @@ -24,20 +25,105 @@ DEFINE_PER_CPU(struct hacdbs, hacdbs_pcp) =3D { =20 inline u64 hdbss_get_ttwl(u64 chunk_size) { u64 hw_lvl =3D ARM64_HW_PGTABLE_LEVELS(ilog2(chunk_size)); =20 return HDBSS_ENTRY_TTWL(3 - hw_lvl); } =20 static __ro_after_init int hacdbsirq =3D -1; =20 +static void hacdbs_start(u64 *hw_entries, int size) +{ + u64 br; + /* Each entry is 8 bytes */ + int size_b =3D size * sizeof(hw_entries[0]); + int size_p2 =3D max(roundup_pow_of_two(size_b), PAGE_SIZE); + + /* If not using the full size of the array, put a stop entry at the end */ + if (size_b < size_p2) + hw_entries[size] =3D HDBSS_ENTRY_VALID | HDBSS_ENTRY_TTWL_RESV; + + sysreg_clear_set_s(SYS_HACDBSCONS_EL2, + HACDBSCONS_EL2_ERR_REASON | HACDBSCONS_EL2_INDEX, 0); + + br =3D (virt_to_phys(hw_entries) & HACDBSBR_EL2_BADDR_MASK) | + FIELD_PREP(HACDBSBR_EL2_SZ, ilog2(size_p2) - 12) | + FIELD_PREP(HACDBSBR_EL2_EN, 1); + + this_cpu_write(hacdbs_pcp.status, HACDBS_RUNNING); + this_cpu_write(hacdbs_pcp.size, size); + write_sysreg_s(br, SYS_HACDBSBR_EL2); + isb(); +} + +static int hacdbs_stop(void) +{ + write_sysreg_s(0, SYS_HACDBSBR_EL2); + isb(); + + if (this_cpu_read(hacdbs_pcp.status) =3D=3D HACDBS_ERROR) { + /* In case of error, HACDBSCONS_EL2.INDEX should point the faulty entry = */ + u64 cons =3D read_sysreg_s(SYS_HACDBSCONS_EL2); + int idx =3D FIELD_GET(HACDBSCONS_EL2_INDEX, cons); + + trace_printk("HACDBS found error %lu in index %d / %d\n", + FIELD_GET(HACDBSCONS_EL2_ERR_REASON, cons), idx, + this_cpu_read(hacdbs_pcp.size)); + + this_cpu_write(hacdbs_pcp.status, HACDBS_IDLE); + + return idx; + } + + return this_cpu_read(hacdbs_pcp.size); +} + +/* + * Clears dirty-bits for an array of pages (hw_entries) using HACDBS + * Returns the number of items cleaned from the array. If returns value < = size, + * there was an error in the processing. + */ +static int dirty_bit_clear(struct kvm *kvm, u64 *hw_entries, int size) +{ + enum hacdbs_status st; + u64 hcr_el2; + int ret; + + preempt_disable(); + + hcr_el2 =3D read_sysreg(HCR_EL2); + write_sysreg(hcr_el2 | HCR_EL2_VM, HCR_EL2); + __load_stage2(&kvm->arch.mmu, kvm->arch.mmu.arch); + + hacdbs_start(hw_entries, size); + + do { + wfi(); + } while ((st =3D this_cpu_read(hacdbs_pcp.status)) =3D=3D HACDBS_RUNNING); + + ret =3D hacdbs_stop(); + + write_sysreg(hcr_el2, HCR_EL2); + isb(); + + /* + * No DSB is needed here, as kvm_flush_remote_tlbs_memslot() that happens + * later in generic dirty-cleaning code already performs a DSB before + * doing the TLBI. + */ + + preempt_enable(); + + return ret; +} + static irqreturn_t hacdbsirq_handler(int irq, void *pcpu) { u64 cons =3D read_sysreg_s(SYS_HACDBSCONS_EL2); unsigned long err =3D FIELD_GET(HACDBSCONS_EL2_ERR_REASON, cons); =20 switch (err) { case HACDBSCONS_EL2_ERR_REASON_NOF: this_cpu_write(hacdbs_pcp.status, HACDBS_IDLE); break; case HACDBSCONS_EL2_ERR_REASON_IPAHACF: --=20 2.54.0 From nobody Tue Jun 16 16:52:56 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 60E8840F8DD; Thu, 30 Apr 2026 11:15:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547754; cv=none; b=OPNknyO+MUCIZhSTyEnPn0OG/MedXsZSQ9Ek4GIsgxJWXIv/QIbNiGrWpxIzm1NldYEvbXOk6Mh90F62HWAem+ZqfwuSbAwodhkYQefZwRJbBoQYZ0w4qSmAPoXcI8jj1Osb6WryBEya+niNVo098M4SPhfX/h5MjYe2UXcK2uE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547754; c=relaxed/simple; bh=ZkdGKyLMclIRLyhqJWVYz05PFVKmF2W5SXHOKUVIcOA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hRkCHfwgFl8LohGLKMMGROhcNu0bPCfR3I1pEo3cU14rHtUIMADgvQnd4u4WDFDzOr0PuRDdfL0TMEynBIaENg65ol9xsooTxz+Yo6soe0X1w8tfP5/heSDGOV7aKtqBvaa2DbU1V8DyCfanHF9KZUl4Uh5DBJskTMn5fUssWpk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=DTy0btji; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="DTy0btji" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 802A13560; Thu, 30 Apr 2026 04:15:46 -0700 (PDT) Received: from devkitleo.cambridge.arm.com (devkitleo.cambridge.arm.com [10.1.196.90]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8A8073F763; Thu, 30 Apr 2026 04:15:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1777547751; bh=ZkdGKyLMclIRLyhqJWVYz05PFVKmF2W5SXHOKUVIcOA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DTy0btjiqfmFNn4qxNqdiFWNbIgLEI7bpRci9Kvb2fZY44VLTAaUKOwe0pOTWa793 P8WHa0X0PMRXQhJswIXFKvgT75GatY47ZfDzy2oKchbItvbABKfxae2ssSB8YjOcyM FpawrYNUnaFmYrHZaEa/eOsaHxdw20L1bnVSQaic= From: Leonardo Bras To: Catalin Marinas , Will Deacon , Leonardo Bras , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Zenghui Yu , "Rafael J. Wysocki" , Len Brown , Saket Dumbre , Paolo Bonzini , Chengwen Feng , Jonathan Cameron , Kees Cook , =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , Ryan Roberts , Yang Shi , Thomas Huth , mrigendrachaubey , Yeoreum Yun , Mark Brown , Kevin Brodsky , James Clark , Ard Biesheuvel , Fuad Tabba , Raghavendra Rao Ananta , Nathan Chancellor , Vincent Donnefort , Lorenzo Pieralisi , Sascha Bischoff , Anshuman Khandual , Tian Zheng , Wei-Lin Chang Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-acpi@vger.kernel.org, acpica-devel@lists.linux.dev, kvm@vger.kernel.org Subject: [PATCH v1 07/12] kvm: Add arch-generic interface for hw-accelerated dirty-bitmap cleaning Date: Thu, 30 Apr 2026 12:14:11 +0100 Message-ID: <20260430111424.3479613-9-leo.bras@arm.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260430111424.3479613-2-leo.bras@arm.com> References: <20260430111424.3479613-2-leo.bras@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce kvm_arch_dirty_log_clear() that allow implementation of arch-specific hardware-accelerated dirty-log routines. A call to that is added on both kvm_get_dirty_log_protect() and kvm_clear_dirty_log_protect() and will fall back to software version if not implemented, or any error was detected in the arch-specific routine. For an arch to implement this function, it's required to provide an asm/kvm_dirty_bit.h and have CONFIG_HAVE_KVM_HW_DIRTY_BIT=3Dy on building. If the arch does not implement it, and thus lack above config, the introduced snippet is expected to be compiled-out and have zero impact at runtime. Signed-off-by: Leonardo Bras --- include/linux/kvm_dirty_bit.h | 27 +++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 13 ++++++++++++- virt/kvm/Kconfig | 3 +++ 3 files changed, 42 insertions(+), 1 deletion(-) create mode 100644 include/linux/kvm_dirty_bit.h diff --git a/include/linux/kvm_dirty_bit.h b/include/linux/kvm_dirty_bit.h new file mode 100644 index 000000000000..fa4f6b67b623 --- /dev/null +++ b/include/linux/kvm_dirty_bit.h @@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2026 ARM Ltd. + * Author: Leonardo Bras + */ + +#ifndef __KVM_DIRTY_BIT_H__ +#define __KVM_DIRTY_BIT_H__ + +#ifndef CONFIG_HAVE_KVM_HW_DIRTY_BIT + +static inline int kvm_arch_dirty_log_clear(struct kvm *kvm, + struct kvm_memory_slot *memslot, + struct kvm_clear_dirty_log *log, + unsigned long *bitmap, + bool *flush) +{ + return -ENXIO; +} + +#else /* CONFIG_HAVE_KVM_HW_DIRTY_BIT */ + +#include + +#endif /* CONFIG_HAVE_KVM_HW_DIRTY_BIT */ + +#endif /* __KVM_DIRTY_BIT_H__ */ diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 89489996fbc1..7f5048ca9a25 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -58,20 +58,21 @@ #include "async_pf.h" #include "kvm_mm.h" #include "vfio.h" =20 #include =20 #define CREATE_TRACE_POINTS #include =20 #include +#include =20 =20 /* Worst case buffer size needed for holding an integer. */ #define ITOA_MAX_LEN 12 =20 MODULE_AUTHOR("Qumranet"); MODULE_DESCRIPTION("Kernel-based Virtual Machine (KVM) Hypervisor"); MODULE_LICENSE("GPL"); =20 /* Architectures should define their poll value according to the halt late= ncy */ @@ -2255,39 +2256,44 @@ static int kvm_get_dirty_log_protect(struct kvm *kv= m, struct kvm_dirty_log *log) * is some code duplication between this function and * kvm_get_dirty_log, but hopefully all architecture * transition to kvm_get_dirty_log_protect and kvm_get_dirty_log * can be eliminated. */ dirty_bitmap_buffer =3D dirty_bitmap; } else { dirty_bitmap_buffer =3D kvm_second_dirty_bitmap(memslot); memset(dirty_bitmap_buffer, 0, n); =20 + if (kvm_arch_dirty_log_clear(kvm, memslot, NULL, + dirty_bitmap_buffer, &flush) >=3D 0) + goto out; + KVM_MMU_LOCK(kvm); for (i =3D 0; i < n / sizeof(long); i++) { unsigned long mask; gfn_t offset; =20 if (!dirty_bitmap[i]) continue; =20 flush =3D true; mask =3D xchg(&dirty_bitmap[i], 0); dirty_bitmap_buffer[i] =3D mask; =20 offset =3D i * BITS_PER_LONG; kvm_arch_mmu_enable_log_dirty_pt_masked(kvm, memslot, offset, mask); } KVM_MMU_UNLOCK(kvm); } =20 +out: if (flush) kvm_flush_remote_tlbs_memslot(kvm, memslot); =20 if (copy_to_user(log->dirty_bitmap, dirty_bitmap_buffer, n)) return -EFAULT; return 0; } =20 =20 /** @@ -2366,45 +2372,50 @@ static int kvm_clear_dirty_log_protect(struct kvm *= kvm, (log->num_pages < memslot->npages - log->first_page && (log->num_page= s & 63))) return -EINVAL; =20 kvm_arch_sync_dirty_log(kvm, memslot); =20 flush =3D false; dirty_bitmap_buffer =3D kvm_second_dirty_bitmap(memslot); if (copy_from_user(dirty_bitmap_buffer, log->dirty_bitmap, n)) return -EFAULT; =20 + if (kvm_arch_dirty_log_clear(kvm, memslot, log, dirty_bitmap_buffer, + &flush) >=3D 0) + goto out; + KVM_MMU_LOCK(kvm); for (offset =3D log->first_page, i =3D offset / BITS_PER_LONG, n =3D DIV_ROUND_UP(log->num_pages, BITS_PER_LONG); n--; i++, offset +=3D BITS_PER_LONG) { unsigned long mask =3D *dirty_bitmap_buffer++; atomic_long_t *p =3D (atomic_long_t *) &dirty_bitmap[i]; if (!mask) continue; =20 mask &=3D atomic_long_fetch_andnot(mask, p); =20 /* * mask contains the bits that really have been cleared. This * never includes any bits beyond the length of the memslot (if * the length is not aligned to 64 pages), therefore it is not * a problem if userspace sets them in log->dirty_bitmap. */ if (mask) { flush =3D true; + kvm_arch_mmu_enable_log_dirty_pt_masked(kvm, memslot, offset, mask); } } KVM_MMU_UNLOCK(kvm); - +out: if (flush) kvm_flush_remote_tlbs_memslot(kvm, memslot); =20 return 0; } =20 static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm, struct kvm_clear_dirty_log *log) { int r; diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig index 794976b88c6f..f8757b5b84b3 100644 --- a/virt/kvm/Kconfig +++ b/virt/kvm/Kconfig @@ -13,20 +13,23 @@ config HAVE_KVM_PFNCACHE =20 config HAVE_KVM_IRQCHIP bool =20 config HAVE_KVM_IRQ_ROUTING bool =20 config HAVE_KVM_DIRTY_RING bool =20 +config HAVE_KVM_HW_DIRTY_BIT + bool + # Only strongly ordered architectures can select this, as it doesn't # put any explicit constraint on userspace ordering. They can also # select the _ACQ_REL version. config HAVE_KVM_DIRTY_RING_TSO bool select HAVE_KVM_DIRTY_RING depends on X86 =20 # Weakly ordered architectures can only select this, advertising # to userspace the additional ordering requirements. --=20 2.54.0 From nobody Tue Jun 16 16:52:56 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 13F92410D30; Thu, 30 Apr 2026 11:15:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547759; cv=none; b=lpGyBb/3F62ZdgdGSkGxmGSDeySYTpJ9hOvRk6n/5OKd0u21W96SkOipqIXPe+R6ybCEUovlSvHSnaxeIt9YflS9xRFIWHcJ4SIbIs1NTTCK0dnUEKyVIXASJbsN+sLp7g7DNe/DeF9TM0QoeXBZdQTVLrtLTkAWn5Kje6Ld2Rw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547759; c=relaxed/simple; bh=lWLmCZ6IeQYFZCg4if3cctWEygJWfnZ45AE44IeI33I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=isQjex0V5Su1zuhiDJl4rGkWwpH2Ox8olpq/GzUvH4eDr3Oi47NTMZSmVSB7JRxpdLY1P1UWRshRUwvoHwiTsKuSNymi0Bm/qMp2OsLj5demdddDal+OA8cqiV0sFpBQa1BFGr2e8xOUwLocSkM9t248/Fi0OHmW2fikAE1eQkM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=HjdpxJzg; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="HjdpxJzg" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 38D6D3561; Thu, 30 Apr 2026 04:15:51 -0700 (PDT) Received: from devkitleo.cambridge.arm.com (devkitleo.cambridge.arm.com [10.1.196.90]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 428443F763; Thu, 30 Apr 2026 04:15:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1777547756; bh=lWLmCZ6IeQYFZCg4if3cctWEygJWfnZ45AE44IeI33I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HjdpxJzgR00VKAVJXL+cAgBoLcsBry1NPl/IFSgTtUrlE1YG8kHctGics1eljcpnd 1R8qiUgOcK8SvxMrSZr1r76Rxur5WdKSGB3iazxHSyFrEZXy442LK4AfqyJFfR5uMG zXDiQ2CmLtfNO0Almm4kkwKNW+pdhPhch/xzNwRw= From: Leonardo Bras To: Catalin Marinas , Will Deacon , Leonardo Bras , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Zenghui Yu , "Rafael J. Wysocki" , Len Brown , Saket Dumbre , Paolo Bonzini , Chengwen Feng , Jonathan Cameron , Kees Cook , =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , Ryan Roberts , Yang Shi , Thomas Huth , mrigendrachaubey , Yeoreum Yun , Mark Brown , Kevin Brodsky , James Clark , Ard Biesheuvel , Fuad Tabba , Raghavendra Rao Ananta , Nathan Chancellor , Vincent Donnefort , Lorenzo Pieralisi , Sascha Bischoff , Anshuman Khandual , Tian Zheng , Wei-Lin Chang Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-acpi@vger.kernel.org, acpica-devel@lists.linux.dev, kvm@vger.kernel.org Subject: [PATCH v1 08/12] KVM: arm64: Add hardware-accelerated dirty-bitmap cleaning routine Date: Thu, 30 Apr 2026 12:14:12 +0100 Message-ID: <20260430111424.3479613-10-leo.bras@arm.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260430111424.3479613-2-leo.bras@arm.com> References: <20260430111424.3479613-2-leo.bras@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement arm64 version of kvm_arch_dirty_log_clear() making use of FEAT_HACDBS. It works by transversing the dirty-bitmap and converting the set bits into HDBSS entries in a 64-page blocks granularity. The resulting HDBSS array is then fed to the HACDBS mechanism that walks the pagetable marking writable-dirty pages as writable-clean. In case of error, rewrite all unprocessed entries, including the faulting one, to the dirty-bitmap and fall back to generic software cleaning. In case of the options to "manual protect + init set" are enabled, do the hugepage splitting in the same fashion as the generic software cleaning, i.e. in 64-page blocks. For that, remove the static qualifier from kvm_mmu_split_huge_pages() and make the function available on kvm_host.h. Signed-off-by: Leonardo Bras --- arch/arm64/include/asm/kvm_dirty_bit.h | 24 ++++ include/linux/kvm_host.h | 3 + arch/arm64/kvm/dirty_bit.c | 146 +++++++++++++++++++++++++ arch/arm64/kvm/mmu.c | 4 +- 4 files changed, 175 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_dirty_bit.h b/arch/arm64/include/as= m/kvm_dirty_bit.h index 904e59f95b7e..3d749f979c67 100644 --- a/arch/arm64/include/asm/kvm_dirty_bit.h +++ b/arch/arm64/include/asm/kvm_dirty_bit.h @@ -20,11 +20,35 @@ struct hacdbs { enum hacdbs_status status; int size; }; =20 DECLARE_PER_CPU(struct hacdbs, hacdbs_pcp); =20 void __init kvm_hacdbs_init(void); void kvm_hacdbs_cpu_up(void); void kvm_hacdbs_cpu_down(void); =20 +int __kvm_arch_dirty_log_clear(struct kvm *kvm, + struct kvm_memory_slot *memslot, + struct kvm_clear_dirty_log *log, + unsigned long *bitmap, + bool *flush); + +static inline bool kvm_arch_dirty_clear_enabled(struct kvm *kvm) +{ + return this_cpu_read(hacdbs_pcp.status) =3D=3D HACDBS_IDLE && + (kvm->arch.mmu.pgt->flags & KVM_PGTABLE_S2_DBM); +} + +static inline int kvm_arch_dirty_log_clear(struct kvm *kvm, + struct kvm_memory_slot *memslot, + struct kvm_clear_dirty_log *log, + unsigned long *bitmap, + bool *flush) +{ + if (!kvm_arch_dirty_clear_enabled(kvm)) + return -EPERM; + + return __kvm_arch_dirty_log_clear(kvm, memslot, log, bitmap, flush); +} + #endif /* __ARM64_KVM_DIRTY_BIT_H__ */ diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 4c14aee1fb06..5e3a3c484dd4 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1662,20 +1662,23 @@ void kvm_arch_disable_virtualization_cpu(void); bool kvm_vcpu_has_events(struct kvm_vcpu *vcpu); int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu); bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu); int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu); bool kvm_arch_dy_runnable(struct kvm_vcpu *vcpu); bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu); bool kvm_arch_vcpu_preempted_in_kernel(struct kvm_vcpu *vcpu); void kvm_arch_pre_destroy_vm(struct kvm *kvm); void kvm_arch_create_vm_debugfs(struct kvm *kvm); =20 +int kvm_mmu_split_huge_pages(struct kvm *kvm, phys_addr_t addr, + phys_addr_t end); + #ifndef __KVM_HAVE_ARCH_VM_ALLOC /* * All architectures that want to use vzalloc currently also * need their own kvm_arch_alloc_vm implementation. */ static inline struct kvm *kvm_arch_alloc_vm(void) { return kzalloc_obj(struct kvm, GFP_KERNEL_ACCOUNT); } #endif diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c index 22e3ed07256a..0b7dcb8467c0 100644 --- a/arch/arm64/kvm/dirty_bit.c +++ b/arch/arm64/kvm/dirty_bit.c @@ -110,20 +110,166 @@ static int dirty_bit_clear(struct kvm *kvm, u64 *hw_= entries, int size) * No DSB is needed here, as kvm_flush_remote_tlbs_memslot() that happens * later in generic dirty-cleaning code already performs a DSB before * doing the TLBI. */ =20 preempt_enable(); =20 return ret; } =20 +static inline void hdbss_to_bitmap(u64 *hdbss_array, int start, int end, + unsigned long *dirty_bitmap, + unsigned long long offset) +{ + u64 w =3D (gpa_to_gfn(hdbss_array[start]) - offset) / BITS_PER_LONG; + u64 mask =3D 0; + int idx =3D start; + + do { + u64 entry =3D (gpa_to_gfn(hdbss_array[idx]) - offset); + + if (entry / BITS_PER_LONG =3D=3D w) { + mask |=3D BIT(entry % BITS_PER_LONG); + } else { + atomic_long_or(mask, (atomic_long_t *)&dirty_bitmap[w]); + w =3D entry / BITS_PER_LONG; + mask =3D BIT(entry % BITS_PER_LONG); + } + } while (++idx < end); + atomic_long_or(mask, (atomic_long_t *)&dirty_bitmap[w]); +} + +static inline int mask_to_hdbss(unsigned long *mask, u64 *hw_entries, cons= t gfn_t offset, + u64 ttwl, int idx, int entries_sz) +{ + while (idx < entries_sz) { + int j =3D __ffs(*mask); + u64 a =3D gfn_to_gpa(offset + j); + + hw_entries[idx++] =3D (a & HDBSS_ENTRY_IPA) | + ttwl | + HDBSS_ENTRY_VALID; + + *mask &=3D ~BIT(j); + if (!*mask) + break; + } + + return idx; +} + +int __kvm_arch_dirty_log_clear(struct kvm *kvm, + struct kvm_memory_slot *memslot, + struct kvm_clear_dirty_log *log, + unsigned long *bitmap, + bool *flush) +{ + int ret =3D 0; + int idx =3D 0; + unsigned long *dirty_bitmap =3D memslot->dirty_bitmap; + u64 *hw_entries; + const int entries_sz =3D PAGE_SIZE / sizeof(*hw_entries); + u64 ttwl; + u64 start, end; + gfn_t base_gfn; + + hw_entries =3D kmalloc_objs(u64, entries_sz, GFP_KERNEL); + if (!hw_entries) + return -ENOMEM; + + ttwl =3D hdbss_get_ttwl(kvm->arch.mmu.split_page_chunk_size); + + if (log) { + start =3D log->first_page / BITS_PER_LONG; + end =3D start + DIV_ROUND_UP(log->num_pages, BITS_PER_LONG); + base_gfn =3D memslot->base_gfn + log->first_page % BITS_PER_LONG; + } else { + start =3D 0; + end =3D kvm_dirty_bitmap_bytes(memslot) / sizeof(long); + base_gfn =3D memslot->base_gfn; + } + + write_lock(&kvm->mmu_lock); + + for (unsigned long i =3D start; i < end; i++) { + unsigned long mask; + gfn_t offset; + atomic_long_t *p; + + if (log) { /* Clean only what is in the input bitmap */ + mask =3D bitmap[i]; + if (!mask) + continue; + + p =3D (atomic_long_t *)&dirty_bitmap[i]; + mask &=3D atomic_long_fetch_andnot(mask, p); + } else { /* Clean everything */ + if (!dirty_bitmap[i]) + continue; + + mask =3D xchg(&dirty_bitmap[i], 0); + bitmap[i] =3D mask; + } + + if (!mask) + continue; + + offset =3D base_gfn + i * BITS_PER_LONG; + + if (kvm_dirty_log_manual_protect_and_init_set(kvm)) + kvm_mmu_split_huge_pages(kvm, + gfn_to_gpa(offset + __ffs(mask)), + gfn_to_gpa(offset + __fls(mask) + 1)); + + do { + idx =3D mask_to_hdbss(&mask, hw_entries, offset, ttwl, idx, entries_sz); + if (idx >=3D entries_sz) { + ret =3D dirty_bit_clear(kvm, hw_entries, idx); + *flush =3D *flush || ret > 0; + if (ret !=3D idx) { + /* Save bits not converted back to bitmap */ + atomic_long_or(mask, (atomic_long_t *)&dirty_bitmap[i]); + goto out_err; + } + idx =3D 0; + } + } while (mask); + } + + if (idx !=3D 0) { + ret =3D dirty_bit_clear(kvm, hw_entries, idx); + *flush =3D *flush || ret > 0; + } +out_err: + if (unlikely(ret !=3D idx)) { + /* + * In case there is an error and not all entries in HACDBS get + * cleaned, we have to mark the dirty bits back in the bitmap, + * as that will be used by the software routine. + * + * Entries should be in order, since they were extraxed from + * the dirty-bitmap, so batching the atomic writes is efficient. + */ + + if (ret < idx) + hdbss_to_bitmap(hw_entries, ret, idx, dirty_bitmap, memslot->base_gfn); + + ret =3D -EAGAIN; + } + + write_unlock(&kvm->mmu_lock); + kfree(hw_entries); + + return ret; +} + static irqreturn_t hacdbsirq_handler(int irq, void *pcpu) { u64 cons =3D read_sysreg_s(SYS_HACDBSCONS_EL2); unsigned long err =3D FIELD_GET(HACDBSCONS_EL2_ERR_REASON, cons); =20 switch (err) { case HACDBSCONS_EL2_ERR_REASON_NOF: this_cpu_write(hacdbs_pcp.status, HACDBS_IDLE); break; case HACDBSCONS_EL2_ERR_REASON_IPAHACF: diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 42c734423253..166720f29138 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -110,22 +110,22 @@ static bool need_split_memcache_topup_or_resched(stru= ct kvm *kvm) =20 if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) return true; =20 chunk_size =3D kvm->arch.mmu.split_page_chunk_size; min =3D kvm_mmu_split_nr_page_tables(chunk_size); cache =3D &kvm->arch.mmu.split_page_cache; return kvm_mmu_memory_cache_nr_free_objects(cache) < min; } =20 -static int kvm_mmu_split_huge_pages(struct kvm *kvm, phys_addr_t addr, - phys_addr_t end) +int kvm_mmu_split_huge_pages(struct kvm *kvm, phys_addr_t addr, + phys_addr_t end) { struct kvm_mmu_memory_cache *cache; struct kvm_pgtable *pgt; int ret, cache_capacity; u64 next, chunk_size; =20 lockdep_assert_held_write(&kvm->mmu_lock); =20 chunk_size =3D kvm->arch.mmu.split_page_chunk_size; cache_capacity =3D kvm_mmu_split_nr_page_tables(chunk_size); --=20 2.54.0 From nobody Tue Jun 16 16:52:56 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id BFE4F41B355; Thu, 30 Apr 2026 11:16:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547763; cv=none; b=mdNu7DGVMyFEGMSR7vQF/BrG6B+g7PQpzswCCEfTzL3CrWP1HP4BXt6D/6+xt6xnFuCyIS0sCc4ayuQACcBVqdoMtabTQg7zfobtGAmouT8d0DBisUUGBafLrnM9gnALam2cd1GjG9BBdprZ3gjajcrHS0yr+Fp36IJlIjMItQc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547763; c=relaxed/simple; bh=Zmni17ql/oXEfVlvG4MsyK+AbWgGCuAnsUIgkHp6qlI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZECN+dkb5RwFJWtlfCzAxbjN3x1BClGp5w2KqvhRVV3dERps9EAI3gwJk7v0bsdtVSb3KizQraj6dJrx+8v/9XiPvCXtER3W0t6l3/H3m1RjQ87GaaB+22VXM7616jj51yWiltKZ+xGXL0aUFu1QMXc6UJgfbkKKbEEGUQxVTQw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=K5yulkkv; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="K5yulkkv" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E5BE33569; Thu, 30 Apr 2026 04:15:55 -0700 (PDT) Received: from devkitleo.cambridge.arm.com (devkitleo.cambridge.arm.com [10.1.196.90]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id F0F6D3F763; Thu, 30 Apr 2026 04:15:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1777547761; bh=Zmni17ql/oXEfVlvG4MsyK+AbWgGCuAnsUIgkHp6qlI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=K5yulkkv6ef2FoG2tHcMYukCH1/RbPACOnFFVdnpwDBETk19/K8H8xLDGqjkyTF2j g8nLkN62ZqybnOHXVqrbrNmC5h91CoSZ6GnSiFoS37HlTAujvYD6bu6R9HVMzGXeZE oIcsvLzH/zsKujlc3ZR9HSynF7dpG7wyQP4Pevw4= From: Leonardo Bras To: Catalin Marinas , Will Deacon , Leonardo Bras , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Zenghui Yu , "Rafael J. Wysocki" , Len Brown , Saket Dumbre , Paolo Bonzini , Chengwen Feng , Jonathan Cameron , Kees Cook , =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , Ryan Roberts , Yang Shi , Thomas Huth , mrigendrachaubey , Yeoreum Yun , Mark Brown , Kevin Brodsky , James Clark , Ard Biesheuvel , Fuad Tabba , Raghavendra Rao Ananta , Nathan Chancellor , Vincent Donnefort , Lorenzo Pieralisi , Sascha Bischoff , Anshuman Khandual , Tian Zheng , Wei-Lin Chang Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-acpi@vger.kernel.org, acpica-devel@lists.linux.dev, kvm@vger.kernel.org Subject: [PATCH v1 09/12] kvm/dirty_ring: Introduce get_memslot and move helpers to header Date: Thu, 30 Apr 2026 12:14:13 +0100 Message-ID: <20260430111424.3479613-11-leo.bras@arm.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260430111424.3479613-2-leo.bras@arm.com> References: <20260430111424.3479613-2-leo.bras@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Dirty-ring entry struct carries a slot number which is used to return a struct memslot*. That struct carry important information such as memory offset of that memory slot, which is used to calculate the guest physical address of the page which needs to be marked clean. In order to fetch that memslot information without duplicating code, split that part of kvm_reset_dirty_gfn() into a new helper kvm_dirty_ring_get_memslot(). Along with that, make it available on kvm_dirty_ring.h other helpers such as kvm_dirty_gfn_harvested() and kvm_dirty_gfn_set_invalid() which will be useful on implementing arch specific dirty-ring cleaning accelerators. Signed-off-by: Leonardo Bras --- include/linux/kvm_dirty_ring.h | 12 ++++++++++++ virt/kvm/dirty_ring.c | 30 ++++++++++++++++-------------- 2 files changed, 28 insertions(+), 14 deletions(-) diff --git a/include/linux/kvm_dirty_ring.h b/include/linux/kvm_dirty_ring.h index eb10d87adf7d..190d97fce4a4 100644 --- a/include/linux/kvm_dirty_ring.h +++ b/include/linux/kvm_dirty_ring.h @@ -77,18 +77,30 @@ bool kvm_use_dirty_bitmap(struct kvm *kvm); bool kvm_arch_allow_write_without_running_vcpu(struct kvm *kvm); u32 kvm_dirty_ring_get_rsvd_entries(struct kvm *kvm); int kvm_dirty_ring_alloc(struct kvm *kvm, struct kvm_dirty_ring *ring, int index, u32 size); int kvm_dirty_ring_reset(struct kvm *kvm, struct kvm_dirty_ring *ring, int *nr_entries_reset); void kvm_dirty_ring_push(struct kvm_vcpu *vcpu, u32 slot, u64 offset); =20 bool kvm_dirty_ring_check_request(struct kvm_vcpu *vcpu); =20 +static inline bool kvm_dirty_gfn_harvested(struct kvm_dirty_gfn *gfn) +{ + return smp_load_acquire(&gfn->flags) & KVM_DIRTY_GFN_F_RESET; +} + +static inline void kvm_dirty_gfn_set_invalid(struct kvm_dirty_gfn *gfn) +{ + smp_store_release(&gfn->flags, 0); +} + +struct kvm_memory_slot *kvm_dirty_ring_get_memslot(struct kvm *kvm, u32 sl= ot); + /* for use in vm_operations_struct */ struct page *kvm_dirty_ring_get_page(struct kvm_dirty_ring *ring, u32 offs= et); =20 void kvm_dirty_ring_free(struct kvm_dirty_ring *ring); =20 #endif /* CONFIG_HAVE_KVM_DIRTY_RING */ =20 #endif /* KVM_DIRTY_RING_H */ diff --git a/virt/kvm/dirty_ring.c b/virt/kvm/dirty_ring.c index 02bc6b00d76c..83ac5ac907c1 100644 --- a/virt/kvm/dirty_ring.c +++ b/virt/kvm/dirty_ring.c @@ -43,32 +43,44 @@ static u32 kvm_dirty_ring_used(struct kvm_dirty_ring *r= ing) static bool kvm_dirty_ring_soft_full(struct kvm_dirty_ring *ring) { return kvm_dirty_ring_used(ring) >=3D ring->soft_limit; } =20 static bool kvm_dirty_ring_full(struct kvm_dirty_ring *ring) { return kvm_dirty_ring_used(ring) >=3D ring->size; } =20 -static void kvm_reset_dirty_gfn(struct kvm *kvm, u32 slot, u64 offset, u64= mask) +static inline struct kvm_memory_slot * +__kvm_dirty_ring_get_memslot(struct kvm *kvm, u32 slot) { - struct kvm_memory_slot *memslot; int as_id, id; =20 as_id =3D slot >> 16; id =3D (u16)slot; =20 if (as_id >=3D kvm_arch_nr_memslot_as_ids(kvm) || id >=3D KVM_USER_MEM_SL= OTS) - return; + return 0; =20 - memslot =3D id_to_memslot(__kvm_memslots(kvm, as_id), id); + return id_to_memslot(__kvm_memslots(kvm, as_id), id); +} + +struct kvm_memory_slot *kvm_dirty_ring_get_memslot(struct kvm *kvm, u32 sl= ot) +{ + return __kvm_dirty_ring_get_memslot(kvm, slot); +} + +static void kvm_reset_dirty_gfn(struct kvm *kvm, u32 slot, u64 offset, u64= mask) +{ + struct kvm_memory_slot *memslot; + + memslot =3D __kvm_dirty_ring_get_memslot(kvm, slot); =20 if (!memslot || (offset + __fls(mask)) >=3D memslot->npages) return; =20 KVM_MMU_LOCK(kvm); kvm_arch_mmu_enable_log_dirty_pt_masked(kvm, memslot, offset, mask); KVM_MMU_UNLOCK(kvm); } =20 int kvm_dirty_ring_alloc(struct kvm *kvm, struct kvm_dirty_ring *ring, @@ -80,35 +92,25 @@ int kvm_dirty_ring_alloc(struct kvm *kvm, struct kvm_di= rty_ring *ring, =20 ring->size =3D size / sizeof(struct kvm_dirty_gfn); ring->soft_limit =3D ring->size - kvm_dirty_ring_get_rsvd_entries(kvm); ring->dirty_index =3D 0; ring->reset_index =3D 0; ring->index =3D index; =20 return 0; } =20 -static inline void kvm_dirty_gfn_set_invalid(struct kvm_dirty_gfn *gfn) -{ - smp_store_release(&gfn->flags, 0); -} - static inline void kvm_dirty_gfn_set_dirtied(struct kvm_dirty_gfn *gfn) { gfn->flags =3D KVM_DIRTY_GFN_F_DIRTY; } =20 -static inline bool kvm_dirty_gfn_harvested(struct kvm_dirty_gfn *gfn) -{ - return smp_load_acquire(&gfn->flags) & KVM_DIRTY_GFN_F_RESET; -} - int kvm_dirty_ring_reset(struct kvm *kvm, struct kvm_dirty_ring *ring, int *nr_entries_reset) { /* * To minimize mmu_lock contention, batch resets for harvested entries * whose gfns are in the same slot, and are within N frame numbers of * each other, where N is the number of bits in an unsigned long. For * simplicity, process the current set of entries when the next entry * can't be included in the batch. * --=20 2.54.0 From nobody Tue Jun 16 16:52:56 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7C7E5421880; Thu, 30 Apr 2026 11:16:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547767; cv=none; b=Bp+IpbljJhYsjUnD2Y2892tUktmEuPCWh0w6e4HniEZk5LDTaFVpRV0uXuOELaytJ+hrCiqrslHLmtY9X4mxejsHdfeFjBTFbyhWyTgekyeu5fPeNkGyBIJap3Ez3WictXwLInfVWUSUmmNrwtcGUs9CVHldfX/+sEspyWaqhkk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547767; c=relaxed/simple; bh=q/NKBboAJbYod0gf/Ix7PhIBx5sQObfz8OK5XkSm43U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RiiiIGjo+RDfEKqriyHQGpfnqtQ/tGN+40nmaNJHJSm6BcNZ61mDpo5wiLDib5/ZEjVQaZcnJb1hy2Qz5XPxCNfcYe9vZjoh31vVIs5yo6p/RwXU7bGTp4vhgsdhEcyWM1iDvtVwyzrExUfb5LhY7RHjhlruuAfYJwFl0dRjE6I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=FSbrttDP; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="FSbrttDP" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9D258356D; Thu, 30 Apr 2026 04:16:00 -0700 (PDT) Received: from devkitleo.cambridge.arm.com (devkitleo.cambridge.arm.com [10.1.196.90]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A74FE3F763; Thu, 30 Apr 2026 04:16:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1777547766; bh=q/NKBboAJbYod0gf/Ix7PhIBx5sQObfz8OK5XkSm43U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FSbrttDP37W00c2im+As8lbB9vFtmE6uJ9NzYPqQ3NMp6zj+bbLmMx0UJzC0mQFqd 6KAv8y+ZoToAfNeKB9S9rTApTp0aGj2Ho03zvSWm/l7x3CUSKbsHT2UXzw6nnhNhpZ rxEKsVBwtyHdo0l3m397C4nOzNUMf1dqUCse7okI= From: Leonardo Bras To: Catalin Marinas , Will Deacon , Leonardo Bras , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Zenghui Yu , "Rafael J. Wysocki" , Len Brown , Saket Dumbre , Paolo Bonzini , Chengwen Feng , Jonathan Cameron , Kees Cook , =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , Ryan Roberts , Yang Shi , Thomas Huth , mrigendrachaubey , Yeoreum Yun , Mark Brown , Kevin Brodsky , James Clark , Ard Biesheuvel , Fuad Tabba , Raghavendra Rao Ananta , Nathan Chancellor , Vincent Donnefort , Lorenzo Pieralisi , Sascha Bischoff , Anshuman Khandual , Tian Zheng , Wei-Lin Chang Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-acpi@vger.kernel.org, acpica-devel@lists.linux.dev, kvm@vger.kernel.org Subject: [PATCH v1 10/12] kvm/dirty_ring: Add arch-generic interface for hw-accelerated dirty-ring cleaning Date: Thu, 30 Apr 2026 12:14:14 +0100 Message-ID: <20260430111424.3479613-12-leo.bras@arm.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260430111424.3479613-2-leo.bras@arm.com> References: <20260430111424.3479613-2-leo.bras@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce kvm_arch_dirty_ring_clear() that allow implementation of arch-specific hardware-accelerated dirty-ring routines. A call to that is added on kvm_dirty_ring_reset() and will fall back to software version if not implemented, or any error was detected in the arch-specific routine. For an arch to implement this function, it's required to provide it in a asm/kvm_dirty_bit.h and have CONFIG_HAVE_KVM_HW_DIRTY_BIT=3Dy on building. If the arch does not implement it, and thus lack above config, the introduced snippet is expected to be compiled-out and have zero impact at runtime. Signed-off-by: Leonardo Bras --- include/linux/kvm_dirty_bit.h | 7 +++++++ virt/kvm/dirty_ring.c | 4 ++++ 2 files changed, 11 insertions(+) diff --git a/include/linux/kvm_dirty_bit.h b/include/linux/kvm_dirty_bit.h index fa4f6b67b623..8492979d694e 100644 --- a/include/linux/kvm_dirty_bit.h +++ b/include/linux/kvm_dirty_bit.h @@ -11,17 +11,24 @@ =20 static inline int kvm_arch_dirty_log_clear(struct kvm *kvm, struct kvm_memory_slot *memslot, struct kvm_clear_dirty_log *log, unsigned long *bitmap, bool *flush) { return -ENXIO; } =20 +static inline int kvm_arch_dirty_ring_clear(struct kvm *kvm, + struct kvm_dirty_ring *ring, + int *nr_entries_reset) +{ + return -ENXIO; +} + #else /* CONFIG_HAVE_KVM_HW_DIRTY_BIT */ =20 #include =20 #endif /* CONFIG_HAVE_KVM_HW_DIRTY_BIT */ =20 #endif /* __KVM_DIRTY_BIT_H__ */ diff --git a/virt/kvm/dirty_ring.c b/virt/kvm/dirty_ring.c index 83ac5ac907c1..a58c811dd461 100644 --- a/virt/kvm/dirty_ring.c +++ b/virt/kvm/dirty_ring.c @@ -1,20 +1,21 @@ // SPDX-License-Identifier: GPL-2.0-only /* * KVM dirty ring implementation * * Copyright 2019 Red Hat, Inc. */ #include #include #include #include +#include #include #include "kvm_mm.h" =20 int __weak kvm_cpu_dirty_log_size(struct kvm *kvm) { return 0; } =20 u32 kvm_dirty_ring_get_rsvd_entries(struct kvm *kvm) { @@ -125,20 +126,23 @@ int kvm_dirty_ring_reset(struct kvm *kvm, struct kvm_= dirty_ring *ring, struct kvm_dirty_gfn *entry; =20 /* * Ensure concurrent calls to KVM_RESET_DIRTY_RINGS are serialized, * e.g. so that KVM fully resets all entries processed by a given call * before returning to userspace. Holding slots_lock also protects * the various memslot accesses. */ lockdep_assert_held(&kvm->slots_lock); =20 + if (kvm_arch_dirty_ring_clear(kvm, ring, nr_entries_reset) >=3D 0) + return 0; + while (likely((*nr_entries_reset) < INT_MAX)) { if (signal_pending(current)) return -EINTR; =20 entry =3D &ring->dirty_gfns[ring->reset_index & (ring->size - 1)]; =20 if (!kvm_dirty_gfn_harvested(entry)) break; =20 next_slot =3D READ_ONCE(entry->slot); --=20 2.54.0 From nobody Tue Jun 16 16:52:56 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 40CBE42189A; Thu, 30 Apr 2026 11:16:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547772; cv=none; b=Bp7l4qm20tAOFE9g7pzz0StmRLCSkiBffXNc/usCz/cyyvJqvuYVk9mQVsXeEcvrVIKgYUe/ZiyyFuKHuA7Jf96pxpKZ5h5fu6s+FQ4oh/KqHG9wcFYOlP5VsnLqvkO+a9z9Brxy/vlNe3hhFBknIZ9773ntIKRbFzfd2JVrHz8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547772; c=relaxed/simple; bh=fPffxhpzXsn8izUh0F0BUdbzOcL0eKYDh++M+0U1pJY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Pao3+JBdq4spi4szNQl7KeZzJ8TgS/VQ8jwb9wavFMd+lh3Mscp16pVaamDKU3RKs0F0EqJe9vRFiEaFjDhxQmBjsgGwaE53LPQ9y4uPvJLfKE93HSWO+eVDi6L5LoZzEjBdZRgMS9Hd8yd6svEiq0pSY1VKmeZrc/KAQTLjC/k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=F2Fribgv; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="F2Fribgv" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5AA86356B; Thu, 30 Apr 2026 04:16:05 -0700 (PDT) Received: from devkitleo.cambridge.arm.com (devkitleo.cambridge.arm.com [10.1.196.90]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 5F1B03F763; Thu, 30 Apr 2026 04:16:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1777547770; bh=fPffxhpzXsn8izUh0F0BUdbzOcL0eKYDh++M+0U1pJY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=F2FribgvepEhj4KmtyowEbRPZ1k2UjwoGhsaeKYWQ/KdYmhB4cIuuC7AG87K6o8Kr NwV01a4Jq8sUYRkmJ/N9azU5WKc96Ax+s+LKNDutqQIndBM0DZSPzZ/CizOJBU6yq8 qFSgzNc/18I0HhkgGm5tOqP81PSmbV2v//O2qyYc= From: Leonardo Bras To: Catalin Marinas , Will Deacon , Leonardo Bras , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Zenghui Yu , "Rafael J. Wysocki" , Len Brown , Saket Dumbre , Paolo Bonzini , Chengwen Feng , Jonathan Cameron , Kees Cook , =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , Ryan Roberts , Yang Shi , Thomas Huth , mrigendrachaubey , Yeoreum Yun , Mark Brown , Kevin Brodsky , James Clark , Ard Biesheuvel , Fuad Tabba , Raghavendra Rao Ananta , Nathan Chancellor , Vincent Donnefort , Lorenzo Pieralisi , Sascha Bischoff , Anshuman Khandual , Tian Zheng , Wei-Lin Chang Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-acpi@vger.kernel.org, acpica-devel@lists.linux.dev, kvm@vger.kernel.org Subject: [PATCH v1 11/12] KVM: arm64: Add hardware-accelerated dirty-ring cleaning routine Date: Thu, 30 Apr 2026 12:14:15 +0100 Message-ID: <20260430111424.3479613-13-leo.bras@arm.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260430111424.3479613-2-leo.bras@arm.com> References: <20260430111424.3479613-2-leo.bras@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement arm64 version of kvm_arch_dirty_ring_clear() making use of FEAT_HACDBS. It works by transversing the dirty-ring and converting its entries into HDBSS entries based on the slot offset. The resulting HDBSS array is then fed to the HACDBS mechanism that walks the pagetable marking writable-dirty pages as writable-clean. Only successfully cleaned entries are set as invalid on the dirty-ring, so in case of error, falling back to generic software cleaning will take care of any remaining entry in the dirty-ring. Signed-off-by: Leonardo Bras --- arch/arm64/include/asm/kvm_dirty_bit.h | 13 +++++ arch/arm64/kvm/dirty_bit.c | 66 ++++++++++++++++++++++++++ 2 files changed, 79 insertions(+) diff --git a/arch/arm64/include/asm/kvm_dirty_bit.h b/arch/arm64/include/as= m/kvm_dirty_bit.h index 3d749f979c67..d76c109937d8 100644 --- a/arch/arm64/include/asm/kvm_dirty_bit.h +++ b/arch/arm64/include/asm/kvm_dirty_bit.h @@ -26,29 +26,42 @@ DECLARE_PER_CPU(struct hacdbs, hacdbs_pcp); void __init kvm_hacdbs_init(void); void kvm_hacdbs_cpu_up(void); void kvm_hacdbs_cpu_down(void); =20 int __kvm_arch_dirty_log_clear(struct kvm *kvm, struct kvm_memory_slot *memslot, struct kvm_clear_dirty_log *log, unsigned long *bitmap, bool *flush); =20 +int __kvm_arch_dirty_ring_clear(struct kvm *kvm, struct kvm_dirty_ring *ri= ng, + int *nr_entries_reset); + static inline bool kvm_arch_dirty_clear_enabled(struct kvm *kvm) { return this_cpu_read(hacdbs_pcp.status) =3D=3D HACDBS_IDLE && (kvm->arch.mmu.pgt->flags & KVM_PGTABLE_S2_DBM); } =20 static inline int kvm_arch_dirty_log_clear(struct kvm *kvm, struct kvm_memory_slot *memslot, struct kvm_clear_dirty_log *log, unsigned long *bitmap, bool *flush) { if (!kvm_arch_dirty_clear_enabled(kvm)) return -EPERM; =20 return __kvm_arch_dirty_log_clear(kvm, memslot, log, bitmap, flush); } =20 +static inline int kvm_arch_dirty_ring_clear(struct kvm *kvm, + struct kvm_dirty_ring *ring, + int *nr_entries_reset) +{ + if (!kvm_arch_dirty_clear_enabled(kvm)) + return -EPERM; + + return __kvm_arch_dirty_ring_clear(kvm, ring, nr_entries_reset); +} + #endif /* __ARM64_KVM_DIRTY_BIT_H__ */ diff --git a/arch/arm64/kvm/dirty_bit.c b/arch/arm64/kvm/dirty_bit.c index 0b7dcb8467c0..b99808a36469 100644 --- a/arch/arm64/kvm/dirty_bit.c +++ b/arch/arm64/kvm/dirty_bit.c @@ -256,20 +256,86 @@ int __kvm_arch_dirty_log_clear(struct kvm *kvm, =20 ret =3D -EAGAIN; } =20 write_unlock(&kvm->mmu_lock); kfree(hw_entries); =20 return ret; } =20 +int __kvm_arch_dirty_ring_clear(struct kvm *kvm, struct kvm_dirty_ring *ri= ng, + int *nr_entries_reset) +{ + u64 *hw_entries; + u64 slot_offset =3D 0; + u64 ttwl; + int i, ret; + u32 slot =3D -1; + + if (signal_pending(current)) + return -EINTR; + + ttwl =3D hdbss_get_ttwl(kvm->arch.mmu.split_page_chunk_size); + + hw_entries =3D kmalloc(max(ring->size * sizeof(u64), PAGE_SIZE), GFP_KERN= EL); + if (!hw_entries) + return -ENOMEM; + + for (i =3D 0; i < ring->size; i++) { + struct kvm_dirty_gfn *entry; + gfn_t gfn; + + entry =3D &ring->dirty_gfns[(ring->reset_index + i) & + (ring->size - 1)]; + + if (!kvm_dirty_gfn_harvested(entry)) + break; + + if (entry->slot !=3D slot) { + struct kvm_memory_slot *memslot; + + memslot =3D kvm_dirty_ring_get_memslot(kvm, entry->slot); + slot =3D entry->slot; + slot_offset =3D memslot->base_gfn; + } + + gfn =3D slot_offset + entry->offset; + + hw_entries[i] =3D (gfn_to_gpa(gfn) & HDBSS_ENTRY_IPA) | + ttwl | HDBSS_ENTRY_VALID; + } + + ret =3D dirty_bit_clear(kvm, hw_entries, i); + + /* Set as invalid all successfully cleaned entries */ + for (int j =3D 0; j < ret; j++) { + struct kvm_dirty_gfn *entry; + + entry =3D &ring->dirty_gfns[(ring->reset_index + j) & + (ring->size - 1)]; + + kvm_dirty_gfn_set_invalid(entry); + } + + /* In case of error, try software cleaning from the faulting entry */ + ring->reset_index +=3D ret; + *nr_entries_reset +=3D ret; + + kfree(hw_entries); + + if (ret < i) + return -EFAULT; + + return ret; +} + static irqreturn_t hacdbsirq_handler(int irq, void *pcpu) { u64 cons =3D read_sysreg_s(SYS_HACDBSCONS_EL2); unsigned long err =3D FIELD_GET(HACDBSCONS_EL2_ERR_REASON, cons); =20 switch (err) { case HACDBSCONS_EL2_ERR_REASON_NOF: this_cpu_write(hacdbs_pcp.status, HACDBS_IDLE); break; case HACDBSCONS_EL2_ERR_REASON_IPAHACF: --=20 2.54.0 From nobody Tue Jun 16 16:52:56 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E66BC42189A; Thu, 30 Apr 2026 11:16:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547777; cv=none; b=IduPVITu9xr3GOaIyfKqmh5q6KxP05KgF2aL8ykTKZ8C9I+lzRoHFQVGU7os1WxZoxq4zkLz61Q6l6mizEbg1NIJhRjajc0ojecEsucY8q2MMDJG2mutEMdPJqSoYoUhPCkdrySFNvPZJZn9I5LmHvh5PmizuCQzOXw0eV5Yv+A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777547777; c=relaxed/simple; bh=/RURBevpFF+4tw+ka7Tw3QDbKLsEWw0aWfJtqGVfHwQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZVhPfrKwqGOVMD7s6GxhYxCIp/wB8UeV3h9wgrWXj5nwJOcA7PMd5cW7xMXpePkPwwzKyZIFWW2oD0OjXLmKXiX1gRfjTnzjRJKG37gfAvDA6tdBIkIDM0zaDiDIoL9eH6ljKVOYLGp5ZH/8yMNtBo2dq2tM1cwe8RBUeFQs8/o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=Dg83rIEn; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="Dg83rIEn" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0DC61356C; Thu, 30 Apr 2026 04:16:10 -0700 (PDT) Received: from devkitleo.cambridge.arm.com (devkitleo.cambridge.arm.com [10.1.196.90]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 18E533F763; Thu, 30 Apr 2026 04:16:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1777547775; bh=/RURBevpFF+4tw+ka7Tw3QDbKLsEWw0aWfJtqGVfHwQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Dg83rIEnCnvnNNd90W6UyyYmALfyt5w3xkYcgsMDx7kJQnSLWr+aI1bKWqOcI7gUb f3YA79bgia7x7C+As0zyTHsMVTPe98UnOnrZcgz57AqdzZNuP3ALca1cAQCPqPWKxP gHZ6XNUy68+rW5XuA+jTKVIrUYB6Ky9olnogHKH4= From: Leonardo Bras To: Catalin Marinas , Will Deacon , Leonardo Bras , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Zenghui Yu , "Rafael J. Wysocki" , Len Brown , Saket Dumbre , Paolo Bonzini , Chengwen Feng , Jonathan Cameron , Kees Cook , =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , Ryan Roberts , Yang Shi , Thomas Huth , mrigendrachaubey , Yeoreum Yun , Mark Brown , Kevin Brodsky , James Clark , Ard Biesheuvel , Fuad Tabba , Raghavendra Rao Ananta , Nathan Chancellor , Vincent Donnefort , Lorenzo Pieralisi , Sascha Bischoff , Anshuman Khandual , Tian Zheng , Wei-Lin Chang Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-acpi@vger.kernel.org, acpica-devel@lists.linux.dev, kvm@vger.kernel.org Subject: [PATCH v1 12/12] KVM: arm64: Enable KVM_HW_DIRTY_BIT Date: Thu, 30 Apr 2026 12:14:16 +0100 Message-ID: <20260430111424.3479613-14-leo.bras@arm.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260430111424.3479613-2-leo.bras@arm.com> References: <20260430111424.3479613-2-leo.bras@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Set the corresponding bit to enable hardware accelerated dirty-bitmap and dirty-ring cleaning for arm64. Actually using acceleration depends on the cpus enabling FEAT_HACDBS as well as the pre-requisite features for it, such as FEAT_HDBSS and FEAT_HAFDBS. Also, add myself as maintainer for the new bits introduced. Signed-off-by: Leonardo Bras --- MAINTAINERS | 7 +++++++ arch/arm64/kvm/Kconfig | 1 + 2 files changed, 8 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index d25342ca8aa1..4b8a2b05b916 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2069,20 +2069,27 @@ F: arch/arm64/boot/dts/Makefile ARM ARCHITECTED TIMER DRIVER M: Mark Rutland M: Marc Zyngier L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) S: Maintained F: arch/arm/include/asm/arch_timer.h F: arch/arm64/include/asm/arch_timer.h F: drivers/clocksource/arm_arch_timer.c F: drivers/clocksource/arm_arch_timer_mmio.c =20 +ARM STAGE-2 DIRTY-BIT TRACKER/CLEANER ACCELERATOR +M: Leonardo Bras +L: kvmarm@lists.linux.dev +S: Maintained +F: arch/arm64/include/asm/kvm_dirty_bit.h +F: arch/arm64/kvm/dirty_bit.c + ARM ETHOS-U NPU DRIVER M: Rob Herring (Arm) M: Tomeu Vizoso L: dri-devel@lists.freedesktop.org S: Supported T: git https://gitlab.freedesktop.org/drm/misc/kernel.git F: drivers/accel/ethosu/ F: include/uapi/drm/ethosu_accel.h =20 ARM GENERIC INTERRUPT CONTROLLER DRIVERS diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 449154f9a485..db8487bf738b 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -27,20 +27,21 @@ menuconfig KVM select VIRT_XFER_TO_GUEST_WORK select KVM_VFIO select HAVE_KVM_DIRTY_RING_ACQ_REL select NEED_KVM_DIRTY_RING_WITH_BITMAP select HAVE_KVM_MSI select HAVE_KVM_IRQCHIP select HAVE_KVM_IRQ_ROUTING select HAVE_KVM_IRQ_BYPASS select HAVE_KVM_READONLY_MEM select HAVE_KVM_VCPU_RUN_PID_CHANGE + select HAVE_KVM_HW_DIRTY_BIT if ACPI select SCHED_INFO select GUEST_PERF_EVENTS if PERF_EVENTS select KVM_GUEST_MEMFD help Support hosting virtualized guest machines. =20 If unsure, say N. =20 if KVM =20 --=20 2.54.0