From nobody Thu Oct 9 11:05:46 2025 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AAA6C2E9EB9 for ; Wed, 18 Jun 2025 11:36:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750246603; cv=none; b=aNa4x4F9dmu9h8euchMmGDdD1R6LPNkiv/FvD03wFz7uItql6oXFkUlnFD/wvP3ejckZdd2iKCYEpshV9jvoSIaBFE9snxmIEXLxsC5PkSOzIMJwLw3TUsOG0iExTcMT7qrwP0erv/1vVv8MEhKAHvg6TkScuxADwRe0P2Sff/A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750246603; c=relaxed/simple; bh=X0N8wDjIxKazhYu/6fsz/Bdqc7YSzKG8rGrYZT7tAuI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=T3Od5PhtA3PQVFZLULcO4yCgP2vyNsoC9urMpvKHnmcZ3cfuFNFzTjD0fXjziX8bqAELwLZ/AKduuNXRVsNmDjbVo1wpJ+82m8t20tntOxhfRpDN3fA7Cq9c5bjSjlSZuOl4A1wUsMIi9iTe8UeM7uGTxj/pGHlcQJOE0RnGt4U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ventanamicro.com; spf=pass smtp.mailfrom=ventanamicro.com; dkim=pass (2048-bit key) header.d=ventanamicro.com header.i=@ventanamicro.com header.b=l6r5mBq3; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ventanamicro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ventanamicro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ventanamicro.com header.i=@ventanamicro.com header.b="l6r5mBq3" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-236470b2dceso60709235ad.0 for ; Wed, 18 Jun 2025 04:36:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ventanamicro.com; s=google; t=1750246600; x=1750851400; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=82BM7Lr/w5wu8VyfjUmTApgJjDkQfTzaQ0EY82JI9q8=; b=l6r5mBq3w0E0mAd+sHX+WGpk8s/HrRm64iFludS1HbgAUIa0f+oUij7sK1kuDxJBJq 0wf7mgw7s5yOq4gyVk7sC5uSkuCxdARCHLZNyas7sbt+y5zYSOr1sAeERJV4RlIwZmQx lKNSrZJH4aMQz3UQPOb9zqksKxvWASp+Pw1YVEKskcpZ3fuAEApUGATSMrYqgVgruSHy LT6ktfpKjJbvcAw+ocHRt9xXk70BDYgVxLwDi7AQDfd8XIZV9sCb3zljd1eVQktYx5Cq HfxQT/CQmsaWqlZCBzbWKCjBQ3BB82vb2n77WDZ/8MVZxS+atiDrntcw4EWn8X+/96f4 3JDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750246600; x=1750851400; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=82BM7Lr/w5wu8VyfjUmTApgJjDkQfTzaQ0EY82JI9q8=; b=RCTc89CJkNXgpO6In16DhmxppqwP09jIp0xlzi+dKWw1tOUOvc0zLel6T700JPiT1A v5cmYIrt91pFal0FOtrCkM+oCgDUyhO513wBGTZYSjrKFJAcFTAMa3cmMGudQfryT3M4 N5ru9IDg5gUimqeJPkvDA6ICWgeBpHDMVmgbLbdf6XECj1QLe69Tw3npnMCoGudkgUo8 p5VCIaLoeRUwPjPdpjSYj46UQ7soUoFDXD9doHN2DeGmEINlKSTr5/DZ3hyifGCDlbeZ OqUyhboKKItDPqePAHIweqE6QoqVwvrCWFz0DqtC6+cD+RktPPKksOqNLIhVwVhYT6h9 EBlw== X-Forwarded-Encrypted: i=1; AJvYcCWUNO5UWbzGTkP2ZeFvlXvmi6eOIXmEsadwLKNA62y7DTO9yokqdnnjDkRrYQyxxipLj31UWO+F4Tfn7jg=@vger.kernel.org X-Gm-Message-State: AOJu0Yzkf3PbdRDzCH9IcV8+TIY8TCh7EiZyxvc0KDpyoQtpS9LqGLQN A6o1p/ifnJV4j5EGJjDLXdBJJpn7diSZajHHUUg2HD/LneL6iLXrFbuJB7CD/+cRCB7W3aV527Z D1O8F X-Gm-Gg: ASbGncsnOHJc02gs1JRuZtsJQKcO+AqrSFE0kqOAk1uu2hWdvSlBzRnw/sQr/1usTOx a9PjhvO0e3Jx4mHSfS2zWZ4Ww2Et80H2Y2iH72kC1C78sKscsj2BZ7mNyqj7xDXDhS6YzCk0G35 FGMHZnv3xTUk0GsXZ3tj47YRI02S4T59S90hreG1ot8t11p0yzyc0RgrYGIBtdvkNheyFDCWobu BQADBa25D2itzqbTCaVn4OuC13twK1LYU8j7fLUjeSNFQrv+GVr8+verpEy22QgHkNP6lKTmSqa U3qwkd2Wot1zVdODUuQ8Uhk/8+pLFiVwrMrai+iYYIMJuB9HlG0HDpciaTYWugOChPHCZm3Q4jE IZ+JQuNiReHuJA621HPjUyFUS2loX X-Google-Smtp-Source: AGHT+IFtTqBIHy1eDXsLGZscx1YuytvMsKNqPDwplEMCMcqmhs6mPxTzAbcafirh9yc0wLabklItOQ== X-Received: by 2002:a17:902:f642:b0:234:c549:d9f1 with SMTP id d9443c01a7336-2366b3df960mr245881045ad.47.1750246599668; Wed, 18 Jun 2025 04:36:39 -0700 (PDT) Received: from localhost.localdomain ([122.171.23.44]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-237c57c63efsm9112475ad.172.2025.06.18.04.36.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Jun 2025 04:36:38 -0700 (PDT) From: Anup Patel To: Atish Patra Cc: Palmer Dabbelt , Paul Walmsley , Alexandre Ghiti , Andrew Jones , Anup Patel , kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Anup Patel Subject: [PATCH v3 11/12] RISC-V: KVM: Factor-out g-stage page table management Date: Wed, 18 Jun 2025 17:05:31 +0530 Message-ID: <20250618113532.471448-12-apatel@ventanamicro.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250618113532.471448-1-apatel@ventanamicro.com> References: <20250618113532.471448-1-apatel@ventanamicro.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The upcoming nested virtualization can share g-stage page table management with the current host g-stage implementation hence factor-out g-stage page table management as separate sources and also use "kvm_riscv_mmu_" prefix for host g-stage functions. Signed-off-by: Anup Patel Reviewed-by: Nutty Liu --- arch/riscv/include/asm/kvm_gstage.h | 72 ++++ arch/riscv/include/asm/kvm_mmu.h | 32 +- arch/riscv/kvm/Makefile | 1 + arch/riscv/kvm/aia_imsic.c | 11 +- arch/riscv/kvm/gstage.c | 337 +++++++++++++++++++ arch/riscv/kvm/main.c | 2 +- arch/riscv/kvm/mmu.c | 492 ++++++---------------------- arch/riscv/kvm/vcpu.c | 4 +- arch/riscv/kvm/vcpu_exit.c | 5 +- arch/riscv/kvm/vm.c | 6 +- 10 files changed, 530 insertions(+), 432 deletions(-) create mode 100644 arch/riscv/include/asm/kvm_gstage.h create mode 100644 arch/riscv/kvm/gstage.c diff --git a/arch/riscv/include/asm/kvm_gstage.h b/arch/riscv/include/asm/k= vm_gstage.h new file mode 100644 index 000000000000..595e2183173e --- /dev/null +++ b/arch/riscv/include/asm/kvm_gstage.h @@ -0,0 +1,72 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2019 Western Digital Corporation or its affiliates. + * Copyright (c) 2025 Ventana Micro Systems Inc. + */ + +#ifndef __RISCV_KVM_GSTAGE_H_ +#define __RISCV_KVM_GSTAGE_H_ + +#include + +struct kvm_gstage { + struct kvm *kvm; + unsigned long flags; +#define KVM_GSTAGE_FLAGS_LOCAL BIT(0) + unsigned long vmid; + pgd_t *pgd; +}; + +struct kvm_gstage_mapping { + gpa_t addr; + pte_t pte; + u32 level; +}; + +#ifdef CONFIG_64BIT +#define kvm_riscv_gstage_index_bits 9 +#else +#define kvm_riscv_gstage_index_bits 10 +#endif + +extern unsigned long kvm_riscv_gstage_mode; +extern unsigned long kvm_riscv_gstage_pgd_levels; + +#define kvm_riscv_gstage_pgd_xbits 2 +#define kvm_riscv_gstage_pgd_size (1UL << (HGATP_PAGE_SHIFT + kvm_riscv_gs= tage_pgd_xbits)) +#define kvm_riscv_gstage_gpa_bits (HGATP_PAGE_SHIFT + \ + (kvm_riscv_gstage_pgd_levels * \ + kvm_riscv_gstage_index_bits) + \ + kvm_riscv_gstage_pgd_xbits) +#define kvm_riscv_gstage_gpa_size ((gpa_t)(1ULL << kvm_riscv_gstage_gpa_bi= ts)) + +bool kvm_riscv_gstage_get_leaf(struct kvm_gstage *gstage, gpa_t addr, + pte_t **ptepp, u32 *ptep_level); + +int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage, + struct kvm_mmu_memory_cache *pcache, + const struct kvm_gstage_mapping *map); + +int kvm_riscv_gstage_map_page(struct kvm_gstage *gstage, + struct kvm_mmu_memory_cache *pcache, + gpa_t gpa, phys_addr_t hpa, unsigned long page_size, + bool page_rdonly, bool page_exec, + struct kvm_gstage_mapping *out_map); + +enum kvm_riscv_gstage_op { + GSTAGE_OP_NOP =3D 0, /* Nothing */ + GSTAGE_OP_CLEAR, /* Clear/Unmap */ + GSTAGE_OP_WP, /* Write-protect */ +}; + +void kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr, + pte_t *ptep, u32 ptep_level, enum kvm_riscv_gstage_op op); + +void kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage, + gpa_t start, gpa_t size, bool may_block); + +void kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa= _t end); + +void kvm_riscv_gstage_mode_detect(void); + +#endif diff --git a/arch/riscv/include/asm/kvm_mmu.h b/arch/riscv/include/asm/kvm_= mmu.h index 91c11e692dc7..5439e76f0a96 100644 --- a/arch/riscv/include/asm/kvm_mmu.h +++ b/arch/riscv/include/asm/kvm_mmu.h @@ -6,28 +6,16 @@ #ifndef __RISCV_KVM_MMU_H_ #define __RISCV_KVM_MMU_H_ =20 -#include +#include =20 -struct kvm_gstage_mapping { - gpa_t addr; - pte_t pte; - u32 level; -}; - -int kvm_riscv_gstage_ioremap(struct kvm *kvm, gpa_t gpa, - phys_addr_t hpa, unsigned long size, - bool writable, bool in_atomic); -void kvm_riscv_gstage_iounmap(struct kvm *kvm, gpa_t gpa, - unsigned long size); -int kvm_riscv_gstage_map(struct kvm_vcpu *vcpu, - struct kvm_memory_slot *memslot, - gpa_t gpa, unsigned long hva, bool is_write, - struct kvm_gstage_mapping *out_map); -int kvm_riscv_gstage_alloc_pgd(struct kvm *kvm); -void kvm_riscv_gstage_free_pgd(struct kvm *kvm); -void kvm_riscv_gstage_update_hgatp(struct kvm_vcpu *vcpu); -void kvm_riscv_gstage_mode_detect(void); -unsigned long kvm_riscv_gstage_mode(void); -int kvm_riscv_gstage_gpa_bits(void); +int kvm_riscv_mmu_ioremap(struct kvm *kvm, gpa_t gpa, phys_addr_t hpa, + unsigned long size, bool writable, bool in_atomic); +void kvm_riscv_mmu_iounmap(struct kvm *kvm, gpa_t gpa, unsigned long size); +int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memsl= ot, + gpa_t gpa, unsigned long hva, bool is_write, + struct kvm_gstage_mapping *out_map); +int kvm_riscv_mmu_alloc_pgd(struct kvm *kvm); +void kvm_riscv_mmu_free_pgd(struct kvm *kvm); +void kvm_riscv_mmu_update_hgatp(struct kvm_vcpu *vcpu); =20 #endif diff --git a/arch/riscv/kvm/Makefile b/arch/riscv/kvm/Makefile index 06e2d52a9b88..07197395750e 100644 --- a/arch/riscv/kvm/Makefile +++ b/arch/riscv/kvm/Makefile @@ -14,6 +14,7 @@ kvm-y +=3D aia.o kvm-y +=3D aia_aplic.o kvm-y +=3D aia_device.o kvm-y +=3D aia_imsic.o +kvm-y +=3D gstage.o kvm-y +=3D main.o kvm-y +=3D mmu.o kvm-y +=3D nacl.o diff --git a/arch/riscv/kvm/aia_imsic.c b/arch/riscv/kvm/aia_imsic.c index 40b469c0a01f..ea1a36836d9c 100644 --- a/arch/riscv/kvm/aia_imsic.c +++ b/arch/riscv/kvm/aia_imsic.c @@ -704,9 +704,8 @@ void kvm_riscv_vcpu_aia_imsic_release(struct kvm_vcpu *= vcpu) */ =20 /* Purge the G-stage mapping */ - kvm_riscv_gstage_iounmap(vcpu->kvm, - vcpu->arch.aia_context.imsic_addr, - IMSIC_MMIO_PAGE_SZ); + kvm_riscv_mmu_iounmap(vcpu->kvm, vcpu->arch.aia_context.imsic_addr, + IMSIC_MMIO_PAGE_SZ); =20 /* TODO: Purge the IOMMU mapping ??? */ =20 @@ -786,9 +785,9 @@ int kvm_riscv_vcpu_aia_imsic_update(struct kvm_vcpu *vc= pu) imsic_vsfile_local_clear(new_vsfile_hgei, imsic->nr_hw_eix); =20 /* Update G-stage mapping for the new IMSIC VS-file */ - ret =3D kvm_riscv_gstage_ioremap(kvm, vcpu->arch.aia_context.imsic_addr, - new_vsfile_pa, IMSIC_MMIO_PAGE_SZ, - true, true); + ret =3D kvm_riscv_mmu_ioremap(kvm, vcpu->arch.aia_context.imsic_addr, + new_vsfile_pa, IMSIC_MMIO_PAGE_SZ, + true, true); if (ret) goto fail_free_vsfile_hgei; =20 diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c new file mode 100644 index 000000000000..9c7c44f09b05 --- /dev/null +++ b/arch/riscv/kvm/gstage.c @@ -0,0 +1,337 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2019 Western Digital Corporation or its affiliates. + * Copyright (c) 2025 Ventana Micro Systems Inc. + */ + +#include +#include +#include +#include +#include +#include + +#ifdef CONFIG_64BIT +unsigned long kvm_riscv_gstage_mode __ro_after_init =3D HGATP_MODE_SV39X4; +unsigned long kvm_riscv_gstage_pgd_levels __ro_after_init =3D 3; +#else +unsigned long kvm_riscv_gstage_mode __ro_after_init =3D HGATP_MODE_SV32X4; +unsigned long kvm_riscv_gstage_pgd_levels __ro_after_init =3D 2; +#endif + +#define gstage_pte_leaf(__ptep) \ + (pte_val(*(__ptep)) & (_PAGE_READ | _PAGE_WRITE | _PAGE_EXEC)) + +static inline unsigned long gstage_pte_index(gpa_t addr, u32 level) +{ + unsigned long mask; + unsigned long shift =3D HGATP_PAGE_SHIFT + (kvm_riscv_gstage_index_bits *= level); + + if (level =3D=3D (kvm_riscv_gstage_pgd_levels - 1)) + mask =3D (PTRS_PER_PTE * (1UL << kvm_riscv_gstage_pgd_xbits)) - 1; + else + mask =3D PTRS_PER_PTE - 1; + + return (addr >> shift) & mask; +} + +static inline unsigned long gstage_pte_page_vaddr(pte_t pte) +{ + return (unsigned long)pfn_to_virt(__page_val_to_pfn(pte_val(pte))); +} + +static int gstage_page_size_to_level(unsigned long page_size, u32 *out_lev= el) +{ + u32 i; + unsigned long psz =3D 1UL << 12; + + for (i =3D 0; i < kvm_riscv_gstage_pgd_levels; i++) { + if (page_size =3D=3D (psz << (i * kvm_riscv_gstage_index_bits))) { + *out_level =3D i; + return 0; + } + } + + return -EINVAL; +} + +static int gstage_level_to_page_order(u32 level, unsigned long *out_pgorde= r) +{ + if (kvm_riscv_gstage_pgd_levels < level) + return -EINVAL; + + *out_pgorder =3D 12 + (level * kvm_riscv_gstage_index_bits); + return 0; +} + +static int gstage_level_to_page_size(u32 level, unsigned long *out_pgsize) +{ + int rc; + unsigned long page_order =3D PAGE_SHIFT; + + rc =3D gstage_level_to_page_order(level, &page_order); + if (rc) + return rc; + + *out_pgsize =3D BIT(page_order); + return 0; +} + +bool kvm_riscv_gstage_get_leaf(struct kvm_gstage *gstage, gpa_t addr, + pte_t **ptepp, u32 *ptep_level) +{ + pte_t *ptep; + u32 current_level =3D kvm_riscv_gstage_pgd_levels - 1; + + *ptep_level =3D current_level; + ptep =3D (pte_t *)gstage->pgd; + ptep =3D &ptep[gstage_pte_index(addr, current_level)]; + while (ptep && pte_val(ptep_get(ptep))) { + if (gstage_pte_leaf(ptep)) { + *ptep_level =3D current_level; + *ptepp =3D ptep; + return true; + } + + if (current_level) { + current_level--; + *ptep_level =3D current_level; + ptep =3D (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); + ptep =3D &ptep[gstage_pte_index(addr, current_level)]; + } else { + ptep =3D NULL; + } + } + + return false; +} + +static void gstage_tlb_flush(struct kvm_gstage *gstage, u32 level, gpa_t a= ddr) +{ + unsigned long order =3D PAGE_SHIFT; + + if (gstage_level_to_page_order(level, &order)) + return; + addr &=3D ~(BIT(order) - 1); + + if (gstage->flags & KVM_GSTAGE_FLAGS_LOCAL) + kvm_riscv_local_hfence_gvma_vmid_gpa(gstage->vmid, addr, BIT(order), ord= er); + else + kvm_riscv_hfence_gvma_vmid_gpa(gstage->kvm, -1UL, 0, addr, BIT(order), o= rder); +} + +int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage, + struct kvm_mmu_memory_cache *pcache, + const struct kvm_gstage_mapping *map) +{ + u32 current_level =3D kvm_riscv_gstage_pgd_levels - 1; + pte_t *next_ptep =3D (pte_t *)gstage->pgd; + pte_t *ptep =3D &next_ptep[gstage_pte_index(map->addr, current_level)]; + + if (current_level < map->level) + return -EINVAL; + + while (current_level !=3D map->level) { + if (gstage_pte_leaf(ptep)) + return -EEXIST; + + if (!pte_val(ptep_get(ptep))) { + if (!pcache) + return -ENOMEM; + next_ptep =3D kvm_mmu_memory_cache_alloc(pcache); + if (!next_ptep) + return -ENOMEM; + set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)), + __pgprot(_PAGE_TABLE))); + } else { + if (gstage_pte_leaf(ptep)) + return -EEXIST; + next_ptep =3D (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); + } + + current_level--; + ptep =3D &next_ptep[gstage_pte_index(map->addr, current_level)]; + } + + if (pte_val(*ptep) !=3D pte_val(map->pte)) { + set_pte(ptep, map->pte); + if (gstage_pte_leaf(ptep)) + gstage_tlb_flush(gstage, current_level, map->addr); + } + + return 0; +} + +int kvm_riscv_gstage_map_page(struct kvm_gstage *gstage, + struct kvm_mmu_memory_cache *pcache, + gpa_t gpa, phys_addr_t hpa, unsigned long page_size, + bool page_rdonly, bool page_exec, + struct kvm_gstage_mapping *out_map) +{ + pgprot_t prot; + int ret; + + out_map->addr =3D gpa; + out_map->level =3D 0; + + ret =3D gstage_page_size_to_level(page_size, &out_map->level); + if (ret) + return ret; + + /* + * A RISC-V implementation can choose to either: + * 1) Update 'A' and 'D' PTE bits in hardware + * 2) Generate page fault when 'A' and/or 'D' bits are not set + * PTE so that software can update these bits. + * + * We support both options mentioned above. To achieve this, we + * always set 'A' and 'D' PTE bits at time of creating G-stage + * mapping. To support KVM dirty page logging with both options + * mentioned above, we will write-protect G-stage PTEs to track + * dirty pages. + */ + + if (page_exec) { + if (page_rdonly) + prot =3D PAGE_READ_EXEC; + else + prot =3D PAGE_WRITE_EXEC; + } else { + if (page_rdonly) + prot =3D PAGE_READ; + else + prot =3D PAGE_WRITE; + } + out_map->pte =3D pfn_pte(PFN_DOWN(hpa), prot); + out_map->pte =3D pte_mkdirty(out_map->pte); + + return kvm_riscv_gstage_set_pte(gstage, pcache, out_map); +} + +void kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr, + pte_t *ptep, u32 ptep_level, enum kvm_riscv_gstage_op op) +{ + int i, ret; + pte_t old_pte, *next_ptep; + u32 next_ptep_level; + unsigned long next_page_size, page_size; + + ret =3D gstage_level_to_page_size(ptep_level, &page_size); + if (ret) + return; + + WARN_ON(addr & (page_size - 1)); + + if (!pte_val(ptep_get(ptep))) + return; + + if (ptep_level && !gstage_pte_leaf(ptep)) { + next_ptep =3D (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); + next_ptep_level =3D ptep_level - 1; + ret =3D gstage_level_to_page_size(next_ptep_level, &next_page_size); + if (ret) + return; + + if (op =3D=3D GSTAGE_OP_CLEAR) + set_pte(ptep, __pte(0)); + for (i =3D 0; i < PTRS_PER_PTE; i++) + kvm_riscv_gstage_op_pte(gstage, addr + i * next_page_size, + &next_ptep[i], next_ptep_level, op); + if (op =3D=3D GSTAGE_OP_CLEAR) + put_page(virt_to_page(next_ptep)); + } else { + old_pte =3D *ptep; + if (op =3D=3D GSTAGE_OP_CLEAR) + set_pte(ptep, __pte(0)); + else if (op =3D=3D GSTAGE_OP_WP) + set_pte(ptep, __pte(pte_val(ptep_get(ptep)) & ~_PAGE_WRITE)); + if (pte_val(*ptep) !=3D pte_val(old_pte)) + gstage_tlb_flush(gstage, ptep_level, addr); + } +} + +void kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage, + gpa_t start, gpa_t size, bool may_block) +{ + int ret; + pte_t *ptep; + u32 ptep_level; + bool found_leaf; + unsigned long page_size; + gpa_t addr =3D start, end =3D start + size; + + while (addr < end) { + found_leaf =3D kvm_riscv_gstage_get_leaf(gstage, addr, &ptep, &ptep_leve= l); + ret =3D gstage_level_to_page_size(ptep_level, &page_size); + if (ret) + break; + + if (!found_leaf) + goto next; + + if (!(addr & (page_size - 1)) && ((end - addr) >=3D page_size)) + kvm_riscv_gstage_op_pte(gstage, addr, ptep, + ptep_level, GSTAGE_OP_CLEAR); + +next: + addr +=3D page_size; + + /* + * If the range is too large, release the kvm->mmu_lock + * to prevent starvation and lockup detector warnings. + */ + if (!(gstage->flags & KVM_GSTAGE_FLAGS_LOCAL) && may_block && addr < end) + cond_resched_lock(&gstage->kvm->mmu_lock); + } +} + +void kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa= _t end) +{ + int ret; + pte_t *ptep; + u32 ptep_level; + bool found_leaf; + gpa_t addr =3D start; + unsigned long page_size; + + while (addr < end) { + found_leaf =3D kvm_riscv_gstage_get_leaf(gstage, addr, &ptep, &ptep_leve= l); + ret =3D gstage_level_to_page_size(ptep_level, &page_size); + if (ret) + break; + + if (!found_leaf) + goto next; + + if (!(addr & (page_size - 1)) && ((end - addr) >=3D page_size)) + kvm_riscv_gstage_op_pte(gstage, addr, ptep, + ptep_level, GSTAGE_OP_WP); + +next: + addr +=3D page_size; + } +} + +void __init kvm_riscv_gstage_mode_detect(void) +{ +#ifdef CONFIG_64BIT + /* Try Sv57x4 G-stage mode */ + csr_write(CSR_HGATP, HGATP_MODE_SV57X4 << HGATP_MODE_SHIFT); + if ((csr_read(CSR_HGATP) >> HGATP_MODE_SHIFT) =3D=3D HGATP_MODE_SV57X4) { + kvm_riscv_gstage_mode =3D HGATP_MODE_SV57X4; + kvm_riscv_gstage_pgd_levels =3D 5; + goto skip_sv48x4_test; + } + + /* Try Sv48x4 G-stage mode */ + csr_write(CSR_HGATP, HGATP_MODE_SV48X4 << HGATP_MODE_SHIFT); + if ((csr_read(CSR_HGATP) >> HGATP_MODE_SHIFT) =3D=3D HGATP_MODE_SV48X4) { + kvm_riscv_gstage_mode =3D HGATP_MODE_SV48X4; + kvm_riscv_gstage_pgd_levels =3D 4; + } +skip_sv48x4_test: + + csr_write(CSR_HGATP, 0); + kvm_riscv_local_hfence_gvma_all(); +#endif +} diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c index b861a5dd7bd9..67c876de74ef 100644 --- a/arch/riscv/kvm/main.c +++ b/arch/riscv/kvm/main.c @@ -135,7 +135,7 @@ static int __init riscv_kvm_init(void) (rc) ? slist : "no features"); } =20 - switch (kvm_riscv_gstage_mode()) { + switch (kvm_riscv_gstage_mode) { case HGATP_MODE_SV32X4: str =3D "Sv32x4"; break; diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 806614b3e46d..9f7dcd8cd741 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -6,9 +6,7 @@ * Anup Patel */ =20 -#include #include -#include #include #include #include @@ -17,342 +15,28 @@ #include #include #include -#include -#include - -#ifdef CONFIG_64BIT -static unsigned long gstage_mode __ro_after_init =3D (HGATP_MODE_SV39X4 <<= HGATP_MODE_SHIFT); -static unsigned long gstage_pgd_levels __ro_after_init =3D 3; -#define gstage_index_bits 9 -#else -static unsigned long gstage_mode __ro_after_init =3D (HGATP_MODE_SV32X4 <<= HGATP_MODE_SHIFT); -static unsigned long gstage_pgd_levels __ro_after_init =3D 2; -#define gstage_index_bits 10 -#endif - -#define gstage_pgd_xbits 2 -#define gstage_pgd_size (1UL << (HGATP_PAGE_SHIFT + gstage_pgd_xbits)) -#define gstage_gpa_bits (HGATP_PAGE_SHIFT + \ - (gstage_pgd_levels * gstage_index_bits) + \ - gstage_pgd_xbits) -#define gstage_gpa_size ((gpa_t)(1ULL << gstage_gpa_bits)) - -#define gstage_pte_leaf(__ptep) \ - (pte_val(*(__ptep)) & (_PAGE_READ | _PAGE_WRITE | _PAGE_EXEC)) - -static inline unsigned long gstage_pte_index(gpa_t addr, u32 level) -{ - unsigned long mask; - unsigned long shift =3D HGATP_PAGE_SHIFT + (gstage_index_bits * level); - - if (level =3D=3D (gstage_pgd_levels - 1)) - mask =3D (PTRS_PER_PTE * (1UL << gstage_pgd_xbits)) - 1; - else - mask =3D PTRS_PER_PTE - 1; - - return (addr >> shift) & mask; -} =20 -static inline unsigned long gstage_pte_page_vaddr(pte_t pte) -{ - return (unsigned long)pfn_to_virt(__page_val_to_pfn(pte_val(pte))); -} - -static int gstage_page_size_to_level(unsigned long page_size, u32 *out_lev= el) -{ - u32 i; - unsigned long psz =3D 1UL << 12; - - for (i =3D 0; i < gstage_pgd_levels; i++) { - if (page_size =3D=3D (psz << (i * gstage_index_bits))) { - *out_level =3D i; - return 0; - } - } - - return -EINVAL; -} - -static int gstage_level_to_page_order(u32 level, unsigned long *out_pgorde= r) -{ - if (gstage_pgd_levels < level) - return -EINVAL; - - *out_pgorder =3D 12 + (level * gstage_index_bits); - return 0; -} - -static int gstage_level_to_page_size(u32 level, unsigned long *out_pgsize) -{ - int rc; - unsigned long page_order =3D PAGE_SHIFT; - - rc =3D gstage_level_to_page_order(level, &page_order); - if (rc) - return rc; - - *out_pgsize =3D BIT(page_order); - return 0; -} - -static bool gstage_get_leaf_entry(struct kvm *kvm, gpa_t addr, - pte_t **ptepp, u32 *ptep_level) -{ - pte_t *ptep; - u32 current_level =3D gstage_pgd_levels - 1; - - *ptep_level =3D current_level; - ptep =3D (pte_t *)kvm->arch.pgd; - ptep =3D &ptep[gstage_pte_index(addr, current_level)]; - while (ptep && pte_val(ptep_get(ptep))) { - if (gstage_pte_leaf(ptep)) { - *ptep_level =3D current_level; - *ptepp =3D ptep; - return true; - } - - if (current_level) { - current_level--; - *ptep_level =3D current_level; - ptep =3D (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); - ptep =3D &ptep[gstage_pte_index(addr, current_level)]; - } else { - ptep =3D NULL; - } - } - - return false; -} - -static void gstage_remote_tlb_flush(struct kvm *kvm, u32 level, gpa_t addr) -{ - unsigned long order =3D PAGE_SHIFT; - - if (gstage_level_to_page_order(level, &order)) - return; - addr &=3D ~(BIT(order) - 1); - - kvm_riscv_hfence_gvma_vmid_gpa(kvm, -1UL, 0, addr, BIT(order), order); -} - -static int gstage_set_pte(struct kvm *kvm, - struct kvm_mmu_memory_cache *pcache, - const struct kvm_gstage_mapping *map) -{ - u32 current_level =3D gstage_pgd_levels - 1; - pte_t *next_ptep =3D (pte_t *)kvm->arch.pgd; - pte_t *ptep =3D &next_ptep[gstage_pte_index(map->addr, current_level)]; - - if (current_level < map->level) - return -EINVAL; - - while (current_level !=3D map->level) { - if (gstage_pte_leaf(ptep)) - return -EEXIST; - - if (!pte_val(ptep_get(ptep))) { - if (!pcache) - return -ENOMEM; - next_ptep =3D kvm_mmu_memory_cache_alloc(pcache); - if (!next_ptep) - return -ENOMEM; - set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)), - __pgprot(_PAGE_TABLE))); - } else { - if (gstage_pte_leaf(ptep)) - return -EEXIST; - next_ptep =3D (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); - } - - current_level--; - ptep =3D &next_ptep[gstage_pte_index(map->addr, current_level)]; - } - - if (pte_val(*ptep) !=3D pte_val(map->pte)) { - set_pte(ptep, map->pte); - if (gstage_pte_leaf(ptep)) - gstage_remote_tlb_flush(kvm, current_level, map->addr); - } - - return 0; -} - -static int gstage_map_page(struct kvm *kvm, - struct kvm_mmu_memory_cache *pcache, - gpa_t gpa, phys_addr_t hpa, - unsigned long page_size, - bool page_rdonly, bool page_exec, - struct kvm_gstage_mapping *out_map) -{ - pgprot_t prot; - int ret; - - out_map->addr =3D gpa; - out_map->level =3D 0; - - ret =3D gstage_page_size_to_level(page_size, &out_map->level); - if (ret) - return ret; - - /* - * A RISC-V implementation can choose to either: - * 1) Update 'A' and 'D' PTE bits in hardware - * 2) Generate page fault when 'A' and/or 'D' bits are not set - * PTE so that software can update these bits. - * - * We support both options mentioned above. To achieve this, we - * always set 'A' and 'D' PTE bits at time of creating G-stage - * mapping. To support KVM dirty page logging with both options - * mentioned above, we will write-protect G-stage PTEs to track - * dirty pages. - */ - - if (page_exec) { - if (page_rdonly) - prot =3D PAGE_READ_EXEC; - else - prot =3D PAGE_WRITE_EXEC; - } else { - if (page_rdonly) - prot =3D PAGE_READ; - else - prot =3D PAGE_WRITE; - } - out_map->pte =3D pfn_pte(PFN_DOWN(hpa), prot); - out_map->pte =3D pte_mkdirty(out_map->pte); - - return gstage_set_pte(kvm, pcache, out_map); -} - -enum gstage_op { - GSTAGE_OP_NOP =3D 0, /* Nothing */ - GSTAGE_OP_CLEAR, /* Clear/Unmap */ - GSTAGE_OP_WP, /* Write-protect */ -}; - -static void gstage_op_pte(struct kvm *kvm, gpa_t addr, - pte_t *ptep, u32 ptep_level, enum gstage_op op) -{ - int i, ret; - pte_t old_pte, *next_ptep; - u32 next_ptep_level; - unsigned long next_page_size, page_size; - - ret =3D gstage_level_to_page_size(ptep_level, &page_size); - if (ret) - return; - - BUG_ON(addr & (page_size - 1)); - - if (!pte_val(ptep_get(ptep))) - return; - - if (ptep_level && !gstage_pte_leaf(ptep)) { - next_ptep =3D (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); - next_ptep_level =3D ptep_level - 1; - ret =3D gstage_level_to_page_size(next_ptep_level, - &next_page_size); - if (ret) - return; - - if (op =3D=3D GSTAGE_OP_CLEAR) - set_pte(ptep, __pte(0)); - for (i =3D 0; i < PTRS_PER_PTE; i++) - gstage_op_pte(kvm, addr + i * next_page_size, - &next_ptep[i], next_ptep_level, op); - if (op =3D=3D GSTAGE_OP_CLEAR) - put_page(virt_to_page(next_ptep)); - } else { - old_pte =3D *ptep; - if (op =3D=3D GSTAGE_OP_CLEAR) - set_pte(ptep, __pte(0)); - else if (op =3D=3D GSTAGE_OP_WP) - set_pte(ptep, __pte(pte_val(ptep_get(ptep)) & ~_PAGE_WRITE)); - if (pte_val(*ptep) !=3D pte_val(old_pte)) - gstage_remote_tlb_flush(kvm, ptep_level, addr); - } -} - -static void gstage_unmap_range(struct kvm *kvm, gpa_t start, - gpa_t size, bool may_block) -{ - int ret; - pte_t *ptep; - u32 ptep_level; - bool found_leaf; - unsigned long page_size; - gpa_t addr =3D start, end =3D start + size; - - while (addr < end) { - found_leaf =3D gstage_get_leaf_entry(kvm, addr, - &ptep, &ptep_level); - ret =3D gstage_level_to_page_size(ptep_level, &page_size); - if (ret) - break; - - if (!found_leaf) - goto next; - - if (!(addr & (page_size - 1)) && ((end - addr) >=3D page_size)) - gstage_op_pte(kvm, addr, ptep, - ptep_level, GSTAGE_OP_CLEAR); - -next: - addr +=3D page_size; - - /* - * If the range is too large, release the kvm->mmu_lock - * to prevent starvation and lockup detector warnings. - */ - if (may_block && addr < end) - cond_resched_lock(&kvm->mmu_lock); - } -} - -static void gstage_wp_range(struct kvm *kvm, gpa_t start, gpa_t end) -{ - int ret; - pte_t *ptep; - u32 ptep_level; - bool found_leaf; - gpa_t addr =3D start; - unsigned long page_size; - - while (addr < end) { - found_leaf =3D gstage_get_leaf_entry(kvm, addr, - &ptep, &ptep_level); - ret =3D gstage_level_to_page_size(ptep_level, &page_size); - if (ret) - break; - - if (!found_leaf) - goto next; - - if (!(addr & (page_size - 1)) && ((end - addr) >=3D page_size)) - gstage_op_pte(kvm, addr, ptep, - ptep_level, GSTAGE_OP_WP); - -next: - addr +=3D page_size; - } -} - -static void gstage_wp_memory_region(struct kvm *kvm, int slot) +static void mmu_wp_memory_region(struct kvm *kvm, int slot) { struct kvm_memslots *slots =3D kvm_memslots(kvm); struct kvm_memory_slot *memslot =3D id_to_memslot(slots, slot); phys_addr_t start =3D memslot->base_gfn << PAGE_SHIFT; phys_addr_t end =3D (memslot->base_gfn + memslot->npages) << PAGE_SHIFT; + struct kvm_gstage gstage; + + gstage.kvm =3D kvm; + gstage.flags =3D 0; + gstage.vmid =3D READ_ONCE(kvm->arch.vmid.vmid); + gstage.pgd =3D kvm->arch.pgd; =20 spin_lock(&kvm->mmu_lock); - gstage_wp_range(kvm, start, end); + kvm_riscv_gstage_wp_range(&gstage, start, end); spin_unlock(&kvm->mmu_lock); kvm_flush_remote_tlbs_memslot(kvm, memslot); } =20 -int kvm_riscv_gstage_ioremap(struct kvm *kvm, gpa_t gpa, - phys_addr_t hpa, unsigned long size, - bool writable, bool in_atomic) +int kvm_riscv_mmu_ioremap(struct kvm *kvm, gpa_t gpa, phys_addr_t hpa, + unsigned long size, bool writable, bool in_atomic) { int ret =3D 0; unsigned long pfn; @@ -362,6 +46,12 @@ int kvm_riscv_gstage_ioremap(struct kvm *kvm, gpa_t gpa, .gfp_zero =3D __GFP_ZERO, }; struct kvm_gstage_mapping map; + struct kvm_gstage gstage; + + gstage.kvm =3D kvm; + gstage.flags =3D 0; + gstage.vmid =3D READ_ONCE(kvm->arch.vmid.vmid); + gstage.pgd =3D kvm->arch.pgd; =20 end =3D (gpa + size + PAGE_SIZE - 1) & PAGE_MASK; pfn =3D __phys_to_pfn(hpa); @@ -374,12 +64,12 @@ int kvm_riscv_gstage_ioremap(struct kvm *kvm, gpa_t gp= a, if (!writable) map.pte =3D pte_wrprotect(map.pte); =20 - ret =3D kvm_mmu_topup_memory_cache(&pcache, gstage_pgd_levels); + ret =3D kvm_mmu_topup_memory_cache(&pcache, kvm_riscv_gstage_pgd_levels); if (ret) goto out; =20 spin_lock(&kvm->mmu_lock); - ret =3D gstage_set_pte(kvm, &pcache, &map); + ret =3D kvm_riscv_gstage_set_pte(&gstage, &pcache, &map); spin_unlock(&kvm->mmu_lock); if (ret) goto out; @@ -392,10 +82,17 @@ int kvm_riscv_gstage_ioremap(struct kvm *kvm, gpa_t gp= a, return ret; } =20 -void kvm_riscv_gstage_iounmap(struct kvm *kvm, gpa_t gpa, unsigned long si= ze) +void kvm_riscv_mmu_iounmap(struct kvm *kvm, gpa_t gpa, unsigned long size) { + struct kvm_gstage gstage; + + gstage.kvm =3D kvm; + gstage.flags =3D 0; + gstage.vmid =3D READ_ONCE(kvm->arch.vmid.vmid); + gstage.pgd =3D kvm->arch.pgd; + spin_lock(&kvm->mmu_lock); - gstage_unmap_range(kvm, gpa, size, false); + kvm_riscv_gstage_unmap_range(&gstage, gpa, size, false); spin_unlock(&kvm->mmu_lock); } =20 @@ -407,8 +104,14 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kv= m *kvm, phys_addr_t base_gfn =3D slot->base_gfn + gfn_offset; phys_addr_t start =3D (base_gfn + __ffs(mask)) << PAGE_SHIFT; phys_addr_t end =3D (base_gfn + __fls(mask) + 1) << PAGE_SHIFT; + struct kvm_gstage gstage; + + gstage.kvm =3D kvm; + gstage.flags =3D 0; + gstage.vmid =3D READ_ONCE(kvm->arch.vmid.vmid); + gstage.pgd =3D kvm->arch.pgd; =20 - gstage_wp_range(kvm, start, end); + kvm_riscv_gstage_wp_range(&gstage, start, end); } =20 void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *mems= lot) @@ -425,7 +128,7 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) =20 void kvm_arch_flush_shadow_all(struct kvm *kvm) { - kvm_riscv_gstage_free_pgd(kvm); + kvm_riscv_mmu_free_pgd(kvm); } =20 void kvm_arch_flush_shadow_memslot(struct kvm *kvm, @@ -433,9 +136,15 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, { gpa_t gpa =3D slot->base_gfn << PAGE_SHIFT; phys_addr_t size =3D slot->npages << PAGE_SHIFT; + struct kvm_gstage gstage; + + gstage.kvm =3D kvm; + gstage.flags =3D 0; + gstage.vmid =3D READ_ONCE(kvm->arch.vmid.vmid); + gstage.pgd =3D kvm->arch.pgd; =20 spin_lock(&kvm->mmu_lock); - gstage_unmap_range(kvm, gpa, size, false); + kvm_riscv_gstage_unmap_range(&gstage, gpa, size, false); spin_unlock(&kvm->mmu_lock); } =20 @@ -450,7 +159,7 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, * the memory slot is write protected. */ if (change !=3D KVM_MR_DELETE && new->flags & KVM_MEM_LOG_DIRTY_PAGES) - gstage_wp_memory_region(kvm, new->id); + mmu_wp_memory_region(kvm, new->id); } =20 int kvm_arch_prepare_memory_region(struct kvm *kvm, @@ -472,7 +181,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, * space addressable by the KVM guest GPA space. */ if ((new->base_gfn + new->npages) >=3D - (gstage_gpa_size >> PAGE_SHIFT)) + (kvm_riscv_gstage_gpa_size >> PAGE_SHIFT)) return -EFAULT; =20 hva =3D new->userspace_addr; @@ -528,9 +237,8 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, goto out; } =20 - ret =3D kvm_riscv_gstage_ioremap(kvm, gpa, pa, - vm_end - vm_start, - writable, false); + ret =3D kvm_riscv_mmu_ioremap(kvm, gpa, pa, vm_end - vm_start, + writable, false); if (ret) break; } @@ -541,7 +249,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, goto out; =20 if (ret) - kvm_riscv_gstage_iounmap(kvm, base_gpa, size); + kvm_riscv_mmu_iounmap(kvm, base_gpa, size); =20 out: mmap_read_unlock(current->mm); @@ -550,12 +258,18 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, =20 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) { + struct kvm_gstage gstage; + if (!kvm->arch.pgd) return false; =20 - gstage_unmap_range(kvm, range->start << PAGE_SHIFT, - (range->end - range->start) << PAGE_SHIFT, - range->may_block); + gstage.kvm =3D kvm; + gstage.flags =3D 0; + gstage.vmid =3D READ_ONCE(kvm->arch.vmid.vmid); + gstage.pgd =3D kvm->arch.pgd; + kvm_riscv_gstage_unmap_range(&gstage, range->start << PAGE_SHIFT, + (range->end - range->start) << PAGE_SHIFT, + range->may_block); return false; } =20 @@ -564,14 +278,19 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_rang= e *range) pte_t *ptep; u32 ptep_level =3D 0; u64 size =3D (range->end - range->start) << PAGE_SHIFT; + struct kvm_gstage gstage; =20 if (!kvm->arch.pgd) return false; =20 WARN_ON(size !=3D PAGE_SIZE && size !=3D PMD_SIZE && size !=3D PUD_SIZE); =20 - if (!gstage_get_leaf_entry(kvm, range->start << PAGE_SHIFT, - &ptep, &ptep_level)) + gstage.kvm =3D kvm; + gstage.flags =3D 0; + gstage.vmid =3D READ_ONCE(kvm->arch.vmid.vmid); + gstage.pgd =3D kvm->arch.pgd; + if (!kvm_riscv_gstage_get_leaf(&gstage, range->start << PAGE_SHIFT, + &ptep, &ptep_level)) return false; =20 return ptep_test_and_clear_young(NULL, 0, ptep); @@ -582,23 +301,27 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn= _range *range) pte_t *ptep; u32 ptep_level =3D 0; u64 size =3D (range->end - range->start) << PAGE_SHIFT; + struct kvm_gstage gstage; =20 if (!kvm->arch.pgd) return false; =20 WARN_ON(size !=3D PAGE_SIZE && size !=3D PMD_SIZE && size !=3D PUD_SIZE); =20 - if (!gstage_get_leaf_entry(kvm, range->start << PAGE_SHIFT, - &ptep, &ptep_level)) + gstage.kvm =3D kvm; + gstage.flags =3D 0; + gstage.vmid =3D READ_ONCE(kvm->arch.vmid.vmid); + gstage.pgd =3D kvm->arch.pgd; + if (!kvm_riscv_gstage_get_leaf(&gstage, range->start << PAGE_SHIFT, + &ptep, &ptep_level)) return false; =20 return pte_young(ptep_get(ptep)); } =20 -int kvm_riscv_gstage_map(struct kvm_vcpu *vcpu, - struct kvm_memory_slot *memslot, - gpa_t gpa, unsigned long hva, bool is_write, - struct kvm_gstage_mapping *out_map) +int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memsl= ot, + gpa_t gpa, unsigned long hva, bool is_write, + struct kvm_gstage_mapping *out_map) { int ret; kvm_pfn_t hfn; @@ -611,13 +334,19 @@ int kvm_riscv_gstage_map(struct kvm_vcpu *vcpu, bool logging =3D (memslot->dirty_bitmap && !(memslot->flags & KVM_MEM_READONLY)) ? true : false; unsigned long vma_pagesize, mmu_seq; + struct kvm_gstage gstage; struct page *page; =20 + gstage.kvm =3D kvm; + gstage.flags =3D 0; + gstage.vmid =3D READ_ONCE(kvm->arch.vmid.vmid); + gstage.pgd =3D kvm->arch.pgd; + /* Setup initial state of output mapping */ memset(out_map, 0, sizeof(*out_map)); =20 /* We need minimum second+third level pages */ - ret =3D kvm_mmu_topup_memory_cache(pcache, gstage_pgd_levels); + ret =3D kvm_mmu_topup_memory_cache(pcache, kvm_riscv_gstage_pgd_levels); if (ret) { kvm_err("Failed to topup G-stage cache\n"); return ret; @@ -684,11 +413,11 @@ int kvm_riscv_gstage_map(struct kvm_vcpu *vcpu, =20 if (writable) { mark_page_dirty(kvm, gfn); - ret =3D gstage_map_page(kvm, pcache, gpa, hfn << PAGE_SHIFT, - vma_pagesize, false, true, out_map); + ret =3D kvm_riscv_gstage_map_page(&gstage, pcache, gpa, hfn << PAGE_SHIF= T, + vma_pagesize, false, true, out_map); } else { - ret =3D gstage_map_page(kvm, pcache, gpa, hfn << PAGE_SHIFT, - vma_pagesize, true, true, out_map); + ret =3D kvm_riscv_gstage_map_page(&gstage, pcache, gpa, hfn << PAGE_SHIF= T, + vma_pagesize, true, true, out_map); } =20 if (ret) @@ -700,7 +429,7 @@ int kvm_riscv_gstage_map(struct kvm_vcpu *vcpu, return ret; } =20 -int kvm_riscv_gstage_alloc_pgd(struct kvm *kvm) +int kvm_riscv_mmu_alloc_pgd(struct kvm *kvm) { struct page *pgd_page; =20 @@ -710,7 +439,7 @@ int kvm_riscv_gstage_alloc_pgd(struct kvm *kvm) } =20 pgd_page =3D alloc_pages(GFP_KERNEL | __GFP_ZERO, - get_order(gstage_pgd_size)); + get_order(kvm_riscv_gstage_pgd_size)); if (!pgd_page) return -ENOMEM; kvm->arch.pgd =3D page_to_virt(pgd_page); @@ -719,13 +448,18 @@ int kvm_riscv_gstage_alloc_pgd(struct kvm *kvm) return 0; } =20 -void kvm_riscv_gstage_free_pgd(struct kvm *kvm) +void kvm_riscv_mmu_free_pgd(struct kvm *kvm) { + struct kvm_gstage gstage; void *pgd =3D NULL; =20 spin_lock(&kvm->mmu_lock); if (kvm->arch.pgd) { - gstage_unmap_range(kvm, 0UL, gstage_gpa_size, false); + gstage.kvm =3D kvm; + gstage.flags =3D 0; + gstage.vmid =3D READ_ONCE(kvm->arch.vmid.vmid); + gstage.pgd =3D kvm->arch.pgd; + kvm_riscv_gstage_unmap_range(&gstage, 0UL, kvm_riscv_gstage_gpa_size, fa= lse); pgd =3D READ_ONCE(kvm->arch.pgd); kvm->arch.pgd =3D NULL; kvm->arch.pgd_phys =3D 0; @@ -733,12 +467,12 @@ void kvm_riscv_gstage_free_pgd(struct kvm *kvm) spin_unlock(&kvm->mmu_lock); =20 if (pgd) - free_pages((unsigned long)pgd, get_order(gstage_pgd_size)); + free_pages((unsigned long)pgd, get_order(kvm_riscv_gstage_pgd_size)); } =20 -void kvm_riscv_gstage_update_hgatp(struct kvm_vcpu *vcpu) +void kvm_riscv_mmu_update_hgatp(struct kvm_vcpu *vcpu) { - unsigned long hgatp =3D gstage_mode; + unsigned long hgatp =3D kvm_riscv_gstage_mode << HGATP_MODE_SHIFT; struct kvm_arch *k =3D &vcpu->kvm->arch; =20 hgatp |=3D (READ_ONCE(k->vmid.vmid) << HGATP_VMID_SHIFT) & HGATP_VMID; @@ -749,37 +483,3 @@ void kvm_riscv_gstage_update_hgatp(struct kvm_vcpu *vc= pu) if (!kvm_riscv_gstage_vmid_bits()) kvm_riscv_local_hfence_gvma_all(); } - -void __init kvm_riscv_gstage_mode_detect(void) -{ -#ifdef CONFIG_64BIT - /* Try Sv57x4 G-stage mode */ - csr_write(CSR_HGATP, HGATP_MODE_SV57X4 << HGATP_MODE_SHIFT); - if ((csr_read(CSR_HGATP) >> HGATP_MODE_SHIFT) =3D=3D HGATP_MODE_SV57X4) { - gstage_mode =3D (HGATP_MODE_SV57X4 << HGATP_MODE_SHIFT); - gstage_pgd_levels =3D 5; - goto skip_sv48x4_test; - } - - /* Try Sv48x4 G-stage mode */ - csr_write(CSR_HGATP, HGATP_MODE_SV48X4 << HGATP_MODE_SHIFT); - if ((csr_read(CSR_HGATP) >> HGATP_MODE_SHIFT) =3D=3D HGATP_MODE_SV48X4) { - gstage_mode =3D (HGATP_MODE_SV48X4 << HGATP_MODE_SHIFT); - gstage_pgd_levels =3D 4; - } -skip_sv48x4_test: - - csr_write(CSR_HGATP, 0); - kvm_riscv_local_hfence_gvma_all(); -#endif -} - -unsigned long __init kvm_riscv_gstage_mode(void) -{ - return gstage_mode >> HGATP_MODE_SHIFT; -} - -int kvm_riscv_gstage_gpa_bits(void) -{ - return gstage_gpa_bits; -} diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c index 8ad7b31f5939..fe028b4274df 100644 --- a/arch/riscv/kvm/vcpu.c +++ b/arch/riscv/kvm/vcpu.c @@ -632,7 +632,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) } } =20 - kvm_riscv_gstage_update_hgatp(vcpu); + kvm_riscv_mmu_update_hgatp(vcpu); =20 kvm_riscv_vcpu_timer_restore(vcpu); =20 @@ -717,7 +717,7 @@ static void kvm_riscv_check_vcpu_requests(struct kvm_vc= pu *vcpu) kvm_riscv_reset_vcpu(vcpu, true); =20 if (kvm_check_request(KVM_REQ_UPDATE_HGATP, vcpu)) - kvm_riscv_gstage_update_hgatp(vcpu); + kvm_riscv_mmu_update_hgatp(vcpu); =20 if (kvm_check_request(KVM_REQ_FENCE_I, vcpu)) kvm_riscv_fence_i_process(vcpu); diff --git a/arch/riscv/kvm/vcpu_exit.c b/arch/riscv/kvm/vcpu_exit.c index 6b4694bc07ea..0bb0c51e3c89 100644 --- a/arch/riscv/kvm/vcpu_exit.c +++ b/arch/riscv/kvm/vcpu_exit.c @@ -43,8 +43,9 @@ static int gstage_page_fault(struct kvm_vcpu *vcpu, struc= t kvm_run *run, }; } =20 - ret =3D kvm_riscv_gstage_map(vcpu, memslot, fault_addr, hva, - (trap->scause =3D=3D EXC_STORE_GUEST_PAGE_FAULT) ? true : false, &host_m= ap); + ret =3D kvm_riscv_mmu_map(vcpu, memslot, fault_addr, hva, + (trap->scause =3D=3D EXC_STORE_GUEST_PAGE_FAULT) ? true : false, + &host_map); if (ret < 0) return ret; =20 diff --git a/arch/riscv/kvm/vm.c b/arch/riscv/kvm/vm.c index 8601cf29e5f8..66d91ae6e9b2 100644 --- a/arch/riscv/kvm/vm.c +++ b/arch/riscv/kvm/vm.c @@ -32,13 +32,13 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long typ= e) { int r; =20 - r =3D kvm_riscv_gstage_alloc_pgd(kvm); + r =3D kvm_riscv_mmu_alloc_pgd(kvm); if (r) return r; =20 r =3D kvm_riscv_gstage_vmid_init(kvm); if (r) { - kvm_riscv_gstage_free_pgd(kvm); + kvm_riscv_mmu_free_pgd(kvm); return r; } =20 @@ -200,7 +200,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long = ext) r =3D KVM_USER_MEM_SLOTS; break; case KVM_CAP_VM_GPA_BITS: - r =3D kvm_riscv_gstage_gpa_bits(); + r =3D kvm_riscv_gstage_gpa_bits; break; default: r =3D 0; --=20 2.43.0