From nobody Tue Nov 26 08:29:58 2024 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B64A199FDE for ; Sun, 20 Oct 2024 19:47:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729453682; cv=none; b=Yy4Ep+Ph57JNXmBAPdzaac1K+5Ye/7OG4TtBPy4LBqjKWw9ORAqjahFrVM0b9B4zIN0Tw93cs6yhV82h+7gIcNI9m54HVBAFrgMb1rSWHX5I4b095zL4UJNj0AxlGkUrb3B0l3fXHnhMRcHAnSB2Rjk8NZY0jubzJvDm1J5t3HA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729453682; c=relaxed/simple; bh=aLBJo1op6p7EbwqsjHLsNImpiFMlqml80zE8MJz+O5c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dRdC1gOhXeeWfwzeSmQ+y/c2kPwJU+e5xf0I1QsuhbuAXQmk1dNl7YpPsafgvlVdHI2WnpUN7d5BXkOJbjujmCW5VV0cAN7RPVlS52tkFDwSzySkw4+X6JIzyUQpg8MOioy3asxY+gpGjhOsnQEnPY+Gm/I82SimiMoWCZgrpKU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ventanamicro.com; spf=pass smtp.mailfrom=ventanamicro.com; dkim=pass (2048-bit key) header.d=ventanamicro.com header.i=@ventanamicro.com header.b=HTG4Vk3V; arc=none smtp.client-ip=209.85.216.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ventanamicro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ventanamicro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ventanamicro.com header.i=@ventanamicro.com header.b="HTG4Vk3V" Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-2e31af47681so3057781a91.2 for ; Sun, 20 Oct 2024 12:47:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ventanamicro.com; s=google; t=1729453679; x=1730058479; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2PNHY6Ej4ODA3l2FaZRmpOmWoo1Wxqh6777J5n1pDa4=; b=HTG4Vk3Vv2Llo6fx72mBdtKZmypTPIfaVasAzNJ76U9a/pEPWdAMDgTTbOb1zPdftw 5WaI1mI4dH7q5V6v7q4PaI6qgZVxCdZY6kXVuzTeCXG+Qc+wYVVTZzoJZgVICPJMZFxZ q2e/1QtmX8ks2x0wtT9RuBcU8Rjx9RtBUQcEA6R6ltqZ5WfC4pbZeusGys0nxTieGrYv ZCXN24v1fQPSHGWKDilHe6hr27XIt8jaimmwrqiBARs3NP5lRILmf5Aq+hWMlHIK80U9 zagy76dKyZqq8KH1nuFPCxSkHhP5V2mkh2iYx5Cy9w8S7mPl4r9QxmE6lNSDabVgIKie A7Ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729453679; x=1730058479; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2PNHY6Ej4ODA3l2FaZRmpOmWoo1Wxqh6777J5n1pDa4=; b=aSlpNJnoGtpndjTDIKD/rZoY8doG3omRv5qfse9plCWBSJwLPCSzI7Iq+X5O+a53Qg F9+NJQKfqQsNk1UgJgftz2ZUAXF+Mz8gF7jSByawpVTZo1D751NwsckrRuQPAu1H+MID Yf0Qrh1R7iM896jM4W7OdzD+BAMWDwc0E+5w6ysHC0mH403YIA+1wlqpz7sbxC3ZNPp4 ZA4QiB+5dISZUZRUGzPg3cCH2mXfBhZzkHkCgmoQvZWOleZxvCnbdEBGguDH0SAikTkY RzZ5ef33Vwv0bSOHFTz2TfthMMicMj2POWhek/4OD3HZkU9f43wKaz9BuHtCn+43UFK7 Z8PQ== X-Forwarded-Encrypted: i=1; AJvYcCUHBunGJROG9jzMTwzZTwUSGZwnx2dp15BchGhF2yPYmw6TiKBEqj/iC9a9r62au1gZha9wPsh3/x0j4zw=@vger.kernel.org X-Gm-Message-State: AOJu0YwU/4++Gr3ovGkUPLlDfVJElIjxxhcFDTj1Hf6hcmXxuywA1VtK 2q0xkRzqparZ3J8AU9d1669OgYlf/ewIq0jYDoDg+k91HTG8UvajZ4tYNQhkPQM= X-Google-Smtp-Source: AGHT+IFl/0jkuzkA2N3qfrZw6TQdQzdowPrIIb9QTb2dp0fwvxYkkYmfrYtD1JWF8OFyEwLW30Af0A== X-Received: by 2002:a17:90a:15c9:b0:2e2:bad3:e393 with SMTP id 98e67ed59e1d1-2e5616de72amr10854988a91.3.1729453679214; Sun, 20 Oct 2024 12:47:59 -0700 (PDT) Received: from anup-ubuntu-vm.localdomain ([50.238.223.131]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e5ad365d4dsm1933188a91.14.2024.10.20.12.47.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Oct 2024 12:47:58 -0700 (PDT) From: Anup Patel To: Palmer Dabbelt , Paul Walmsley Cc: Atish Patra , Andrew Jones , Anup Patel , kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Anup Patel , Atish Patra Subject: [PATCH v2 08/13] RISC-V: KVM: Add common nested acceleration support Date: Mon, 21 Oct 2024 01:17:29 +0530 Message-ID: <20241020194734.58686-9-apatel@ventanamicro.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241020194734.58686-1-apatel@ventanamicro.com> References: <20241020194734.58686-1-apatel@ventanamicro.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a common nested acceleration support which will be shared by all parts of KVM RISC-V. This nested acceleration support detects and enables SBI NACL extension usage based on static keys which ensures minimum impact on the non-nested scenario. Signed-off-by: Anup Patel Reviewed-by: Atish Patra --- arch/riscv/include/asm/kvm_nacl.h | 239 ++++++++++++++++++++++++++++++ arch/riscv/kvm/Makefile | 1 + arch/riscv/kvm/main.c | 51 ++++++- arch/riscv/kvm/nacl.c | 152 +++++++++++++++++++ 4 files changed, 441 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/include/asm/kvm_nacl.h create mode 100644 arch/riscv/kvm/nacl.c diff --git a/arch/riscv/include/asm/kvm_nacl.h b/arch/riscv/include/asm/kvm= _nacl.h new file mode 100644 index 000000000000..8f3e3ebf5017 --- /dev/null +++ b/arch/riscv/include/asm/kvm_nacl.h @@ -0,0 +1,239 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2024 Ventana Micro Systems Inc. + */ + +#ifndef __KVM_NACL_H +#define __KVM_NACL_H + +#include +#include +#include +#include +#include + +DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_available); +#define kvm_riscv_nacl_available() \ + static_branch_unlikely(&kvm_riscv_nacl_available) + +DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_csr_available); +#define kvm_riscv_nacl_sync_csr_available() \ + static_branch_unlikely(&kvm_riscv_nacl_sync_csr_available) + +DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_hfence_available); +#define kvm_riscv_nacl_sync_hfence_available() \ + static_branch_unlikely(&kvm_riscv_nacl_sync_hfence_available) + +DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_sret_available); +#define kvm_riscv_nacl_sync_sret_available() \ + static_branch_unlikely(&kvm_riscv_nacl_sync_sret_available) + +DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_autoswap_csr_available); +#define kvm_riscv_nacl_autoswap_csr_available() \ + static_branch_unlikely(&kvm_riscv_nacl_autoswap_csr_available) + +struct kvm_riscv_nacl { + void *shmem; + phys_addr_t shmem_phys; +}; +DECLARE_PER_CPU(struct kvm_riscv_nacl, kvm_riscv_nacl); + +void __kvm_riscv_nacl_hfence(void *shmem, + unsigned long control, + unsigned long page_num, + unsigned long page_count); + +int kvm_riscv_nacl_enable(void); + +void kvm_riscv_nacl_disable(void); + +void kvm_riscv_nacl_exit(void); + +int kvm_riscv_nacl_init(void); + +#ifdef CONFIG_32BIT +#define lelong_to_cpu(__x) le32_to_cpu(__x) +#define cpu_to_lelong(__x) cpu_to_le32(__x) +#else +#define lelong_to_cpu(__x) le64_to_cpu(__x) +#define cpu_to_lelong(__x) cpu_to_le64(__x) +#endif + +#define nacl_shmem() \ + this_cpu_ptr(&kvm_riscv_nacl)->shmem + +#define nacl_scratch_read_long(__shmem, __offset) \ +({ \ + unsigned long *__p =3D (__shmem) + \ + SBI_NACL_SHMEM_SCRATCH_OFFSET + \ + (__offset); \ + lelong_to_cpu(*__p); \ +}) + +#define nacl_scratch_write_long(__shmem, __offset, __val) \ +do { \ + unsigned long *__p =3D (__shmem) + \ + SBI_NACL_SHMEM_SCRATCH_OFFSET + \ + (__offset); \ + *__p =3D cpu_to_lelong(__val); \ +} while (0) + +#define nacl_scratch_write_longs(__shmem, __offset, __array, __count) \ +do { \ + unsigned int __i; \ + unsigned long *__p =3D (__shmem) + \ + SBI_NACL_SHMEM_SCRATCH_OFFSET + \ + (__offset); \ + for (__i =3D 0; __i < (__count); __i++) \ + __p[__i] =3D cpu_to_lelong((__array)[__i]); \ +} while (0) + +#define nacl_sync_hfence(__e) \ + sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SYNC_HFENCE, \ + (__e), 0, 0, 0, 0, 0) + +#define nacl_hfence_mkconfig(__type, __order, __vmid, __asid) \ +({ \ + unsigned long __c =3D SBI_NACL_SHMEM_HFENCE_CONFIG_PEND; \ + __c |=3D ((__type) & SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_MASK) \ + << SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_SHIFT; \ + __c |=3D (((__order) - SBI_NACL_SHMEM_HFENCE_ORDER_BASE) & \ + SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_MASK) \ + << SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_SHIFT; \ + __c |=3D ((__vmid) & SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_MASK) \ + << SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_SHIFT; \ + __c |=3D ((__asid) & SBI_NACL_SHMEM_HFENCE_CONFIG_ASID_MASK); \ + __c; \ +}) + +#define nacl_hfence_mkpnum(__order, __addr) \ + ((__addr) >> (__order)) + +#define nacl_hfence_mkpcount(__order, __size) \ + ((__size) >> (__order)) + +#define nacl_hfence_gvma(__shmem, __gpa, __gpsz, __order) \ +__kvm_riscv_nacl_hfence(__shmem, \ + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_GVMA, \ + __order, 0, 0), \ + nacl_hfence_mkpnum(__order, __gpa), \ + nacl_hfence_mkpcount(__order, __gpsz)) + +#define nacl_hfence_gvma_all(__shmem) \ +__kvm_riscv_nacl_hfence(__shmem, \ + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_ALL, \ + 0, 0, 0), 0, 0) + +#define nacl_hfence_gvma_vmid(__shmem, __vmid, __gpa, __gpsz, __order) \ +__kvm_riscv_nacl_hfence(__shmem, \ + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_VMID, \ + __order, __vmid, 0), \ + nacl_hfence_mkpnum(__order, __gpa), \ + nacl_hfence_mkpcount(__order, __gpsz)) + +#define nacl_hfence_gvma_vmid_all(__shmem, __vmid) \ +__kvm_riscv_nacl_hfence(__shmem, \ + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_VMID_ALL, \ + 0, __vmid, 0), 0, 0) + +#define nacl_hfence_vvma(__shmem, __vmid, __gva, __gvsz, __order) \ +__kvm_riscv_nacl_hfence(__shmem, \ + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_VVMA, \ + __order, __vmid, 0), \ + nacl_hfence_mkpnum(__order, __gva), \ + nacl_hfence_mkpcount(__order, __gvsz)) + +#define nacl_hfence_vvma_all(__shmem, __vmid) \ +__kvm_riscv_nacl_hfence(__shmem, \ + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ALL, \ + 0, __vmid, 0), 0, 0) + +#define nacl_hfence_vvma_asid(__shmem, __vmid, __asid, __gva, __gvsz, __or= der)\ +__kvm_riscv_nacl_hfence(__shmem, \ + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ASID, \ + __order, __vmid, __asid), \ + nacl_hfence_mkpnum(__order, __gva), \ + nacl_hfence_mkpcount(__order, __gvsz)) + +#define nacl_hfence_vvma_asid_all(__shmem, __vmid, __asid) \ +__kvm_riscv_nacl_hfence(__shmem, \ + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ASID_ALL, \ + 0, __vmid, __asid), 0, 0) + +#define nacl_csr_read(__shmem, __csr) \ +({ \ + unsigned long *__a =3D (__shmem) + SBI_NACL_SHMEM_CSR_OFFSET; \ + lelong_to_cpu(__a[SBI_NACL_SHMEM_CSR_INDEX(__csr)]); \ +}) + +#define nacl_csr_write(__shmem, __csr, __val) \ +do { \ + void *__s =3D (__shmem); \ + unsigned int __i =3D SBI_NACL_SHMEM_CSR_INDEX(__csr); \ + unsigned long *__a =3D (__s) + SBI_NACL_SHMEM_CSR_OFFSET; \ + u8 *__b =3D (__s) + SBI_NACL_SHMEM_DBITMAP_OFFSET; \ + __a[__i] =3D cpu_to_lelong(__val); \ + __b[__i >> 3] |=3D 1U << (__i & 0x7); \ +} while (0) + +#define nacl_csr_swap(__shmem, __csr, __val) \ +({ \ + void *__s =3D (__shmem); \ + unsigned int __i =3D SBI_NACL_SHMEM_CSR_INDEX(__csr); \ + unsigned long *__a =3D (__s) + SBI_NACL_SHMEM_CSR_OFFSET; \ + u8 *__b =3D (__s) + SBI_NACL_SHMEM_DBITMAP_OFFSET; \ + unsigned long __r =3D lelong_to_cpu(__a[__i]); \ + __a[__i] =3D cpu_to_lelong(__val); \ + __b[__i >> 3] |=3D 1U << (__i & 0x7); \ + __r; \ +}) + +#define nacl_sync_csr(__csr) \ + sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SYNC_CSR, \ + (__csr), 0, 0, 0, 0, 0) + +/* + * Each ncsr_xyz() macro defined below has it's own static-branch so every + * use of ncsr_xyz() macro emits a patchable direct jump. This means multi= ple + * back-to-back ncsr_xyz() macro usage will emit multiple patchable direct + * jumps which is sub-optimal. + * + * Based on the above, it is recommended to avoid multiple back-to-back + * ncsr_xyz() macro usage. + */ + +#define ncsr_read(__csr) \ +({ \ + unsigned long __r; \ + if (kvm_riscv_nacl_available()) \ + __r =3D nacl_csr_read(nacl_shmem(), __csr); \ + else \ + __r =3D csr_read(__csr); \ + __r; \ +}) + +#define ncsr_write(__csr, __val) \ +do { \ + if (kvm_riscv_nacl_sync_csr_available()) \ + nacl_csr_write(nacl_shmem(), __csr, __val); \ + else \ + csr_write(__csr, __val); \ +} while (0) + +#define ncsr_swap(__csr, __val) \ +({ \ + unsigned long __r; \ + if (kvm_riscv_nacl_sync_csr_available()) \ + __r =3D nacl_csr_swap(nacl_shmem(), __csr, __val); \ + else \ + __r =3D csr_swap(__csr, __val); \ + __r; \ +}) + +#define nsync_csr(__csr) \ +do { \ + if (kvm_riscv_nacl_sync_csr_available()) \ + nacl_sync_csr(__csr); \ +} while (0) + +#endif diff --git a/arch/riscv/kvm/Makefile b/arch/riscv/kvm/Makefile index c1eac0d093de..0fb1840c3e0a 100644 --- a/arch/riscv/kvm/Makefile +++ b/arch/riscv/kvm/Makefile @@ -16,6 +16,7 @@ kvm-y +=3D aia_device.o kvm-y +=3D aia_imsic.o kvm-y +=3D main.o kvm-y +=3D mmu.o +kvm-y +=3D nacl.o kvm-y +=3D tlb.o kvm-y +=3D vcpu.o kvm-y +=3D vcpu_exit.o diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c index 5682e338ae6d..1fa8be5ee509 100644 --- a/arch/riscv/kvm/main.c +++ b/arch/riscv/kvm/main.c @@ -10,8 +10,8 @@ #include #include #include -#include #include +#include #include =20 long kvm_arch_dev_ioctl(struct file *filp, @@ -22,6 +22,12 @@ long kvm_arch_dev_ioctl(struct file *filp, =20 int kvm_arch_enable_virtualization_cpu(void) { + int rc; + + rc =3D kvm_riscv_nacl_enable(); + if (rc) + return rc; + csr_write(CSR_HEDELEG, KVM_HEDELEG_DEFAULT); csr_write(CSR_HIDELEG, KVM_HIDELEG_DEFAULT); =20 @@ -49,17 +55,21 @@ void kvm_arch_disable_virtualization_cpu(void) csr_write(CSR_HVIP, 0); csr_write(CSR_HEDELEG, 0); csr_write(CSR_HIDELEG, 0); + + kvm_riscv_nacl_disable(); } =20 static void kvm_riscv_teardown(void) { kvm_riscv_aia_exit(); + kvm_riscv_nacl_exit(); kvm_unregister_perf_callbacks(); } =20 static int __init riscv_kvm_init(void) { int rc; + char slist[64]; const char *str; =20 if (!riscv_isa_extension_available(NULL, h)) { @@ -77,16 +87,53 @@ static int __init riscv_kvm_init(void) return -ENODEV; } =20 + rc =3D kvm_riscv_nacl_init(); + if (rc && rc !=3D -ENODEV) + return rc; + kvm_riscv_gstage_mode_detect(); =20 kvm_riscv_gstage_vmid_detect(); =20 rc =3D kvm_riscv_aia_init(); - if (rc && rc !=3D -ENODEV) + if (rc && rc !=3D -ENODEV) { + kvm_riscv_nacl_exit(); return rc; + } =20 kvm_info("hypervisor extension available\n"); =20 + if (kvm_riscv_nacl_available()) { + rc =3D 0; + slist[0] =3D '\0'; + if (kvm_riscv_nacl_sync_csr_available()) { + if (rc) + strcat(slist, ", "); + strcat(slist, "sync_csr"); + rc++; + } + if (kvm_riscv_nacl_sync_hfence_available()) { + if (rc) + strcat(slist, ", "); + strcat(slist, "sync_hfence"); + rc++; + } + if (kvm_riscv_nacl_sync_sret_available()) { + if (rc) + strcat(slist, ", "); + strcat(slist, "sync_sret"); + rc++; + } + if (kvm_riscv_nacl_autoswap_csr_available()) { + if (rc) + strcat(slist, ", "); + strcat(slist, "autoswap_csr"); + rc++; + } + kvm_info("using SBI nested acceleration with %s\n", + (rc) ? slist : "no features"); + } + switch (kvm_riscv_gstage_mode()) { case HGATP_MODE_SV32X4: str =3D "Sv32x4"; diff --git a/arch/riscv/kvm/nacl.c b/arch/riscv/kvm/nacl.c new file mode 100644 index 000000000000..08a95ad9ada2 --- /dev/null +++ b/arch/riscv/kvm/nacl.c @@ -0,0 +1,152 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2024 Ventana Micro Systems Inc. + */ + +#include +#include +#include + +DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_available); +DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_csr_available); +DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_hfence_available); +DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_sret_available); +DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_autoswap_csr_available); +DEFINE_PER_CPU(struct kvm_riscv_nacl, kvm_riscv_nacl); + +void __kvm_riscv_nacl_hfence(void *shmem, + unsigned long control, + unsigned long page_num, + unsigned long page_count) +{ + int i, ent =3D -1, try_count =3D 5; + unsigned long *entp; + +again: + for (i =3D 0; i < SBI_NACL_SHMEM_HFENCE_ENTRY_MAX; i++) { + entp =3D shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_CONFIG(i); + if (lelong_to_cpu(*entp) & SBI_NACL_SHMEM_HFENCE_CONFIG_PEND) + continue; + + ent =3D i; + break; + } + + if (ent < 0) { + if (try_count) { + nacl_sync_hfence(-1UL); + goto again; + } else { + pr_warn("KVM: No free entry in NACL shared memory\n"); + return; + } + } + + entp =3D shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_CONFIG(i); + *entp =3D cpu_to_lelong(control); + entp =3D shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_PNUM(i); + *entp =3D cpu_to_lelong(page_num); + entp =3D shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_PCOUNT(i); + *entp =3D cpu_to_lelong(page_count); +} + +int kvm_riscv_nacl_enable(void) +{ + int rc; + struct sbiret ret; + struct kvm_riscv_nacl *nacl; + + if (!kvm_riscv_nacl_available()) + return 0; + nacl =3D this_cpu_ptr(&kvm_riscv_nacl); + + ret =3D sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SET_SHMEM, + nacl->shmem_phys, 0, 0, 0, 0, 0); + rc =3D sbi_err_map_linux_errno(ret.error); + if (rc) + return rc; + + return 0; +} + +void kvm_riscv_nacl_disable(void) +{ + if (!kvm_riscv_nacl_available()) + return; + + sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SET_SHMEM, + SBI_SHMEM_DISABLE, SBI_SHMEM_DISABLE, 0, 0, 0, 0); +} + +void kvm_riscv_nacl_exit(void) +{ + int cpu; + struct kvm_riscv_nacl *nacl; + + if (!kvm_riscv_nacl_available()) + return; + + /* Allocate per-CPU shared memory */ + for_each_possible_cpu(cpu) { + nacl =3D per_cpu_ptr(&kvm_riscv_nacl, cpu); + if (!nacl->shmem) + continue; + + free_pages((unsigned long)nacl->shmem, + get_order(SBI_NACL_SHMEM_SIZE)); + nacl->shmem =3D NULL; + nacl->shmem_phys =3D 0; + } +} + +static long nacl_probe_feature(long feature_id) +{ + struct sbiret ret; + + if (!kvm_riscv_nacl_available()) + return 0; + + ret =3D sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_PROBE_FEATURE, + feature_id, 0, 0, 0, 0, 0); + return ret.value; +} + +int kvm_riscv_nacl_init(void) +{ + int cpu; + struct page *shmem_page; + struct kvm_riscv_nacl *nacl; + + if (sbi_spec_version < sbi_mk_version(1, 0) || + sbi_probe_extension(SBI_EXT_NACL) <=3D 0) + return -ENODEV; + + /* Enable NACL support */ + static_branch_enable(&kvm_riscv_nacl_available); + + /* Probe NACL features */ + if (nacl_probe_feature(SBI_NACL_FEAT_SYNC_CSR)) + static_branch_enable(&kvm_riscv_nacl_sync_csr_available); + if (nacl_probe_feature(SBI_NACL_FEAT_SYNC_HFENCE)) + static_branch_enable(&kvm_riscv_nacl_sync_hfence_available); + if (nacl_probe_feature(SBI_NACL_FEAT_SYNC_SRET)) + static_branch_enable(&kvm_riscv_nacl_sync_sret_available); + if (nacl_probe_feature(SBI_NACL_FEAT_AUTOSWAP_CSR)) + static_branch_enable(&kvm_riscv_nacl_autoswap_csr_available); + + /* Allocate per-CPU shared memory */ + for_each_possible_cpu(cpu) { + nacl =3D per_cpu_ptr(&kvm_riscv_nacl, cpu); + + shmem_page =3D alloc_pages(GFP_KERNEL | __GFP_ZERO, + get_order(SBI_NACL_SHMEM_SIZE)); + if (!shmem_page) { + kvm_riscv_nacl_exit(); + return -ENOMEM; + } + nacl->shmem =3D page_to_virt(shmem_page); + nacl->shmem_phys =3D page_to_phys(shmem_page); + } + + return 0; +} --=20 2.43.0