From nobody Sat Nov 30 05:24:32 2024 Received: from smtp-fw-52002.amazon.com (smtp-fw-52002.amazon.com [52.119.213.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 60F4A1A4B6E for ; Wed, 11 Sep 2024 14:38:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=52.119.213.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726065523; cv=none; b=mtgc/OMcmwLh1fBian0tyQOFra9OHNO6Bqruuwrj09lg8LLpn8U0dQlVioIXW0U/GBI8jzBnN6UvIdrMytHUaSmDeUn/PJKU91pmO2NpQeog6J1R01YRyiAqx7feNN+wAuj3KjCNjgYkvxDH6BunZbs22c/kyMPAVsYQrQc+F7Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726065523; c=relaxed/simple; bh=y69GLusekEpOSkfF6e4m3oTOacotg5cYQYvn2myDj2w=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=o6YcJR5I/M8X4kUJWj3nkSUKTJEl4VczIRakONNNj6FslLlMU6SF9oVGYyUxZpVqCG1kSbOMcRTtGGFLQKSKK/W78UAdtCNriEvXvlPGlKkKpdjsBHj6h9O9kHuWNX9RN8uN2UFZkOVivsg18e8p3bcaYL3f9ms7QJSJdDXUSlE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.de; spf=pass smtp.mailfrom=amazon.de; dkim=pass (1024-bit key) header.d=amazon.de header.i=@amazon.de header.b=LIeLRz3s; arc=none smtp.client-ip=52.119.213.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=amazon.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amazon.de header.i=@amazon.de header.b="LIeLRz3s" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.de; i=@amazon.de; q=dns/txt; s=amazon201209; t=1726065520; x=1757601520; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=y7aHv3GtJyRu8tFW/46gCzZ/hJvvqzg3ZHRgqRHdqdU=; b=LIeLRz3sAPyO7Y9WMDYHaD3IyoPwLRe+Ta1QhWRAWRY+BoKAblG4VPKP TqMCiAHCF7VxoY23617uT2vE1vv55vSRr7NfCY5H4nBMTohsHkPP0HsO7 yvfibcs11gQHX2+Fu+hmTLQE0HP0j6l87hUbolN8HKo14ZlxXFk6aChpr w=; X-IronPort-AV: E=Sophos;i="6.10,220,1719878400"; d="scan'208";a="658274111" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO smtpout.prod.us-west-2.prod.farcaster.email.amazon.dev) ([10.43.8.6]) by smtp-border-fw-52002.iad7.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Sep 2024 14:37:14 +0000 Received: from EX19MTAEUC002.ant.amazon.com [10.0.43.254:16521] by smtpin.naws.eu-west-1.prod.farcaster.email.amazon.dev [10.0.27.59:2525] with esmtp (Farcaster) id c7fcd23e-81e6-4810-ad32-43bfceb7c5f9; Wed, 11 Sep 2024 14:37:13 +0000 (UTC) X-Farcaster-Flow-ID: c7fcd23e-81e6-4810-ad32-43bfceb7c5f9 Received: from EX19D007EUA004.ant.amazon.com (10.252.50.76) by EX19MTAEUC002.ant.amazon.com (10.252.51.245) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Wed, 11 Sep 2024 14:37:12 +0000 Received: from EX19MTAUEC001.ant.amazon.com (10.252.135.222) by EX19D007EUA004.ant.amazon.com (10.252.50.76) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.35; Wed, 11 Sep 2024 14:37:12 +0000 Received: from dev-dsk-faresx-1b-27755bf1.eu-west-1.amazon.com (10.253.79.181) by mail-relay.amazon.com (10.252.135.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34 via Frontend Transport; Wed, 11 Sep 2024 14:37:09 +0000 From: Fares Mehanna To: CC: , Fares Mehanna , "Marc Zyngier" , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , "Will Deacon" , Andrew Morton , "Kemeng Shi" , =?UTF-8?q?Pierre-Cl=C3=A9ment=20Tosi?= , Ard Biesheuvel , Mark Rutland , Javier Martinez Canillas , "Arnd Bergmann" , Fuad Tabba , Mark Brown , Joey Gouly , Kristina Martsenko , Randy Dunlap , "Bjorn Helgaas" , Jean-Philippe Brucker , "Mike Rapoport (IBM)" , "David Hildenbrand" , Roman Kagan , "moderated list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , "open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , open list , "open list:MEMORY MANAGEMENT" Subject: [RFC PATCH 5/7] arm64: KVM: Allocate vCPU gp-regs dynamically on VHE and KERNEL_SECRETMEM enabled systems Date: Wed, 11 Sep 2024 14:34:04 +0000 Message-ID: <20240911143421.85612-6-faresx@amazon.de> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240911143421.85612-1-faresx@amazon.de> References: <20240911143421.85612-1-faresx@amazon.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To allocate the vCPU gp-regs using secret memory, we need to dynamically allocate the vCPU gp-regs first. This is tricky with NVHE (Non-Virtualization Host Extensions) since it will require adjusting the virtual address on every access. With a large shared codebase between the OS and the hypervisor, it would be cumbersome to dupli= cate the code with one version using `kern_hyp_va()`. To avoid this issue, and since the secret memory feature will not be enable= d on NVHE systems, we're introducing the following changes: 1. Maintain a `struct user_pt_regs regs_storage` in the vCPU context struct= as a fallback storage for the vCPU gp-regs. 2. Introduce a pointer `struct user_pt_regs *regs` in the vCPU context stru= ct to hold the dynamically allocated vCPU gp-regs. If we are on an NVHE system or a VHE (Virtualization Host Extensions) system that doesn't support `KERNEL_SECRETMEM`, we will use `ctxt_storage`. Access= ing the context in this case will not require a de-reference operation. If we are on a VHE system with support for `KERNEL_SECRETMEM`, we will use = the `regs` pointer. In this case, we will add one de-reference operation every = time the vCPU gp-reg is accessed. Accessing the gp-regs embedded in the vCPU context without de-reference is = done as: add \regs, \ctxt, #CPU_USER_PT_REGS_STRG Accessing the dynamically allocated gp-regs with de-reference is done as: ldr \regs, [\ctxt, #CPU_USER_PT_REGS] By default, we are using the first version. If we are booting on a system t= hat supports VHE and `KERNEL_SECRETMEM`, we switch to the second version. We are also allocating the needed gp-regs allocations for vCPU, kvm_hyp_ctx= t and kvm_host_data structs when needed. Signed-off-by: Fares Mehanna --- arch/arm64/include/asm/kvm_asm.h | 4 +- arch/arm64/include/asm/kvm_host.h | 24 +++++++++++- arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kvm/arm.c | 63 ++++++++++++++++++++++++++++++- arch/arm64/kvm/va_layout.c | 23 +++++++++++ 6 files changed, 112 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_= asm.h index fa4fb642a5f5..1d6de0806dbd 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -314,7 +314,9 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, = u64 spsr, u64 elr_virt, .endm =20 .macro get_ctxt_gp_regs ctxt, regs - add \regs, \ctxt, #CPU_USER_PT_REGS +alternative_cb ARM64_HAS_VIRT_HOST_EXTN, kvm_update_ctxt_gp_regs + add \regs, \ctxt, #CPU_USER_PT_REGS_STRG +alternative_cb_end .endm =20 /* diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index 31cbd62a5d06..23a10178d1b0 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -541,7 +541,9 @@ struct kvm_sysreg_masks { }; =20 struct kvm_cpu_context { - struct user_pt_regs regs; /* sp =3D sp_el0 */ + struct user_pt_regs *regs; /* sp =3D sp_el0 */ + struct user_pt_regs regs_storage; + struct secretmem_area *regs_area; =20 u64 spsr_abt; u64 spsr_und; @@ -946,7 +948,25 @@ struct kvm_vcpu_arch { #define vcpu_clear_on_unsupported_cpu(vcpu) \ vcpu_clear_flag(vcpu, ON_UNSUPPORTED_CPU) =20 -#define ctxt_gp_regs(ctxt) (&(ctxt)->regs) +/* Static allocation is used if NVHE-host or if KERNEL_SECRETMEM is not en= abled */ +static __inline bool kvm_use_dynamic_regs(void) +{ +#ifndef CONFIG_KERNEL_SECRETMEM + return false; +#endif + return cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN); +} + +static __always_inline struct user_pt_regs *ctxt_gp_regs(const struct kvm_= cpu_context *ctxt) +{ + struct user_pt_regs *regs =3D (void *) ctxt; + asm volatile(ALTERNATIVE_CB("add %0, %0, %1\n", + ARM64_HAS_VIRT_HOST_EXTN, + kvm_update_ctxt_gp_regs) + : "+r" (regs) + : "I" (offsetof(struct kvm_cpu_context, regs_storage))); + return regs; +} #define vcpu_gp_regs(v) (ctxt_gp_regs(&(v)->arch.ctxt)) =20 /* diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offset= s.c index 27de1dddb0ab..275d480f5e65 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -128,6 +128,7 @@ int main(void) DEFINE(VCPU_FAULT_DISR, offsetof(struct kvm_vcpu, arch.fault.disr_el1)); DEFINE(VCPU_HCR_EL2, offsetof(struct kvm_vcpu, arch.hcr_el2)); DEFINE(CPU_USER_PT_REGS, offsetof(struct kvm_cpu_context, regs)); + DEFINE(CPU_USER_PT_REGS_STRG, offsetof(struct kvm_cpu_context, regs_stor= age)); DEFINE(CPU_ELR_EL2, offsetof(struct kvm_cpu_context, sys_regs[ELR_EL2])= ); DEFINE(CPU_RGSR_EL1, offsetof(struct kvm_cpu_context, sys_regs[RGSR_EL1= ])); DEFINE(CPU_GCR_EL1, offsetof(struct kvm_cpu_context, sys_regs[GCR_EL1])= ); diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 8f5422ed1b75..e3bb626e299c 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -86,6 +86,7 @@ KVM_NVHE_ALIAS(kvm_patch_vector_branch); KVM_NVHE_ALIAS(kvm_update_va_mask); KVM_NVHE_ALIAS(kvm_get_kimage_voffset); KVM_NVHE_ALIAS(kvm_compute_final_ctr_el0); +KVM_NVHE_ALIAS(kvm_update_ctxt_gp_regs); KVM_NVHE_ALIAS(spectre_bhb_patch_loop_iter); KVM_NVHE_ALIAS(spectre_bhb_patch_loop_mitigation_enable); KVM_NVHE_ALIAS(spectre_bhb_patch_wa3); diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 9bef7638342e..78c562a060de 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include #include @@ -452,6 +453,7 @@ int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned i= nt id) =20 int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) { + unsigned long pages_needed; int err; =20 spin_lock_init(&vcpu->arch.mp_state_lock); @@ -469,6 +471,14 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) =20 vcpu->arch.mmu_page_cache.gfp_zero =3D __GFP_ZERO; =20 + if (kvm_use_dynamic_regs()) { + pages_needed =3D (sizeof(*vcpu_gp_regs(vcpu)) + PAGE_SIZE - 1) / PAGE_SI= ZE; + vcpu->arch.ctxt.regs_area =3D secretmem_allocate_pages(fls(pages_needed = - 1)); + if (!vcpu->arch.ctxt.regs_area) + return -ENOMEM; + vcpu->arch.ctxt.regs =3D vcpu->arch.ctxt.regs_area->ptr; + } + /* Set up the timer */ kvm_timer_vcpu_init(vcpu); =20 @@ -489,9 +499,14 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) =20 err =3D kvm_vgic_vcpu_init(vcpu); if (err) - return err; + goto free_vcpu_ctxt; =20 return kvm_share_hyp(vcpu, vcpu + 1); + +free_vcpu_ctxt: + if (kvm_use_dynamic_regs()) + secretmem_release_pages(vcpu->arch.ctxt.regs_area); + return err; } =20 void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu) @@ -508,6 +523,9 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu) kvm_pmu_vcpu_destroy(vcpu); kvm_vgic_vcpu_destroy(vcpu); kvm_arm_vcpu_destroy(vcpu); + + if (kvm_use_dynamic_regs()) + secretmem_release_pages(vcpu->arch.ctxt.regs_area); } =20 void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) @@ -2683,6 +2701,45 @@ static int __init init_hyp_mode(void) return err; } =20 +static int init_hyp_hve_mode(void) +{ + int cpu; + int err =3D 0; + + if (!kvm_use_dynamic_regs()) + return 0; + + /* Allocate gp-regs */ + for_each_possible_cpu(cpu) { + void *hyp_ctxt_regs; + void *kvm_host_data_regs; + + hyp_ctxt_regs =3D kzalloc(sizeof(struct user_pt_regs), GFP_KERNEL); + if (!hyp_ctxt_regs) { + err =3D -ENOMEM; + goto free_regs; + } + per_cpu(kvm_hyp_ctxt, cpu).regs =3D hyp_ctxt_regs; + + kvm_host_data_regs =3D kzalloc(sizeof(struct user_pt_regs), GFP_KERNEL); + if (!kvm_host_data_regs) { + err =3D -ENOMEM; + goto free_regs; + } + per_cpu(kvm_host_data, cpu).host_ctxt.regs =3D kvm_host_data_regs; + } + + return 0; + +free_regs: + for_each_possible_cpu(cpu) { + kfree(per_cpu(kvm_hyp_ctxt, cpu).regs); + kfree(per_cpu(kvm_host_data, cpu).host_ctxt.regs); + } + + return err; +} + struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr) { struct kvm_vcpu *vcpu =3D NULL; @@ -2806,6 +2863,10 @@ static __init int kvm_arm_init(void) err =3D init_hyp_mode(); if (err) goto out_err; + } else { + err =3D init_hyp_hve_mode(); + if (err) + goto out_err; } =20 err =3D kvm_init_vector_slots(); diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c index 91b22a014610..fcef7e89d042 100644 --- a/arch/arm64/kvm/va_layout.c +++ b/arch/arm64/kvm/va_layout.c @@ -185,6 +185,29 @@ void __init kvm_update_va_mask(struct alt_instr *alt, } } =20 +void __init kvm_update_ctxt_gp_regs(struct alt_instr *alt, + __le32 *origptr, __le32 *updptr, int nr_inst) +{ + u32 rd, rn, imm, insn, oinsn; + + BUG_ON(nr_inst !=3D 1); + + if (!kvm_use_dynamic_regs()) + return; + + oinsn =3D le32_to_cpu(origptr[0]); + rd =3D aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RD, oinsn); + rn =3D aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RN, oinsn); + imm =3D offsetof(struct kvm_cpu_context, regs); + + insn =3D aarch64_insn_gen_load_store_imm(rd, rn, imm, + AARCH64_INSN_SIZE_64, + AARCH64_INSN_LDST_LOAD_IMM_OFFSET); + BUG_ON(insn =3D=3D AARCH64_BREAK_FAULT); + + updptr[0] =3D cpu_to_le32(insn); +} + void kvm_patch_vector_branch(struct alt_instr *alt, __le32 *origptr, __le32 *updptr, int nr_inst) { --=20 2.40.1 Amazon Web Services Development Center Germany GmbH Krausenstr. 38 10117 Berlin Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B Sitz: Berlin Ust-ID: DE 365 538 597