From nobody Sun Nov 24 00:54:03 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1728444121; cv=none; d=zohomail.com; s=zohoarc; b=KSLAV/IlrghoATCSooRcbbTMOvyzJTc+1tpTU3vrdXPdIsaxOnDCQy9rSCCWUecW0t8CohrbRILk24C0xpSuHlRvOU7qACpzK+T+t9xwzx+wcwpmI2TO2WpOOU5HtU8eUYQLN74+ePoqux8jybxg0dNZh2aBMuTv4SH/LTRFXwA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1728444121; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=Jj0uN7kPc/t9wC6+Q32tzsrJ/j6YGIFDMC/1LoRCBC0=; b=gmqPoVqlYH72GzFSytxkvNXEbovoSW8QjNRZoOg/CkoCxrkK8jOl9bSBIlf+QLCdCNgtRRarl4lTAZCcvk6LkRfSNkity+AXhl0V4Zn9qbhZiGPI23OpxDryqszfrPUx3Ykx0p/z1y9jhvSnJi3H5lmsME8IRqYOw/iL5NNvbHo= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1728444121867920.0475425336504; Tue, 8 Oct 2024 20:22:01 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1syNGf-0006OE-RV; Tue, 08 Oct 2024 23:21:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1syNGd-0006HR-Uu; Tue, 08 Oct 2024 23:21:39 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1syNGb-0004Xd-IZ; Tue, 08 Oct 2024 23:21:39 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XNdTB75TXz6K98r; Wed, 9 Oct 2024 11:21:18 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 9C4D2140447; Wed, 9 Oct 2024 11:21:35 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.126.173.89) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Wed, 9 Oct 2024 05:21:16 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V4 06/33] arm/virt, kvm: Pre-create disabled possible vCPUs @machine init Date: Wed, 9 Oct 2024 04:17:48 +0100 Message-ID: <20241009031815.250096-7-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241009031815.250096-1-salil.mehta@huawei.com> References: <20241009031815.250096-1-salil.mehta@huawei.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.126.173.89] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta From: Salil Mehta via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1728444123906116600 Content-Type: text/plain; charset="utf-8" In the ARMv8 architecture, the GIC must know all the CPUs it is connected to during its initialization, and this cannot change afterward. This must be ensured during the initialization of the VGIC in KVM, which requires all vC= PUs to be created and present during its initialization. This is necessary beca= use: 1. The association between GICC and MPIDR must be fixed at VM initialization time. This is represented by the register `GIC_TYPER(mp_affinity, proc_n= um)`. 2. GICC (CPU interfaces), GICR (redistributors), etc., must all be initiali= zed at boot time. 3. Memory regions associated with GICR, etc., cannot be changed (added, del= eted, or modified) after the VM has been initialized. This patch adds support to pre-create all possible vCPUs within the host us= ing the KVM interface as part of the virtual machine initialization. These vCPU= s can later be attached to QOM/ACPI when they are actually hot-plugged and made present. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta Reported-by: Vishnu Pajjuri [VP: Identified CPU stall issue & suggested probable fix] --- hw/arm/virt.c | 69 ++++++++++++++++++++++++++++++++++++++----- include/hw/core/cpu.h | 1 + target/arm/cpu64.c | 1 + target/arm/kvm.c | 41 ++++++++++++++++++++++++- target/arm/kvm_arm.h | 11 +++++++ 5 files changed, 114 insertions(+), 9 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 3db4769289..badde5ed7a 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2364,17 +2364,12 @@ static void machvirt_init(MachineState *machine) =20 assert(possible_cpus->len =3D=3D max_cpus); for (n =3D 0; n < possible_cpus->len; n++) { + CPUArchId *cpu_slot; Object *cpuobj; CPUState *cs; =20 - if (n >=3D smp_cpus) { - break; - } - cpuobj =3D object_new(possible_cpus->cpus[n].type); - cs =3D CPU(cpuobj); - cs->cpu_index =3D n; =20 aarch64 &=3D object_property_get_bool(cpuobj, "aarch64", NULL); object_property_set_int(cpuobj, "socket-id", @@ -2386,8 +2381,57 @@ static void machvirt_init(MachineState *machine) object_property_set_int(cpuobj, "thread-id", virt_get_thread_id(machine, n), NULL); =20 - qdev_realize(DEVICE(cpuobj), NULL, &error_fatal); - object_unref(cpuobj); + if (n < smp_cpus) { + qdev_realize(DEVICE(cpuobj), NULL, &error_fatal); + object_unref(cpuobj); + } else { + /* handling for vCPUs which are yet-to-be hot-plugged */ + cs->cpu_index =3D n; + cpu_slot =3D virt_find_cpu_slot(cs); + + /* GICv3 will need `mp-affinity` to derive `gicr_typer` */ + virt_cpu_set_properties(cpuobj, cpu_slot, &error_fatal); + + /* + * For KVM, we shall be pre-creating the now disabled/un-plugg= ed + * possbile host vcpus and park them till the time they are + * actually hot plugged. This is required to pre-size the host + * GICC and GICR with the all possible vcpus for this VM. + */ + if (kvm_enabled()) { + kvm_arm_create_host_vcpu(ARM_CPU(cs)); + /* + * Override the default architecture ID with the one fetch= ed + * from KVM. After initialization, we will destroy the CPU= State + * for disabled vCPUs; however, the CPU slot and its assoc= iation + * with the architecture ID (and consequently the vCPU ID)= will + * remain fixed for the entire lifetime of QEMU and cannot= be + * altered. This is also an ARM CPU architectural constrai= nt. + */ + cpu_slot->arch_id =3D arm_cpu_mp_affinity(ARM_CPU(cs)); + } + /* + * Add disabled vCPU to CPU slot during the init phase of the = virt + * machine + * 1. We need this ARMCPU object during the GIC init. This obj= ect + * will facilitate in pre-realizing the GIC. Any info like + * mp-affinity(required to derive gicr_type) etc. could sti= ll be + * fetched while preserving QOM abstraction akin to realized + * vCPUs. + * 2. Now, after initialization of the virt machine is complet= e we + * could use two approaches to deal with this ARMCPU object: + * (i) re-use this ARMCPU object during hotplug of this vCP= U. + * OR + * (ii) defer release this ARMCPU object after gic has been + * initialized or during pre-plug phase when a vCPU is + * hotplugged. + * + * We will use the (ii) approach and release the ARMCPU obj= ects + * after GIC and machine has been fully initialized during + * machine_init_done() phase. + */ + cpu_slot->cpu =3D cs; + } } =20 /* Now we've created the CPUs we can see if they have the hypvirt time= r */ @@ -2990,6 +3034,15 @@ static void virt_cpu_plug(HotplugHandler *hotplug_de= v, DeviceState *dev, /* insert the cold/hot-plugged vcpu in the slot */ cpu_slot =3D virt_find_cpu_slot(cs); cpu_slot->cpu =3D CPU(dev); + if (kvm_enabled()) { + /* + * Override the default architecture ID with the one fetched from = KVM + * Currently, KVM derives the architecture ID from the vCPU ID spe= cified + * by QEMU. In the future, we might implement a change where the e= ntire + * architecture ID can be configured directly by QEMU. + */ + cpu_slot->arch_id =3D arm_cpu_mp_affinity(ARM_CPU(cs)); + } =20 cs->disabled =3D false; } diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index 73a4e4cce1..bcc62fbf0c 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -530,6 +530,7 @@ struct CPUState { uint64_t dirty_pages; int kvm_vcpu_stats_fd; bool vcpu_dirty; + VMChangeStateEntry *vmcse; =20 /* Use by accel-block: CPU is executing an ioctl() */ QemuLockCnt in_ioctl_lock; diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index d36d3519df..2a517fdb9f 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -793,6 +793,7 @@ static void aarch64_cpu_initfn(Object *obj) * explicitly enabled. */ cs->disabled =3D true; + cs->thread_id =3D 0; } =20 static void aarch64_cpu_finalizefn(Object *obj) diff --git a/target/arm/kvm.c b/target/arm/kvm.c index 849e2e21b3..8ed160cbca 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -1003,6 +1003,38 @@ void kvm_arm_reset_vcpu(ARMCPU *cpu) write_list_to_cpustate(cpu); } =20 +void kvm_arm_create_host_vcpu(ARMCPU *cpu) +{ + CPUState *cs =3D CPU(cpu); + unsigned long vcpu_id =3D cs->cpu_index; + int ret; + + ret =3D kvm_create_vcpu(cs); + if (ret < 0) { + error_report("Failed to create host vcpu %ld", vcpu_id); + abort(); + } + + /* + * Initialize the vCPU in the host. This will reset the sys regs + * for this vCPU and related registers like MPIDR_EL1 etc. also + * gets programmed during this call to host. These are referred + * later while setting device attributes of the GICR during GICv3 + * reset + */ + ret =3D kvm_arch_init_vcpu(cs); + if (ret < 0) { + error_report("Failed to initialize host vcpu %ld", vcpu_id); + abort(); + } + + /* + * park the created vCPU. shall be used during kvm_get_vcpu() when + * threads are created during realization of ARM vCPUs. + */ + kvm_park_vcpu(cs); +} + /* * Update KVM's MP_STATE based on what QEMU thinks it is */ @@ -1874,7 +1906,14 @@ int kvm_arch_init_vcpu(CPUState *cs) return -EINVAL; } =20 - qemu_add_vm_change_state_handler(kvm_arm_vm_state_change, cpu); + /* + * Install VM change handler only when vCPU thread has been spawned + * i.e. vCPU is being realized + */ + if (cs->thread_id) { + cs->vmcse =3D qemu_add_vm_change_state_handler(kvm_arm_vm_state_ch= ange, + cpu); + } =20 /* Determine init features for this CPU */ memset(cpu->kvm_init_features, 0, sizeof(cpu->kvm_init_features)); diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h index cfaa0d9bc7..0be7e896d2 100644 --- a/target/arm/kvm_arm.h +++ b/target/arm/kvm_arm.h @@ -96,6 +96,17 @@ void kvm_arm_cpu_post_load(ARMCPU *cpu); */ void kvm_arm_reset_vcpu(ARMCPU *cpu); =20 +/** + * kvm_arm_create_host_vcpu: + * @cpu: ARMCPU + * + * Called at to pre create all possible kvm vCPUs within the the host at t= he + * virt machine init time. This will also init this pre-created vCPU and + * hence result in vCPU reset at host. These pre created and inited vCPUs + * shall be parked for use when ARM vCPUs are actually realized. + */ +void kvm_arm_create_host_vcpu(ARMCPU *cpu); + #ifdef CONFIG_KVM /** * kvm_arm_create_scratch_host_vcpu: --=20 2.34.1