From nobody Mon Nov 25 03:41:40 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1718322045; cv=none; d=zohomail.com; s=zohoarc; b=ctSbReSJhiK6Ri50JUUdwHBIu1HgxfNYRiqWaTWQrWx3t0vIZJOV6HP6vdnw82cwUVy+meofJKWJiM+DV6jn82kMgRIkNaHoix0W4YTL0sHR/cD2ZyWtHoTH/b2acx0abuu/H5hSjIbddqqtNxSpXKu/pJLLJa3bnd/r8Q4nqGE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1718322045; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=1WqnPCx7a+wQq2Icw+lRB/hvpTrs/NIkDFLNFWJkmKs=; b=jbd2F5yORs+ZJKAttGFqyVtYwBUSKFVDMUItiwrii6fseXmr+rhtv1d89YBG0WsNo5kxX5tggrOJpCOUL7bYi7cZXQjpzAO612GmIApUWcaIVjeAPzq8CobzssNodUNAP/Z0FXRZ4LJXBWqn9oZ3D9IYk4hFXSSnkgMf6bTKKY0= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1718322045167905.3329169221939; Thu, 13 Jun 2024 16:40:45 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu3Y-0008A6-DV; Thu, 13 Jun 2024 19:40:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu3I-0007kb-CZ; Thu, 13 Jun 2024 19:40:22 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu3D-0003KP-NT; Thu, 13 Jun 2024 19:40:17 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f0c0zWJz6HJTY; Fri, 14 Jun 2024 07:35:28 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 6C9A8140594; Fri, 14 Jun 2024 07:40:10 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:39:48 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 06/29] arm/virt, kvm: Pre-create disabled possible vCPUs @machine init Date: Fri, 14 Jun 2024 00:36:16 +0100 Message-ID: <20240613233639.202896-7-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta From: Salil Mehta via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1718322047065100003 Content-Type: text/plain; charset="utf-8" In the ARMv8 architecture, the GIC must know all the CPUs it is connected to during its initialization, and this cannot change afterward. This must be ensured during the initialization of the VGIC as well in KVM, which require= s all vCPUs to be created and present during its initialization. This is necessary because: 1. The association between GICC and MPIDR must be fixed at VM initialization time. This is represented by the register `GIC_TYPER(mp_affinity, proc_n= um)`. 2. GICC (CPU interfaces), GICR (redistributors), etc., must all be initiali= zed at boot time. 3. Memory regions associated with GICR, etc., cannot be changed (added, del= eted, or modified) after the VM has been initialized. This patch adds support to pre-create all possible vCPUs within the host us= ing the KVM interface as part of the virtual machine initialization. These vCPU= s can later be attached to QOM/ACPI when they are actually hot-plugged and made present. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta Reported-by: Vishnu Pajjuri [VP: Identified CPU stall issue & suggested probable fix] --- hw/arm/virt.c | 56 +++++++++++++++++++++++++++++++++++-------- include/hw/core/cpu.h | 1 + target/arm/cpu64.c | 1 + target/arm/kvm.c | 41 ++++++++++++++++++++++++++++++- target/arm/kvm_arm.h | 11 +++++++++ 5 files changed, 99 insertions(+), 11 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index a285139165..81e7a27786 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2383,14 +2383,8 @@ static void machvirt_init(MachineState *machine) Object *cpuobj; CPUState *cs; =20 - if (n >=3D smp_cpus) { - break; - } - cpuobj =3D object_new(possible_cpus->cpus[n].type); - cs =3D CPU(cpuobj); - cs->cpu_index =3D n; =20 aarch64 &=3D object_property_get_bool(cpuobj, "aarch64", NULL); object_property_set_int(cpuobj, "socket-id", @@ -2402,11 +2396,53 @@ static void machvirt_init(MachineState *machine) object_property_set_int(cpuobj, "thread-id", virt_get_thread_id(machine, n), NULL); =20 - cpu_slot =3D virt_find_cpu_slot(machine, cs->cpu_index); - virt_cpu_set_properties(cpuobj, cpu_slot, &error_fatal); + if (n < smp_cpus) { + qdev_realize(DEVICE(cpuobj), NULL, &error_fatal); + object_unref(cpuobj); + } else { + /* handling for vcpus which are yet to be hot-plugged */ + cs->cpu_index =3D n; + cpu_slot =3D virt_find_cpu_slot(machine, cs->cpu_index); =20 - qdev_realize(DEVICE(cpuobj), NULL, &error_fatal); - object_unref(cpuobj); + /* + * ARM host vCPU features need to be fixed at the boot time. B= ut as + * per current approach this CPU object will be destroyed duri= ng + * cpu_post_init(). During hotplug of vCPUs these properties a= re + * initialized again. + */ + virt_cpu_set_properties(cpuobj, cpu_slot, &error_fatal); + + /* + * For KVM, we shall be pre-creating the now disabled/un-plugg= ed + * possbile host vcpus and park them till the time they are + * actually hot plugged. This is required to pre-size the host + * GICC and GICR with the all possible vcpus for this VM. + */ + if (kvm_enabled()) { + kvm_arm_create_host_vcpu(ARM_CPU(cs)); + } + /* + * Add disabled vCPU to CPU slot during the init phase of the = virt + * machine + * 1. We need this ARMCPU object during the GIC init. This obj= ect + * will facilitate in pre-realizing the GIC. Any info like + * mp-affinity(required to derive gicr_type) etc. could sti= ll be + * fetched while preserving QOM abstraction akin to realized + * vCPUs. + * 2. Now, after initialization of the virt machine is complet= e we + * could use two approaches to deal with this ARMCPU object: + * (i) re-use this ARMCPU object during hotplug of this vCP= U. + * OR + * (ii) defer release this ARMCPU object after gic has been + * initialized or during pre-plug phase when a vCPU is + * hotplugged. + * + * We will use the (ii) approach and release the ARMCPU obj= ects + * after GIC and machine has been fully initialized during + * machine_init_done() phase. + */ + cpu_slot->cpu =3D cs; + } } =20 /* Now we've created the CPUs we can see if they have the hypvirt time= r */ diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index 60b4778da9..62e68611c0 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -520,6 +520,7 @@ struct CPUState { uint64_t dirty_pages; int kvm_vcpu_stats_fd; bool vcpu_dirty; + VMChangeStateEntry *vmcse; =20 /* Use by accel-block: CPU is executing an ioctl() */ QemuLockCnt in_ioctl_lock; diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index d6b48b3424..9b7e8b032c 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -789,6 +789,7 @@ static void aarch64_cpu_initfn(Object *obj) * enabled explicitly */ cs->disabled =3D true; + cs->thread_id =3D 0; } =20 static void aarch64_cpu_finalizefn(Object *obj) diff --git a/target/arm/kvm.c b/target/arm/kvm.c index 7cf5cf31de..01c83c1994 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -1003,6 +1003,38 @@ void kvm_arm_reset_vcpu(ARMCPU *cpu) write_list_to_cpustate(cpu); } =20 +void kvm_arm_create_host_vcpu(ARMCPU *cpu) +{ + CPUState *cs =3D CPU(cpu); + unsigned long vcpu_id =3D cs->cpu_index; + int ret; + + ret =3D kvm_create_vcpu(cs); + if (ret < 0) { + error_report("Failed to create host vcpu %ld", vcpu_id); + abort(); + } + + /* + * Initialize the vCPU in the host. This will reset the sys regs + * for this vCPU and related registers like MPIDR_EL1 etc. also + * gets programmed during this call to host. These are referred + * later while setting device attributes of the GICR during GICv3 + * reset + */ + ret =3D kvm_arch_init_vcpu(cs); + if (ret < 0) { + error_report("Failed to initialize host vcpu %ld", vcpu_id); + abort(); + } + + /* + * park the created vCPU. shall be used during kvm_get_vcpu() when + * threads are created during realization of ARM vCPUs. + */ + kvm_park_vcpu(cs); +} + /* * Update KVM's MP_STATE based on what QEMU thinks it is */ @@ -1874,7 +1906,14 @@ int kvm_arch_init_vcpu(CPUState *cs) return -EINVAL; } =20 - qemu_add_vm_change_state_handler(kvm_arm_vm_state_change, cpu); + /* + * Install VM change handler only when vCPU thread has been spawned + * i.e. vCPU is being realized + */ + if (cs->thread_id) { + cs->vmcse =3D qemu_add_vm_change_state_handler(kvm_arm_vm_state_ch= ange, + cpu); + } =20 /* Determine init features for this CPU */ memset(cpu->kvm_init_features, 0, sizeof(cpu->kvm_init_features)); diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h index cfaa0d9bc7..0be7e896d2 100644 --- a/target/arm/kvm_arm.h +++ b/target/arm/kvm_arm.h @@ -96,6 +96,17 @@ void kvm_arm_cpu_post_load(ARMCPU *cpu); */ void kvm_arm_reset_vcpu(ARMCPU *cpu); =20 +/** + * kvm_arm_create_host_vcpu: + * @cpu: ARMCPU + * + * Called at to pre create all possible kvm vCPUs within the the host at t= he + * virt machine init time. This will also init this pre-created vCPU and + * hence result in vCPU reset at host. These pre created and inited vCPUs + * shall be parked for use when ARM vCPUs are actually realized. + */ +void kvm_arm_create_host_vcpu(ARMCPU *cpu); + #ifdef CONFIG_KVM /** * kvm_arm_create_scratch_host_vcpu: --=20 2.34.1