From nobody Wed Oct 23 00:36:46 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1728986598; cv=none; d=zohomail.com; s=zohoarc; b=Ru3B5wPv4Tr+/r33Bq0ua/W/wklV82gU2ghma3QDr0LZoa9O68XpUsyn0h1DfoEhzhLnBhXY2mFx6IeTmlPwKXKA15XJY+jwiD1b7QdsNk8doupDZP04SHz82i3dAzRr5+7CMkwD6hYW16pl+sUsV0KbK2BqiAMqKeB0MCb4080= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1728986598; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=lGk/UP+qtzibj/mRCKdjrYu6YcvO/SH2PWHIzYlYnUk=; b=A2NecLLJ40P3tL1pFv3hnIJLk6/ZfCvv1+IpowRLx9s98QlQdlYoc9YHCal6FaK+/uv0E29/ASpycLASMgv/seLWFkbn99EAqxqHaPXLjxgTAyqYaRTYu4wT5oOvWoXn6riw3NCpZcNcuqhu1rG+BZhXUpzJoSn8eURiamx5J0o= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1728986598361369.9116007950347; Tue, 15 Oct 2024 03:03:18 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eOW-0001FW-95; Tue, 15 Oct 2024 06:03:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eO9-0000kW-66; Tue, 15 Oct 2024 06:02:50 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eO6-0000wm-WB; Tue, 15 Oct 2024 06:02:48 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV3k5NW4z6FGmc; Tue, 15 Oct 2024 18:01:06 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id E3F2D1404F9; Tue, 15 Oct 2024 18:02:43 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:02:24 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 05/30] arm/virt, kvm: Pre-create KVM vCPUs for all unplugged QOM vCPUs @machine init Date: Tue, 15 Oct 2024 10:59:47 +0100 Message-ID: <20241015100012.254223-6-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta From: Salil Mehta via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1728986600252116600 Content-Type: text/plain; charset="utf-8" ARM CPU architecture does not allows CPU to be plugged after system has initialized. This is an constraint. Hence, Kernel must know all the CPUs be= ing booted during its initialization. This applies to the Guest Kernel as well = and therefore, number of KVM vCPUs need to be fixed at the VM initialization ti= me. Also, The GIC must know all the CPUs it is connected to during its initialization, and this cannot change afterward. This must also be ensured during the initialization of the VGIC in KVM. This is necessary because: 1. The association between GICR and MPIDR must be fixed at VM initialization time. This is represented by the register `GICR_TYPER(mp_affinity, proc_= num)` 2. Memory regions associated with GICR, etc., cannot be changed (added, del= eted, or modified) after the VM has been initialized. This is not an ARM architectural constraint but rather invites a difficult and messy change= in VGIC data structures. This patch adds support to pre-create all possible vCPUs within the KVM host as part of the virtual machine initialization. These vCPUs can later be unp= arked and attached to the hotplugged QOM vCPU. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta Reported-by: Vishnu Pajjuri [VP: Identified CPU stall issue & suggested probable fix] --- hw/arm/virt.c | 74 ++++++++++++++++++++++++++++++++++++++----- include/hw/core/cpu.h | 1 + target/arm/cpu64.c | 9 ++++++ target/arm/kvm.c | 41 +++++++++++++++++++++++- target/arm/kvm_arm.h | 10 ++++++ 5 files changed, 126 insertions(+), 9 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 48102d5a4c..858a19bc8b 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2366,17 +2366,12 @@ static void machvirt_init(MachineState *machine) =20 assert(possible_cpus->len =3D=3D max_cpus); for (n =3D 0; n < possible_cpus->len; n++) { + CPUArchId *cpu_slot; Object *cpuobj; CPUState *cs; =20 - if (n >=3D smp_cpus) { - break; - } - cpuobj =3D object_new(possible_cpus->cpus[n].type); - cs =3D CPU(cpuobj); - cs->cpu_index =3D n; =20 aarch64 &=3D object_property_get_bool(cpuobj, "aarch64", NULL); object_property_set_int(cpuobj, "socket-id", @@ -2388,8 +2383,61 @@ static void machvirt_init(MachineState *machine) object_property_set_int(cpuobj, "thread-id", virt_get_thread_id(machine, n), NULL); =20 - qdev_realize(DEVICE(cpuobj), NULL, &error_fatal); - object_unref(cpuobj); + if (n < smp_cpus) { + qdev_realize(DEVICE(cpuobj), NULL, &error_fatal); + object_unref(cpuobj); + } else { + /* + * Handling vCPUs that are yet to be hot-plugged requires the + * unrealized `ARMCPU` object for the following purposes: + * + * 1. To create the corresponding host KVM vCPU. + * 2. During the GICv3 realization phase, the `GICR_TYPER` val= ue is + * derived using the fetched MPIDR/mp-affinity. It's worth + * considering modifying the GICv3 realization code to dire= ctly + * fetch the `arch-id`/mp-affinity from the possible vCPUs. + * 3. Additionally, the `ARMCPU` object must be retained until + * `virt_cpu_post_init`, as there may be late per-vCPU + * initializations. + * + * Once these tasks are completed, the initialized `ARMCPU` ob= ject + * can be safely released as those are not required and will be + * recreated when they are {hot,cold}-plugged later. + */ + cs->cpu_index =3D n; + cpu_slot =3D virt_find_cpu_slot(cs); + + /* + * We will pre-create the KVM vCPUs corresponding to the curre= ntly + * unplugged but possible QOM vCPUs and park them until they a= re + * actually hot-plugged. The ARM architecture does not allow n= ew + * CPUs to be plugged after the system has been initialized, a= nd + * this constraint is also reflected in KVM. + */ + if (kvm_enabled()) { + kvm_arm_create_host_vcpu(ARM_CPU(cs)); + /* + * Override the default architecture ID with the one fetch= ed + * from KVM. After initialization, we will destroy the CPU= State + * for disabled vCPUs; however, the CPU slot and its assoc= iation + * with the architecture ID (and consequently the vCPU ID)= will + * remain fixed for the entire lifetime of QEMU. + */ + cpu_slot->arch_id =3D arm_cpu_mp_affinity(ARM_CPU(cs)); + } + + /* + * GICv3 realization will need `mp-affinity` to derive `gicr_ty= per` + */ + object_property_set_int(cpuobj, "mp-affinity", cpu_slot->arch_= id, + NULL); + + /* + * Add the unplugged vCPU to the vCPU slot temporarily. It wil= l be + * released later in the virt_post_init_cpu() function + */ + cpu_slot->cpu =3D cs; + } } =20 /* Now we've created the CPUs we can see if they have the hypvirt time= r */ @@ -2992,6 +3040,16 @@ static void virt_cpu_plug(HotplugHandler *hotplug_de= v, DeviceState *dev, /* insert the cold/hot-plugged vcpu in the slot */ cpu_slot =3D virt_find_cpu_slot(cs); cpu_slot->cpu =3D CPU(dev); + + if (kvm_enabled()) { + /* + * Override the default architecture ID with the one fetched from = KVM + * Currently, KVM derives the architecture ID from the vCPU ID spe= cified + * by QEMU. In the future, we might implement a change where the e= ntire + * architecture ID can be configured directly by QEMU. + */ + cpu_slot->arch_id =3D arm_cpu_mp_affinity(ARM_CPU(cs)); + } } =20 static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index 299e96c45b..4e0cb325a0 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -534,6 +534,7 @@ struct CPUState { uint64_t dirty_pages; int kvm_vcpu_stats_fd; bool vcpu_dirty; + VMChangeStateEntry *vmcse; =20 /* Use by accel-block: CPU is executing an ioctl() */ QemuLockCnt in_ioctl_lock; diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index 0e217f827e..d2f4624d61 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -791,6 +791,14 @@ static void aarch64_cpu_set_aarch64(Object *obj, bool = value, Error **errp) } } =20 +static void aarch64_cpu_initfn(Object *obj) +{ + CPUState *cs =3D CPU(obj); + + /* TODO: re-check if this is necessary still */ + cs->thread_id =3D 0; +} + static void aarch64_cpu_finalizefn(Object *obj) { } @@ -850,6 +858,7 @@ void aarch64_cpu_register(const ARMCPUInfo *info) static const TypeInfo aarch64_cpu_type_info =3D { .name =3D TYPE_AARCH64_CPU, .parent =3D TYPE_ARM_CPU, + .instance_init =3D aarch64_cpu_initfn, .instance_finalize =3D aarch64_cpu_finalizefn, .abstract =3D true, .class_init =3D aarch64_cpu_class_init, diff --git a/target/arm/kvm.c b/target/arm/kvm.c index f1f1b5b375..e82cb2aa8b 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -1003,6 +1003,38 @@ void kvm_arm_reset_vcpu(ARMCPU *cpu) write_list_to_cpustate(cpu); } =20 +void kvm_arm_create_host_vcpu(ARMCPU *cpu) +{ + CPUState *cs =3D CPU(cpu); + unsigned long vcpu_id =3D cs->cpu_index; + int ret; + + ret =3D kvm_create_vcpu(cs); + if (ret < 0) { + error_report("Failed to create host vcpu %ld", vcpu_id); + abort(); + } + + /* + * Initialize the vCPU in the host. This will reset the sys regs + * for this vCPU and related registers like MPIDR_EL1 etc. also + * gets programmed during this call to host. These are referred + * later while setting device attributes of the GICR during GICv3 + * reset + */ + ret =3D kvm_arch_init_vcpu(cs); + if (ret < 0) { + error_report("Failed to initialize host vcpu %ld", vcpu_id); + abort(); + } + + /* + * park the created vCPU. shall be used during kvm_get_vcpu() when + * threads are created during realization of ARM vCPUs. + */ + kvm_park_vcpu(cs); +} + /* * Update KVM's MP_STATE based on what QEMU thinks it is */ @@ -1874,7 +1906,14 @@ int kvm_arch_init_vcpu(CPUState *cs) return -EINVAL; } =20 - qemu_add_vm_change_state_handler(kvm_arm_vm_state_change, cpu); + /* + * Install VM change handler only when vCPU thread has been spawned + * i.e. vCPU is being realized + */ + if (cs->thread_id) { + cs->vmcse =3D qemu_add_vm_change_state_handler(kvm_arm_vm_state_ch= ange, + cpu); + } =20 /* Determine init features for this CPU */ memset(cpu->kvm_init_features, 0, sizeof(cpu->kvm_init_features)); diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h index cfaa0d9bc7..93d12eeb74 100644 --- a/target/arm/kvm_arm.h +++ b/target/arm/kvm_arm.h @@ -96,6 +96,16 @@ void kvm_arm_cpu_post_load(ARMCPU *cpu); */ void kvm_arm_reset_vcpu(ARMCPU *cpu); =20 +/** + * kvm_arm_create_host_vcpu: + * @cpu: ARMCPU + * + * Called to pre-create possible KVM vCPU within the host during the + * `virt_machine` initialization phase. This pre-created vCPU will be park= ed and + * will be reused when ARM QOM vCPU is actually hotplugged. + */ +void kvm_arm_create_host_vcpu(ARMCPU *cpu); + #ifdef CONFIG_KVM /** * kvm_arm_create_scratch_host_vcpu: --=20 2.34.1