From nobody Mon Nov 25 03:32:44 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1718322446; cv=none; d=zohomail.com; s=zohoarc; b=PT5cJy9hPmJk/lzeYS1D6eV/MMe/adxYdBs1zce/ay1FhZwW+0+qq6KNmVrNmG6DUcTAcSYRZKYLH9IK/3HMNRXQXxk/tGVTm68bdWDLi2KNaSq7Ge7xISkL4qcn7XQnGqd0RLMH0CBugQL4SDmYwdRc/61helX16/UBN6c2uh8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1718322446; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=AEcJedHVz4FCPSBXV6KLJpHI1rsa+Ek6tedwEO7yh3c=; b=cc6cgG+j9axF/3DImAVqKbJNx8Zqnhd53QV3iqx2Hx4exJNRnrC8Lz1XTDGKFnAhsLTK0wF72MKJB8+jxw+kALc0IvKlDjTKPwqnSnqGPhjM/1I4D/hPc4os1yON7ShjU00zqN/og/tIXm2CF9DCAGOxsYDwKvspkY4er1vNhKs= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1718322446300984.609509256942; Thu, 13 Jun 2024 16:47:26 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sHu9q-0005zV-JB; Thu, 13 Jun 2024 19:47:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu9n-0005q6-1u; Thu, 13 Jun 2024 19:47:03 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sHu9k-0004kI-OV; Thu, 13 Jun 2024 19:47:02 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4W0f8S0zM6z67j6n; Fri, 14 Jun 2024 07:42:16 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 7279014065C; Fri, 14 Jun 2024 07:46:58 +0800 (CST) Received: from 00293818-MRGF.china.huawei.com (10.195.245.24) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 14 Jun 2024 00:46:35 +0100 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V3 24/29] target/arm: Add support of *unrealize* ARMCPU during vCPU Hot-unplug Date: Fri, 14 Jun 2024 00:36:34 +0100 Message-ID: <20240613233639.202896-25-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240613233639.202896-1-salil.mehta@huawei.com> References: <20240613233639.202896-1-salil.mehta@huawei.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.195.245.24] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To lhrpeml500001.china.huawei.com (7.191.163.213) Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta From: Salil Mehta via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1718322446904100001 Content-Type: text/plain; charset="utf-8" vCPU Hot-unplug will result in QOM CPU object unrealization which will do a= way with all the vCPU thread creations, allocations, registrations that happened as part of the realization process. This change introduces the ARM CPU unre= alize function taking care of exactly that. Note, initialized KVM vCPUs are not destroyed in host KVM but their Qemu co= ntext is parked at the QEMU KVM layer. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta Reported-by: Vishnu Pajjuri [VP: Identified CPU stall issue & suggested probable fix] Signed-off-by: Salil Mehta --- target/arm/cpu.c | 101 +++++++++++++++++++++++++++++++++++++++++ target/arm/cpu.h | 14 ++++++ target/arm/gdbstub.c | 6 +++ target/arm/helper.c | 25 ++++++++++ target/arm/internals.h | 3 ++ target/arm/kvm.c | 5 ++ 6 files changed, 154 insertions(+) diff --git a/target/arm/cpu.c b/target/arm/cpu.c index c92162fa97..a3dc669309 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -157,6 +157,16 @@ void arm_register_pre_el_change_hook(ARMCPU *cpu, ARME= LChangeHookFn *hook, QLIST_INSERT_HEAD(&cpu->pre_el_change_hooks, entry, node); } =20 +void arm_unregister_pre_el_change_hooks(ARMCPU *cpu) +{ + ARMELChangeHook *entry, *next; + + QLIST_FOREACH_SAFE(entry, &cpu->pre_el_change_hooks, node, next) { + QLIST_REMOVE(entry, node); + g_free(entry); + } +} + void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, void *opaque) { @@ -168,6 +178,16 @@ void arm_register_el_change_hook(ARMCPU *cpu, ARMELCha= ngeHookFn *hook, QLIST_INSERT_HEAD(&cpu->el_change_hooks, entry, node); } =20 +void arm_unregister_el_change_hooks(ARMCPU *cpu) +{ + ARMELChangeHook *entry, *next; + + QLIST_FOREACH_SAFE(entry, &cpu->el_change_hooks, node, next) { + QLIST_REMOVE(entry, node); + g_free(entry); + } +} + static void cp_reg_reset(gpointer key, gpointer value, gpointer opaque) { /* Reset a single ARMCPRegInfo register */ @@ -2552,6 +2572,85 @@ static void arm_cpu_realizefn(DeviceState *dev, Erro= r **errp) acc->parent_realize(dev, errp); } =20 +static void arm_cpu_unrealizefn(DeviceState *dev) +{ + ARMCPUClass *acc =3D ARM_CPU_GET_CLASS(dev); + ARMCPU *cpu =3D ARM_CPU(dev); + CPUARMState *env =3D &cpu->env; + CPUState *cs =3D CPU(dev); + bool has_secure; + + has_secure =3D cpu->has_el3 || arm_feature(env, ARM_FEATURE_M_SECURITY= ); + + /* rock 'n' un-roll, whatever happened in the arm_cpu_realizefn cleanl= y */ + cpu_address_space_destroy(cs, ARMASIdx_NS); + + if (cpu->tag_memory !=3D NULL) { + cpu_address_space_destroy(cs, ARMASIdx_TagNS); + if (has_secure) { + cpu_address_space_destroy(cs, ARMASIdx_TagS); + } + } + + if (has_secure) { + cpu_address_space_destroy(cs, ARMASIdx_S); + } + + destroy_cpreg_list(cpu); + arm_cpu_unregister_gdb_regs(cpu); + unregister_cp_regs_for_features(cpu); + + if (cpu->sau_sregion && arm_feature(env, ARM_FEATURE_M_SECURITY)) { + g_free(env->sau.rbar); + g_free(env->sau.rlar); + } + + if (arm_feature(env, ARM_FEATURE_PMSA) && + arm_feature(env, ARM_FEATURE_V7) && + cpu->pmsav7_dregion) { + if (arm_feature(env, ARM_FEATURE_V8)) { + g_free(env->pmsav8.rbar[M_REG_NS]); + g_free(env->pmsav8.rlar[M_REG_NS]); + if (arm_feature(env, ARM_FEATURE_M_SECURITY)) { + g_free(env->pmsav8.rbar[M_REG_S]); + g_free(env->pmsav8.rlar[M_REG_S]); + } + } else { + g_free(env->pmsav7.drbar); + g_free(env->pmsav7.drsr); + g_free(env->pmsav7.dracr); + } + if (cpu->pmsav8r_hdregion) { + g_free(env->pmsav8.hprbar); + g_free(env->pmsav8.hprlar); + } + } + + if (arm_feature(env, ARM_FEATURE_PMU)) { + if (!kvm_enabled()) { + arm_unregister_pre_el_change_hooks(cpu); + arm_unregister_el_change_hooks(cpu); + } + +#ifndef CONFIG_USER_ONLY + if (cpu->pmu_timer) { + timer_del(cpu->pmu_timer); + } +#endif + } + + cpu_remove_sync(CPU(dev)); + acc->parent_unrealize(dev); + +#ifndef CONFIG_USER_ONLY + timer_del(cpu->gt_timer[GTIMER_PHYS]); + timer_del(cpu->gt_timer[GTIMER_VIRT]); + timer_del(cpu->gt_timer[GTIMER_HYP]); + timer_del(cpu->gt_timer[GTIMER_SEC]); + timer_del(cpu->gt_timer[GTIMER_HYPVIRT]); +#endif +} + static ObjectClass *arm_cpu_class_by_name(const char *cpu_model) { ObjectClass *oc; @@ -2654,6 +2753,8 @@ static void arm_cpu_class_init(ObjectClass *oc, void = *data) =20 device_class_set_parent_realize(dc, arm_cpu_realizefn, &acc->parent_realize); + device_class_set_parent_unrealize(dc, arm_cpu_unrealizefn, + &acc->parent_unrealize); =20 device_class_set_props(dc, arm_cpu_properties); =20 diff --git a/target/arm/cpu.h b/target/arm/cpu.h index 208c719db3..a4a7555f7e 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -1118,6 +1118,7 @@ struct ARMCPUClass { =20 const ARMCPUInfo *info; DeviceRealize parent_realize; + DeviceUnrealize parent_unrealize; ResettablePhases parent_phases; }; =20 @@ -3228,6 +3229,13 @@ static inline AddressSpace *arm_addressspace(CPUStat= e *cs, MemTxAttrs attrs) */ void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, void *opaque); +/** + * arm_unregister_pre_el_change_hook: + * unregister all pre EL change hook functions. Generally called during + * unrealize'ing leg + */ +void arm_unregister_pre_el_change_hooks(ARMCPU *cpu); + /** * arm_register_el_change_hook: * Register a hook function which will be called immediately after this @@ -3240,6 +3248,12 @@ void arm_register_pre_el_change_hook(ARMCPU *cpu, AR= MELChangeHookFn *hook, */ void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, void *opaque); +/** + * arm_unregister_el_change_hook: + * unregister all EL change hook functions. Generally called during + * unrealize'ing leg + */ +void arm_unregister_el_change_hooks(ARMCPU *cpu); =20 /** * arm_rebuild_hflags: diff --git a/target/arm/gdbstub.c b/target/arm/gdbstub.c index a3bb73cfa7..948e40b981 100644 --- a/target/arm/gdbstub.c +++ b/target/arm/gdbstub.c @@ -555,3 +555,9 @@ void arm_cpu_register_gdb_regs_for_features(ARMCPU *cpu) } #endif /* CONFIG_TCG */ } + +void arm_cpu_unregister_gdb_regs(ARMCPU *cpu) +{ + CPUState *cs =3D CPU(cpu); + gdb_unregister_coprocessor_all(cs); +} diff --git a/target/arm/helper.c b/target/arm/helper.c index 7587635960..9a2468347a 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -264,6 +264,19 @@ void init_cpreg_list(ARMCPU *cpu) g_list_free(keys); } =20 +void destroy_cpreg_list(ARMCPU *cpu) +{ + assert(cpu->cpreg_indexes); + assert(cpu->cpreg_values); + assert(cpu->cpreg_vmstate_indexes); + assert(cpu->cpreg_vmstate_values); + + g_free(cpu->cpreg_indexes); + g_free(cpu->cpreg_values); + g_free(cpu->cpreg_vmstate_indexes); + g_free(cpu->cpreg_vmstate_values); +} + static bool arm_pan_enabled(CPUARMState *env) { if (is_a64(env)) { @@ -9987,6 +10000,18 @@ void register_cp_regs_for_features(ARMCPU *cpu) #endif } =20 +void unregister_cp_regs_for_features(ARMCPU *cpu) +{ + CPUARMState *env =3D &cpu->env; + if (arm_feature(env, ARM_FEATURE_M)) { + /* M profile has no coprocessor registers */ + return; + } + + /* empty it all. unregister all the coprocessor registers */ + g_hash_table_remove_all(cpu->cp_regs); +} + /* * Private utility function for define_one_arm_cp_reg_with_opaque(): * add a single reginfo struct to the hash table. diff --git a/target/arm/internals.h b/target/arm/internals.h index ee3ebd383e..34dab0bb02 100644 --- a/target/arm/internals.h +++ b/target/arm/internals.h @@ -353,9 +353,12 @@ void arm_cpu_register(const ARMCPUInfo *info); void aarch64_cpu_register(const ARMCPUInfo *info); =20 void register_cp_regs_for_features(ARMCPU *cpu); +void unregister_cp_regs_for_features(ARMCPU *cpu); void init_cpreg_list(ARMCPU *cpu); +void destroy_cpreg_list(ARMCPU *cpu); =20 void arm_cpu_register_gdb_regs_for_features(ARMCPU *cpu); +void arm_cpu_unregister_gdb_regs(ARMCPU *cpu); void arm_translate_init(void); =20 void arm_restore_state_to_opc(CPUState *cs, diff --git a/target/arm/kvm.c b/target/arm/kvm.c index 01c83c1994..1121771c4a 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -1988,6 +1988,11 @@ int kvm_arch_init_vcpu(CPUState *cs) =20 int kvm_arch_destroy_vcpu(CPUState *cs) { + /* vCPUs which are yet to be realized will not have handler */ + if (cs->thread_id) { + qemu_del_vm_change_state_handler(cs->vmcse); + } + return 0; } =20 --=20 2.34.1