From nobody Sun Sep 28 17:13:24 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1757667474858697.0544592600614; Fri, 12 Sep 2025 01:57:54 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uwzZq-0001u3-9F; Fri, 12 Sep 2025 04:56:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwzZl-0001g6-Ag; Fri, 12 Sep 2025 04:56:13 -0400 Received: from [218.76.62.146] (helo=ultrarisc.com) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uwzZh-0007fr-AW; Fri, 12 Sep 2025 04:56:12 -0400 Received: from ur-dp1000.. (unknown [192.168.100.1]) by localhost.localdomain (Coremail) with SMTP id AQAAfwCXENBN4MNofEEtAA--.43821S3; Fri, 12 Sep 2025 16:57:01 +0800 (CST) From: Xie Bo To: qemu-devel@nongnu.org Cc: ajones@ventanamicro.com, qemu-riscv@nongnu.org, mjt@tls.msk.ru, pbonzini@redhat.com, anup@brainfault.org, alistair.francis@wdc.com, rkrcmar@ventanamicro.com, palmer@dabbelt.com, xiamy@ultrarisc.com, Xie Bo Subject: [PATCH v8 for v10.0.0 1/2] Set KVM initial privilege mode and mp_state Date: Fri, 12 Sep 2025 16:55:32 +0800 Message-ID: <20250912085535.1649347-2-xb@ultrarisc.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250912085535.1649347-1-xb@ultrarisc.com> References: <20250912085535.1649347-1-xb@ultrarisc.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: AQAAfwCXENBN4MNofEEtAA--.43821S3 X-Coremail-Antispam: 1UD129KBjvJXoWxAF4UJr15AFy3ZF45Xw4rGrg_yoW5XF4xpF 4kCw4akrWkAa97Jw4fJryvgr1ruw4kGrWUC397ZrWxZFsxArWYgF1ktFyUCFyDWFW5Arya 9a90vr1rAa1UZ3JanT9S1TB71UUUUUDqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUml14x267AKxVW5JVWrJwAFc2x0x2IEx4CE42xK8VAvwI8IcIk0 rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2048vs2IY020E87I2jVAFwI0_Jr4l82xGYIkIc2 x26xkF7I0E14v26r4j6ryUM28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48ve4kI8wA2z4x0 Y4vE2Ix0cI8IcVAFwI0_Jr0_JF4l84ACjcxK6xIIjxv20xvEc7CjxVAFwI0_Gr0_Cr1l84 ACjcxK6I8E87Iv67AKxVW8JVWxJwA2z4x0Y4vEx4A2jsIEc7CjxVAFwI0_Gr1j6F4UJwAS 0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I80ewAv7VC0I7IYx2 IY67AKxVWUGVWUXwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFVCjc4AY6r1j6r4UM4x0 Y48IcxkI7VAKI48JM4x0x7Aq67IIx4CEVc8vx2IErcIFxwACI402YVCY1x02628vn2kIc2 xKxwCY1x0262kKe7AKxVWUtVW8ZwCY02Avz4vE-syl42xK82IYc2Ij64vIr41l4I8I3I0E 4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGV WUWwC2zVAF1VAY17CE14v26r1q6r43MIIYrxkI7VAKI48JMIIF0xvE2Ix0cI8IcVAFwI0_ Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I0E14v26r4j6F4UMIIF0xvE42xK8VAvwI8IcIk0rV WUJVWUCwCI42IY6I8E87Iv67AKxVWUJVW8JwCI42IY6I8E87Iv6xkF7I0E14v26r4j6r4U JbIYCTnIWIevJa73UjIFyTuYvjfUnZXrUUUUU X-CM-SenderInfo: l0e63zxwud2x1vfou0bp/1tbiAQADB2jDefAANwAAsQ X-Host-Lookup-Failed: Reverse DNS lookup failed for 218.76.62.146 (failed) Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=218.76.62.146; envelope-from=xb@ultrarisc.com; helo=ultrarisc.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, RDNS_NONE=0.793, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1757667477629116600 Content-Type: text/plain; charset="utf-8" For KVM mode, the privilege mode should not include M-mode, and the=20 initial value should be set to S-mode. Additionally, a following patch=20 adds the implementation of putting the vCPU privilege mode to KVM.=20 When the vCPU runs for the first time, QEMU will first put the privilege=20 state to KVM. If the initial value is set to M-mode, KVM will encounter=20 an error. In addition, this patch introduces the 'mp_state' field to RISC-V=20 vCPUs, following the convention used by KVM on x86. The 'mp_state'=20 reflects the multiprocessor state of a vCPU, and is used to control=20 whether the vCPU is runnable by KVM. Randomly select one CPU as the=20 boot CPU. Since each CPU executes the riscv_cpu_reset_hold() function=20 and CPU0 executes first, only CPU0 randomly selects the boot CPU. Reviewed-by: Andrew Jones Signed-off-by: Xie Bo --- target/riscv/cpu.c | 17 ++++++++++++++++- target/riscv/cpu.h | 2 ++ 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c index 09ded6829a..f6c787ebdc 100644 --- a/target/riscv/cpu.c +++ b/target/riscv/cpu.c @@ -38,6 +38,7 @@ #include "kvm/kvm_riscv.h" #include "tcg/tcg-cpu.h" #include "tcg/tcg.h" +#include "hw/boards.h" =20 /* RISC-V CPU definitions */ static const char riscv_single_letter_exts[] =3D "IEMAFDQCBPVH"; @@ -1031,18 +1032,32 @@ static void riscv_cpu_reset_hold(Object *obj, Reset= Type type) #ifndef CONFIG_USER_ONLY uint8_t iprio; int i, irq, rdzero; + static int boot_cpu_index; #endif CPUState *cs =3D CPU(obj); RISCVCPU *cpu =3D RISCV_CPU(cs); RISCVCPUClass *mcc =3D RISCV_CPU_GET_CLASS(obj); CPURISCVState *env =3D &cpu->env; + MachineState *ms =3D MACHINE(qdev_get_machine()); =20 if (mcc->parent_phases.hold) { mcc->parent_phases.hold(obj, type); } #ifndef CONFIG_USER_ONLY env->misa_mxl =3D mcc->misa_mxl_max; - env->priv =3D PRV_M; + if (kvm_enabled()) { + env->priv =3D PRV_S; + } else { + env->priv =3D PRV_M; + } + if (cs->cpu_index =3D=3D 0) { + boot_cpu_index =3D g_random_int_range(0, ms->smp.cpus); + } + if (cs->cpu_index =3D=3D boot_cpu_index) { + env->mp_state =3D KVM_MP_STATE_RUNNABLE; + } else { + env->mp_state =3D KVM_MP_STATE_STOPPED; + } env->mstatus &=3D ~(MSTATUS_MIE | MSTATUS_MPRV); if (env->misa_mxl > MXL_RV32) { /* diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h index 51e49e03de..4b1c5bf0e4 100644 --- a/target/riscv/cpu.h +++ b/target/riscv/cpu.h @@ -256,6 +256,8 @@ struct CPUArchState { #endif =20 target_ulong priv; + /* Current multiprocessor state of this vCPU. */ + uint32_t mp_state; /* CSRs for execution environment configuration */ uint64_t menvcfg; target_ulong senvcfg; --=20 2.43.0 From nobody Sun Sep 28 17:13:24 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1757667477054415.993537745456; Fri, 12 Sep 2025 01:57:57 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uwzZt-00023x-DE; Fri, 12 Sep 2025 04:56:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwzZm-0001mp-J8; Fri, 12 Sep 2025 04:56:14 -0400 Received: from [218.76.62.146] (helo=ultrarisc.com) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uwzZh-0007hs-Pr; Fri, 12 Sep 2025 04:56:13 -0400 Received: from ur-dp1000.. (unknown [192.168.100.1]) by localhost.localdomain (Coremail) with SMTP id AQAAfwCXENBN4MNofEEtAA--.43821S4; Fri, 12 Sep 2025 16:57:02 +0800 (CST) From: Xie Bo To: qemu-devel@nongnu.org Cc: ajones@ventanamicro.com, qemu-riscv@nongnu.org, mjt@tls.msk.ru, pbonzini@redhat.com, anup@brainfault.org, alistair.francis@wdc.com, rkrcmar@ventanamicro.com, palmer@dabbelt.com, xiamy@ultrarisc.com, Xie Bo Subject: [PATCH v8 for v10.0.0 2/2] Fix VM resume after QEMU+KVM migration Date: Fri, 12 Sep 2025 16:55:33 +0800 Message-ID: <20250912085535.1649347-3-xb@ultrarisc.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250912085535.1649347-1-xb@ultrarisc.com> References: <20250912085535.1649347-1-xb@ultrarisc.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: AQAAfwCXENBN4MNofEEtAA--.43821S4 X-Coremail-Antispam: 1UD129KBjvJXoWxtF4kZw1rKF4rWw1kJr4xJFb_yoW7ArW7pr s8GFZ8CryxGrWxX34fJ34DXFn5Gw47GanxC3y09r4akF45GrZ09r1kKay2yrs5G348Ar12 9F45AFy3ua15tFJanT9S1TB71UUUUUDqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUml14x267AKxVWrJVCq3wAFc2x0x2IEx4CE42xK8VAvwI8IcIk0 rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2048vs2IY020E87I2jVAFwI0_Jryl82xGYIkIc2 x26xkF7I0E14v26ryj6s0DM28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48ve4kI8wA2z4x0 Y4vE2Ix0cI8IcVAFwI0_JFI_Gr1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI0_Gr0_Cr1l84 ACjcxK6I8E87Iv67AKxVW8JVWxJwA2z4x0Y4vEx4A2jsIEc7CjxVAFwI0_Gr1j6F4UJwAS 0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I80ewAv7VC0I7IYx2 IY67AKxVWUGVWUXwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFVCjc4AY6r1j6r4UM4x0 Y48IcxkI7VAKI48JM4x0x7Aq67IIx4CEVc8vx2IErcIFxwACI402YVCY1x02628vn2kIc2 xKxwCY1x0262kKe7AKxVWUtVW8ZwCY02Avz4vE-syl42xK82IYc2Ij64vIr41l4I8I3I0E 4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGV WUWwC2zVAF1VAY17CE14v26r1q6r43MIIYrxkI7VAKI48JMIIF0xvE2Ix0cI8IcVAFwI0_ JFI_Gr1lIxAIcVC0I7IYx2IY6xkF7I0E14v26r4j6F4UMIIF0xvE42xK8VAvwI8IcIk0rV WUJVWUCwCI42IY6I8E87Iv67AKxVWUJVW8JwCI42IY6I8E87Iv6xkF7I0E14v26r4j6r4U JbIYCTnIWIevJa73UjIFyTuYvjfUF5rcDUUUU X-CM-SenderInfo: l0e63zxwud2x1vfou0bp/1tbiAQADB2jDefAAOgAAsd X-Host-Lookup-Failed: Reverse DNS lookup failed for 218.76.62.146 (failed) Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=218.76.62.146; envelope-from=xb@ultrarisc.com; helo=ultrarisc.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, RDNS_NONE=0.793, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1757667479515116600 Fix two migration issues for virtual machines in KVM mode: 1.It saves and restores the vCPU's privilege mode to ensure that the=20 vCPU's privilege mode is correct after migration. 2.It saves and restores the vCPU's mp_state (runnable or stopped) and=20 includes this state in the migration sequence, upgrading the vmstate=20 version to ensure that the vCPU's mp_state is correct after migration. KVM_PUT_RUNTIME_STATE only synchronizes the vCPU=E2=80=99s runtime-modified=20 state (such as registers), whereas mp_state is related to system boot,=20 multi-core initialization, and is not modified during normal operation.=20 Therefore, mp_state is only synchronized to KVM during KVM_PUT_RESET_STATE=20 and KVM_PUT_FULL_STATE. Reviewed-by: Andrew Jones Signed-off-by: Xie Bo --- target/riscv/kvm/kvm-cpu.c | 60 ++++++++++++++++++++++++++++-------- target/riscv/kvm/kvm_riscv.h | 3 +- target/riscv/machine.c | 5 +-- 3 files changed, 52 insertions(+), 16 deletions(-) diff --git a/target/riscv/kvm/kvm-cpu.c b/target/riscv/kvm/kvm-cpu.c index 0f4997a918..1434dd1c51 100644 --- a/target/riscv/kvm/kvm-cpu.c +++ b/target/riscv/kvm/kvm-cpu.c @@ -576,6 +576,12 @@ static int kvm_riscv_get_regs_core(CPUState *cs) } env->pc =3D reg; =20 + ret =3D kvm_get_one_reg(cs, RISCV_CORE_REG(env, mode), ®); + if (ret) { + return ret; + } + env->priv =3D reg; + for (i =3D 1; i < 32; i++) { uint64_t id =3D kvm_riscv_reg_id_ulong(env, KVM_REG_RISCV_CORE, i); ret =3D kvm_get_one_reg(cs, id, ®); @@ -601,6 +607,12 @@ static int kvm_riscv_put_regs_core(CPUState *cs) return ret; } =20 + reg =3D env->priv; + ret =3D kvm_set_one_reg(cs, RISCV_CORE_REG(env, mode), ®); + if (ret) { + return ret; + } + for (i =3D 1; i < 32; i++) { uint64_t id =3D kvm_riscv_reg_id_ulong(env, KVM_REG_RISCV_CORE, i); reg =3D env->gpr[i]; @@ -1244,25 +1256,52 @@ int kvm_arch_get_registers(CPUState *cs, Error **er= rp) return ret; } =20 + ret =3D kvm_riscv_sync_mpstate_to_qemu(cs); + if (ret) { + return ret; + } + return ret; } =20 -int kvm_riscv_sync_mpstate_to_kvm(RISCVCPU *cpu, int state) +int kvm_riscv_sync_mpstate_to_kvm(CPUState *cs) { + int ret =3D 0; + CPURISCVState *env =3D &RISCV_CPU(cs)->env; + if (cap_has_mp_state) { struct kvm_mp_state mp_state =3D { - .mp_state =3D state + .mp_state =3D env->mp_state }; =20 - int ret =3D kvm_vcpu_ioctl(CPU(cpu), KVM_SET_MP_STATE, &mp_state); + ret =3D kvm_vcpu_ioctl(cs, KVM_SET_MP_STATE, &mp_state); if (ret) { - fprintf(stderr, "%s: failed to sync MP_STATE %d/%s\n", + fprintf(stderr, "%s: failed to sync MP_STATE to KVM %d/%s\n", __func__, ret, strerror(-ret)); - return -1; } } =20 - return 0; + return ret; +} + +int kvm_riscv_sync_mpstate_to_qemu(CPUState *cs) +{ + int ret =3D 0; + CPURISCVState *env =3D &RISCV_CPU(cs)->env; + + if (cap_has_mp_state) { + struct kvm_mp_state mp_state; + + ret =3D kvm_vcpu_ioctl(cs, KVM_GET_MP_STATE, &mp_state); + if (ret) { + fprintf(stderr, "%s: failed to sync MP_STATE to QEMU %d/%s\n", + __func__, ret, strerror(-ret)); + return ret; + } + env->mp_state =3D mp_state.mp_state; + } + + return ret; } =20 int kvm_arch_put_registers(CPUState *cs, int level, Error **errp) @@ -1289,13 +1328,8 @@ int kvm_arch_put_registers(CPUState *cs, int level, = Error **errp) return ret; } =20 - if (KVM_PUT_RESET_STATE =3D=3D level) { - RISCVCPU *cpu =3D RISCV_CPU(cs); - if (cs->cpu_index =3D=3D 0) { - ret =3D kvm_riscv_sync_mpstate_to_kvm(cpu, KVM_MP_STATE_RUNNAB= LE); - } else { - ret =3D kvm_riscv_sync_mpstate_to_kvm(cpu, KVM_MP_STATE_STOPPE= D); - } + if (level >=3D KVM_PUT_RESET_STATE) { + ret =3D kvm_riscv_sync_mpstate_to_kvm(cs); if (ret) { return ret; } diff --git a/target/riscv/kvm/kvm_riscv.h b/target/riscv/kvm/kvm_riscv.h index b2bcd1041f..953db94160 100644 --- a/target/riscv/kvm/kvm_riscv.h +++ b/target/riscv/kvm/kvm_riscv.h @@ -28,7 +28,8 @@ void kvm_riscv_aia_create(MachineState *machine, uint64_t= group_shift, uint64_t aplic_base, uint64_t imsic_base, uint64_t guest_num); void riscv_kvm_aplic_request(void *opaque, int irq, int level); -int kvm_riscv_sync_mpstate_to_kvm(RISCVCPU *cpu, int state); +int kvm_riscv_sync_mpstate_to_kvm(CPUState *cs); +int kvm_riscv_sync_mpstate_to_qemu(CPUState *cs); void riscv_kvm_cpu_finalize_features(RISCVCPU *cpu, Error **errp); uint64_t kvm_riscv_get_timebase_frequency(RISCVCPU *cpu); =20 diff --git a/target/riscv/machine.c b/target/riscv/machine.c index 889e2b6570..8562a0a1d6 100644 --- a/target/riscv/machine.c +++ b/target/riscv/machine.c @@ -401,8 +401,8 @@ static const VMStateDescription vmstate_ssp =3D { =20 const VMStateDescription vmstate_riscv_cpu =3D { .name =3D "cpu", - .version_id =3D 10, - .minimum_version_id =3D 10, + .version_id =3D 11, + .minimum_version_id =3D 11, .post_load =3D riscv_cpu_post_load, .fields =3D (const VMStateField[]) { VMSTATE_UINTTL_ARRAY(env.gpr, RISCVCPU, 32), @@ -422,6 +422,7 @@ const VMStateDescription vmstate_riscv_cpu =3D { VMSTATE_UNUSED(4), VMSTATE_UINT32(env.misa_ext_mask, RISCVCPU), VMSTATE_UINTTL(env.priv, RISCVCPU), + VMSTATE_UINT32(env.mp_state, RISCVCPU), VMSTATE_BOOL(env.virt_enabled, RISCVCPU), VMSTATE_UINT64(env.resetvec, RISCVCPU), VMSTATE_UINTTL(env.mhartid, RISCVCPU), --=20 2.43.0