From nobody Sun Sep 28 16:38:11 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1757920258643767.2471238743434; Mon, 15 Sep 2025 00:10:58 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uy3Kn-0005g3-Kb; Mon, 15 Sep 2025 03:09:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uy3Kf-0005eJ-Iw; Mon, 15 Sep 2025 03:09:03 -0400 Received: from [218.76.62.146] (helo=ultrarisc.com) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uy3KT-0007Lc-LG; Mon, 15 Sep 2025 03:09:01 -0400 Received: from ur-dp1000.. (unknown [192.168.100.1]) by localhost.localdomain (Coremail) with SMTP id AQAAfwCXwdChu8do+wkuAA--.45058S3; Mon, 15 Sep 2025 15:09:38 +0800 (CST) From: Xie Bo To: qemu-devel@nongnu.org Cc: ajones@ventanamicro.com, qemu-riscv@nongnu.org, alistair23@gmail.com, pbonzini@redhat.com, anup@brainfault.org, alistair.francis@wdc.com, rkrcmar@ventanamicro.com, palmer@dabbelt.com, xiamy@ultrarisc.com, Xie Bo Subject: [PATCH v9 1/2] Set KVM initial privilege mode and mp_state Date: Mon, 15 Sep 2025 15:08:07 +0800 Message-ID: <20250915070811.3422578-2-xb@ultrarisc.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250915070811.3422578-1-xb@ultrarisc.com> References: <20250915070811.3422578-1-xb@ultrarisc.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: AQAAfwCXwdChu8do+wkuAA--.45058S3 X-Coremail-Antispam: 1UD129KBjvJXoWxAF4UJr15AFy3ZF45Xw4rGrg_yoW5Xr4UpF WkCw43CrykXa97Jw4fJrWvgr109w4kGrWUCws7ZrWxZFsxArWYgF4ktFy7CFyDWFZ8ArWS 9a90yr1rAa1UZ3JanT9S1TB71UUUUUJqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUQS14x267AKxVW5JVWrJwAFc2x0x2IEx4CE42xK8VAvwI8IcIk0 rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2048vs2IY020E87I2jVAFwI0_Jr4l82xGYIkIc2 x26xkF7I0E14v26r4j6ryUM28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48ve4kI8wA2z4x0 Y4vE2Ix0cI8IcVAFwI0_JFI_Gr1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI0_Gr0_Cr1l84 ACjcxK6I8E87Iv67AKxVW8JVWxJwA2z4x0Y4vEx4A2jsIEc7CjxVAFwI0_Gr0_Gr1UM2kK e7AKxVWUXVWUAwAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I 80ewAv7VC0I7IYx2IY67AKxVWUGVWUXwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFVCj c4AY6r1j6r4UM4x0Y48IcxkI7VAKI48JM4x0x7Aq67IIx4CEVc8vx2IErcIFxwACI402YV CY1x02628vn2kIc2xKxwCY1x0262kKe7AKxVWUtVW8ZwCY02Avz4vE-syl42xK82IYc2Ij 64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x 8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r1q6r43MIIYrxkI7VAKI48JMIIF0xvE 2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I0E14v26r4j6F4UMIIF0xvE42 xK8VAvwI8IcIk0rVWUJVWUCwCI42IY6I8E87Iv67AKxVWUJVW8JwCI42IY6I8E87Iv6xkF 7I0E14v26r4j6r4UJbIYCTnIWIevJa73UjIFyTuYvjfU8tC7UUUUU X-CM-SenderInfo: l0e63zxwud2x1vfou0bp/1tbiAQAGB2jHbnEAJwAAsX X-Host-Lookup-Failed: Reverse DNS lookup failed for 218.76.62.146 (failed) Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=218.76.62.146; envelope-from=xb@ultrarisc.com; helo=ultrarisc.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, RDNS_NONE=0.793, SPF_PASS=-0.001, T_SPF_HELO_TEMPERROR=0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1757920260958116600 Content-Type: text/plain; charset="utf-8" For KVM mode, the privilege mode should not include M-mode, and the initial value should be set to S-mode. Additionally, a following patch adds the implementation of putting the vCPU privilege mode to KVM. When the vCPU runs for the first time, QEMU will first put the privilege state to KVM. If the initial value is set to M-mode, KVM will encounter an error. In addition, this patch introduces the 'mp_state' field to RISC-V vCPUs, following the convention used by KVM on x86. The 'mp_state' reflects the multiprocessor state of a vCPU, and is used to control whether the vCPU is runnable by KVM. Randomly select one CPU as the boot CPU. Since each CPU executes the riscv_cpu_reset_hold() function and CPU0 executes first, only CPU0 randomly selects the boot CPU. Signed-off-by: Xie Bo Reviewed-by: Andrew Jones --- target/riscv/cpu.c | 17 ++++++++++++++++- target/riscv/cpu.h | 2 ++ 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c index d055ddf462..55892a2fc7 100644 --- a/target/riscv/cpu.c +++ b/target/riscv/cpu.c @@ -37,6 +37,7 @@ #include "kvm/kvm_riscv.h" #include "tcg/tcg-cpu.h" #include "tcg/tcg.h" +#include "hw/boards.h" =20 /* RISC-V CPU definitions */ static const char riscv_single_letter_exts[] =3D "IEMAFDQCBPVH"; @@ -685,18 +686,32 @@ static void riscv_cpu_reset_hold(Object *obj, ResetTy= pe type) #ifndef CONFIG_USER_ONLY uint8_t iprio; int i, irq, rdzero; + static int boot_cpu_index; #endif CPUState *cs =3D CPU(obj); RISCVCPU *cpu =3D RISCV_CPU(cs); RISCVCPUClass *mcc =3D RISCV_CPU_GET_CLASS(obj); CPURISCVState *env =3D &cpu->env; + MachineState *ms =3D MACHINE(qdev_get_machine()); =20 if (mcc->parent_phases.hold) { mcc->parent_phases.hold(obj, type); } #ifndef CONFIG_USER_ONLY env->misa_mxl =3D mcc->def->misa_mxl_max; - env->priv =3D PRV_M; + if (kvm_enabled()) { + env->priv =3D PRV_S; + } else { + env->priv =3D PRV_M; + } + if (cs->cpu_index =3D=3D 0) { + boot_cpu_index =3D g_random_int_range(0, ms->smp.cpus); + } + if (cs->cpu_index =3D=3D boot_cpu_index) { + env->mp_state =3D KVM_MP_STATE_RUNNABLE; + } else { + env->mp_state =3D KVM_MP_STATE_STOPPED; + } env->mstatus &=3D ~(MSTATUS_MIE | MSTATUS_MPRV); if (env->misa_mxl > MXL_RV32) { /* diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h index 738e68fa6e..7ea4859de7 100644 --- a/target/riscv/cpu.h +++ b/target/riscv/cpu.h @@ -269,6 +269,8 @@ struct CPUArchState { #endif =20 target_ulong priv; + /* Current multiprocessor state of this vCPU. */ + uint32_t mp_state; /* CSRs for execution environment configuration */ uint64_t menvcfg; target_ulong senvcfg; --=20 2.43.0 From nobody Sun Sep 28 16:38:11 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1757920226423521.2212866642725; Mon, 15 Sep 2025 00:10:26 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uy3Kh-0005dk-7b; Mon, 15 Sep 2025 03:09:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uy3KX-0005dF-HP; Mon, 15 Sep 2025 03:08:53 -0400 Received: from [218.76.62.146] (helo=ultrarisc.com) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uy3KN-0007M2-G8; Mon, 15 Sep 2025 03:08:51 -0400 Received: from ur-dp1000.. (unknown [192.168.100.1]) by localhost.localdomain (Coremail) with SMTP id AQAAfwCXwdChu8do+wkuAA--.45058S4; Mon, 15 Sep 2025 15:09:39 +0800 (CST) From: Xie Bo To: qemu-devel@nongnu.org Cc: ajones@ventanamicro.com, qemu-riscv@nongnu.org, alistair23@gmail.com, pbonzini@redhat.com, anup@brainfault.org, alistair.francis@wdc.com, rkrcmar@ventanamicro.com, palmer@dabbelt.com, xiamy@ultrarisc.com, Xie Bo Subject: [PATCH v9 2/2] Fix VM resume after QEMU+KVM migration Date: Mon, 15 Sep 2025 15:08:08 +0800 Message-ID: <20250915070811.3422578-3-xb@ultrarisc.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250915070811.3422578-1-xb@ultrarisc.com> References: <20250915070811.3422578-1-xb@ultrarisc.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: AQAAfwCXwdChu8do+wkuAA--.45058S4 X-Coremail-Antispam: 1UD129KBjvJXoWxtF4fKr4kXr1xArWfZFy5urg_yoW7WFW8pr s8GFZxCr93GrWxXw1fJ34DXFn5Gw47GwsxC3y8ur4akF45GrZ09r1kKay2yrs5G348Ar12 9F45AFy3ua15tFJanT9S1TB71UUUUUJqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUQS14x267AKxVWrJVCq3wAFc2x0x2IEx4CE42xK8VAvwI8IcIk0 rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2048vs2IY020E87I2jVAFwI0_Jryl82xGYIkIc2 x26xkF7I0E14v26ryj6s0DM28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48ve4kI8wA2z4x0 Y4vE2Ix0cI8IcVAFwI0_JFI_Gr1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI0_Gr0_Cr1l84 ACjcxK6I8E87Iv67AKxVW8JVWxJwA2z4x0Y4vEx4A2jsIEc7CjxVAFwI0_Gr0_Gr1UM2kK e7AKxVWUXVWUAwAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I 80ewAv7VC0I7IYx2IY67AKxVWUGVWUXwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFVCj c4AY6r1j6r4UM4x0Y48IcxkI7VAKI48JM4x0x7Aq67IIx4CEVc8vx2IErcIFxwACI402YV CY1x02628vn2kIc2xKxwCY1x0262kKe7AKxVWUtVW8ZwCY02Avz4vE-syl42xK82IYc2Ij 64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x 8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r1q6r43MIIYrxkI7VAKI48JMIIF0xvE 2Ix0cI8IcVAFwI0_JFI_Gr1lIxAIcVC0I7IYx2IY6xkF7I0E14v26r4j6F4UMIIF0xvE42 xK8VAvwI8IcIk0rVWUJVWUCwCI42IY6I8E87Iv67AKxVWUJVW8JwCI42IY6I8E87Iv6xkF 7I0E14v26r4j6r4UJbIYCTnIWIevJa73UjIFyTuYvjfUoOzVDUUUU X-CM-SenderInfo: l0e63zxwud2x1vfou0bp/1tbiAQAGB2jHbnEAKQAAsZ X-Host-Lookup-Failed: Reverse DNS lookup failed for 218.76.62.146 (failed) Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=218.76.62.146; envelope-from=xb@ultrarisc.com; helo=ultrarisc.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, RDNS_NONE=0.793, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1757920229182116600 Fix two migration issues for virtual machines in KVM mode: 1.It saves and restores the vCPU's privilege mode to ensure that the vCPU's privilege mode is correct after migration. 2.It saves and restores the vCPU's mp_state (runnable or stopped) and includes this state in the migration sequence, upgrading the vmstate version to ensure that the vCPU's mp_state is correct after migration. KVM_PUT_RUNTIME_STATE only synchronizes the vCPU=E2=80=99s runtime-modified state (such as registers), whereas mp_state is related to system boot, multi-core initialization, and is not modified during normal operation. Therefore, mp_state is only synchronized to KVM during KVM_PUT_RESET_STATE and KVM_PUT_FULL_STATE. Signed-off-by: Xie Bo Reviewed-by: Andrew Jones Reviewed-by: Radim Kr=C4=8Dm=C3=A1=C5=99 --- target/riscv/kvm/kvm-cpu.c | 60 ++++++++++++++++++++++++++++-------- target/riscv/kvm/kvm_riscv.h | 3 +- target/riscv/machine.c | 5 +-- 3 files changed, 52 insertions(+), 16 deletions(-) diff --git a/target/riscv/kvm/kvm-cpu.c b/target/riscv/kvm/kvm-cpu.c index 5c19062c19..1fa755bd96 100644 --- a/target/riscv/kvm/kvm-cpu.c +++ b/target/riscv/kvm/kvm-cpu.c @@ -594,6 +594,12 @@ static int kvm_riscv_get_regs_core(CPUState *cs) } env->pc =3D reg; =20 + ret =3D kvm_get_one_reg(cs, RISCV_CORE_REG(mode), ®); + if (ret) { + return ret; + } + env->priv =3D reg; + for (i =3D 1; i < 32; i++) { uint64_t id =3D KVM_RISCV_REG_ID_ULONG(KVM_REG_RISCV_CORE, i); ret =3D kvm_get_one_reg(cs, id, ®); @@ -619,6 +625,12 @@ static int kvm_riscv_put_regs_core(CPUState *cs) return ret; } =20 + reg =3D env->priv; + ret =3D kvm_set_one_reg(cs, RISCV_CORE_REG(mode), ®); + if (ret) { + return ret; + } + for (i =3D 1; i < 32; i++) { uint64_t id =3D KVM_RISCV_REG_ID_ULONG(KVM_REG_RISCV_CORE, i); reg =3D env->gpr[i]; @@ -1348,25 +1360,52 @@ int kvm_arch_get_registers(CPUState *cs, Error **er= rp) return ret; } =20 + ret =3D kvm_riscv_sync_mpstate_to_qemu(cs); + if (ret) { + return ret; + } + return ret; } =20 -int kvm_riscv_sync_mpstate_to_kvm(RISCVCPU *cpu, int state) +int kvm_riscv_sync_mpstate_to_kvm(CPUState *cs) { + int ret =3D 0; + CPURISCVState *env =3D &RISCV_CPU(cs)->env; + if (cap_has_mp_state) { struct kvm_mp_state mp_state =3D { - .mp_state =3D state + .mp_state =3D env->mp_state }; =20 - int ret =3D kvm_vcpu_ioctl(CPU(cpu), KVM_SET_MP_STATE, &mp_state); + ret =3D kvm_vcpu_ioctl(cs, KVM_SET_MP_STATE, &mp_state); if (ret) { - fprintf(stderr, "%s: failed to sync MP_STATE %d/%s\n", + fprintf(stderr, "%s: failed to sync MP_STATE to KVM %d/%s\n", __func__, ret, strerror(-ret)); - return -1; } } =20 - return 0; + return ret; +} + +int kvm_riscv_sync_mpstate_to_qemu(CPUState *cs) +{ + int ret =3D 0; + CPURISCVState *env =3D &RISCV_CPU(cs)->env; + + if (cap_has_mp_state) { + struct kvm_mp_state mp_state; + + ret =3D kvm_vcpu_ioctl(cs, KVM_GET_MP_STATE, &mp_state); + if (ret) { + fprintf(stderr, "%s: failed to sync MP_STATE to QEMU %d/%s\n", + __func__, ret, strerror(-ret)); + return ret; + } + env->mp_state =3D mp_state.mp_state; + } + + return ret; } =20 int kvm_arch_put_registers(CPUState *cs, int level, Error **errp) @@ -1393,13 +1432,8 @@ int kvm_arch_put_registers(CPUState *cs, int level, = Error **errp) return ret; } =20 - if (KVM_PUT_RESET_STATE =3D=3D level) { - RISCVCPU *cpu =3D RISCV_CPU(cs); - if (cs->cpu_index =3D=3D 0) { - ret =3D kvm_riscv_sync_mpstate_to_kvm(cpu, KVM_MP_STATE_RUNNAB= LE); - } else { - ret =3D kvm_riscv_sync_mpstate_to_kvm(cpu, KVM_MP_STATE_STOPPE= D); - } + if (level >=3D KVM_PUT_RESET_STATE) { + ret =3D kvm_riscv_sync_mpstate_to_kvm(cs); if (ret) { return ret; } diff --git a/target/riscv/kvm/kvm_riscv.h b/target/riscv/kvm/kvm_riscv.h index b2bcd1041f..953db94160 100644 --- a/target/riscv/kvm/kvm_riscv.h +++ b/target/riscv/kvm/kvm_riscv.h @@ -28,7 +28,8 @@ void kvm_riscv_aia_create(MachineState *machine, uint64_t= group_shift, uint64_t aplic_base, uint64_t imsic_base, uint64_t guest_num); void riscv_kvm_aplic_request(void *opaque, int irq, int level); -int kvm_riscv_sync_mpstate_to_kvm(RISCVCPU *cpu, int state); +int kvm_riscv_sync_mpstate_to_kvm(CPUState *cs); +int kvm_riscv_sync_mpstate_to_qemu(CPUState *cs); void riscv_kvm_cpu_finalize_features(RISCVCPU *cpu, Error **errp); uint64_t kvm_riscv_get_timebase_frequency(RISCVCPU *cpu); =20 diff --git a/target/riscv/machine.c b/target/riscv/machine.c index 51e0567ed3..6e5be17c27 100644 --- a/target/riscv/machine.c +++ b/target/riscv/machine.c @@ -426,8 +426,8 @@ static const VMStateDescription vmstate_sstc =3D { =20 const VMStateDescription vmstate_riscv_cpu =3D { .name =3D "cpu", - .version_id =3D 10, - .minimum_version_id =3D 10, + .version_id =3D 11, + .minimum_version_id =3D 11, .post_load =3D riscv_cpu_post_load, .fields =3D (const VMStateField[]) { VMSTATE_UINTTL_ARRAY(env.gpr, RISCVCPU, 32), @@ -447,6 +447,7 @@ const VMStateDescription vmstate_riscv_cpu =3D { VMSTATE_UNUSED(4), VMSTATE_UINT32(env.misa_ext_mask, RISCVCPU), VMSTATE_UINTTL(env.priv, RISCVCPU), + VMSTATE_UINT32(env.mp_state, RISCVCPU), VMSTATE_BOOL(env.virt_enabled, RISCVCPU), VMSTATE_UINT64(env.resetvec, RISCVCPU), VMSTATE_UINTTL(env.mhartid, RISCVCPU), --=20 2.43.0