target/riscv/cpu.h | 4 +++ target/riscv/kvm/kvm-cpu.c | 62 ++++++++++++++++++++++++++---------- target/riscv/kvm/kvm_riscv.h | 2 +- target/riscv/machine.c | 45 ++++++++++++++++++++++++++ 4 files changed, 96 insertions(+), 17 deletions(-)
This series revives v9, which was applied to riscv-to-apply.next but did not reach QEMU master. RISC-V KVM migration currently loses two pieces of vCPU state: the privilege mode core register and KVM MP state. The latter leaves secondary vCPUs in their reset STOPPED state on the destination. Synchronize the privilege mode through the existing generic CPU VMState, and carry MP state in a capability-gated KVM-only subsection. Keeping MP state in a subsection avoids changing the generic RISC-V CPU VMState version or adding KVM state to TCG migration streams. Migration streams from older QEMU versions that lack the subsection can still be loaded, retaining their previous destination reset behavior. When KVM_CAP_MP_STATE is available, streams produced by this series require a destination that recognizes the new subsection; an older destination rejects the stream instead of silently losing MP state. Changes from v9: - Rebase on QEMU master at e1705a25aff3. - Drop the reset-to-S-mode change that is already present upstream. - Keep the generic RISC-V CPU VMState version unchanged. - Save MP state in a KVM-only subsection and restore it only when the subsection was loaded. - Reset the subsection-loaded flag in the parent VMState pre-load callback. Testing on a native riscv64 host with Linux 6.6.20 and GCC 13.3.0: - KVM build: riscv64-softmmu, --enable-kvm. - Non-KVM build: riscv64-softmmu, --disable-kvm. - Four-vCPU KVM migration on unmodified master: all vCPUs failed to run after migration and the guest reported RCU stalls. - The same migration with this series: CPU0 through CPU3 all ran after migration. - Four user-mode busy workers, one pinned to each vCPU, all made forward progress across migration to exercise privilege-mode restoration. Xie Bo (2): target/riscv/kvm: Synchronize privilege mode target/riscv/kvm: Preserve MP state across migration target/riscv/cpu.h | 4 +++ target/riscv/kvm/kvm-cpu.c | 62 ++++++++++++++++++++++++++---------- target/riscv/kvm/kvm_riscv.h | 2 +- target/riscv/machine.c | 45 ++++++++++++++++++++++++++ 4 files changed, 96 insertions(+), 17 deletions(-) -- 2.17.1
This series revives v9, which was applied to riscv-to-apply.next but did not reach QEMU master. RISC-V KVM migration currently loses two pieces of vCPU state: the privilege mode core register and KVM MP state. The latter leaves secondary vCPUs in their reset STOPPED state on the destination. Synchronize the privilege mode through the existing generic CPU VMState, and carry MP state in a capability-gated KVM-only subsection. The subsection keeps KVM state out of TCG migration streams. Bump the generic RISC-V CPU VMState version and minimum version to 12 as required by the new pre_load hook and subsection. Changes from v10: - Bump the RISC-V CPU VMState version_id and minimum_version_id to 12. - Add Daniel's Reviewed-by tags. Testing on a native riscv64 host with Linux 6.6.20 and GCC 13.3.0: - KVM build: riscv64-softmmu, --enable-kvm. - Non-KVM build: riscv64-softmmu, --disable-kvm. - Four-vCPU KVM migration on unmodified master: all vCPUs failed to run after migration and the guest reported RCU stalls. - The same migration with this series: CPU0 through CPU3 all ran after migration. - Four user-mode busy workers, one pinned to each vCPU, all made forward progress across migration to exercise privilege-mode restoration. Xie Bo (2): target/riscv/kvm: Synchronize privilege mode target/riscv/kvm: Preserve MP state across migration target/riscv/cpu.h | 4 +++ target/riscv/kvm/kvm-cpu.c | 62 ++++++++++++++++++++++++++---------- target/riscv/kvm/kvm_riscv.h | 2 +- target/riscv/machine.c | 49 ++++++++++++++++++++++++++-- 4 files changed, 98 insertions(+), 19 deletions(-) -- 2.17.1
Hi, Now that this series is in master, could it also be considered for stable-11.1? Both commits are needed together: 7df9aa5 target/riscv/kvm: Synchronize privilege mode ddac7cf target/riscv/kvm: Preserve MP state across migration They fix RISC-V KVM live migration with multiple vCPUs. Without the second fix, secondary vCPUs can remain stopped after migration; without the first, the vCPU privilege mode may be restored incorrectly. I tested the series on native riscv64 hardware with KVM enabled, including multi-vCPU live migration, a 24-hour post-migration run, and multiple repeated migrations across two guests. I did not observe guest hangs or soft lockups. Thanks, Xie Bo
On 8/25/26 10:03, Xie Bo wrote: > Hi, > > Now that this series is in master, could it also be considered for > stable-11.1? > > Both commits are needed together: > > 7df9aa5 target/riscv/kvm: Synchronize privilege mode > ddac7cf target/riscv/kvm: Preserve MP state across migration Heh. I remember that big riscv pull request. I processed it before 11.1.1, so changes from it are included in 11.1.1. However, these two changes didn't have neither Fixes: nor Resolves: tags, -- this is the criteria for riscv fixes for stable - so I skipped them :) Picked these two up now, but it will be about a month before the next stable release. Yes, as Alistar noted, it is best to include Cc: qemu-stable@ when submitting a patch which should be picked up for the stable series. I understand sometimes it is not obvious, so yes, Cc'ing qemu-stable@ after the fact works too. Thank you! /mjt
On Fri, 2026-08-28 at 10:04 +0300, Michael Tokarev wrote: > On 8/25/26 10:03, Xie Bo wrote: > > Hi, > > > > Now that this series is in master, could it also be considered for > > stable-11.1? > > > > Both commits are needed together: > > > > 7df9aa5 target/riscv/kvm: Synchronize privilege mode > > ddac7cf target/riscv/kvm: Preserve MP state across migration > > Heh. I remember that big riscv pull request. I processed it before > 11.1.1, so changes from it are included in 11.1.1. > > However, these two changes didn't have neither Fixes: nor Resolves: > tags, -- this is the criteria for riscv fixes for stable - so I > skipped them :) > > Picked these two up now, but it will be about a month before the > next stable release. Thanks! > > Yes, as Alistar noted, it is best to include Cc: qemu-stable@ when > submitting a patch which should be picked up for the stable series. > I understand sometimes it is not obvious, so yes, Cc'ing qemu-stable@ > after the fact works too. Yes please. I do skim the PRs to check for patches to include, but these ones didn't originally jump out Alistair > > Thank you! > > /mjt
On Tue, 2026-08-25 at 15:03 +0800, Xie Bo wrote: > Hi, > > Now that this series is in master, could it also be considered for > stable-11.1? > When sending patches that should be backported please include """ Cc: qemu-stable@nongnu.org """ in the commit message. Then it would have been picked up Alistair > Both commits are needed together: > > 7df9aa5 target/riscv/kvm: Synchronize privilege mode > ddac7cf target/riscv/kvm: Preserve MP state across migration > > They fix RISC-V KVM live migration with multiple vCPUs. Without the > second fix, secondary vCPUs can remain stopped after migration; > without > the first, the vCPU privilege mode may be restored incorrectly. > > I tested the series on native riscv64 hardware with KVM enabled, > including multi-vCPU live migration, a 24-hour post-migration run, > and > multiple repeated migrations across two guests. I did not observe > guest > hangs or soft lockups. > > Thanks, > Xie Bo >
On Sat, 2026-08-08 at 20:51 +0800, Xie Bo wrote: > This series revives v9, which was applied to riscv-to-apply.next but > did > not reach QEMU master. > > RISC-V KVM migration currently loses two pieces of vCPU state: the > privilege mode core register and KVM MP state. The latter leaves > secondary > vCPUs in their reset STOPPED state on the destination. > > Synchronize the privilege mode through the existing generic CPU > VMState, > and carry MP state in a capability-gated KVM-only subsection. The > subsection > keeps KVM state out of TCG migration streams. Bump the generic RISC-V > CPU > VMState version and minimum version to 12 as required by the new > pre_load > hook and subsection. > > Changes from v10: > - Bump the RISC-V CPU VMState version_id and minimum_version_id to > 12. > - Add Daniel's Reviewed-by tags. > > Testing on a native riscv64 host with Linux 6.6.20 and GCC 13.3.0: > - KVM build: riscv64-softmmu, --enable-kvm. > - Non-KVM build: riscv64-softmmu, --disable-kvm. > - Four-vCPU KVM migration on unmodified master: all vCPUs failed to > run > after migration and the guest reported RCU stalls. > - The same migration with this series: CPU0 through CPU3 all ran > after > migration. > - Four user-mode busy workers, one pinned to each vCPU, all made > forward > progress across migration to exercise privilege-mode restoration. Thanks! Applied to riscv-to-apply.next Alistair > > Xie Bo (2): > target/riscv/kvm: Synchronize privilege mode > target/riscv/kvm: Preserve MP state across migration > > target/riscv/cpu.h | 4 +++ > target/riscv/kvm/kvm-cpu.c | 62 ++++++++++++++++++++++++++-------- > -- > target/riscv/kvm/kvm_riscv.h | 2 +- > target/riscv/machine.c | 49 ++++++++++++++++++++++++++-- > 4 files changed, 98 insertions(+), 19 deletions(-)
The KVM core register synchronization currently omits the vCPU privilege
mode. As a result, env.priv can be stale when the migration stream is saved
and the destination can restore the vCPU in the wrong mode.
Read and write the KVM core mode register together with the other core
registers. The generic RISC-V CPU VMState already carries env.priv, so no
migration format change is required.
Signed-off-by: Xie Bo <xb@ultrarisc.com>
Reviewed-by: Daniel Henrique Barboza <daniel.barboza@oss.qualcomm.com>
---
target/riscv/kvm/kvm-cpu.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/target/riscv/kvm/kvm-cpu.c b/target/riscv/kvm/kvm-cpu.c
index 495cb42dc8..8218832fbe 100644
--- a/target/riscv/kvm/kvm-cpu.c
+++ b/target/riscv/kvm/kvm-cpu.c
@@ -603,6 +603,12 @@ static int kvm_riscv_get_regs_core(CPUState *cs)
}
env->pc = reg;
+ ret = kvm_get_one_reg(cs, RISCV_CORE_REG(mode), ®);
+ if (ret) {
+ return ret;
+ }
+ env->priv = reg;
+
for (i = 1; i < 32; i++) {
uint64_t id = KVM_RISCV_REG_ID_ULONG(KVM_REG_RISCV_CORE, i);
ret = kvm_get_one_reg(cs, id, ®);
@@ -628,6 +634,12 @@ static int kvm_riscv_put_regs_core(CPUState *cs)
return ret;
}
+ reg = env->priv;
+ ret = kvm_set_one_reg(cs, RISCV_CORE_REG(mode), ®);
+ if (ret) {
+ return ret;
+ }
+
for (i = 1; i < 32; i++) {
uint64_t id = KVM_RISCV_REG_ID_ULONG(KVM_REG_RISCV_CORE, i);
reg = env->gpr[i];
--
2.17.1RISC-V KVM initializes secondary vCPUs in KVM_MP_STATE_STOPPED, but QEMU
does not save their runtime MP state. A destination therefore retains reset
MP state after migration and cannot reliably resume all vCPUs.
Save KVM_GET_MP_STATE in a capability-gated KVM VMState subsection and
restore it on KVM_PUT_FULL_STATE. Keep the existing reset initialization
path unchanged. Track whether the subsection was loaded so streams where
the subsection is absent retain the destination reset behavior.
Bump the RISC-V CPU VMState version and minimum version to 12 for the new
pre_load hook and KVM MP-state subsection. Keep the subsection out of KVM
migration streams when the host does not support the MP-state capability.
Signed-off-by: Xie Bo <xb@ultrarisc.com>
Reviewed-by: Daniel Henrique Barboza <daniel.barboza@oss.qualcomm.com>
---
target/riscv/cpu.h | 4 +++
target/riscv/kvm/kvm-cpu.c | 50 ++++++++++++++++++++++++------------
target/riscv/kvm/kvm_riscv.h | 2 +-
target/riscv/machine.c | 49 +++++++++++++++++++++++++++++++++--
4 files changed, 86 insertions(+), 19 deletions(-)
diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h
index c9dfa7daff..e980b5964d 100644
--- a/target/riscv/cpu.h
+++ b/target/riscv/cpu.h
@@ -538,6 +538,10 @@ struct CPUArchState {
uint64_t kvm_timer_compare;
uint64_t kvm_timer_state;
uint64_t kvm_timer_frequency;
+
+ /* KVM multiprocessor state */
+ uint32_t kvm_mp_state;
+ bool kvm_mp_state_loaded;
#endif /* CONFIG_KVM */
};
diff --git a/target/riscv/kvm/kvm-cpu.c b/target/riscv/kvm/kvm-cpu.c
index 8218832fbe..cc5655d46d 100644
--- a/target/riscv/kvm/kvm-cpu.c
+++ b/target/riscv/kvm/kvm-cpu.c
@@ -1374,25 +1374,35 @@ int kvm_arch_get_registers(CPUState *cs, Error **errp)
return ret;
}
+ if (cap_has_mp_state) {
+ struct kvm_mp_state mp_state;
+
+ ret = kvm_vcpu_ioctl(cs, KVM_GET_MP_STATE, &mp_state);
+ if (ret) {
+ return ret;
+ }
+ RISCV_CPU(cs)->env.kvm_mp_state = mp_state.mp_state;
+ }
+
return ret;
}
-int kvm_riscv_sync_mpstate_to_kvm(RISCVCPU *cpu, int state)
+bool kvm_riscv_has_mp_state(void)
{
- if (cap_has_mp_state) {
- struct kvm_mp_state mp_state = {
- .mp_state = state
- };
+ return cap_has_mp_state;
+}
- int ret = kvm_vcpu_ioctl(CPU(cpu), KVM_SET_MP_STATE, &mp_state);
- if (ret) {
- fprintf(stderr, "%s: failed to sync MP_STATE %d/%s\n",
- __func__, ret, strerror(-ret));
- return -1;
- }
+static int kvm_riscv_put_mp_state(CPUState *cs)
+{
+ struct kvm_mp_state mp_state = {
+ .mp_state = RISCV_CPU(cs)->env.kvm_mp_state,
+ };
+
+ if (!cap_has_mp_state) {
+ return 0;
}
- return 0;
+ return kvm_vcpu_ioctl(cs, KVM_SET_MP_STATE, &mp_state);
}
int kvm_arch_put_registers(CPUState *cs, KvmPutState level, Error **errp)
@@ -1431,10 +1441,18 @@ int kvm_arch_put_registers(CPUState *cs, KvmPutState level, Error **errp)
}
if (KVM_PUT_RESET_STATE == level) {
- RISCVCPU *cpu = RISCV_CPU(cs);
- int state = cs->cpu_index == 0 ? KVM_MP_STATE_RUNNABLE
- : KVM_MP_STATE_STOPPED;
- ret = kvm_riscv_sync_mpstate_to_kvm(cpu, state);
+ CPURISCVState *env = &RISCV_CPU(cs)->env;
+
+ env->kvm_mp_state = cs->cpu_index == 0 ? KVM_MP_STATE_RUNNABLE
+ : KVM_MP_STATE_STOPPED;
+ env->kvm_mp_state_loaded = false;
+ ret = kvm_riscv_put_mp_state(cs);
+ if (ret) {
+ return ret;
+ }
+ } else if (KVM_PUT_FULL_STATE == level &&
+ RISCV_CPU(cs)->env.kvm_mp_state_loaded) {
+ ret = kvm_riscv_put_mp_state(cs);
if (ret) {
return ret;
}
diff --git a/target/riscv/kvm/kvm_riscv.h b/target/riscv/kvm/kvm_riscv.h
index b2bcd1041f..61eaa12443 100644
--- a/target/riscv/kvm/kvm_riscv.h
+++ b/target/riscv/kvm/kvm_riscv.h
@@ -28,7 +28,7 @@ void kvm_riscv_aia_create(MachineState *machine, uint64_t group_shift,
uint64_t aplic_base, uint64_t imsic_base,
uint64_t guest_num);
void riscv_kvm_aplic_request(void *opaque, int irq, int level);
-int kvm_riscv_sync_mpstate_to_kvm(RISCVCPU *cpu, int state);
+bool kvm_riscv_has_mp_state(void);
void riscv_kvm_cpu_finalize_features(RISCVCPU *cpu, Error **errp);
uint64_t kvm_riscv_get_timebase_frequency(RISCVCPU *cpu);
diff --git a/target/riscv/machine.c b/target/riscv/machine.c
index 0ab613a298..31c49ca3e6 100644
--- a/target/riscv/machine.c
+++ b/target/riscv/machine.c
@@ -25,6 +25,9 @@
#include "exec/icount.h"
#include "target/riscv/tcg/debug.h"
#include "hw/riscv/machines-qom.h"
+#ifdef CONFIG_KVM
+#include "kvm/kvm_riscv.h"
+#endif
static bool pmp_needed(void *opaque)
{
@@ -222,6 +225,44 @@ static const VMStateDescription vmstate_kvmtimer = {
VMSTATE_END_OF_LIST()
}
};
+
+static int riscv_cpu_kvm_pre_load(void *opaque)
+{
+ RISCVCPU *cpu = opaque;
+
+ cpu->env.kvm_mp_state_loaded = false;
+ return 0;
+}
+
+static bool kvm_mp_state_needed(void *opaque)
+{
+ return kvm_enabled() && kvm_riscv_has_mp_state();
+}
+
+static int kvm_mp_state_post_load(void *opaque, int version_id)
+{
+ RISCVCPU *cpu = opaque;
+ CPURISCVState *env = &cpu->env;
+
+ if (!kvm_enabled() || !kvm_riscv_has_mp_state()) {
+ return -ENOTSUP;
+ }
+
+ env->kvm_mp_state_loaded = true;
+ return 0;
+}
+
+static const VMStateDescription vmstate_kvm_mp_state = {
+ .name = "cpu/kvm-mp-state",
+ .version_id = 1,
+ .minimum_version_id = 1,
+ .needed = kvm_mp_state_needed,
+ .post_load = kvm_mp_state_post_load,
+ .fields = (const VMStateField[]) {
+ VMSTATE_UINT32(env.kvm_mp_state, RISCVCPU),
+ VMSTATE_END_OF_LIST()
+ }
+};
#endif
static bool debug_needed(void *opaque)
@@ -457,8 +498,11 @@ static const VMStateDescription vmstate_mseccfg = {
const VMStateDescription vmstate_riscv_cpu = {
.name = "cpu",
- .version_id = 11,
- .minimum_version_id = 11,
+ .version_id = 12,
+ .minimum_version_id = 12,
+#ifdef CONFIG_KVM
+ .pre_load = riscv_cpu_kvm_pre_load,
+#endif
.post_load = riscv_cpu_post_load,
.fields = (const VMStateField[]) {
VMSTATE_UINT64_ARRAY(env.gpr, RISCVCPU, 32),
@@ -522,6 +566,7 @@ const VMStateDescription vmstate_riscv_cpu = {
&vmstate_rv128,
#ifdef CONFIG_KVM
&vmstate_kvmtimer,
+ &vmstate_kvm_mp_state,
#endif
&vmstate_envcfg,
&vmstate_debug,
--
2.17.1© 2016 - 2026 Red Hat, Inc.