[v9] target/riscv：Fix riscv64 kvm migration

[PATCH v9 1/2] Set KVM initial privilege mode and mp_state

Posted by Xie Bo 4 months, 4 weeks ago

For KVM mode, the privilege mode should not include M-mode, and the
initial value should be set to S-mode. Additionally, a following patch
adds the implementation of putting the vCPU privilege mode to KVM.
When the vCPU runs for the first time, QEMU will first put the privilege
state to KVM. If the initial value is set to M-mode, KVM will encounter
an error.

In addition, this patch introduces the 'mp_state' field to RISC-V
vCPUs, following the convention used by KVM on x86. The 'mp_state'
reflects the multiprocessor state of a vCPU, and is used to control
whether the vCPU is runnable by KVM. Randomly select one CPU as the
boot CPU. Since each CPU executes the riscv_cpu_reset_hold() function
and CPU0 executes first, only CPU0 randomly selects the boot CPU.

Signed-off-by: Xie Bo <xb@ultrarisc.com>
---
 target/riscv/cpu.c | 17 ++++++++++++++++-
 target/riscv/cpu.h |  2 ++
 2 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
index d055ddf462..55892a2fc7 100644
--- a/target/riscv/cpu.c
+++ b/target/riscv/cpu.c
@@ -37,6 +37,7 @@
 #include "kvm/kvm_riscv.h"
 #include "tcg/tcg-cpu.h"
 #include "tcg/tcg.h"
+#include "hw/boards.h"
 
 /* RISC-V CPU definitions */
 static const char riscv_single_letter_exts[] = "IEMAFDQCBPVH";
@@ -685,18 +686,32 @@ static void riscv_cpu_reset_hold(Object *obj, ResetType type)
 #ifndef CONFIG_USER_ONLY
     uint8_t iprio;
     int i, irq, rdzero;
+    static int boot_cpu_index;
 #endif
     CPUState *cs = CPU(obj);
     RISCVCPU *cpu = RISCV_CPU(cs);
     RISCVCPUClass *mcc = RISCV_CPU_GET_CLASS(obj);
     CPURISCVState *env = &cpu->env;
+    MachineState *ms = MACHINE(qdev_get_machine());
 
     if (mcc->parent_phases.hold) {
         mcc->parent_phases.hold(obj, type);
     }
 #ifndef CONFIG_USER_ONLY
     env->misa_mxl = mcc->def->misa_mxl_max;
-    env->priv = PRV_M;
+    if (kvm_enabled()) {
+        env->priv = PRV_S;
+    } else {
+        env->priv = PRV_M;
+    }
+    if (cs->cpu_index == 0) {
+        boot_cpu_index = g_random_int_range(0, ms->smp.cpus);
+    }
+    if (cs->cpu_index == boot_cpu_index) {
+        env->mp_state = KVM_MP_STATE_RUNNABLE;
+    } else {
+        env->mp_state = KVM_MP_STATE_STOPPED;
+    }
     env->mstatus &= ~(MSTATUS_MIE | MSTATUS_MPRV);
     if (env->misa_mxl > MXL_RV32) {
         /*
diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h
index 738e68fa6e..7ea4859de7 100644
--- a/target/riscv/cpu.h
+++ b/target/riscv/cpu.h
@@ -269,6 +269,8 @@ struct CPUArchState {
 #endif
 
     target_ulong priv;
+    /* Current multiprocessor state of this vCPU. */
+    uint32_t mp_state;
     /* CSRs for execution environment configuration */
     uint64_t menvcfg;
     target_ulong senvcfg;
-- 
2.43.0

Re: [PATCH v9 1/2] Set KVM initial privilege mode and mp_state

Posted by Daniel Henrique Barboza 3 months, 3 weeks ago

Hi,

On 9/15/25 4:08 AM, Xie Bo wrote:
> For KVM mode, the privilege mode should not include M-mode, and the
> initial value should be set to S-mode. Additionally, a following patch
> adds the implementation of putting the vCPU privilege mode to KVM.
> When the vCPU runs for the first time, QEMU will first put the privilege
> state to KVM. If the initial value is set to M-mode, KVM will encounter
> an error.
> 
> In addition, this patch introduces the 'mp_state' field to RISC-V
> vCPUs, following the convention used by KVM on x86. The 'mp_state'
> reflects the multiprocessor state of a vCPU, and is used to control
> whether the vCPU is runnable by KVM. Randomly select one CPU as the
> boot CPU. Since each CPU executes the riscv_cpu_reset_hold() function
> and CPU0 executes first, only CPU0 randomly selects the boot CPU.
> 
> Signed-off-by: Xie Bo <xb@ultrarisc.com>
> ---
>   target/riscv/cpu.c | 17 ++++++++++++++++-
>   target/riscv/cpu.h |  2 ++
>   2 files changed, 18 insertions(+), 1 deletion(-)
> 
> diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
> index d055ddf462..55892a2fc7 100644
> --- a/target/riscv/cpu.c
> +++ b/target/riscv/cpu.c
> @@ -37,6 +37,7 @@
>   #include "kvm/kvm_riscv.h"
>   #include "tcg/tcg-cpu.h"
>   #include "tcg/tcg.h"
> +#include "hw/boards.h"
>   
>   /* RISC-V CPU definitions */
>   static const char riscv_single_letter_exts[] = "IEMAFDQCBPVH";
> @@ -685,18 +686,32 @@ static void riscv_cpu_reset_hold(Object *obj, ResetType type)
>   #ifndef CONFIG_USER_ONLY
>       uint8_t iprio;
>       int i, irq, rdzero;
> +    static int boot_cpu_index;
>   #endif
>       CPUState *cs = CPU(obj);
>       RISCVCPU *cpu = RISCV_CPU(cs);
>       RISCVCPUClass *mcc = RISCV_CPU_GET_CLASS(obj);
>       CPURISCVState *env = &cpu->env;
> +    MachineState *ms = MACHINE(qdev_get_machine());
>   
>       if (mcc->parent_phases.hold) {
>           mcc->parent_phases.hold(obj, type);
>       }
>   #ifndef CONFIG_USER_ONLY
>       env->misa_mxl = mcc->def->misa_mxl_max;
> -    env->priv = PRV_M;
> +    if (kvm_enabled()) {
> +        env->priv = PRV_S;
> +    } else {
> +        env->priv = PRV_M;
> +    }

Sorry for the late v9 review. riscv_cpu_reset_hold() is reserved for TCG initialization
only. All kvm specific initialization is done in kvm_riscv_reset_vcpu(), called in the
end of this function:

     if (kvm_enabled()) {
         kvm_riscv_reset_vcpu(cpu);
     }

Everything that is KVM specific should be put in kvm_riscv_reset_vcpu() and other KVM
related helpers in target/riscv/kvm/kvm-cpu.c.


As for the except above, I just sent a patch that is fixing it inside kvm_riscv_reset_regs_csr():

("[PATCH] target/riscv/kvm: fix env->priv setting in reset_regs_csr()")
https://lore.kernel.org/qemu-riscv/20251022111105.483992-1-dbarboza@ventanamicro.com/


The reason I went ahead with that patch is because it is related to an opened Gitlab
issue (https://gitlab.com/qemu-project/qemu/-/issues/2991) and we already missed the
window for 10.1 to fix it.

In case you send another version of this series you can either drop this particular
change or move it to kvm_riscv_reset_regs_csr(). We'll keep whatever is merged first.


Thanks,

Daniel



> +    if (cs->cpu_index == 0) {
> +        boot_cpu_index = g_random_int_range(0, ms->smp.cpus);
> +    }
> +    if (cs->cpu_index == boot_cpu_index) {
> +        env->mp_state = KVM_MP_STATE_RUNNABLE;
> +    } else {
> +        env->mp_state = KVM_MP_STATE_STOPPED;
> +    }
>       env->mstatus &= ~(MSTATUS_MIE | MSTATUS_MPRV);
>       if (env->misa_mxl > MXL_RV32) {
>           /*
> diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h
> index 738e68fa6e..7ea4859de7 100644
> --- a/target/riscv/cpu.h
> +++ b/target/riscv/cpu.h
> @@ -269,6 +269,8 @@ struct CPUArchState {
>   #endif
>   
>       target_ulong priv;
> +    /* Current multiprocessor state of this vCPU. */
> +    uint32_t mp_state;
>       /* CSRs for execution environment configuration */
>       uint64_t menvcfg;
>       target_ulong senvcfg;

Re: [PATCH v9 1/2] Set KVM initial privilege mode and mp_state

Posted by Alistair Francis 4 months, 2 weeks ago

On Mon, Sep 15, 2025 at 5:08 PM Xie Bo <xb@ultrarisc.com> wrote:
>
> For KVM mode, the privilege mode should not include M-mode, and the
> initial value should be set to S-mode. Additionally, a following patch
> adds the implementation of putting the vCPU privilege mode to KVM.
> When the vCPU runs for the first time, QEMU will first put the privilege
> state to KVM. If the initial value is set to M-mode, KVM will encounter
> an error.
>
> In addition, this patch introduces the 'mp_state' field to RISC-V
> vCPUs, following the convention used by KVM on x86. The 'mp_state'
> reflects the multiprocessor state of a vCPU, and is used to control
> whether the vCPU is runnable by KVM. Randomly select one CPU as the
> boot CPU. Since each CPU executes the riscv_cpu_reset_hold() function
> and CPU0 executes first, only CPU0 randomly selects the boot CPU.
>
> Signed-off-by: Xie Bo <xb@ultrarisc.com>
> ---
>  target/riscv/cpu.c | 17 ++++++++++++++++-
>  target/riscv/cpu.h |  2 ++
>  2 files changed, 18 insertions(+), 1 deletion(-)

This fails to build with the following error, it seems an include is missing

../target/riscv/cpu.c: In function ‘riscv_cpu_reset_hold’:
../target/riscv/cpu.c:711:25: error: ‘KVM_MP_STATE_RUNNABLE’
undeclared (first use in this function)
 711 |         env->mp_state = KVM_MP_STATE_RUNNABLE;
     |                         ^~~~~~~~~~~~~~~~~~~~~
../target/riscv/cpu.c:711:25: note: each undeclared identifier is
reported only once for each function it appears in
../target/riscv/cpu.c:713:25: error: ‘KVM_MP_STATE_STOPPED’ undeclared
(first use in this function); did you mean ‘S390_CPU_STATE_STOPPED’?
 713 |         env->mp_state = KVM_MP_STATE_STOPPED;
     |                         ^~~~~~~~~~~~~~~~~~~~
     |                         S390_CPU_STATE_STO

Alistair

>
> diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
> index d055ddf462..55892a2fc7 100644
> --- a/target/riscv/cpu.c
> +++ b/target/riscv/cpu.c
> @@ -37,6 +37,7 @@
>  #include "kvm/kvm_riscv.h"
>  #include "tcg/tcg-cpu.h"
>  #include "tcg/tcg.h"
> +#include "hw/boards.h"
>
>  /* RISC-V CPU definitions */
>  static const char riscv_single_letter_exts[] = "IEMAFDQCBPVH";
> @@ -685,18 +686,32 @@ static void riscv_cpu_reset_hold(Object *obj, ResetType type)
>  #ifndef CONFIG_USER_ONLY
>      uint8_t iprio;
>      int i, irq, rdzero;
> +    static int boot_cpu_index;
>  #endif
>      CPUState *cs = CPU(obj);
>      RISCVCPU *cpu = RISCV_CPU(cs);
>      RISCVCPUClass *mcc = RISCV_CPU_GET_CLASS(obj);
>      CPURISCVState *env = &cpu->env;
> +    MachineState *ms = MACHINE(qdev_get_machine());
>
>      if (mcc->parent_phases.hold) {
>          mcc->parent_phases.hold(obj, type);
>      }
>  #ifndef CONFIG_USER_ONLY
>      env->misa_mxl = mcc->def->misa_mxl_max;
> -    env->priv = PRV_M;
> +    if (kvm_enabled()) {
> +        env->priv = PRV_S;
> +    } else {
> +        env->priv = PRV_M;
> +    }
> +    if (cs->cpu_index == 0) {
> +        boot_cpu_index = g_random_int_range(0, ms->smp.cpus);
> +    }
> +    if (cs->cpu_index == boot_cpu_index) {
> +        env->mp_state = KVM_MP_STATE_RUNNABLE;
> +    } else {
> +        env->mp_state = KVM_MP_STATE_STOPPED;
> +    }
>      env->mstatus &= ~(MSTATUS_MIE | MSTATUS_MPRV);
>      if (env->misa_mxl > MXL_RV32) {
>          /*
> diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h
> index 738e68fa6e..7ea4859de7 100644
> --- a/target/riscv/cpu.h
> +++ b/target/riscv/cpu.h
> @@ -269,6 +269,8 @@ struct CPUArchState {
>  #endif
>
>      target_ulong priv;
> +    /* Current multiprocessor state of this vCPU. */
> +    uint32_t mp_state;
>      /* CSRs for execution environment configuration */
>      uint64_t menvcfg;
>      target_ulong senvcfg;
> --
> 2.43.0
>

Re: [PATCH v9 1/2] Set KVM initial privilege mode and mp_state

Posted by Radim Krčmář 4 months, 1 week ago

2025-09-29T11:48:08+10:00, Alistair Francis <alistair23@gmail.com>:
> On Mon, Sep 15, 2025 at 5:08 PM Xie Bo <xb@ultrarisc.com> wrote:
>>
>> For KVM mode, the privilege mode should not include M-mode, and the
>> initial value should be set to S-mode. Additionally, a following patch
>> adds the implementation of putting the vCPU privilege mode to KVM.
>> When the vCPU runs for the first time, QEMU will first put the privilege
>> state to KVM. If the initial value is set to M-mode, KVM will encounter
>> an error.
>>
>> In addition, this patch introduces the 'mp_state' field to RISC-V
>> vCPUs, following the convention used by KVM on x86. The 'mp_state'
>> reflects the multiprocessor state of a vCPU, and is used to control
>> whether the vCPU is runnable by KVM. Randomly select one CPU as the
>> boot CPU. Since each CPU executes the riscv_cpu_reset_hold() function
>> and CPU0 executes first, only CPU0 randomly selects the boot CPU.
>>
>> Signed-off-by: Xie Bo <xb@ultrarisc.com>
>> ---
>>  target/riscv/cpu.c | 17 ++++++++++++++++-
>>  target/riscv/cpu.h |  2 ++
>>  2 files changed, 18 insertions(+), 1 deletion(-)
>
> This fails to build with the following error, it seems an include is missing
>
> ../target/riscv/cpu.c: In function ‘riscv_cpu_reset_hold’:
> ../target/riscv/cpu.c:711:25: error: ‘KVM_MP_STATE_RUNNABLE’
> undeclared (first use in this function)
>  711 |         env->mp_state = KVM_MP_STATE_RUNNABLE;
>      |                         ^~~~~~~~~~~~~~~~~~~~~
> ../target/riscv/cpu.c:711:25: note: each undeclared identifier is
> reported only once for each function it appears in
> ../target/riscv/cpu.c:713:25: error: ‘KVM_MP_STATE_STOPPED’ undeclared
> (first use in this function); did you mean ‘S390_CPU_STATE_STOPPED’?
>  713 |         env->mp_state = KVM_MP_STATE_STOPPED;
>      |                         ^~~~~~~~~~~~~~~~~~~~
>      |                         S390_CPU_STATE_STO

I think this is because the code belongs in kvm_riscv_reset_vcpu().

>> diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
>> +    static int boot_cpu_index;
>> +    MachineState *ms = MACHINE(qdev_get_machine());
>> +    if (cs->cpu_index == 0) {
>> +        boot_cpu_index = g_random_int_range(0, ms->smp.cpus);
>> +    }

If we're already touching the code, can we also express boot_cpu_index
in a VM state?  Using a static variable is pretty scary.

Thanks.

Re: [PATCH v9 1/2] Set KVM initial privilege mode and mp_state

Posted by Radim Krčmář 4 months, 3 weeks ago

2025-09-15T15:08:07+08:00, Xie Bo <xb@ultrarisc.com>:
> For KVM mode, the privilege mode should not include M-mode, and the
> initial value should be set to S-mode. Additionally, a following patch
> adds the implementation of putting the vCPU privilege mode to KVM.
> When the vCPU runs for the first time, QEMU will first put the privilege
> state to KVM. If the initial value is set to M-mode, KVM will encounter
> an error.
>
> In addition, this patch introduces the 'mp_state' field to RISC-V
> vCPUs, following the convention used by KVM on x86. The 'mp_state'
> reflects the multiprocessor state of a vCPU, and is used to control
> whether the vCPU is runnable by KVM. Randomly select one CPU as the
> boot CPU. Since each CPU executes the riscv_cpu_reset_hold() function
> and CPU0 executes first, only CPU0 randomly selects the boot CPU.

This could really be two patches, as changing the boot 

> Signed-off-by: Xie Bo <xb@ultrarisc.com>
> ---
> diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
> @@ -685,18 +686,32 @@ static void riscv_cpu_reset_hold(Object *obj, ResetType type)
> +    static int boot_cpu_index;
       ^^^^^^
!

> +    if (kvm_enabled()) {
> +        env->priv = PRV_S;
> +    } else {
> +        env->priv = PRV_M;
> +    }

(I think changing the priv belongs to a separate patch.)

> +    if (cs->cpu_index == 0) {
> +        boot_cpu_index = g_random_int_range(0, ms->smp.cpus);
> +    }

This adds an assumption that vcpu_index == 0 is executed first.
Is that always going to be true?

If we reset the VCPUs in a different order (or in parallel), we might
also online zero or two VCPUs.

Performing the selection just once, in the reset initiator, would allow
us to avoid the dreaded static variable by putting it in machine arch
state.

> +    if (cs->cpu_index == boot_cpu_index) {
> +        env->mp_state = KVM_MP_STATE_RUNNABLE;
> +    } else {
> +        env->mp_state = KVM_MP_STATE_STOPPED;
> +    }

Thanks.