RISC-V: KVM: Read HGEIP CSR on the correct cpu

[PATCH] RISC-V: KVM: Read HGEIP CSR on the correct cpu

Posted by fangyu.yu@linux.alibaba.com 3 months, 3 weeks ago

From: Fangyu Yu <fangyu.yu@linux.alibaba.com>

When executing kvm_riscv_vcpu_aia_has_interrupts, the vCPU may have
migrated and the IMSIC VS-file have not been updated yet, currently
the HGEIP CSR should be read from the imsic->vsfile_cpu ( the pCPU
before migration ) via on_each_cpu_mask, but this will trigger an
IPI call and repeated IPI within a period of time is expensive in
a many-core systems.

Just let the vCPU execute and update the correct IMSIC VS-file via
kvm_riscv_vcpu_aia_imsic_update may be a simple solution.

Fixes: 4cec89db80ba ("RISC-V: KVM: Move HGEI[E|P] CSR access to IMSIC virtualization")
Signed-off-by: Fangyu Yu <fangyu.yu@linux.alibaba.com>
---
 arch/riscv/kvm/aia_imsic.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/arch/riscv/kvm/aia_imsic.c b/arch/riscv/kvm/aia_imsic.c
index fda0346f0ea1..168c02ad0a78 100644
--- a/arch/riscv/kvm/aia_imsic.c
+++ b/arch/riscv/kvm/aia_imsic.c
@@ -689,8 +689,12 @@ bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu)
 	 */
 
 	read_lock_irqsave(&imsic->vsfile_lock, flags);
-	if (imsic->vsfile_cpu > -1)
-		ret = !!(csr_read(CSR_HGEIP) & BIT(imsic->vsfile_hgei));
+	if (imsic->vsfile_cpu > -1) {
+		if (imsic->vsfile_cpu != smp_processor_id())
+			ret = true;
+		else
+			ret = !!(csr_read(CSR_HGEIP) & BIT(imsic->vsfile_hgei));
+	}
 	read_unlock_irqrestore(&imsic->vsfile_lock, flags);
 
 	return ret;
-- 
2.50.1

Re: [PATCH] RISC-V: KVM: Read HGEIP CSR on the correct cpu

Posted by Anup Patel 3 months, 3 weeks ago

On Thu, Oct 16, 2025 at 6:57 AM <fangyu.yu@linux.alibaba.com> wrote:
>
> From: Fangyu Yu <fangyu.yu@linux.alibaba.com>
>
> When executing kvm_riscv_vcpu_aia_has_interrupts, the vCPU may have
> migrated and the IMSIC VS-file have not been updated yet, currently
> the HGEIP CSR should be read from the imsic->vsfile_cpu ( the pCPU
> before migration ) via on_each_cpu_mask, but this will trigger an
> IPI call and repeated IPI within a period of time is expensive in
> a many-core systems.
>
> Just let the vCPU execute and update the correct IMSIC VS-file via
> kvm_riscv_vcpu_aia_imsic_update may be a simple solution.
>
> Fixes: 4cec89db80ba ("RISC-V: KVM: Move HGEI[E|P] CSR access to IMSIC virtualization")
> Signed-off-by: Fangyu Yu <fangyu.yu@linux.alibaba.com>
> ---
>  arch/riscv/kvm/aia_imsic.c | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/arch/riscv/kvm/aia_imsic.c b/arch/riscv/kvm/aia_imsic.c
> index fda0346f0ea1..168c02ad0a78 100644
> --- a/arch/riscv/kvm/aia_imsic.c
> +++ b/arch/riscv/kvm/aia_imsic.c
> @@ -689,8 +689,12 @@ bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu)
>          */
>
>         read_lock_irqsave(&imsic->vsfile_lock, flags);
> -       if (imsic->vsfile_cpu > -1)
> -               ret = !!(csr_read(CSR_HGEIP) & BIT(imsic->vsfile_hgei));
> +       if (imsic->vsfile_cpu > -1) {
> +               if (imsic->vsfile_cpu != smp_processor_id())

Good catch !!!

I agree with Guo Ren. We should use "vcpu->cpu" over here
instead of smp_processor_id(). Also, I think we should add
some comments for future reference. I will take care of this
at the time of merging this patch.

Queued this as fixes for Linux-6.18

Thanks,
Anup

> +                       ret = true;
> +               else
> +                       ret = !!(csr_read(CSR_HGEIP) & BIT(imsic->vsfile_hgei));
> +       }
>         read_unlock_irqrestore(&imsic->vsfile_lock, flags);
>
>         return ret;
> --
> 2.50.1
>

Re: [PATCH] RISC-V: KVM: Read HGEIP CSR on the correct cpu

Posted by Guo Ren 3 months, 3 weeks ago

On Fri, Oct 17, 2025 at 2:59 PM Anup Patel <anup@brainfault.org> wrote:
>
> On Thu, Oct 16, 2025 at 6:57 AM <fangyu.yu@linux.alibaba.com> wrote:
> >
> > From: Fangyu Yu <fangyu.yu@linux.alibaba.com>
> >
> > When executing kvm_riscv_vcpu_aia_has_interrupts, the vCPU may have
> > migrated and the IMSIC VS-file have not been updated yet, currently
> > the HGEIP CSR should be read from the imsic->vsfile_cpu ( the pCPU
> > before migration ) via on_each_cpu_mask, but this will trigger an
> > IPI call and repeated IPI within a period of time is expensive in
> > a many-core systems.
> >
> > Just let the vCPU execute and update the correct IMSIC VS-file via
> > kvm_riscv_vcpu_aia_imsic_update may be a simple solution.
> >
> > Fixes: 4cec89db80ba ("RISC-V: KVM: Move HGEI[E|P] CSR access to IMSIC virtualization")
> > Signed-off-by: Fangyu Yu <fangyu.yu@linux.alibaba.com>
> > ---
> >  arch/riscv/kvm/aia_imsic.c | 8 ++++++--
> >  1 file changed, 6 insertions(+), 2 deletions(-)
> >
> > diff --git a/arch/riscv/kvm/aia_imsic.c b/arch/riscv/kvm/aia_imsic.c
> > index fda0346f0ea1..168c02ad0a78 100644
> > --- a/arch/riscv/kvm/aia_imsic.c
> > +++ b/arch/riscv/kvm/aia_imsic.c
> > @@ -689,8 +689,12 @@ bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu)
> >          */
> >
> >         read_lock_irqsave(&imsic->vsfile_lock, flags);
> > -       if (imsic->vsfile_cpu > -1)
> > -               ret = !!(csr_read(CSR_HGEIP) & BIT(imsic->vsfile_hgei));
> > +       if (imsic->vsfile_cpu > -1) {
> > +               if (imsic->vsfile_cpu != smp_processor_id())
>
> Good catch !!!
>
> I agree with Guo Ren. We should use "vcpu->cpu" over here
> instead of smp_processor_id(). Also, I think we should add
> some comments for future reference. I will take care of this
> at the time of merging this patch.
>
> Queued this as fixes for Linux-6.18
Thx Anup, then Fangyu needn't update v2.

This patch also reveals the idea of our CSR_HGEIP design - Per-CPU access.
Although arm64 & x86 don't have a similar problem by using a shared
memory address map, it would be a complex hw design. The RISC-V
CSR_HGEIP design is much simpler for hw, and vCPU migration on the
pCPUs could tolerate one additional VM_enter/exit cost for an idle
vcpu.

>
> Thanks,
> Anup
>
> > +                       ret = true;
> > +               else
> > +                       ret = !!(csr_read(CSR_HGEIP) & BIT(imsic->vsfile_hgei));
> > +       }
> >         read_unlock_irqrestore(&imsic->vsfile_lock, flags);
> >
> >         return ret;
> > --
> > 2.50.1
> >



-- 
Best Regards
 Guo Ren

Re: [PATCH] RISC-V: KVM: Read HGEIP CSR on the correct cpu

Posted by Guo Ren 3 months, 3 weeks ago

On Thu, Oct 16, 2025 at 9:27 AM <fangyu.yu@linux.alibaba.com> wrote:
>
> From: Fangyu Yu <fangyu.yu@linux.alibaba.com>
>
> When executing kvm_riscv_vcpu_aia_has_interrupts, the vCPU may have
> migrated and the IMSIC VS-file have not been updated yet, currently
> the HGEIP CSR should be read from the imsic->vsfile_cpu ( the pCPU
> before migration ) via on_each_cpu_mask, but this will trigger an
> IPI call and repeated IPI within a period of time is expensive in
> a many-core systems.
>
> Just let the vCPU execute and update the correct IMSIC VS-file via
> kvm_riscv_vcpu_aia_imsic_update may be a simple solution.
>
> Fixes: 4cec89db80ba ("RISC-V: KVM: Move HGEI[E|P] CSR access to IMSIC virtualization")
> Signed-off-by: Fangyu Yu <fangyu.yu@linux.alibaba.com>
> ---
>  arch/riscv/kvm/aia_imsic.c | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/arch/riscv/kvm/aia_imsic.c b/arch/riscv/kvm/aia_imsic.c
> index fda0346f0ea1..168c02ad0a78 100644
> --- a/arch/riscv/kvm/aia_imsic.c
> +++ b/arch/riscv/kvm/aia_imsic.c
> @@ -689,8 +689,12 @@ bool kvm_riscv_vcpu_aia_imsic_has_interrupt(struct kvm_vcpu *vcpu)
>          */
>
>         read_lock_irqsave(&imsic->vsfile_lock, flags);
> -       if (imsic->vsfile_cpu > -1)
> -               ret = !!(csr_read(CSR_HGEIP) & BIT(imsic->vsfile_hgei));
> +       if (imsic->vsfile_cpu > -1) {
> +               if (imsic->vsfile_cpu != smp_processor_id())
Why not if (imsic->vsfile_cpu != vcpu->cpu)?
if pcpu changed, the kvm_sched_in() would update "vcpu->cpu by
raw_smp_processor_id()".

The smp_processor_id() may involve DEBUG_PREEMPT code, and
check_preemption_disabled() may cause a problem? Anyway, a full
smp_processor_id() is not needed here, vcpu->cpu is enough.

If (imsic->vsfile_cpu != vcpu->cpu), then:

Reviewed-by: Guo Ren <guoren@kernel.org>

> +                       ret = true;
Agree!

When a vCPU migrates to another pCPU, we could assume the interrupt
has come in to prevent the missing interrupt.

To illustrate why ret = true, consider this example:

1. Guest WFI VM_exit.

2. Host Trap -> kvm_riscv_vcpu_exit() -> kvm_riscv_vcpu_virtual_insn()
->kvm_vcpu_halt() ->
    kvm_vcpu_check_block() -> kvm_riscv_vcpu_aia_imsic_has_interrupt() loop:
        for (;;) {
                set_current_state(TASK_INTERRUPTIBLE);

                if (kvm_vcpu_check_block(vcpu) < 0)
                        break;

                waited = true;
                schedule(); // if pcpu changed, the kvm_sched_in()
would update "vcpu->cpu = raw_smp_processor_id()".
        }

 3. Back to the outer loop in kvm/vcpu.c:
        while (ret > 0) {
                /* Check conditions before entering the guest */
                ret = xfer_to_guest_mode_handle_work(vcpu);
                if (ret)
                        continue;
                ret = 1;

                kvm_riscv_gstage_vmid_update(vcpu);

                kvm_riscv_check_vcpu_requests(vcpu);

                preempt_disable();

                /* Update AIA HW state before entering guest */
                ret = kvm_riscv_vcpu_aia_update(vcpu); // Update
imsic->vsfile_cpu with vcpu->cpu;

4. VM_enter Guest WFI
    If there are no changes, then Guest WFI VM_exit again.

5. Host Trap again, but the vsfile_cpu & vcpu->cpu &
raw_smp_processor_id() are synchronized.



> +               else
> +                       ret = !!(csr_read(CSR_HGEIP) & BIT(imsic->vsfile_hgei));
> +       }
>         read_unlock_irqrestore(&imsic->vsfile_lock, flags);
>
>         return ret;
> --
> 2.50.1
>


--
Best Regards
 Guo Ren