arch/riscv/include/asm/kvm_host.h | 1 + 1 file changed, 1 insertion(+)
Delegate illegal instruction fault to VS mode in default to avoid such
exceptions being trapped to HS and redirected back to VS.
Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
---
arch/riscv/include/asm/kvm_host.h | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h
index 85cfebc32e4cf..97cc2c0dba73a 100644
--- a/arch/riscv/include/asm/kvm_host.h
+++ b/arch/riscv/include/asm/kvm_host.h
@@ -48,6 +48,7 @@
BIT(EXC_SYSCALL) | \
BIT(EXC_INST_PAGE_FAULT) | \
BIT(EXC_LOAD_PAGE_FAULT) | \
+ BIT(EXC_INST_ILLEGAL) | \
BIT(EXC_STORE_PAGE_FAULT))
#define KVM_HIDELEG_DEFAULT (BIT(IRQ_VS_SOFT) | \
--
2.20.12025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>: > Delegate illegal instruction fault to VS mode in default to avoid such > exceptions being trapped to HS and redirected back to VS. > > Signed-off-by: Xu Lu <luxu.kernel@bytedance.com> > --- > diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h > @@ -48,6 +48,7 @@ > + BIT(EXC_INST_ILLEGAL) | \ You should also remove the dead code in kvm_riscv_vcpu_exit. And why not delegate the others as well? (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS, EXC_STORE_ACCESS, and EXC_INST_ACCESS.) Thanks.
On 20/06/2025 14:04, Radim Krčmář wrote: > 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>: >> Delegate illegal instruction fault to VS mode in default to avoid such >> exceptions being trapped to HS and redirected back to VS. >> >> Signed-off-by: Xu Lu <luxu.kernel@bytedance.com> >> --- >> diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h >> @@ -48,6 +48,7 @@ >> + BIT(EXC_INST_ILLEGAL) | \ > > You should also remove the dead code in kvm_riscv_vcpu_exit. > > And why not delegate the others as well? > (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS, > EXC_STORE_ACCESS, and EXC_INST_ACCESS.) Currently, OpenSBI does not delegate misaligned exception by default and handles misaligned access by itself, this is (partially) why we added the FWFT SBI extension to request such delegation. Since some supervisor software expect that default, they do not have code to handle misaligned accesses emulation. So they should not be delegated by default. Thanks, Clément > > Thanks. > > _______________________________________________ > linux-riscv mailing list > linux-riscv@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-riscv
Hi Clément, On Mon, Jun 23, 2025 at 4:05 PM Clément Léger <cleger@rivosinc.com> wrote: > > > > On 20/06/2025 14:04, Radim Krčmář wrote: > > 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>: > >> Delegate illegal instruction fault to VS mode in default to avoid such > >> exceptions being trapped to HS and redirected back to VS. > >> > >> Signed-off-by: Xu Lu <luxu.kernel@bytedance.com> > >> --- > >> diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h > >> @@ -48,6 +48,7 @@ > >> + BIT(EXC_INST_ILLEGAL) | \ > > > > You should also remove the dead code in kvm_riscv_vcpu_exit. > > > > And why not delegate the others as well? > > (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS, > > EXC_STORE_ACCESS, and EXC_INST_ACCESS.) > > Currently, OpenSBI does not delegate misaligned exception by default and > handles misaligned access by itself, this is (partially) why we added > the FWFT SBI extension to request such delegation. Since some supervisor > software expect that default, they do not have code to handle misaligned > accesses emulation. So they should not be delegated by default. It doesn't matter whether these exceptions are delegated in medeleg. KVM in HS-mode does not handle illegal instruction or misaligned access and only redirects them back to VS-mode. Delegating such exceptions in hedeleg helps save CPU usage even when they are not delegated in medeleg: opensbi will check whether these exceptions are delegated to VS-mode and redirect them to VS-mode if possible. There seems to be no conflicts with SSE implementation. Please correct me if I missed anything. Best Regards, Xu Lu > > Thanks, > > Clément > > > > > Thanks. > > > > _______________________________________________ > > linux-riscv mailing list > > linux-riscv@lists.infradead.org > > http://lists.infradead.org/mailman/listinfo/linux-riscv >
On 23/06/2025 14:12, Xu Lu wrote: > Hi Clément, > > On Mon, Jun 23, 2025 at 4:05 PM Clément Léger <cleger@rivosinc.com> wrote: >> >> >> >> On 20/06/2025 14:04, Radim Krčmář wrote: >>> 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>: >>>> Delegate illegal instruction fault to VS mode in default to avoid such >>>> exceptions being trapped to HS and redirected back to VS. >>>> >>>> Signed-off-by: Xu Lu <luxu.kernel@bytedance.com> >>>> --- >>>> diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h >>>> @@ -48,6 +48,7 @@ >>>> + BIT(EXC_INST_ILLEGAL) | \ >>> >>> You should also remove the dead code in kvm_riscv_vcpu_exit. >>> >>> And why not delegate the others as well? >>> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS, >>> EXC_STORE_ACCESS, and EXC_INST_ACCESS.) >> >> Currently, OpenSBI does not delegate misaligned exception by default and >> handles misaligned access by itself, this is (partially) why we added >> the FWFT SBI extension to request such delegation. Since some supervisor >> software expect that default, they do not have code to handle misaligned >> accesses emulation. So they should not be delegated by default. > > It doesn't matter whether these exceptions are delegated in medeleg. Not sure to totally understand, but if the exceptions are not delegated in medeleg, they won't be delegated to VS-mode even though hedeleg bit is set right ? The spec says: A synchronous trap that has been delegated to HS-mode (using medeleg) is further delegated to VS-mode if V=1 before the trap and the corresponding hedeleg bit is set. > KVM in HS-mode does not handle illegal instruction or misaligned > access and only redirects them back to VS-mode. Delegating such > exceptions in hedeleg helps save CPU usage even when they are not > delegated in medeleg: opensbi will check whether these exceptions are > delegated to VS-mode and redirect them to VS-mode if possible. There > seems to be no conflicts with SSE implementation. Please correct me if > I missed anything. AFAIU, this means that since medeleg bit for misaligned accesses were not delegated up to the introduction of the FWFT extension, VS-mode generated misaligned accesses were handled by OpenSBI right ? Now that we are requesting openSBI to delegate misaligned accesses, HS-mode handles it's own misaligned accesses through the trap handler. With that configuration, if VS-mode generate a misaligned access, it will end up being redirected to VS-mode and won't be handle by HS-mode. To summarize, prior to FWFT, medeleg wasn't delegating misaligned accesses to S-mode: - VS-mode misaligned access -> trap to M-mode -> OpenSBI handle it -> Back to VS-mode, misaligned access fixed up by OpenSBI Now that Linux uses SBI FWFT to delegates misaligned accesses (without hedeleg being set for misaligned delegation, but that doesn't really matter, the outcome is the same): - VS-mode misaligned access -> trap to HS-mode -> redirection to VS-mode, needs to handle the misaligned access by itself This means that previously, misaligned access were silently fixed up by OpenSBI for VS-mode and now that FWFT is used for delegation, this isn't true anymore. So, old kernel or sueprvisor software that included code to handle misaligned accesses will crash. Did I missed something ? Note: this is not directly related to your series but my introduction of FWFT ! Thanks, Clément > > Best Regards, > Xu Lu > >> >> Thanks, >> >> Clément >> >>> >>> Thanks. >>> >>> _______________________________________________ >>> linux-riscv mailing list >>> linux-riscv@lists.infradead.org >>> http://lists.infradead.org/mailman/listinfo/linux-riscv >>
Hi Clément, On Mon, Jun 23, 2025 at 8:35 PM Clément Léger <cleger@rivosinc.com> wrote: > > > > On 23/06/2025 14:12, Xu Lu wrote: > > Hi Clément, > > > > On Mon, Jun 23, 2025 at 4:05 PM Clément Léger <cleger@rivosinc.com> wrote: > >> > >> > >> > >> On 20/06/2025 14:04, Radim Krčmář wrote: > >>> 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>: > >>>> Delegate illegal instruction fault to VS mode in default to avoid such > >>>> exceptions being trapped to HS and redirected back to VS. > >>>> > >>>> Signed-off-by: Xu Lu <luxu.kernel@bytedance.com> > >>>> --- > >>>> diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h > >>>> @@ -48,6 +48,7 @@ > >>>> + BIT(EXC_INST_ILLEGAL) | \ > >>> > >>> You should also remove the dead code in kvm_riscv_vcpu_exit. > >>> > >>> And why not delegate the others as well? > >>> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS, > >>> EXC_STORE_ACCESS, and EXC_INST_ACCESS.) > >> > >> Currently, OpenSBI does not delegate misaligned exception by default and > >> handles misaligned access by itself, this is (partially) why we added > >> the FWFT SBI extension to request such delegation. Since some supervisor > >> software expect that default, they do not have code to handle misaligned > >> accesses emulation. So they should not be delegated by default. > > > > It doesn't matter whether these exceptions are delegated in medeleg. > > Not sure to totally understand, but if the exceptions are not delegated > in medeleg, they won't be delegated to VS-mode even though hedeleg bit > is set right ? The spec says: > > A synchronous trap that has been delegated to HS-mode (using medeleg) > is further delegated to VS-mode if V=1 before the trap and the > corresponding hedeleg bit is set. Yes, you are right. The illegal insn exception is still trapped in M mode if it is not delegated in medeleg. But delegating it in hedeleg is still useful. The opensbi will check CSR_HEDELEG in the function sbi_trap_redirect. If the exception has been delegated to VS-mode in CSR_HEDLEG, opensbi can directly return back to VS-mode, without the overhead of going back to HS-mode and then going back to VS-mode. > > > > > KVM in HS-mode does not handle illegal instruction or misaligned > > access and only redirects them back to VS-mode. Delegating such > > exceptions in hedeleg helps save CPU usage even when they are not > > delegated in medeleg: opensbi will check whether these exceptions are > > delegated to VS-mode and redirect them to VS-mode if possible. There > > seems to be no conflicts with SSE implementation. Please correct me if > > I missed anything. > > AFAIU, this means that since medeleg bit for misaligned accesses were > not delegated up to the introduction of the FWFT extension, VS-mode > generated misaligned accesses were handled by OpenSBI right ? Now that > we are requesting openSBI to delegate misaligned accesses, HS-mode > handles it's own misaligned accesses through the trap handler. With that > configuration, if VS-mode generate a misaligned access, it will end up > being redirected to VS-mode and won't be handle by HS-mode. > > To summarize, prior to FWFT, medeleg wasn't delegating misaligned > accesses to S-mode: > > - VS-mode misaligned access -> trap to M-mode -> OpenSBI handle it -> > Back to VS-mode, misaligned access fixed up by OpenSBI Yes, this is what I want the procedure of handling illegal insn exceptions to be. Actually it now behaves as: VS-mode illegal insn exception -> trap to M-mode -> OpenSBI handles it -> Back to HS-mode, does nothing -> Back to VS-mode. I want to avoid going through HS-mode. > > Now that Linux uses SBI FWFT to delegates misaligned accesses (without > hedeleg being set for misaligned delegation, but that doesn't really > matter, the outcome is the same): > > - VS-mode misaligned access -> trap to HS-mode -> redirection to > VS-mode, needs to handle the misaligned access by itself > > > This means that previously, misaligned access were silently fixed up by > OpenSBI for VS-mode and now that FWFT is used for delegation, this isn't > true anymore. So, old kernel or sueprvisor software that included code > to handle misaligned accesses will crash. Did I missed something ? Great! You make it very clear! Thanks for your explanation. But even when misalign exceptions are delegated to HS-mode, KVM seems to do nothing but redirect to VS-mode when VM get trapped due to misalign exceptions. So maybe we can directly delegate the misaligned exceptions in hedeleg too before running VCPU and retrieve them after VCPU exists. And then the handling procedure will be: VS-mode misaligned exception -> trap to VS-mode -> VS handles it -> Back to VU-mode. Please correct me if I missed anything. Best Regards, Xu Lu > > Note: this is not directly related to your series but my introduction of > FWFT ! > > Thanks, > > Clément > > > > > Best Regards, > > Xu Lu > > > >> > >> Thanks, > >> > >> Clément > >> > >>> > >>> Thanks. > >>> > >>> _______________________________________________ > >>> linux-riscv mailing list > >>> linux-riscv@lists.infradead.org > >>> http://lists.infradead.org/mailman/listinfo/linux-riscv > >> >
On 23/06/2025 15:30, Xu Lu wrote: > Hi Clément, > > On Mon, Jun 23, 2025 at 8:35 PM Clément Léger <cleger@rivosinc.com> wrote: >> >> >> >> On 23/06/2025 14:12, Xu Lu wrote: >>> Hi Clément, >>> >>> On Mon, Jun 23, 2025 at 4:05 PM Clément Léger <cleger@rivosinc.com> wrote: >>>> >>>> >>>> >>>> On 20/06/2025 14:04, Radim Krčmář wrote: >>>>> 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>: >>>>>> Delegate illegal instruction fault to VS mode in default to avoid such >>>>>> exceptions being trapped to HS and redirected back to VS. >>>>>> >>>>>> Signed-off-by: Xu Lu <luxu.kernel@bytedance.com> >>>>>> --- >>>>>> diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h >>>>>> @@ -48,6 +48,7 @@ >>>>>> + BIT(EXC_INST_ILLEGAL) | \ >>>>> >>>>> You should also remove the dead code in kvm_riscv_vcpu_exit. >>>>> >>>>> And why not delegate the others as well? >>>>> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS, >>>>> EXC_STORE_ACCESS, and EXC_INST_ACCESS.) >>>> >>>> Currently, OpenSBI does not delegate misaligned exception by default and >>>> handles misaligned access by itself, this is (partially) why we added >>>> the FWFT SBI extension to request such delegation. Since some supervisor >>>> software expect that default, they do not have code to handle misaligned >>>> accesses emulation. So they should not be delegated by default. >>> >>> It doesn't matter whether these exceptions are delegated in medeleg. >> >> Not sure to totally understand, but if the exceptions are not delegated >> in medeleg, they won't be delegated to VS-mode even though hedeleg bit >> is set right ? The spec says: >> >> A synchronous trap that has been delegated to HS-mode (using medeleg) >> is further delegated to VS-mode if V=1 before the trap and the >> corresponding hedeleg bit is set. > > Yes, you are right. The illegal insn exception is still trapped in M > mode if it is not delegated in medeleg. But delegating it in hedeleg > is still useful. The opensbi will check CSR_HEDELEG in the function > sbi_trap_redirect. If the exception has been delegated to VS-mode in > CSR_HEDLEG, opensbi can directly return back to VS-mode, without the > overhead of going back to HS-mode and then going back to VS-mode. > >> >> >> >>> KVM in HS-mode does not handle illegal instruction or misaligned >>> access and only redirects them back to VS-mode. Delegating such >>> exceptions in hedeleg helps save CPU usage even when they are not >>> delegated in medeleg: opensbi will check whether these exceptions are >>> delegated to VS-mode and redirect them to VS-mode if possible. There >>> seems to be no conflicts with SSE implementation. Please correct me if >>> I missed anything. >> >> AFAIU, this means that since medeleg bit for misaligned accesses were >> not delegated up to the introduction of the FWFT extension, VS-mode >> generated misaligned accesses were handled by OpenSBI right ? Now that >> we are requesting openSBI to delegate misaligned accesses, HS-mode >> handles it's own misaligned accesses through the trap handler. With that >> configuration, if VS-mode generate a misaligned access, it will end up >> being redirected to VS-mode and won't be handle by HS-mode. >> >> To summarize, prior to FWFT, medeleg wasn't delegating misaligned >> accesses to S-mode: >> >> - VS-mode misaligned access -> trap to M-mode -> OpenSBI handle it -> >> Back to VS-mode, misaligned access fixed up by OpenSBI > > Yes, this is what I want the procedure of handling illegal insn > exceptions to be. Actually it now behaves as: > > VS-mode illegal insn exception -> trap to M-mode -> OpenSBI handles it > -> Back to HS-mode, does nothing -> Back to VS-mode. > > I want to avoid going through HS-mode. Hi Xu, Yeah, that make sense as well but that should only happen if the VS-mode requested misaligned access delegation via KVM SBI FWFT interface. I know that currently KVM does do anything useful from the misaligned exception except redirecting it to VS-mode but IMHO, that's a regression I introduced with FWFT misaligned requested delegation... > >> >> Now that Linux uses SBI FWFT to delegates misaligned accesses (without >> hedeleg being set for misaligned delegation, but that doesn't really >> matter, the outcome is the same): >> >> - VS-mode misaligned access -> trap to HS-mode -> redirection to >> VS-mode, needs to handle the misaligned access by itself >> >> >> This means that previously, misaligned access were silently fixed up by >> OpenSBI for VS-mode and now that FWFT is used for delegation, this isn't >> true anymore. So, old kernel or sueprvisor software that included code >> to handle misaligned accesses will crash. Did I missed something ? > > Great! You make it very clear! Thanks for your explanation. But even > when misalign exceptions are delegated to HS-mode, KVM seems to do > nothing but redirect to VS-mode when VM get trapped due to misalign > exceptions. Exactly, which is why I said that either setting hedeleg by default or not will lead to the same outcome, ie: VS-mode needs to handle access by itself (which is a regression introduced by FWFT usage). > So maybe we can directly delegate the misaligned > exceptions in hedeleg too before running VCPU and retrieve them after > VCPU exists. And then the handling procedure will be: > > VS-mode misaligned exception -> trap to VS-mode -> VS handles it -> > Back to VU-mode. I'd better want to let the HS-mode handle the misaligned accesses if not requested via the KVM SBI FWFT interface by VS-mode to keep HS-mode expected behavior. As you pointed out, this is not currently the case and the misaligned exceptions are directly redirected to VS-mode, this differs from what was actually done previously without FWFT (ie OpenSBI handles the misaligned access). To summarize, I think HS-mode should fixup VS-mode misaligned accesses unless requested via KVM SBI FWFT interface, in which case it will delegates them (which is done by the FWFT series). This would match the HS-mode <-> OpenSBI behavior. Thanks, Clément > > Please correct me if I missed anything. > > Best Regards, > > Xu Lu > >> >> Note: this is not directly related to your series but my introduction of >> FWFT ! >> >> Thanks, >> >> Clément >> >>> >>> Best Regards, >>> Xu Lu >>> >>>> >>>> Thanks, >>>> >>>> Clément >>>> >>>>> >>>>> Thanks. >>>>> >>>>> _______________________________________________ >>>>> linux-riscv mailing list >>>>> linux-riscv@lists.infradead.org >>>>> http://lists.infradead.org/mailman/listinfo/linux-riscv >>>> >>
On Mon, Jun 23, 2025 at 9:42 PM Clément Léger <cleger@rivosinc.com> wrote: > > > > On 23/06/2025 15:30, Xu Lu wrote: > > Hi Clément, > > > > On Mon, Jun 23, 2025 at 8:35 PM Clément Léger <cleger@rivosinc.com> wrote: > >> > >> > >> > >> On 23/06/2025 14:12, Xu Lu wrote: > >>> Hi Clément, > >>> > >>> On Mon, Jun 23, 2025 at 4:05 PM Clément Léger <cleger@rivosinc.com> wrote: > >>>> > >>>> > >>>> > >>>> On 20/06/2025 14:04, Radim Krčmář wrote: > >>>>> 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>: > >>>>>> Delegate illegal instruction fault to VS mode in default to avoid such > >>>>>> exceptions being trapped to HS and redirected back to VS. > >>>>>> > >>>>>> Signed-off-by: Xu Lu <luxu.kernel@bytedance.com> > >>>>>> --- > >>>>>> diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h > >>>>>> @@ -48,6 +48,7 @@ > >>>>>> + BIT(EXC_INST_ILLEGAL) | \ > >>>>> > >>>>> You should also remove the dead code in kvm_riscv_vcpu_exit. > >>>>> > >>>>> And why not delegate the others as well? > >>>>> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS, > >>>>> EXC_STORE_ACCESS, and EXC_INST_ACCESS.) > >>>> > >>>> Currently, OpenSBI does not delegate misaligned exception by default and > >>>> handles misaligned access by itself, this is (partially) why we added > >>>> the FWFT SBI extension to request such delegation. Since some supervisor > >>>> software expect that default, they do not have code to handle misaligned > >>>> accesses emulation. So they should not be delegated by default. > >>> > >>> It doesn't matter whether these exceptions are delegated in medeleg. > >> > >> Not sure to totally understand, but if the exceptions are not delegated > >> in medeleg, they won't be delegated to VS-mode even though hedeleg bit > >> is set right ? The spec says: > >> > >> A synchronous trap that has been delegated to HS-mode (using medeleg) > >> is further delegated to VS-mode if V=1 before the trap and the > >> corresponding hedeleg bit is set. > > > > Yes, you are right. The illegal insn exception is still trapped in M > > mode if it is not delegated in medeleg. But delegating it in hedeleg > > is still useful. The opensbi will check CSR_HEDELEG in the function > > sbi_trap_redirect. If the exception has been delegated to VS-mode in > > CSR_HEDLEG, opensbi can directly return back to VS-mode, without the > > overhead of going back to HS-mode and then going back to VS-mode. > > > >> > >> > >> > >>> KVM in HS-mode does not handle illegal instruction or misaligned > >>> access and only redirects them back to VS-mode. Delegating such > >>> exceptions in hedeleg helps save CPU usage even when they are not > >>> delegated in medeleg: opensbi will check whether these exceptions are > >>> delegated to VS-mode and redirect them to VS-mode if possible. There > >>> seems to be no conflicts with SSE implementation. Please correct me if > >>> I missed anything. > >> > >> AFAIU, this means that since medeleg bit for misaligned accesses were > >> not delegated up to the introduction of the FWFT extension, VS-mode > >> generated misaligned accesses were handled by OpenSBI right ? Now that > >> we are requesting openSBI to delegate misaligned accesses, HS-mode > >> handles it's own misaligned accesses through the trap handler. With that > >> configuration, if VS-mode generate a misaligned access, it will end up > >> being redirected to VS-mode and won't be handle by HS-mode. > >> > >> To summarize, prior to FWFT, medeleg wasn't delegating misaligned > >> accesses to S-mode: > >> > >> - VS-mode misaligned access -> trap to M-mode -> OpenSBI handle it -> > >> Back to VS-mode, misaligned access fixed up by OpenSBI > > > > Yes, this is what I want the procedure of handling illegal insn > > exceptions to be. Actually it now behaves as: > > > > VS-mode illegal insn exception -> trap to M-mode -> OpenSBI handles it > > -> Back to HS-mode, does nothing -> Back to VS-mode. > > > > I want to avoid going through HS-mode. > > Hi Xu, > > Yeah, that make sense as well but that should only happen if the VS-mode > requested misaligned access delegation via KVM SBI FWFT interface. I > know that currently KVM does do anything useful from the misaligned > exception except redirecting it to VS-mode but IMHO, that's a regression > I introduced with FWFT misaligned requested delegation... > > > > >> > >> Now that Linux uses SBI FWFT to delegates misaligned accesses (without > >> hedeleg being set for misaligned delegation, but that doesn't really > >> matter, the outcome is the same): > >> > >> - VS-mode misaligned access -> trap to HS-mode -> redirection to > >> VS-mode, needs to handle the misaligned access by itself > >> > >> > >> This means that previously, misaligned access were silently fixed up by > >> OpenSBI for VS-mode and now that FWFT is used for delegation, this isn't > >> true anymore. So, old kernel or sueprvisor software that included code > >> to handle misaligned accesses will crash. Did I missed something ? > > > > Great! You make it very clear! Thanks for your explanation. But even > > when misalign exceptions are delegated to HS-mode, KVM seems to do > > nothing but redirect to VS-mode when VM get trapped due to misalign > > exceptions. > > Exactly, which is why I said that either setting hedeleg by default or > not will lead to the same outcome, ie: VS-mode needs to handle access by > itself (which is a regression introduced by FWFT usage). > > > > So maybe we can directly delegate the misaligned > > exceptions in hedeleg too before running VCPU and retrieve them after > > VCPU exists. And then the handling procedure will be: > > > > VS-mode misaligned exception -> trap to VS-mode -> VS handles it -> > > Back to VU-mode. > > I'd better want to let the HS-mode handle the misaligned accesses if not > requested via the KVM SBI FWFT interface by VS-mode to keep HS-mode > expected behavior. As you pointed out, this is not currently the case > and the misaligned exceptions are directly redirected to VS-mode, this > differs from what was actually done previously without FWFT (ie OpenSBI > handles the misaligned access). > > To summarize, I think HS-mode should fixup VS-mode misaligned accesses > unless requested via KVM SBI FWFT interface, in which case it will > delegates them (which is done by the FWFT series). This would match the > HS-mode <-> OpenSBI behavior. Great! Roger that. Hope it can be fixed in the future. > > Thanks, > > Clément > > > > > Please correct me if I missed anything. > > > > Best Regards, > > > > Xu Lu > > > >> > >> Note: this is not directly related to your series but my introduction of > >> FWFT ! > >> > >> Thanks, > >> > >> Clément > >> > >>> > >>> Best Regards, > >>> Xu Lu > >>> > >>>> > >>>> Thanks, > >>>> > >>>> Clément > >>>> > >>>>> > >>>>> Thanks. > >>>>> > >>>>> _______________________________________________ > >>>>> linux-riscv mailing list > >>>>> linux-riscv@lists.infradead.org > >>>>> http://lists.infradead.org/mailman/listinfo/linux-riscv > >>>> > >> >
On 23/06/2025 16:09, Xu Lu wrote: > On Mon, Jun 23, 2025 at 9:42 PM Clément Léger <cleger@rivosinc.com> wrote: >> >> >> >> On 23/06/2025 15:30, Xu Lu wrote: >>> Hi Clément, >>> >>> On Mon, Jun 23, 2025 at 8:35 PM Clément Léger <cleger@rivosinc.com> wrote: >>>> >>>> >>>> >>>> On 23/06/2025 14:12, Xu Lu wrote: >>>>> Hi Clément, >>>>> >>>>> On Mon, Jun 23, 2025 at 4:05 PM Clément Léger <cleger@rivosinc.com> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 20/06/2025 14:04, Radim Krčmář wrote: >>>>>>> 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>: >>>>>>>> Delegate illegal instruction fault to VS mode in default to avoid such >>>>>>>> exceptions being trapped to HS and redirected back to VS. >>>>>>>> >>>>>>>> Signed-off-by: Xu Lu <luxu.kernel@bytedance.com> >>>>>>>> --- >>>>>>>> diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h >>>>>>>> @@ -48,6 +48,7 @@ >>>>>>>> + BIT(EXC_INST_ILLEGAL) | \ >>>>>>> >>>>>>> You should also remove the dead code in kvm_riscv_vcpu_exit. >>>>>>> >>>>>>> And why not delegate the others as well? >>>>>>> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS, >>>>>>> EXC_STORE_ACCESS, and EXC_INST_ACCESS.) >>>>>> >>>>>> Currently, OpenSBI does not delegate misaligned exception by default and >>>>>> handles misaligned access by itself, this is (partially) why we added >>>>>> the FWFT SBI extension to request such delegation. Since some supervisor >>>>>> software expect that default, they do not have code to handle misaligned >>>>>> accesses emulation. So they should not be delegated by default. >>>>> >>>>> It doesn't matter whether these exceptions are delegated in medeleg. >>>> >>>> Not sure to totally understand, but if the exceptions are not delegated >>>> in medeleg, they won't be delegated to VS-mode even though hedeleg bit >>>> is set right ? The spec says: >>>> >>>> A synchronous trap that has been delegated to HS-mode (using medeleg) >>>> is further delegated to VS-mode if V=1 before the trap and the >>>> corresponding hedeleg bit is set. >>> >>> Yes, you are right. The illegal insn exception is still trapped in M >>> mode if it is not delegated in medeleg. But delegating it in hedeleg >>> is still useful. The opensbi will check CSR_HEDELEG in the function >>> sbi_trap_redirect. If the exception has been delegated to VS-mode in >>> CSR_HEDLEG, opensbi can directly return back to VS-mode, without the >>> overhead of going back to HS-mode and then going back to VS-mode. >>> >>>> >>>> >>>> >>>>> KVM in HS-mode does not handle illegal instruction or misaligned >>>>> access and only redirects them back to VS-mode. Delegating such >>>>> exceptions in hedeleg helps save CPU usage even when they are not >>>>> delegated in medeleg: opensbi will check whether these exceptions are >>>>> delegated to VS-mode and redirect them to VS-mode if possible. There >>>>> seems to be no conflicts with SSE implementation. Please correct me if >>>>> I missed anything. >>>> >>>> AFAIU, this means that since medeleg bit for misaligned accesses were >>>> not delegated up to the introduction of the FWFT extension, VS-mode >>>> generated misaligned accesses were handled by OpenSBI right ? Now that >>>> we are requesting openSBI to delegate misaligned accesses, HS-mode >>>> handles it's own misaligned accesses through the trap handler. With that >>>> configuration, if VS-mode generate a misaligned access, it will end up >>>> being redirected to VS-mode and won't be handle by HS-mode. >>>> >>>> To summarize, prior to FWFT, medeleg wasn't delegating misaligned >>>> accesses to S-mode: >>>> >>>> - VS-mode misaligned access -> trap to M-mode -> OpenSBI handle it -> >>>> Back to VS-mode, misaligned access fixed up by OpenSBI >>> >>> Yes, this is what I want the procedure of handling illegal insn >>> exceptions to be. Actually it now behaves as: >>> >>> VS-mode illegal insn exception -> trap to M-mode -> OpenSBI handles it >>> -> Back to HS-mode, does nothing -> Back to VS-mode. >>> >>> I want to avoid going through HS-mode. >> >> Hi Xu, >> >> Yeah, that make sense as well but that should only happen if the VS-mode >> requested misaligned access delegation via KVM SBI FWFT interface. I >> know that currently KVM does do anything useful from the misaligned >> exception except redirecting it to VS-mode but IMHO, that's a regression >> I introduced with FWFT misaligned requested delegation... >> >>> >>>> >>>> Now that Linux uses SBI FWFT to delegates misaligned accesses (without >>>> hedeleg being set for misaligned delegation, but that doesn't really >>>> matter, the outcome is the same): >>>> >>>> - VS-mode misaligned access -> trap to HS-mode -> redirection to >>>> VS-mode, needs to handle the misaligned access by itself >>>> >>>> >>>> This means that previously, misaligned access were silently fixed up by >>>> OpenSBI for VS-mode and now that FWFT is used for delegation, this isn't >>>> true anymore. So, old kernel or sueprvisor software that included code >>>> to handle misaligned accesses will crash. Did I missed something ? >>> >>> Great! You make it very clear! Thanks for your explanation. But even >>> when misalign exceptions are delegated to HS-mode, KVM seems to do >>> nothing but redirect to VS-mode when VM get trapped due to misalign >>> exceptions. >> >> Exactly, which is why I said that either setting hedeleg by default or >> not will lead to the same outcome, ie: VS-mode needs to handle access by >> itself (which is a regression introduced by FWFT usage). >> >> >>> So maybe we can directly delegate the misaligned >>> exceptions in hedeleg too before running VCPU and retrieve them after >>> VCPU exists. And then the handling procedure will be: >>> >>> VS-mode misaligned exception -> trap to VS-mode -> VS handles it -> >>> Back to VU-mode. >> >> I'd better want to let the HS-mode handle the misaligned accesses if not >> requested via the KVM SBI FWFT interface by VS-mode to keep HS-mode >> expected behavior. As you pointed out, this is not currently the case >> and the misaligned exceptions are directly redirected to VS-mode, this >> differs from what was actually done previously without FWFT (ie OpenSBI >> handles the misaligned access). >> >> To summarize, I think HS-mode should fixup VS-mode misaligned accesses >> unless requested via KVM SBI FWFT interface, in which case it will >> delegates them (which is done by the FWFT series). This would match the >> HS-mode <-> OpenSBI behavior. > > Great! Roger that. Hope it can be fixed in the future. Ahah ! I'll take a look at it if everyone agrees that it's a regression. Thanks, Clément > >> >> Thanks, >> >> Clément >> >>> >>> Please correct me if I missed anything. >>> >>> Best Regards, >>> >>> Xu Lu >>> >>>> >>>> Note: this is not directly related to your series but my introduction of >>>> FWFT ! >>>> >>>> Thanks, >>>> >>>> Clément >>>> >>>>> >>>>> Best Regards, >>>>> Xu Lu >>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Clément >>>>>> >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> _______________________________________________ >>>>>>> linux-riscv mailing list >>>>>>> linux-riscv@lists.infradead.org >>>>>>> http://lists.infradead.org/mailman/listinfo/linux-riscv >>>>>> >>>> >>
2025-06-23T10:04:45+02:00, Clément Léger <cleger@rivosinc.com>: > On 20/06/2025 14:04, Radim Krčmář wrote: >> And why not delegate the others as well? >> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS, >> EXC_STORE_ACCESS, and EXC_INST_ACCESS.) > > Currently, OpenSBI does not delegate misaligned exception by default and > handles misaligned access by itself, this is (partially) why we added > the FWFT SBI extension to request such delegation. Since some supervisor > software expect that default, they do not have code to handle misaligned > accesses emulation. So they should not be delegated by default. Yeah, I forgot about your patches that conflict with the change, thanks. (The current KVM exception handler only forwards all the listed exceptions, so the only observable difference this change would make is that the KVM SBI PMU event is not counted.)
Hi Radim, On Fri, Jun 20, 2025 at 8:04 PM Radim Krčmář <rkrcmar@ventanamicro.com> wrote: > > 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>: > > Delegate illegal instruction fault to VS mode in default to avoid such > > exceptions being trapped to HS and redirected back to VS. > > > > Signed-off-by: Xu Lu <luxu.kernel@bytedance.com> > > --- > > diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h > > @@ -48,6 +48,7 @@ > > + BIT(EXC_INST_ILLEGAL) | \ > > You should also remove the dead code in kvm_riscv_vcpu_exit. I only want to delegate it by default. And KVM may still want to delegate different exceptions for different VMs like what it does for EXC_BREAKPOINT. So maybe it is better to reserve these codes? > > And why not delegate the others as well? > (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS, > EXC_STORE_ACCESS, and EXC_INST_ACCESS.) Thanks for the reminder. I will have a test and resend the patch if it works. > > Thanks. Best Regards, Xu Lu
2025-06-22T18:11:49+08:00, Xu Lu <luxu.kernel@bytedance.com>: > Hi Radim, > > On Fri, Jun 20, 2025 at 8:04 PM Radim Krčmář <rkrcmar@ventanamicro.com> wrote: >> >> 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>: >> > Delegate illegal instruction fault to VS mode in default to avoid such >> > exceptions being trapped to HS and redirected back to VS. >> > >> > Signed-off-by: Xu Lu <luxu.kernel@bytedance.com> >> > --- >> > diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h >> > @@ -48,6 +48,7 @@ >> > + BIT(EXC_INST_ILLEGAL) | \ >> >> You should also remove the dead code in kvm_riscv_vcpu_exit. > > I only want to delegate it by default. And KVM may still want to > delegate different exceptions for different VMs like what it does for > EXC_BREAKPOINT. (I think we could easily reintroduce the code if KVM wants to do that in the future. I also think that it's bad that this patch is doing an observable change without userspace involvement -- the counting of KVM SBI PMU events, but others will probably disagree with me on this.) > So maybe it is better to reserve these codes? Possibly, the current is acceptable if you have considered the implications on PMU events. >> And why not delegate the others as well? >> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS, >> EXC_STORE_ACCESS, and EXC_INST_ACCESS.) > > Thanks for the reminder. I will have a test and resend the patch if it works. The misaligned exceptions are already being worked on, so don't waste your time on them, sorry.
Hi Radim, On Mon, Jun 23, 2025 at 6:04 PM Radim Krčmář <rkrcmar@ventanamicro.com> wrote: > > 2025-06-22T18:11:49+08:00, Xu Lu <luxu.kernel@bytedance.com>: > > Hi Radim, > > > > On Fri, Jun 20, 2025 at 8:04 PM Radim Krčmář <rkrcmar@ventanamicro.com> wrote: > >> > >> 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>: > >> > Delegate illegal instruction fault to VS mode in default to avoid such > >> > exceptions being trapped to HS and redirected back to VS. > >> > > >> > Signed-off-by: Xu Lu <luxu.kernel@bytedance.com> > >> > --- > >> > diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h > >> > @@ -48,6 +48,7 @@ > >> > + BIT(EXC_INST_ILLEGAL) | \ > >> > >> You should also remove the dead code in kvm_riscv_vcpu_exit. > > > > I only want to delegate it by default. And KVM may still want to > > delegate different exceptions for different VMs like what it does for > > EXC_BREAKPOINT. > > (I think we could easily reintroduce the code if KVM wants to do that in > the future. I also think that it's bad that this patch is doing an > observable change without userspace involvement -- the counting of KVM > SBI PMU events, but others will probably disagree with me on this.) > > > So maybe it is better to reserve these codes? > > Possibly, the current is acceptable if you have considered the > implications on PMU events. So maybe it comes back to our discussion on the difference between vs insn fault and illegal insn fault again~ In my personal opinion, it seems to be a waste of CPU resources to trap illegal instruction to HS-mode hypervisor, which does nothing but redirect it back to VS-mode guest kernel. I think it is OK (and maybe it should) to record 0 illegal instruction exits in KVM PMU. If someone wants illegal insn to trigger an vcpu exit, then an ioctl can be provided to remove the delegation like what KVM_SET_GUEST_DEBUG does. > > >> And why not delegate the others as well? > >> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS, > >> EXC_STORE_ACCESS, and EXC_INST_ACCESS.) > > > > Thanks for the reminder. I will have a test and resend the patch if it works. > > The misaligned exceptions are already being worked on, so don't waste > your time on them, sorry. Thanks for the reminder too. I did not consider this before. I will leave the MISALIGNED faults alone. Best Regards, Xu Lu
© 2016 - 2025 Red Hat, Inc.