Upgrade the SBI version to v3.0 so that corresponding features
can be enabled in the guest.

Signed-off-by: Atish Patra <atishp@rivosinc.com>
---
 arch/riscv/include/asm/kvm_vcpu_sbi.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/include/asm/kvm_vcpu_sbi.h
index 4ed6203cdd30..194299e0ab0e 100644
--- a/arch/riscv/include/asm/kvm_vcpu_sbi.h
+++ b/arch/riscv/include/asm/kvm_vcpu_sbi.h
@@ -11,7 +11,7 @@
 
 #define KVM_SBI_IMPID 3
 
-#define KVM_SBI_VERSION_MAJOR 2
+#define KVM_SBI_VERSION_MAJOR 3
 #define KVM_SBI_VERSION_MINOR 0
 
 enum kvm_riscv_sbi_ext_status {

-- 
2.43.0

On Fri, May 23, 2025 at 12:33 AM Atish Patra <atishp@rivosinc.com> wrote:
>
> Upgrade the SBI version to v3.0 so that corresponding features
> can be enabled in the guest.
>
> Signed-off-by: Atish Patra <atishp@rivosinc.com>

Extending the ONE_REG interface to allow KVM user-space select
SBI version can be done as a separate series.

For this series, we can go ahead with this patch.

Reviewed-by: Anup Patel <anup@brainfault.org>

Regards,
Anup

> ---
>  arch/riscv/include/asm/kvm_vcpu_sbi.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/include/asm/kvm_vcpu_sbi.h
> index 4ed6203cdd30..194299e0ab0e 100644
> --- a/arch/riscv/include/asm/kvm_vcpu_sbi.h
> +++ b/arch/riscv/include/asm/kvm_vcpu_sbi.h
> @@ -11,7 +11,7 @@
>
>  #define KVM_SBI_IMPID 3
>
> -#define KVM_SBI_VERSION_MAJOR 2
> +#define KVM_SBI_VERSION_MAJOR 3
>  #define KVM_SBI_VERSION_MINOR 0
>
>  enum kvm_riscv_sbi_ext_status {
>
> --
> 2.43.0
>

2025-05-22T12:03:43-07:00, Atish Patra <atishp@rivosinc.com>:
> Upgrade the SBI version to v3.0 so that corresponding features
> can be enabled in the guest.
>
> Signed-off-by: Atish Patra <atishp@rivosinc.com>
> ---
> diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/include/asm/kvm_vcpu_sbi.h
> -#define KVM_SBI_VERSION_MAJOR 2
> +#define KVM_SBI_VERSION_MAJOR 3

I think it's time to add versioning to KVM SBI implementation.
Userspace should be able to select the desired SBI version and KVM would
tell the guest that newer features are not supported.

We could somewhat get away with the userspace_sbi patch I posted,
because userspace would at least be in control of the SBI version, but
it would still be incorrect without a KVM enforcement, because a
misbehaving guest could use features that should not be supported.

On 5/23/25 6:31 AM, Radim Krčmář wrote:
> 2025-05-22T12:03:43-07:00, Atish Patra <atishp@rivosinc.com>:
>> Upgrade the SBI version to v3.0 so that corresponding features
>> can be enabled in the guest.
>>
>> Signed-off-by: Atish Patra <atishp@rivosinc.com>
>> ---
>> diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/include/asm/kvm_vcpu_sbi.h
>> -#define KVM_SBI_VERSION_MAJOR 2
>> +#define KVM_SBI_VERSION_MAJOR 3
> I think it's time to add versioning to KVM SBI implementation.
> Userspace should be able to select the desired SBI version and KVM would
> tell the guest that newer features are not supported.

We can achieve that through onereg interface by disabling individual SBI 
extensions.
We can extend the existing onereg interface to disable a specific SBI 
version directly
instead of individual ones to save those IOCTL as well.

> We could somewhat get away with the userspace_sbi patch I posted,
> because userspace would at least be in control of the SBI version, but
> it would still be incorrect without a KVM enforcement, because a
> misbehaving guest could use features that should not be supported.

2025-05-23T10:16:11-07:00, Atish Patra <atish.patra@linux.dev>:
> On 5/23/25 6:31 AM, Radim Krčmář wrote:
>> 2025-05-22T12:03:43-07:00, Atish Patra <atishp@rivosinc.com>:
>>> Upgrade the SBI version to v3.0 so that corresponding features
>>> can be enabled in the guest.
>>>
>>> Signed-off-by: Atish Patra <atishp@rivosinc.com>
>>> ---
>>> diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/include/asm/kvm_vcpu_sbi.h
>>> -#define KVM_SBI_VERSION_MAJOR 2
>>> +#define KVM_SBI_VERSION_MAJOR 3
>> I think it's time to add versioning to KVM SBI implementation.
>> Userspace should be able to select the desired SBI version and KVM would
>> tell the guest that newer features are not supported.
>
> We can achieve that through onereg interface by disabling individual SBI 
> extensions.
> We can extend the existing onereg interface to disable a specific SBI 
> version directly
> instead of individual ones to save those IOCTL as well.

Yes, I am all in favor of letting userspace provide all values in the
BASE extension.

On Mon, May 26, 2025 at 11:00:30AM +0200, Radim Krčmář wrote:
> 2025-05-23T10:16:11-07:00, Atish Patra <atish.patra@linux.dev>:
> > On 5/23/25 6:31 AM, Radim Krčmář wrote:
> >> 2025-05-22T12:03:43-07:00, Atish Patra <atishp@rivosinc.com>:
> >>> Upgrade the SBI version to v3.0 so that corresponding features
> >>> can be enabled in the guest.
> >>>
> >>> Signed-off-by: Atish Patra <atishp@rivosinc.com>
> >>> ---
> >>> diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/include/asm/kvm_vcpu_sbi.h
> >>> -#define KVM_SBI_VERSION_MAJOR 2
> >>> +#define KVM_SBI_VERSION_MAJOR 3
> >> I think it's time to add versioning to KVM SBI implementation.
> >> Userspace should be able to select the desired SBI version and KVM would
> >> tell the guest that newer features are not supported.

We need new code for this, but it's a good idea.

> >
> > We can achieve that through onereg interface by disabling individual SBI 
> > extensions.
> > We can extend the existing onereg interface to disable a specific SBI 
> > version directly
> > instead of individual ones to save those IOCTL as well.
> 
> Yes, I am all in favor of letting userspace provide all values in the
> BASE extension.

This is covered by your recent patch that provides userspace_sbi.
With that, userspace can disable all extensions that aren't
supported by a given spec version, disable BASE and then provide
a BASE that advertises the version it wants. The new code is needed
for extensions that userspace still wants KVM to accelerate, but then
KVM needs to be informed it should deny all functions not included in
the selected spec version.

Thanks,
drew

On 5/26/25 4:13 AM, Andrew Jones wrote:
> On Mon, May 26, 2025 at 11:00:30AM +0200, Radim Krčmář wrote:
>> 2025-05-23T10:16:11-07:00, Atish Patra <atish.patra@linux.dev>:
>>> On 5/23/25 6:31 AM, Radim Krčmář wrote:
>>>> 2025-05-22T12:03:43-07:00, Atish Patra <atishp@rivosinc.com>:
>>>>> Upgrade the SBI version to v3.0 so that corresponding features
>>>>> can be enabled in the guest.
>>>>>
>>>>> Signed-off-by: Atish Patra <atishp@rivosinc.com>
>>>>> ---
>>>>> diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/include/asm/kvm_vcpu_sbi.h
>>>>> -#define KVM_SBI_VERSION_MAJOR 2
>>>>> +#define KVM_SBI_VERSION_MAJOR 3
>>>> I think it's time to add versioning to KVM SBI implementation.
>>>> Userspace should be able to select the desired SBI version and KVM would
>>>> tell the guest that newer features are not supported.
> 
> We need new code for this, but it's a good idea.
> 
>>>
>>> We can achieve that through onereg interface by disabling individual SBI
>>> extensions.
>>> We can extend the existing onereg interface to disable a specific SBI
>>> version directly
>>> instead of individual ones to save those IOCTL as well.
>>
>> Yes, I am all in favor of letting userspace provide all values in the
>> BASE extension.
> 

We already support vendorid/archid/impid through one reg. I think we 
just need to add the SBI version support to that so that user space can 
set it.

> This is covered by your recent patch that provides userspace_sbi.

Why do we need to invent new IOCTL for this ? Once the user space sets 
the SBI version, KVM can enforce it.

> With that, userspace can disable all extensions that aren't
> supported by a given spec version, disable BASE and then provide
> a BASE that advertises the version it wants. The new code is needed
> for extensions that userspace still wants KVM to accelerate, but then
> KVM needs to be informed it should deny all functions not included in
> the selected spec version.
> 
> Thanks,
> drew
> 
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv

On Wed, May 28, 2025 at 07:16:11AM -0700, Atish Patra wrote:
> On 5/26/25 4:13 AM, Andrew Jones wrote:
> > On Mon, May 26, 2025 at 11:00:30AM +0200, Radim Krčmář wrote:
> > > 2025-05-23T10:16:11-07:00, Atish Patra <atish.patra@linux.dev>:
> > > > On 5/23/25 6:31 AM, Radim Krčmář wrote:
> > > > > 2025-05-22T12:03:43-07:00, Atish Patra <atishp@rivosinc.com>:
> > > > > > Upgrade the SBI version to v3.0 so that corresponding features
> > > > > > can be enabled in the guest.
> > > > > > 
> > > > > > Signed-off-by: Atish Patra <atishp@rivosinc.com>
> > > > > > ---
> > > > > > diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/include/asm/kvm_vcpu_sbi.h
> > > > > > -#define KVM_SBI_VERSION_MAJOR 2
> > > > > > +#define KVM_SBI_VERSION_MAJOR 3
> > > > > I think it's time to add versioning to KVM SBI implementation.
> > > > > Userspace should be able to select the desired SBI version and KVM would
> > > > > tell the guest that newer features are not supported.
> > 
> > We need new code for this, but it's a good idea.
> > 
> > > > 
> > > > We can achieve that through onereg interface by disabling individual SBI
> > > > extensions.
> > > > We can extend the existing onereg interface to disable a specific SBI
> > > > version directly
> > > > instead of individual ones to save those IOCTL as well.
> > > 
> > > Yes, I am all in favor of letting userspace provide all values in the
> > > BASE extension.
> > 
> 
> We already support vendorid/archid/impid through one reg. I think we just
> need to add the SBI version support to that so that user space can set it.
> 
> > This is covered by your recent patch that provides userspace_sbi.
> 
> Why do we need to invent new IOCTL for this ? Once the user space sets the
> SBI version, KVM can enforce it.

If an SBI spec version provides an extension that can be emulated by
userspace, then userspace could choose to advertise that spec version,
implement a BASE probe function that advertises the extension, and
implement the extension, even if the KVM version running is older
and unaware of it. But, in order to do that, we need KVM to exit to
userspace for all unknown SBI calls and to allow BASE to be overridden
by userspace. The new KVM CAP ioctl allows opting into that new behavior.

The old KVM with new VMM configuration isn't totally far-fetched. While
host kernels tend to get updated regularly to include security fixes,
enterprise kernels tend to stop adding features at some point in order
to maximize stability. While enterprise VMMs would also eventually stop
adding features, enterprise consumers are always free to use their own
VMMs (at their own risk). So, there's a real chance we could have
deployments with older, stable KVM where users want to enable later SBI
extensions, and, in some cases, that should be possible by just updating
the VMM -- but only if KVM is only acting as an SBI implementation
accelerator and not as a userspace SBI implementation gatekeeper.

Thanks,
drew

> 
> > With that, userspace can disable all extensions that aren't
> > supported by a given spec version, disable BASE and then provide
> > a BASE that advertises the version it wants. The new code is needed
> > for extensions that userspace still wants KVM to accelerate, but then
> > KVM needs to be informed it should deny all functions not included in
> > the selected spec version.
> > 
> > Thanks,
> > drew
> > 
> > _______________________________________________
> > linux-riscv mailing list
> > linux-riscv@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-riscv
> 

<Removing Palmer's rivos email address to avoid bouncing>

On 5/28/25 8:09 AM, Andrew Jones wrote:
> On Wed, May 28, 2025 at 07:16:11AM -0700, Atish Patra wrote:
>> On 5/26/25 4:13 AM, Andrew Jones wrote:
>>> On Mon, May 26, 2025 at 11:00:30AM +0200, Radim Krčmář wrote:
>>>> 2025-05-23T10:16:11-07:00, Atish Patra <atish.patra@linux.dev>:
>>>>> On 5/23/25 6:31 AM, Radim Krčmář wrote:
>>>>>> 2025-05-22T12:03:43-07:00, Atish Patra <atishp@rivosinc.com>:
>>>>>>> Upgrade the SBI version to v3.0 so that corresponding features
>>>>>>> can be enabled in the guest.
>>>>>>>
>>>>>>> Signed-off-by: Atish Patra <atishp@rivosinc.com>
>>>>>>> ---
>>>>>>> diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/include/asm/kvm_vcpu_sbi.h
>>>>>>> -#define KVM_SBI_VERSION_MAJOR 2
>>>>>>> +#define KVM_SBI_VERSION_MAJOR 3
>>>>>> I think it's time to add versioning to KVM SBI implementation.
>>>>>> Userspace should be able to select the desired SBI version and KVM would
>>>>>> tell the guest that newer features are not supported.
>>> We need new code for this, but it's a good idea.
>>>
>>>>> We can achieve that through onereg interface by disabling individual SBI
>>>>> extensions.
>>>>> We can extend the existing onereg interface to disable a specific SBI
>>>>> version directly
>>>>> instead of individual ones to save those IOCTL as well.
>>>> Yes, I am all in favor of letting userspace provide all values in the
>>>> BASE extension.
>> We already support vendorid/archid/impid through one reg. I think we just
>> need to add the SBI version support to that so that user space can set it.
>>
>>> This is covered by your recent patch that provides userspace_sbi.
>> Why do we need to invent new IOCTL for this ? Once the user space sets the
>> SBI version, KVM can enforce it.
> If an SBI spec version provides an extension that can be emulated by
> userspace, then userspace could choose to advertise that spec version,
> implement a BASE probe function that advertises the extension, and
> implement the extension, even if the KVM version running is older
> and unaware of it. But, in order to do that, we need KVM to exit to
> userspace for all unknown SBI calls and to allow BASE to be overridden
You mean only the version field in BASE - Correct ?

We already support vendorid/archid/impid through one reg. I don't see the
point of overriding SBI implementation ID & version.

> by userspace. The new KVM CAP ioctl allows opting into that new behavior.

But why we need a new IOCTL for that ? We can achieve that with existing
one reg interface with improvements.

> The old KVM with new VMM configuration isn't totally far-fetched. While
> host kernels tend to get updated regularly to include security fixes,
> enterprise kernels tend to stop adding features at some point in order
> to maximize stability. While enterprise VMMs would also eventually stop
> adding features, enterprise consumers are always free to use their own
> VMMs (at their own risk). So, there's a real chance we could have

I think we are years away from that happening (if it happens). My 
suggestion was not to
try to build a world where no body lives ;). When we get to that 
scenario, the default KVM
shipped will have many extension implemented. So there won't be much 
advantage to
reimplement them in the user space. We can also take an informed 
decision at that time
if the current selective forwarding approach is better or we need to 
blindly forward any
unknown SBI calls to the user space.

> deployments with older, stable KVM where users want to enable later SBI
> extensions, and, in some cases, that should be possible by just updating
> the VMM -- but only if KVM is only acting as an SBI implementation
> accelerator and not as a userspace SBI implementation gatekeeper.

But some of the SBI extensions are so fundamental that it must be 
implemented in KVM
for various reasons pointed by Anup on other thread.

> Thanks,
> drew
>
>>> With that, userspace can disable all extensions that aren't
>>> supported by a given spec version, disable BASE and then provide
>>> a BASE that advertises the version it wants. The new code is needed
>>> for extensions that userspace still wants KVM to accelerate, but then
>>> KVM needs to be informed it should deny all functions not included in
>>> the selected spec version.
>>>
>>> Thanks,
>>> drew
>>>
>>> _______________________________________________
>>> linux-riscv mailing list
>>> linux-riscv@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-riscv

I originally gave up on the idea, but I feel kinda bad for Drew now, so
trying again:

2025-05-28T12:21:59-07:00, Atish Patra <atish.patra@linux.dev>:
> On 5/28/25 8:09 AM, Andrew Jones wrote:
>> On Wed, May 28, 2025 at 07:16:11AM -0700, Atish Patra wrote:
>>> On 5/26/25 4:13 AM, Andrew Jones wrote:
>>>> On Mon, May 26, 2025 at 11:00:30AM +0200, Radim Krčmář wrote:
>>>>> 2025-05-23T10:16:11-07:00, Atish Patra <atish.patra@linux.dev>:
>>>>>> On 5/23/25 6:31 AM, Radim Krčmář wrote:
>>>>>>> 2025-05-22T12:03:43-07:00, Atish Patra <atishp@rivosinc.com>:
>>>>>>>> Upgrade the SBI version to v3.0 so that corresponding features
>>>>>>>> can be enabled in the guest.
>>>>>>>>
>>>>>>>> Signed-off-by: Atish Patra <atishp@rivosinc.com>
>>>>>>>> ---
>>>>>>>> diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/include/asm/kvm_vcpu_sbi.h
>>>>>>>> -#define KVM_SBI_VERSION_MAJOR 2
>>>>>>>> +#define KVM_SBI_VERSION_MAJOR 3
>>>>>>> I think it's time to add versioning to KVM SBI implementation.
>>>>>>> Userspace should be able to select the desired SBI version and KVM would
>>>>>>> tell the guest that newer features are not supported.
>>>> We need new code for this, but it's a good idea.
>>>>
>>>>>> We can achieve that through onereg interface by disabling individual SBI
>>>>>> extensions.
>>>>>> We can extend the existing onereg interface to disable a specific SBI
>>>>>> version directly
>>>>>> instead of individual ones to save those IOCTL as well.
>>>>> Yes, I am all in favor of letting userspace provide all values in the
>>>>> BASE extension.
>>> We already support vendorid/archid/impid through one reg. I think we just
>>> need to add the SBI version support to that so that user space can set it.
>>>
>>>> This is covered by your recent patch that provides userspace_sbi.
>>> Why do we need to invent new IOCTL for this ? Once the user space sets the
>>> SBI version, KVM can enforce it.
>> If an SBI spec version provides an extension that can be emulated by
>> userspace, then userspace could choose to advertise that spec version,
>> implement a BASE probe function that advertises the extension, and
>> implement the extension, even if the KVM version running is older
>> and unaware of it. But, in order to do that, we need KVM to exit to
>> userspace for all unknown SBI calls and to allow BASE to be overridden
> You mean only the version field in BASE - Correct ?

No, "BASE probe function" is the sbi_probe_extension() ecall.

>> by userspace. The new KVM CAP ioctl allows opting into that new behavior.
>
> But why we need a new IOCTL for that ? We can achieve that with existing
> one reg interface with improvements.

It's an existing IOCTL with a new data payload, but I can easily use
ONE_REG if you want to do everything through that.

KVM doesn't really need any other IOCTL than ONE_REGs, it's just
sometimes more reasonable to use a different IOCTL, like ENABLE_CAP.

>> The old KVM with new VMM configuration isn't totally far-fetched. While
>> host kernels tend to get updated regularly to include security fixes,
>> enterprise kernels tend to stop adding features at some point in order
>> to maximize stability. While enterprise VMMs would also eventually stop
>> adding features, enterprise consumers are always free to use their own
>> VMMs (at their own risk). So, there's a real chance we could have
>
> I think we are years away from that happening (if it happens). My 
> suggestion was not to
> try to build a world where no body lives ;). When we get to that 
> scenario, the default KVM
> shipped will have many extension implemented. So there won't be much 
> advantage to
> reimplement them in the user space. We can also take an informed 
> decision at that time
> if the current selective forwarding approach is better

Please don't repeat the design of SUSP/SRST/DBCN.
Seeing them is one of the reasons why I proposed the new interface.

"Blindly" forwarding DBCN to userspace is even a minor optimization. :)

>                                                        or we need to 
> blindly forward any
> unknown SBI calls to the user space.

Yes, KVM has to do what userpace configures it to do.

I don't think that implementing unsupported SBI extensions in KVM is
important -- they should not be a hot path.

>> deployments with older, stable KVM where users want to enable later SBI
>> extensions, and, in some cases, that should be possible by just updating
>> the VMM -- but only if KVM is only acting as an SBI implementation
>> accelerator and not as a userspace SBI implementation gatekeeper.
>
> But some of the SBI extensions are so fundamental that it must be 
> implemented in KVM
> for various reasons pointed by Anup on other thread.

No, SBI does not have to be implemented in KVM at all.

We do have a deep disagreement on what is virtualization and the role of
KVM in it.  I think that userspace wants a generic ISA accelerator.

Even if userspace wants SBI for the M-mode interface, security minded
userspace aims for as little kernel code as possible.
Userspace might want to accelerate some SBI extension in KVM, but it
should not be KVM who decides what userspace wants.

On 5/29/25 3:24 AM, Radim Krčmář wrote:
> I originally gave up on the idea, but I feel kinda bad for Drew now, so
> trying again:

I am sorry if some of my replies came across in the wrong way. That was 
never
the intention.


> 2025-05-28T12:21:59-07:00, Atish Patra <atish.patra@linux.dev>:
>> On 5/28/25 8:09 AM, Andrew Jones wrote:
>>> On Wed, May 28, 2025 at 07:16:11AM -0700, Atish Patra wrote:
>>>> On 5/26/25 4:13 AM, Andrew Jones wrote:
>>>>> On Mon, May 26, 2025 at 11:00:30AM +0200, Radim Krčmář wrote:
>>>>>> 2025-05-23T10:16:11-07:00, Atish Patra <atish.patra@linux.dev>:
>>>>>>> On 5/23/25 6:31 AM, Radim Krčmář wrote:
>>>>>>>> 2025-05-22T12:03:43-07:00, Atish Patra <atishp@rivosinc.com>:
>>>>>>>>> Upgrade the SBI version to v3.0 so that corresponding features
>>>>>>>>> can be enabled in the guest.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Atish Patra <atishp@rivosinc.com>
>>>>>>>>> ---
>>>>>>>>> diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/include/asm/kvm_vcpu_sbi.h
>>>>>>>>> -#define KVM_SBI_VERSION_MAJOR 2
>>>>>>>>> +#define KVM_SBI_VERSION_MAJOR 3
>>>>>>>> I think it's time to add versioning to KVM SBI implementation.
>>>>>>>> Userspace should be able to select the desired SBI version and KVM would
>>>>>>>> tell the guest that newer features are not supported.
>>>>> We need new code for this, but it's a good idea.
>>>>>
>>>>>>> We can achieve that through onereg interface by disabling individual SBI
>>>>>>> extensions.
>>>>>>> We can extend the existing onereg interface to disable a specific SBI
>>>>>>> version directly
>>>>>>> instead of individual ones to save those IOCTL as well.
>>>>>> Yes, I am all in favor of letting userspace provide all values in the
>>>>>> BASE extension.
>>>> We already support vendorid/archid/impid through one reg. I think we just
>>>> need to add the SBI version support to that so that user space can set it.
>>>>
>>>>> This is covered by your recent patch that provides userspace_sbi.
>>>> Why do we need to invent new IOCTL for this ? Once the user space sets the
>>>> SBI version, KVM can enforce it.
>>> If an SBI spec version provides an extension that can be emulated by
>>> userspace, then userspace could choose to advertise that spec version,
>>> implement a BASE probe function that advertises the extension, and
>>> implement the extension, even if the KVM version running is older
>>> and unaware of it. But, in order to do that, we need KVM to exit to
>>> userspace for all unknown SBI calls and to allow BASE to be overridden
>> You mean only the version field in BASE - Correct ?
> No, "BASE probe function" is the sbi_probe_extension() ecall.
>
>>> by userspace. The new KVM CAP ioctl allows opting into that new behavior.
>> But why we need a new IOCTL for that ? We can achieve that with existing
>> one reg interface with improvements.
> It's an existing IOCTL with a new data payload, but I can easily use
> ONE_REG if you want to do everything through that.
>
> KVM doesn't really need any other IOCTL than ONE_REGs, it's just
> sometimes more reasonable to use a different IOCTL, like ENABLE_CAP.
>
>>> The old KVM with new VMM configuration isn't totally far-fetched. While
>>> host kernels tend to get updated regularly to include security fixes,
>>> enterprise kernels tend to stop adding features at some point in order
>>> to maximize stability. While enterprise VMMs would also eventually stop
>>> adding features, enterprise consumers are always free to use their own
>>> VMMs (at their own risk). So, there's a real chance we could have
>> I think we are years away from that happening (if it happens). My
>> suggestion was not to
>> try to build a world where no body lives ;). When we get to that
>> scenario, the default KVM
>> shipped will have many extension implemented. So there won't be much
>> advantage to
>> reimplement them in the user space. We can also take an informed
>> decision at that time
>> if the current selective forwarding approach is better
> Please don't repeat the design of SUSP/SRST/DBCN.
> Seeing them is one of the reasons why I proposed the new interface.
>
> "Blindly" forwarding DBCN to userspace is even a minor optimization. :)
>
>>                                                         or we need to
>> blindly forward any
>> unknown SBI calls to the user space.
> Yes, KVM has to do what userpace configures it to do.
>
> I don't think that implementing unsupported SBI extensions in KVM is
> important -- they should not be a hot path.
>
>>> deployments with older, stable KVM where users want to enable later SBI
>>> extensions, and, in some cases, that should be possible by just updating
>>> the VMM -- but only if KVM is only acting as an SBI implementation
>>> accelerator and not as a userspace SBI implementation gatekeeper.
>> But some of the SBI extensions are so fundamental that it must be
>> implemented in KVM
>> for various reasons pointed by Anup on other thread.
> No, SBI does not have to be implemented in KVM at all.
>
> We do have a deep disagreement on what is virtualization and the role of
> KVM in it.  I think that userspace wants a generic ISA accelerator.

I think the disagreement is the role of SBI in KVM virtualization rather 
than
a generic virtualization and the role of KVM in it. I completely agree 
that KVM should act as an accelerator and defer the control to the user 
space in most of the cases
such e.g I/O operations or system related functionalities. However, SBI 
specification solves
much wider problems than those. Broadly we can categorize SBI 
functionalities into the following
areas

1. Bridging ISA GAP
2. Higher Privilege Assistance
3. Virtualization
4. Platform abstraction
5. Confidential computing

For #1, #3 and #5, I believe user space shouldn't be involved in 
implementation
some of them are in hot path as well. For #4 and #2, there are some 
opportunities which
can be implemented in user space depending on the exact need. I am still 
not clear what is the exact
motivation /right now/ to pursue such a path. May be I missed something.
As per my understanding from our discussion threads, there are two use 
cases possible

1. userspace wants to update more states in HSM. What are the states 
user space should care about scounteren (fixed already in usptream) ?
2. VMM vs KVM version difference - this may be true in the future 
depending on the speed of RISC-V virtualization adoption in the industry.
But we are definitely not there yet. Please let me know if I 
misunderstood any use cases.

> Even if userspace wants SBI for the M-mode interface, security minded
This is probably a 3rd one ? Why we want M-mode interface in the user 
space ?
> userspace aims for as little kernel code as possible.

We trust VMM code more than KVM code ?

> Userspace might want to accelerate some SBI extension in KVM, but it
> should not be KVM who decides what userspace wants.

2025-05-29T11:44:38-07:00, Atish Patra <atish.patra@linux.dev>:
> On 5/29/25 3:24 AM, Radim Krčmář wrote:
>> I originally gave up on the idea, but I feel kinda bad for Drew now, so
>> trying again:
>
> I am sorry if some of my replies came across in the wrong way. That was 
> never
> the intention.

I didn't mean to accuse you, my apologies.  I agree with Drew's
positions, so to expand on a question that wasn't touched in his mail:

>> Even if userspace wants SBI for the M-mode interface, security minded
> This is probably a 3rd one ? Why we want M-mode interface in the user 
> space ?

It is about turning KVM into an ISA accelerator.

A guest thinks it is running in S/HS-mode.
The ecall instruction traps to M-mode.  RISC-V H extension doesn't
accelerate M-mode, so we have to emulate the trap in software.

The ISA doesn't say that M-mode means SBI.  We try really hard to have
SBI on all RISC-V, but I think KVM is taking it a bit too far.

We can discuss how best to describe SBI, so userspace can choose to
accelerate the M-mode in KVM, but I think that the ability to emulate
M-mode in userspace should be provided.

On 5/30/25 4:09 AM, Radim Krčmář wrote:
> 2025-05-29T11:44:38-07:00, Atish Patra <atish.patra@linux.dev>:
>> On 5/29/25 3:24 AM, Radim Krčmář wrote:
>>> I originally gave up on the idea, but I feel kinda bad for Drew now, so
>>> trying again:
>> I am sorry if some of my replies came across in the wrong way. That was
>> never
>> the intention.
> I didn't mean to accuse you, my apologies.  I agree with Drew's
> positions, so to expand on a question that wasn't touched in his mail:
>
>>> Even if userspace wants SBI for the M-mode interface, security minded
>> This is probably a 3rd one ? Why we want M-mode interface in the user
>> space ?
> It is about turning KVM into an ISA accelerator.
>
> A guest thinks it is running in S/HS-mode.
> The ecall instruction traps to M-mode.  RISC-V H extension doesn't
> accelerate M-mode, so we have to emulate the trap in software.
We don't need to accelerate M-mode. That's the beauty of the RISC-V H 
extension.
The ISA is designed in such a way that the SBI is the interface between 
the supervisor environment (VS/HS)
and the supervisor execution environment (HS/M).


>
> The ISA doesn't say that M-mode means SBI.  We try really hard to have
> SBI on all RISC-V, but I think KVM is taking it a bit too far.
>
> We can discuss how best to describe SBI, so userspace can choose to
> accelerate the M-mode in KVM, but I think that the ability to emulate
> M-mode in userspace should be provided.
I am still trying to understand the advantages of emulating the M-mode 
in the user space.
Can you please elaborate ?
I am assuming you are not hinting Nested virtualization which can be 
achieved with existing
ISA provided mechanisms and accelerated by SBI NACL.

2025-05-30T12:29:30-07:00, Atish Patra <atish.patra@linux.dev>:
> On 5/30/25 4:09 AM, Radim Krčmář wrote:
>> 2025-05-29T11:44:38-07:00, Atish Patra <atish.patra@linux.dev>:
>>> On 5/29/25 3:24 AM, Radim Krčmář wrote:
>>>> I originally gave up on the idea, but I feel kinda bad for Drew now, so
>>>> trying again:
>>> I am sorry if some of my replies came across in the wrong way. That was
>>> never
>>> the intention.
>> I didn't mean to accuse you, my apologies.  I agree with Drew's
>> positions, so to expand on a question that wasn't touched in his mail:
>>
>>>> Even if userspace wants SBI for the M-mode interface, security minded
>>> This is probably a 3rd one ? Why we want M-mode interface in the user
>>> space ?
>> It is about turning KVM into an ISA accelerator.
>>
>> A guest thinks it is running in S/HS-mode.
>> The ecall instruction traps to M-mode.  RISC-V H extension doesn't
>> accelerate M-mode, so we have to emulate the trap in software.
> We don't need to accelerate M-mode. That's the beauty of the RISC-V H 
> extension.

(It is a gap to me. :])

> The ISA is designed in such a way that the SBI is the interface between 
> the supervisor environment (VS/HS)
> and the supervisor execution environment (HS/M).

The ISA says nothing about the implementation of said interface.

Returning 42 in x21 as a response to an ecall with 0x10 in a7 and 0x3 in
a6 is perfectly valid RISC-V implementation that KVM currently cannot
virtualize.

>> The ISA doesn't say that M-mode means SBI.  We try really hard to have
>> SBI on all RISC-V, but I think KVM is taking it a bit too far.
>>
>> We can discuss how best to describe SBI, so userspace can choose to
>> accelerate the M-mode in KVM, but I think that the ability to emulate
>> M-mode in userspace should be provided.
> I am still trying to understand the advantages of emulating the M-mode 
> in the user space.
> Can you please elaborate ?

This thread already has a lot of them, so to avoid repeating them, I
have to go into quite niche use-cases:
When developing M-mode software on RISC-V (when RISC-V has more useful
implementations than QEMU), a developer might want to accelerate the
S/U-modes in KVM.
It is also simpler to implement an old SBI interface (especially with
bugs/quirks) if virtualization just executes the old M-mode binary.

Why must KVM prevent userspace from virtualizing RISC-V?

> I am assuming you are not hinting Nested virtualization which can be 
> achieved with existing
> ISA provided mechanisms and accelerated by SBI NACL.

Right, I am talking about virtualization of RISC-V, because I don't have
a crystal ball to figure out what users will want.

On 6/3/25 4:40 AM, Radim Krčmář wrote:
> 2025-05-30T12:29:30-07:00, Atish Patra <atish.patra@linux.dev>:
>> On 5/30/25 4:09 AM, Radim Krčmář wrote:
>>> 2025-05-29T11:44:38-07:00, Atish Patra <atish.patra@linux.dev>:
>>>> On 5/29/25 3:24 AM, Radim Krčmář wrote:
>>>>> I originally gave up on the idea, but I feel kinda bad for Drew now, so
>>>>> trying again:
>>>> I am sorry if some of my replies came across in the wrong way. That was
>>>> never
>>>> the intention.
>>> I didn't mean to accuse you, my apologies.  I agree with Drew's
>>> positions, so to expand on a question that wasn't touched in his mail:
>>>
>>>>> Even if userspace wants SBI for the M-mode interface, security minded
>>>> This is probably a 3rd one ? Why we want M-mode interface in the user
>>>> space ?
>>> It is about turning KVM into an ISA accelerator.
>>>
>>> A guest thinks it is running in S/HS-mode.
>>> The ecall instruction traps to M-mode.  RISC-V H extension doesn't
>>> accelerate M-mode, so we have to emulate the trap in software.
>> We don't need to accelerate M-mode. That's the beauty of the RISC-V H
>> extension.
> (It is a gap to me. :])
RISC-V H extension is designed to virtualize S-mode and U-mode. Not M-mode.
I don't think retrofitting M-mode virtualization has absolutely any 
benefit. It has
many challenges that will probably result in poor performance. It can be 
a hobby project
but I am not sure if it can be adopted in production.

Are there any similar use cases in other ISAs ? Does anybody support 
virtualizaing EL3 in ARM64 ?

>> The ISA is designed in such a way that the SBI is the interface between
>> the supervisor environment (VS/HS)
>> and the supervisor execution environment (HS/M).
> The ISA says nothing about the implementation of said interface.
>
> Returning 42 in x21 as a response to an ecall with 0x10 in a7 and 0x3 in
> a6 is perfectly valid RISC-V implementation that KVM currently cannot
> virtualize.

If the concern is only supporting an older version of SBI version, we 
can support that with onereg
interface today. I think I already agreed on that earlier in this thread 
and revise this series to have
it ready for review.


>>> The ISA doesn't say that M-mode means SBI.  We try really hard to have
>>> SBI on all RISC-V, but I think KVM is taking it a bit too far.
>>>
>>> We can discuss how best to describe SBI, so userspace can choose to
>>> accelerate the M-mode in KVM, but I think that the ability to emulate
>>> M-mode in userspace should be provided.
>> I am still trying to understand the advantages of emulating the M-mode
>> in the user space.
>> Can you please elaborate ?
> This thread already has a lot of them, so to avoid repeating them, I
> have to go into quite niche use-cases:
> When developing M-mode software on RISC-V (when RISC-V has more useful
> implementations than QEMU), a developer might want to accelerate the
> S/U-modes in KVM.
> It is also simpler to implement an old SBI interface (especially with
> bugs/quirks) if virtualization just executes the old M-mode binary.
>
> Why must KVM prevent userspace from virtualizing RISC-V?

If there is a valid use case that can be put into production or
if you have any prototype that it has better performance then we can 
have it.
In absence of either, isn't it better to spend our energy on things that 
actually matter
right now and improve RISC-V virtualization performance rather than 
something that
may or may not be possible in the very far future.

>> I am assuming you are not hinting Nested virtualization which can be
>> achieved with existing
>> ISA provided mechanisms and accelerated by SBI NACL.
> Right, I am talking about virtualization of RISC-V, because I don't have
> a crystal ball to figure out what users will want.

On Thu, May 29, 2025 at 11:44:38AM -0700, Atish Patra wrote:
> 
> On 5/29/25 3:24 AM, Radim Krčmář wrote:
> > I originally gave up on the idea, but I feel kinda bad for Drew now, so
> > trying again:
> 
> I am sorry if some of my replies came across in the wrong way. That was
> never
> the intention.

Not at all. Radim only meant that I was defending his patches, even though
he wasn't :-)

> 
> 
> > 2025-05-28T12:21:59-07:00, Atish Patra <atish.patra@linux.dev>:
> > > On 5/28/25 8:09 AM, Andrew Jones wrote:
> > > > On Wed, May 28, 2025 at 07:16:11AM -0700, Atish Patra wrote:
> > > > > On 5/26/25 4:13 AM, Andrew Jones wrote:
> > > > > > On Mon, May 26, 2025 at 11:00:30AM +0200, Radim Krčmář wrote:
> > > > > > > 2025-05-23T10:16:11-07:00, Atish Patra <atish.patra@linux.dev>:
> > > > > > > > On 5/23/25 6:31 AM, Radim Krčmář wrote:
> > > > > > > > > 2025-05-22T12:03:43-07:00, Atish Patra <atishp@rivosinc.com>:
> > > > > > > > > > Upgrade the SBI version to v3.0 so that corresponding features
> > > > > > > > > > can be enabled in the guest.
> > > > > > > > > > 
> > > > > > > > > > Signed-off-by: Atish Patra <atishp@rivosinc.com>
> > > > > > > > > > ---
> > > > > > > > > > diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/include/asm/kvm_vcpu_sbi.h
> > > > > > > > > > -#define KVM_SBI_VERSION_MAJOR 2
> > > > > > > > > > +#define KVM_SBI_VERSION_MAJOR 3
> > > > > > > > > I think it's time to add versioning to KVM SBI implementation.
> > > > > > > > > Userspace should be able to select the desired SBI version and KVM would
> > > > > > > > > tell the guest that newer features are not supported.
> > > > > > We need new code for this, but it's a good idea.
> > > > > > 
> > > > > > > > We can achieve that through onereg interface by disabling individual SBI
> > > > > > > > extensions.
> > > > > > > > We can extend the existing onereg interface to disable a specific SBI
> > > > > > > > version directly
> > > > > > > > instead of individual ones to save those IOCTL as well.
> > > > > > > Yes, I am all in favor of letting userspace provide all values in the
> > > > > > > BASE extension.
> > > > > We already support vendorid/archid/impid through one reg. I think we just
> > > > > need to add the SBI version support to that so that user space can set it.
> > > > > 
> > > > > > This is covered by your recent patch that provides userspace_sbi.
> > > > > Why do we need to invent new IOCTL for this ? Once the user space sets the
> > > > > SBI version, KVM can enforce it.
> > > > If an SBI spec version provides an extension that can be emulated by
> > > > userspace, then userspace could choose to advertise that spec version,
> > > > implement a BASE probe function that advertises the extension, and
> > > > implement the extension, even if the KVM version running is older
> > > > and unaware of it. But, in order to do that, we need KVM to exit to
> > > > userspace for all unknown SBI calls and to allow BASE to be overridden
> > > You mean only the version field in BASE - Correct ?
> > No, "BASE probe function" is the sbi_probe_extension() ecall.
> > 
> > > > by userspace. The new KVM CAP ioctl allows opting into that new behavior.
> > > But why we need a new IOCTL for that ? We can achieve that with existing
> > > one reg interface with improvements.
> > It's an existing IOCTL with a new data payload, but I can easily use
> > ONE_REG if you want to do everything through that.
> > 
> > KVM doesn't really need any other IOCTL than ONE_REGs, it's just
> > sometimes more reasonable to use a different IOCTL, like ENABLE_CAP.
> > 
> > > > The old KVM with new VMM configuration isn't totally far-fetched. While
> > > > host kernels tend to get updated regularly to include security fixes,
> > > > enterprise kernels tend to stop adding features at some point in order
> > > > to maximize stability. While enterprise VMMs would also eventually stop
> > > > adding features, enterprise consumers are always free to use their own
> > > > VMMs (at their own risk). So, there's a real chance we could have
> > > I think we are years away from that happening (if it happens). My
> > > suggestion was not to
> > > try to build a world where no body lives ;). When we get to that
> > > scenario, the default KVM
> > > shipped will have many extension implemented. So there won't be much
> > > advantage to
> > > reimplement them in the user space. We can also take an informed
> > > decision at that time
> > > if the current selective forwarding approach is better
> > Please don't repeat the design of SUSP/SRST/DBCN.
> > Seeing them is one of the reasons why I proposed the new interface.
> > 
> > "Blindly" forwarding DBCN to userspace is even a minor optimization. :)
> > 
> > >                                                         or we need to
> > > blindly forward any
> > > unknown SBI calls to the user space.
> > Yes, KVM has to do what userpace configures it to do.
> > 
> > I don't think that implementing unsupported SBI extensions in KVM is
> > important -- they should not be a hot path.
> > 
> > > > deployments with older, stable KVM where users want to enable later SBI
> > > > extensions, and, in some cases, that should be possible by just updating
> > > > the VMM -- but only if KVM is only acting as an SBI implementation
> > > > accelerator and not as a userspace SBI implementation gatekeeper.
> > > But some of the SBI extensions are so fundamental that it must be
> > > implemented in KVM
> > > for various reasons pointed by Anup on other thread.
> > No, SBI does not have to be implemented in KVM at all.
> > 
> > We do have a deep disagreement on what is virtualization and the role of
> > KVM in it.  I think that userspace wants a generic ISA accelerator.
> 
> I think the disagreement is the role of SBI in KVM virtualization rather
> than
> a generic virtualization and the role of KVM in it. I completely agree that
> KVM should act as an accelerator and defer the control to the user space in
> most of the cases
> such e.g I/O operations or system related functionalities. However, SBI
> specification solves
> much wider problems than those. Broadly we can categorize SBI
> functionalities into the following
> areas
> 
> 1. Bridging ISA GAP
> 2. Higher Privilege Assistance
> 3. Virtualization
> 4. Platform abstraction
> 5. Confidential computing
> 
> For #1, #3 and #5, I believe user space shouldn't be involved in
> implementation
> some of them are in hot path as well.

IMO, userspace should still be in control of whether or not it's involved
in #1, #3, and #5. It may make little sense for it to be involved, but the
choice should still be its.

> For #4 and #2, there are some
> opportunities which
> can be implemented in user space depending on the exact need. I am still not
> clear what is the exact
> motivation /right now/ to pursue such a path. May be I missed something.
> As per my understanding from our discussion threads, there are two use cases
> possible
> 
> 1. userspace wants to update more states in HSM. What are the states user
> space should care about scounteren (fixed already in usptream) ?
> 2. VMM vs KVM version difference - this may be true in the future depending
> on the speed of RISC-V virtualization adoption in the industry.
> But we are definitely not there yet. Please let me know if I misunderstood
> any use cases.

That's what I'm aware of as well, but I see giving userspace back full
control of what gets accelerated by KVM, and what doesn't, as a fix, which
is why I wouldn't want to delay it any longer.

> 
> > Even if userspace wants SBI for the M-mode interface, security minded
> This is probably a 3rd one ? Why we want M-mode interface in the user space
> ?
> > userspace aims for as little kernel code as possible.
> 
> We trust VMM code more than KVM code ?

We should be skeptical of both, which is why we'd rather put as much code
in userspace as possible. Insecure/faulty userspace will hopefully have
exploits/bugs contained to the single process. An insecure/faulty KVM
means the host is compromised/crashed. On x86, Google put a lot of effort
into moving instruction emulation out of KVM for security concerns[1]. In
general, if it's not a hot path and there's a way to do it in userspace,
then it should be done in userspace (or at least there should be an
option to use userspace -- each use case can choose what's best for
itself).

[1] https://www.linux-kvm.org/images/3/3d/01x02-Steve_Rutherford-Performant_Security_Hardening_of_KVM.pdf

Thanks,
drew

> 
> > Userspace might want to accelerate some SBI extension in KVM, but it
> > should not be KVM who decides what userspace wants.

On Fri, May 30, 2025 at 12:44 AM Andrew Jones <ajones@ventanamicro.com> wrote:
>
> On Thu, May 29, 2025 at 11:44:38AM -0700, Atish Patra wrote:
> >
> > On 5/29/25 3:24 AM, Radim Krčmář wrote:
> > > I originally gave up on the idea, but I feel kinda bad for Drew now, so
> > > trying again:
> >
> > I am sorry if some of my replies came across in the wrong way. That was
> > never
> > the intention.
>
> Not at all. Radim only meant that I was defending his patches, even though
> he wasn't :-)
>
> >
> >
> > > 2025-05-28T12:21:59-07:00, Atish Patra <atish.patra@linux.dev>:
> > > > On 5/28/25 8:09 AM, Andrew Jones wrote:
> > > > > On Wed, May 28, 2025 at 07:16:11AM -0700, Atish Patra wrote:
> > > > > > On 5/26/25 4:13 AM, Andrew Jones wrote:
> > > > > > > On Mon, May 26, 2025 at 11:00:30AM +0200, Radim Krčmář wrote:
> > > > > > > > 2025-05-23T10:16:11-07:00, Atish Patra <atish.patra@linux.dev>:
> > > > > > > > > On 5/23/25 6:31 AM, Radim Krčmář wrote:
> > > > > > > > > > 2025-05-22T12:03:43-07:00, Atish Patra <atishp@rivosinc.com>:
> > > > > > > > > > > Upgrade the SBI version to v3.0 so that corresponding features
> > > > > > > > > > > can be enabled in the guest.
> > > > > > > > > > >
> > > > > > > > > > > Signed-off-by: Atish Patra <atishp@rivosinc.com>
> > > > > > > > > > > ---
> > > > > > > > > > > diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/include/asm/kvm_vcpu_sbi.h
> > > > > > > > > > > -#define KVM_SBI_VERSION_MAJOR 2
> > > > > > > > > > > +#define KVM_SBI_VERSION_MAJOR 3
> > > > > > > > > > I think it's time to add versioning to KVM SBI implementation.
> > > > > > > > > > Userspace should be able to select the desired SBI version and KVM would
> > > > > > > > > > tell the guest that newer features are not supported.
> > > > > > > We need new code for this, but it's a good idea.
> > > > > > >
> > > > > > > > > We can achieve that through onereg interface by disabling individual SBI
> > > > > > > > > extensions.
> > > > > > > > > We can extend the existing onereg interface to disable a specific SBI
> > > > > > > > > version directly
> > > > > > > > > instead of individual ones to save those IOCTL as well.
> > > > > > > > Yes, I am all in favor of letting userspace provide all values in the
> > > > > > > > BASE extension.
> > > > > > We already support vendorid/archid/impid through one reg. I think we just
> > > > > > need to add the SBI version support to that so that user space can set it.
> > > > > >
> > > > > > > This is covered by your recent patch that provides userspace_sbi.
> > > > > > Why do we need to invent new IOCTL for this ? Once the user space sets the
> > > > > > SBI version, KVM can enforce it.
> > > > > If an SBI spec version provides an extension that can be emulated by
> > > > > userspace, then userspace could choose to advertise that spec version,
> > > > > implement a BASE probe function that advertises the extension, and
> > > > > implement the extension, even if the KVM version running is older
> > > > > and unaware of it. But, in order to do that, we need KVM to exit to
> > > > > userspace for all unknown SBI calls and to allow BASE to be overridden
> > > > You mean only the version field in BASE - Correct ?
> > > No, "BASE probe function" is the sbi_probe_extension() ecall.
> > >
> > > > > by userspace. The new KVM CAP ioctl allows opting into that new behavior.
> > > > But why we need a new IOCTL for that ? We can achieve that with existing
> > > > one reg interface with improvements.
> > > It's an existing IOCTL with a new data payload, but I can easily use
> > > ONE_REG if you want to do everything through that.
> > >
> > > KVM doesn't really need any other IOCTL than ONE_REGs, it's just
> > > sometimes more reasonable to use a different IOCTL, like ENABLE_CAP.
> > >
> > > > > The old KVM with new VMM configuration isn't totally far-fetched. While
> > > > > host kernels tend to get updated regularly to include security fixes,
> > > > > enterprise kernels tend to stop adding features at some point in order
> > > > > to maximize stability. While enterprise VMMs would also eventually stop
> > > > > adding features, enterprise consumers are always free to use their own
> > > > > VMMs (at their own risk). So, there's a real chance we could have
> > > > I think we are years away from that happening (if it happens). My
> > > > suggestion was not to
> > > > try to build a world where no body lives ;). When we get to that
> > > > scenario, the default KVM
> > > > shipped will have many extension implemented. So there won't be much
> > > > advantage to
> > > > reimplement them in the user space. We can also take an informed
> > > > decision at that time
> > > > if the current selective forwarding approach is better
> > > Please don't repeat the design of SUSP/SRST/DBCN.
> > > Seeing them is one of the reasons why I proposed the new interface.
> > >
> > > "Blindly" forwarding DBCN to userspace is even a minor optimization. :)
> > >
> > > >                                                         or we need to
> > > > blindly forward any
> > > > unknown SBI calls to the user space.
> > > Yes, KVM has to do what userpace configures it to do.
> > >
> > > I don't think that implementing unsupported SBI extensions in KVM is
> > > important -- they should not be a hot path.
> > >
> > > > > deployments with older, stable KVM where users want to enable later SBI
> > > > > extensions, and, in some cases, that should be possible by just updating
> > > > > the VMM -- but only if KVM is only acting as an SBI implementation
> > > > > accelerator and not as a userspace SBI implementation gatekeeper.
> > > > But some of the SBI extensions are so fundamental that it must be
> > > > implemented in KVM
> > > > for various reasons pointed by Anup on other thread.
> > > No, SBI does not have to be implemented in KVM at all.
> > >
> > > We do have a deep disagreement on what is virtualization and the role of
> > > KVM in it.  I think that userspace wants a generic ISA accelerator.
> >
> > I think the disagreement is the role of SBI in KVM virtualization rather
> > than
> > a generic virtualization and the role of KVM in it. I completely agree that
> > KVM should act as an accelerator and defer the control to the user space in
> > most of the cases
> > such e.g I/O operations or system related functionalities. However, SBI
> > specification solves
> > much wider problems than those. Broadly we can categorize SBI
> > functionalities into the following
> > areas
> >
> > 1. Bridging ISA GAP
> > 2. Higher Privilege Assistance
> > 3. Virtualization
> > 4. Platform abstraction
> > 5. Confidential computing
> >
> > For #1, #3 and #5, I believe user space shouldn't be involved in
> > implementation
> > some of them are in hot path as well.
>
> IMO, userspace should still be in control of whether or not it's involved
> in #1, #3, and #5. It may make little sense for it to be involved, but the
> choice should still be its.
>
> > For #4 and #2, there are some
> > opportunities which
> > can be implemented in user space depending on the exact need. I am still not
> > clear what is the exact
> > motivation /right now/ to pursue such a path. May be I missed something.
> > As per my understanding from our discussion threads, there are two use cases
> > possible
> >
> > 1. userspace wants to update more states in HSM. What are the states user
> > space should care about scounteren (fixed already in usptream) ?
> > 2. VMM vs KVM version difference - this may be true in the future depending
> > on the speed of RISC-V virtualization adoption in the industry.
> > But we are definitely not there yet. Please let me know if I misunderstood
> > any use cases.
>
> That's what I'm aware of as well, but I see giving userspace back full
> control of what gets accelerated by KVM, and what doesn't, as a fix, which
> is why I wouldn't want to delay it any longer.
>
> >
> > > Even if userspace wants SBI for the M-mode interface, security minded
> > This is probably a 3rd one ? Why we want M-mode interface in the user space
> > ?
> > > userspace aims for as little kernel code as possible.
> >
> > We trust VMM code more than KVM code ?
>
> We should be skeptical of both, which is why we'd rather put as much code
> in userspace as possible. Insecure/faulty userspace will hopefully have
> exploits/bugs contained to the single process. An insecure/faulty KVM
> means the host is compromised/crashed. On x86, Google put a lot of effort
> into moving instruction emulation out of KVM for security concerns[1]. In
> general, if it's not a hot path and there's a way to do it in userspace,
> then it should be done in userspace (or at least there should be an
> option to use userspace -- each use case can choose what's best for
> itself).
>
> [1] https://www.linux-kvm.org/images/3/3d/01x02-Steve_Rutherford-Performant_Security_Hardening_of_KVM.pdf
>

We are already forwarding a few of the category #2 and all
of category #4 SBI extensions to KVM user space which are
not in critical or hot-path.

Majority of SBI extensions in categories #1, #2, #3, and #5
provide critical per-VCPU functionality and many of these
are also in hotpath (such as #1, #3, and #5) hence
implemented in kernel space.

Further, KVM user space lacks required functionality (CSRs,
instructions, or ioctls) to implement many critical SBI extensions
in user space so blanket forwarding of all SBI extensions to
KVM user space is not going to fly.
(Note: Previously, I have already provided many examples)

In short, a hybrid approach (current implementation) is the
best thing where only non-critical and non-hotpath SBI
extensions (few of them) are forwarded to KVM user space
while critical / hot-path SBI extensions (majority of them)
are in kernel space.

Regards,
Anup

On 5/28/25 12:21 PM, Atish Patra wrote:
> <Removing Palmer's rivos email address to avoid bouncing>
> 
> On 5/28/25 8:09 AM, Andrew Jones wrote:
>> On Wed, May 28, 2025 at 07:16:11AM -0700, Atish Patra wrote:
>>> On 5/26/25 4:13 AM, Andrew Jones wrote:
>>>> On Mon, May 26, 2025 at 11:00:30AM +0200, Radim Krčmář wrote:
>>>>> 2025-05-23T10:16:11-07:00, Atish Patra <atish.patra@linux.dev>:
>>>>>> On 5/23/25 6:31 AM, Radim Krčmář wrote:
>>>>>>> 2025-05-22T12:03:43-07:00, Atish Patra <atishp@rivosinc.com>:
>>>>>>>> Upgrade the SBI version to v3.0 so that corresponding features
>>>>>>>> can be enabled in the guest.
>>>>>>>>
>>>>>>>> Signed-off-by: Atish Patra <atishp@rivosinc.com>
>>>>>>>> ---
>>>>>>>> diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/ 
>>>>>>>> include/asm/kvm_vcpu_sbi.h
>>>>>>>> -#define KVM_SBI_VERSION_MAJOR 2
>>>>>>>> +#define KVM_SBI_VERSION_MAJOR 3
>>>>>>> I think it's time to add versioning to KVM SBI implementation.
>>>>>>> Userspace should be able to select the desired SBI version and 
>>>>>>> KVM would
>>>>>>> tell the guest that newer features are not supported.
>>>> We need new code for this, but it's a good idea.
>>>>
>>>>>> We can achieve that through onereg interface by disabling 
>>>>>> individual SBI
>>>>>> extensions.
>>>>>> We can extend the existing onereg interface to disable a specific SBI
>>>>>> version directly
>>>>>> instead of individual ones to save those IOCTL as well.
>>>>> Yes, I am all in favor of letting userspace provide all values in the
>>>>> BASE extension.
>>> We already support vendorid/archid/impid through one reg. I think we 
>>> just
>>> need to add the SBI version support to that so that user space can 
>>> set it.
>>>
>>>> This is covered by your recent patch that provides userspace_sbi.
>>> Why do we need to invent new IOCTL for this ? Once the user space 
>>> sets the
>>> SBI version, KVM can enforce it.
>> If an SBI spec version provides an extension that can be emulated by
>> userspace, then userspace could choose to advertise that spec version,
>> implement a BASE probe function that advertises the extension, and
>> implement the extension, even if the KVM version running is older
>> and unaware of it. But, in order to do that, we need KVM to exit to
>> userspace for all unknown SBI calls and to allow BASE to be overridden
> You mean only the version field in BASE - Correct ?
> 
> We already support vendorid/archid/impid through one reg. I don't see the
> point of overriding SBI implementation ID & version.
> 
>> by userspace. The new KVM CAP ioctl allows opting into that new behavior.
> 
> But why we need a new IOCTL for that ? We can achieve that with existing
> one reg interface with improvements.
> 
>> The old KVM with new VMM configuration isn't totally far-fetched. While
>> host kernels tend to get updated regularly to include security fixes,
>> enterprise kernels tend to stop adding features at some point in order
>> to maximize stability. While enterprise VMMs would also eventually stop
>> adding features, enterprise consumers are always free to use their own
>> VMMs (at their own risk). So, there's a real chance we could have
> 
> I think we are years away from that happening (if it happens). My 
> suggestion was not to
> try to build a world where no body lives ;). When we get to that 

We also support KVM as a kernel module. So it is relatively easier to 
update the RISC-V KVM module for enterprise consumers.

> scenario, the default KVM
> shipped will have many extension implemented. So there won't be much 
> advantage to
> reimplement them in the user space. We can also take an informed 
> decision at that time
> if the current selective forwarding approach is better or we need to 
> blindly forward any
> unknown SBI calls to the user space.
> 
>> deployments with older, stable KVM where users want to enable later SBI
>> extensions, and, in some cases, that should be possible by just updating
>> the VMM -- but only if KVM is only acting as an SBI implementation
>> accelerator and not as a userspace SBI implementation gatekeeper.
> 
> But some of the SBI extensions are so fundamental that it must be 
> implemented in KVM
> for various reasons pointed by Anup on other thread.
> 
>> Thanks,
>> drew
>>
>>>> With that, userspace can disable all extensions that aren't
>>>> supported by a given spec version, disable BASE and then provide
>>>> a BASE that advertises the version it wants. The new code is needed
>>>> for extensions that userspace still wants KVM to accelerate, but then
>>>> KVM needs to be informed it should deny all functions not included in
>>>> the selected spec version.
>>>>
>>>> Thanks,
>>>> drew
>>>>
>>>> _______________________________________________
>>>> linux-riscv mailing list
>>>> linux-riscv@lists.infradead.org
>>>> http://lists.infradead.org/mailman/listinfo/linux-riscv