[v2] virtio-net: Fix probe of virtio-net on kvmtool

[PATCH v2] virtio-net: Fix probe of virtio-net on kvmtool

Posted by Rob Bradford via B4 Relay 2 years, 6 months ago

From: Rob Bradford <rbradford@rivosinc.com>

kvmtool does not support the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature
but does advertise the VIRTIO_NET_F_GUEST_TSO{4,6} features. Check that
the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature is present before setting
the NETIF_F_GRO_HW feature bit as otherwise an attempt will be made to
program the virtio-net device using the ctrl queue which will fail.

This resolves the following error when running on kvmtool:

[    1.865992] net eth0: Fail to set guest offload.
[    1.872491] virtio_net virtio2 eth0: set_features() failed (-22); wanted 0x0000000000134829, left 0x0080000000134829

Signed-off-by: Rob Bradford <rbradford@rivosinc.com>
---
Changes in v2:
- Use parentheses to group logical OR of features 
- Link to v1:
  https://lore.kernel.org/r/20230223-virtio-net-kvmtool-v1-1-fc23d29b9d7a@rivosinc.com
---
 drivers/net/virtio_net.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 61e33e4dd0cd..f8341d1a4ccd 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -3780,10 +3780,9 @@ static int virtnet_probe(struct virtio_device *vdev)
 	}
 	if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_CSUM))
 		dev->features |= NETIF_F_RXCSUM;
-	if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
-	    virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6))
-		dev->features |= NETIF_F_GRO_HW;
-	if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))
+	if ((virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
+	    virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6)) &&
+	    virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))
 		dev->hw_features |= NETIF_F_GRO_HW;
 
 	dev->vlan_features = dev->features;

---
base-commit: c39cea6f38eefe356d64d0bc1e1f2267e282cdd3
change-id: 20230223-virtio-net-kvmtool-87f37515be22

Best regards,
-- 
Rob Bradford <rbradford@rivosinc.com>

Re: [PATCH v2] virtio-net: Fix probe of virtio-net on kvmtool

Posted by Michael S. Tsirkin 2 years, 6 months ago

On Thu, Feb 23, 2023 at 07:38:25PM +0000, Rob Bradford via B4 Relay wrote:
> From: Rob Bradford <rbradford@rivosinc.com>
> 
> kvmtool does not support the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature
> but does advertise the VIRTIO_NET_F_GUEST_TSO{4,6} features. Check that
> the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature is present before setting
> the NETIF_F_GRO_HW feature bit as otherwise an attempt will be made to
> program the virtio-net device using the ctrl queue which will fail.
> 
> This resolves the following error when running on kvmtool:
> 
> [    1.865992] net eth0: Fail to set guest offload.
> [    1.872491] virtio_net virtio2 eth0: set_features() failed (-22); wanted 0x0000000000134829, left 0x0080000000134829
> 
> Signed-off-by: Rob Bradford <rbradford@rivosinc.com>
> ---
> Changes in v2:
> - Use parentheses to group logical OR of features 
> - Link to v1:
>   https://lore.kernel.org/r/20230223-virtio-net-kvmtool-v1-1-fc23d29b9d7a@rivosinc.com
> ---
>  drivers/net/virtio_net.c | 7 +++----
>  1 file changed, 3 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 61e33e4dd0cd..f8341d1a4ccd 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -3780,10 +3780,9 @@ static int virtnet_probe(struct virtio_device *vdev)
>  	}
>  	if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_CSUM))
>  		dev->features |= NETIF_F_RXCSUM;
> -	if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
> -	    virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6))
> -		dev->features |= NETIF_F_GRO_HW;
> -	if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))
> +	if ((virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
> +	    virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6)) &&
> +	    virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))
>  		dev->hw_features |= NETIF_F_GRO_HW;

This will disable GRO/LRO on kvmtool completely causing a significant
performance regression.

Jason, isn't this what
	commit dbcf24d153884439dad30484a0e3f02350692e4c
	Author: Jason Wang <jasowang@redhat.com>
	Date:   Tue Aug 17 16:06:59 2021 +0800

	    virtio-net: use NETIF_F_GRO_HW instead of NETIF_F_LRO

was supposed to address?


And apropos this:
    
    Fix this by using NETIF_F_GRO_HW instead. Though the spec does not
    guarantee packets to be re-segmented as the original ones,
    we can add that to the spec, possibly with a flag for devices to
    differentiate between GRO and LRO.

this never happened. What's the plan exactly?




>  	dev->vlan_features = dev->features;
> 
> ---
> base-commit: c39cea6f38eefe356d64d0bc1e1f2267e282cdd3
> change-id: 20230223-virtio-net-kvmtool-87f37515be22
> 
> Best regards,
> -- 
> Rob Bradford <rbradford@rivosinc.com>

Re: [PATCH v2] virtio-net: Fix probe of virtio-net on kvmtool

Posted by Jason Wang 2 years, 6 months ago

On Fri, Feb 24, 2023 at 4:25 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Thu, Feb 23, 2023 at 07:38:25PM +0000, Rob Bradford via B4 Relay wrote:
> > From: Rob Bradford <rbradford@rivosinc.com>
> >
> > kvmtool does not support the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature
> > but does advertise the VIRTIO_NET_F_GUEST_TSO{4,6} features. Check that
> > the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature is present before setting
> > the NETIF_F_GRO_HW feature bit as otherwise an attempt will be made to
> > program the virtio-net device using the ctrl queue which will fail.
> >
> > This resolves the following error when running on kvmtool:
> >
> > [    1.865992] net eth0: Fail to set guest offload.
> > [    1.872491] virtio_net virtio2 eth0: set_features() failed (-22); wanted 0x0000000000134829, left 0x0080000000134829
> >
> > Signed-off-by: Rob Bradford <rbradford@rivosinc.com>
> > ---
> > Changes in v2:
> > - Use parentheses to group logical OR of features
> > - Link to v1:
> >   https://lore.kernel.org/r/20230223-virtio-net-kvmtool-v1-1-fc23d29b9d7a@rivosinc.com
> > ---
> >  drivers/net/virtio_net.c | 7 +++----
> >  1 file changed, 3 insertions(+), 4 deletions(-)
> >
> > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > index 61e33e4dd0cd..f8341d1a4ccd 100644
> > --- a/drivers/net/virtio_net.c
> > +++ b/drivers/net/virtio_net.c
> > @@ -3780,10 +3780,9 @@ static int virtnet_probe(struct virtio_device *vdev)
> >       }
> >       if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_CSUM))
> >               dev->features |= NETIF_F_RXCSUM;
> > -     if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
> > -         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6))
> > -             dev->features |= NETIF_F_GRO_HW;
> > -     if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))
> > +     if ((virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
> > +         virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6)) &&
> > +         virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))
> >               dev->hw_features |= NETIF_F_GRO_HW;
>
> This will disable GRO/LRO on kvmtool completely causing a significant
> performance regression.
>
> Jason, isn't this what
>         commit dbcf24d153884439dad30484a0e3f02350692e4c
>         Author: Jason Wang <jasowang@redhat.com>
>         Date:   Tue Aug 17 16:06:59 2021 +0800
>
>             virtio-net: use NETIF_F_GRO_HW instead of NETIF_F_LRO
>
> was supposed to address?
>

Yes, I've asked a similar question in another thread.

>
> And apropos this:
>
>     Fix this by using NETIF_F_GRO_HW instead. Though the spec does not
>     guarantee packets to be re-segmented as the original ones,
>     we can add that to the spec, possibly with a flag for devices to
>     differentiate between GRO and LRO.
>
> this never happened. What's the plan exactly?

It's in the backlog, but I'm out of bandwidth for doing that now.

Thanks

>
>
>
>
> >       dev->vlan_features = dev->features;
> >
> > ---
> > base-commit: c39cea6f38eefe356d64d0bc1e1f2267e282cdd3
> > change-id: 20230223-virtio-net-kvmtool-87f37515be22
> >
> > Best regards,
> > --
> > Rob Bradford <rbradford@rivosinc.com>
>

Re: [PATCH v2] virtio-net: Fix probe of virtio-net on kvmtool

Posted by Jason Wang 2 years, 6 months ago

On Fri, Feb 24, 2023 at 3:38 AM Rob Bradford via B4 Relay
<devnull+rbradford.rivosinc.com@kernel.org> wrote:
>
> From: Rob Bradford <rbradford@rivosinc.com>
>
> kvmtool does not support the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature
> but does advertise the VIRTIO_NET_F_GUEST_TSO{4,6} features. Check that
> the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature is present before setting
> the NETIF_F_GRO_HW feature bit as otherwise an attempt will be made to
> program the virtio-net device using the ctrl queue which will fail.
>
> This resolves the following error when running on kvmtool:
>
> [    1.865992] net eth0: Fail to set guest offload.
> [    1.872491] virtio_net virtio2 eth0: set_features() failed (-22); wanted 0x0000000000134829, left 0x0080000000134829
>
> Signed-off-by: Rob Bradford <rbradford@rivosinc.com>
> ---
> Changes in v2:
> - Use parentheses to group logical OR of features
> - Link to v1:
>   https://lore.kernel.org/r/20230223-virtio-net-kvmtool-v1-1-fc23d29b9d7a@rivosinc.com
> ---
>  drivers/net/virtio_net.c | 7 +++----
>  1 file changed, 3 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 61e33e4dd0cd..f8341d1a4ccd 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -3780,10 +3780,9 @@ static int virtnet_probe(struct virtio_device *vdev)
>         }
>         if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_CSUM))
>                 dev->features |= NETIF_F_RXCSUM;
> -       if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
> -           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6))
> -               dev->features |= NETIF_F_GRO_HW;
> -       if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))
> +       if ((virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
> +           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6)) &&
> +           virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))
>                 dev->hw_features |= NETIF_F_GRO_HW;

Does this mean we won't have NETIF_F_GRO_HW when only TSO4/TSO6 are
supported but not GUEST_OFFLOADS?

Is this intended?

Thanks

>
>         dev->vlan_features = dev->features;
>
> ---
> base-commit: c39cea6f38eefe356d64d0bc1e1f2267e282cdd3
> change-id: 20230223-virtio-net-kvmtool-87f37515be22
>
> Best regards,
> --
> Rob Bradford <rbradford@rivosinc.com>
>

Re: [PATCH v2] virtio-net: Fix probe of virtio-net on kvmtool

Posted by Xuan Zhuo 2 years, 6 months ago

On Fri, 24 Feb 2023 11:11:37 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Fri, Feb 24, 2023 at 3:38 AM Rob Bradford via B4 Relay
> <devnull+rbradford.rivosinc.com@kernel.org> wrote:
> >
> > From: Rob Bradford <rbradford@rivosinc.com>
> >
> > kvmtool does not support the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature
> > but does advertise the VIRTIO_NET_F_GUEST_TSO{4,6} features. Check that
> > the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature is present before setting
> > the NETIF_F_GRO_HW feature bit as otherwise

Here are settings for dev->features and dev->hw_features.


> > an attempt will be made to
> > program the virtio-net device using the ctrl queue which will fail.
> >
> > This resolves the following error when running on kvmtool:

Can you talk about it in detail what it did?

Thanks.

> >
> > [    1.865992] net eth0: Fail to set guest offload.
> > [    1.872491] virtio_net virtio2 eth0: set_features() failed (-22); wanted 0x0000000000134829, left 0x0080000000134829
> >
> > Signed-off-by: Rob Bradford <rbradford@rivosinc.com>
> > ---
> > Changes in v2:
> > - Use parentheses to group logical OR of features
> > - Link to v1:
> >   https://lore.kernel.org/r/20230223-virtio-net-kvmtool-v1-1-fc23d29b9d7a@rivosinc.com
> > ---
> >  drivers/net/virtio_net.c | 7 +++----
> >  1 file changed, 3 insertions(+), 4 deletions(-)
> >
> > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > index 61e33e4dd0cd..f8341d1a4ccd 100644
> > --- a/drivers/net/virtio_net.c
> > +++ b/drivers/net/virtio_net.c
> > @@ -3780,10 +3780,9 @@ static int virtnet_probe(struct virtio_device *vdev)
> >         }
> >         if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_CSUM))
> >                 dev->features |= NETIF_F_RXCSUM;
> > -       if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
> > -           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6))
> > -               dev->features |= NETIF_F_GRO_HW;
> > -       if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))
> > +       if ((virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
> > +           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6)) &&
> > +           virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))
> >                 dev->hw_features |= NETIF_F_GRO_HW;
>
> Does this mean we won't have NETIF_F_GRO_HW when only TSO4/TSO6 are
> supported but not GUEST_OFFLOADS?
>
> Is this intended?
>
> Thanks
>
> >
> >         dev->vlan_features = dev->features;
> >
> > ---
> > base-commit: c39cea6f38eefe356d64d0bc1e1f2267e282cdd3
> > change-id: 20230223-virtio-net-kvmtool-87f37515be22
> >
> > Best regards,
> > --
> > Rob Bradford <rbradford@rivosinc.com>
> >
>
> _______________________________________________
> Virtualization mailing list
> Virtualization@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH v2] virtio-net: Fix probe of virtio-net on kvmtool

Posted by Rob Bradford 2 years, 6 months ago

On Tue, 28 Feb 2023 at 10:08, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> On Fri, 24 Feb 2023 11:11:37 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > On Fri, Feb 24, 2023 at 3:38 AM Rob Bradford via B4 Relay
> > <devnull+rbradford.rivosinc.com@kernel.org> wrote:
> > >
> > > From: Rob Bradford <rbradford@rivosinc.com>
> > >
> > > kvmtool does not support the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature
> > > but does advertise the VIRTIO_NET_F_GUEST_TSO{4,6} features. Check that
> > > the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature is present before setting
> > > the NETIF_F_GRO_HW feature bit as otherwise
>
> Here are settings for dev->features and dev->hw_features.
>
>
> > > an attempt will be made to
> > > program the virtio-net device using the ctrl queue which will fail.
> > >
> > > This resolves the following error when running on kvmtool:
>
> Can you talk about it in detail what it did?

In the updated v3 version of the patch I went into a lot more detail,
particularly of why the bit was being cleared and triggering the
control queue reprogramming. Based on that I also adjusted the
conditional check.

Cheers,

Rob

Re: [PATCH v2] virtio-net: Fix probe of virtio-net on kvmtool

Posted by Xuan Zhuo 2 years, 6 months ago

On Tue, 28 Feb 2023 18:06:38 +0800, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> On Fri, 24 Feb 2023 11:11:37 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > On Fri, Feb 24, 2023 at 3:38 AM Rob Bradford via B4 Relay
> > <devnull+rbradford.rivosinc.com@kernel.org> wrote:
> > >
> > > From: Rob Bradford <rbradford@rivosinc.com>
> > >
> > > kvmtool does not support the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature
> > > but does advertise the VIRTIO_NET_F_GUEST_TSO{4,6} features. Check that
> > > the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature is present before setting
> > > the NETIF_F_GRO_HW feature bit as otherwise
>
> Here are settings for dev->features and dev->hw_features.

What I want to say is that in normal circumstances, ethtool will identify it and
will not directly modify the backend, if there is no VIRTIO_NET_F_CTRL_GUEST_OFFLOADS.

Thanks.

>
>
> > > an attempt will be made to
> > > program the virtio-net device using the ctrl queue which will fail.
> > >
> > > This resolves the following error when running on kvmtool:
>
> Can you talk about it in detail what it did?
>
> Thanks.
>
> > >
> > > [    1.865992] net eth0: Fail to set guest offload.
> > > [    1.872491] virtio_net virtio2 eth0: set_features() failed (-22); wanted 0x0000000000134829, left 0x0080000000134829
> > >
> > > Signed-off-by: Rob Bradford <rbradford@rivosinc.com>
> > > ---
> > > Changes in v2:
> > > - Use parentheses to group logical OR of features
> > > - Link to v1:
> > >   https://lore.kernel.org/r/20230223-virtio-net-kvmtool-v1-1-fc23d29b9d7a@rivosinc.com
> > > ---
> > >  drivers/net/virtio_net.c | 7 +++----
> > >  1 file changed, 3 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > > index 61e33e4dd0cd..f8341d1a4ccd 100644
> > > --- a/drivers/net/virtio_net.c
> > > +++ b/drivers/net/virtio_net.c
> > > @@ -3780,10 +3780,9 @@ static int virtnet_probe(struct virtio_device *vdev)
> > >         }
> > >         if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_CSUM))
> > >                 dev->features |= NETIF_F_RXCSUM;
> > > -       if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
> > > -           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6))
> > > -               dev->features |= NETIF_F_GRO_HW;
> > > -       if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))
> > > +       if ((virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
> > > +           virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6)) &&
> > > +           virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS))
> > >                 dev->hw_features |= NETIF_F_GRO_HW;
> >
> > Does this mean we won't have NETIF_F_GRO_HW when only TSO4/TSO6 are
> > supported but not GUEST_OFFLOADS?
> >
> > Is this intended?
> >
> > Thanks
> >
> > >
> > >         dev->vlan_features = dev->features;
> > >
> > > ---
> > > base-commit: c39cea6f38eefe356d64d0bc1e1f2267e282cdd3
> > > change-id: 20230223-virtio-net-kvmtool-87f37515be22
> > >
> > > Best regards,
> > > --
> > > Rob Bradford <rbradford@rivosinc.com>
> > >
> >
> > _______________________________________________
> > Virtualization mailing list
> > Virtualization@lists.linux-foundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/virtualization