Hi All:

In some cases, the virtqueue could be backed by different devices. One
example is that in the case of vDPA some parent may emualte virtqueue
via vringh. In this case, it would be wrong if we stick with the
physical DMA device for software emulated device, since there's no
easy way for vringh to know about the hardware IOMMU mappings.

So this series tries to introduce per virtqueue DMA device, then
software virtqueues can utilize the transport specific method to
assign appropirate DMA device.

This fixes the crash of mlx5_vdpa + virtio_vdpa when platform IOMMU is
enabled but not in the passthrough mode. The reason for the crash is
that the virito_ring tries to map the control virtqueue into platform
IOMMU but the vringh assumes a direct mapping (PA as IOVA). This is
fixed by advetise the vDPA device that doesnt do DMA (without a DMA
ops). So DMA API can go with the direct mapping then the vringh will
be happy since mlx5_vdpa assuems a direct/identical mapping by
default.

Please review.

Thanks

Jason Wang (5):
  virtio_ring: per virtqueue dma device
  vdpa: introduce get_vq_dma_device()
  virtio-vdpa: support per vq dma device
  vdpa: set dma mask for vDPA device
  vdpa: mlx5: support per virtqueue dma device

 drivers/vdpa/mlx5/net/mlx5_vnet.c |  11 +++
 drivers/vdpa/vdpa.c               |   5 ++
 drivers/virtio/virtio_ring.c      | 133 +++++++++++++++++++++---------
 drivers/virtio/virtio_vdpa.c      |  13 ++-
 include/linux/vdpa.h              |   6 ++
 include/linux/virtio_ring.h       |  16 ++++
 6 files changed, 141 insertions(+), 43 deletions(-)

-- 
2.25.1

> From: Jason Wang <jasowang@redhat.com>
> Sent: Wednesday, 11 January 2023 8:28
> To: mst@redhat.com; jasowang@redhat.com
> Cc: Eli Cohen <elic@nvidia.com>; gdawar@amd.com;
> virtualization@lists.linux-foundation.org; linux-kernel@vger.kernel.org;
> tanuj.kamde@amd.com
> Subject: [PATCH 0/5] virtio_ring: per virtqueue DMA device
> 
> Hi All:
> 
> In some cases, the virtqueue could be backed by different devices. One
> example is that in the case of vDPA some parent may emualte virtqueue
> via vringh. In this case, it would be wrong if we stick with the
> physical DMA device for software emulated device, since there's no
> easy way for vringh to know about the hardware IOMMU mappings.
> 
> So this series tries to introduce per virtqueue DMA device, then
> software virtqueues can utilize the transport specific method to
> assign appropirate DMA device.
> 
> This fixes the crash of mlx5_vdpa + virtio_vdpa when platform IOMMU is
> enabled but not in the passthrough mode. The reason for the crash is
> that the virito_ring tries to map the control virtqueue into platform
> IOMMU but the vringh assumes a direct mapping (PA as IOVA). This is
> fixed by advetise the vDPA device that doesnt do DMA (without a DMA
> ops). So DMA API can go with the direct mapping then the vringh will
> be happy since mlx5_vdpa assuems a direct/identical mapping by
> default.
> 

Could you provide instructions how to reproduce the crash you were seeing?

> Please review.
> 
> Thanks
> 
> Jason Wang (5):
>   virtio_ring: per virtqueue dma device
>   vdpa: introduce get_vq_dma_device()
>   virtio-vdpa: support per vq dma device
>   vdpa: set dma mask for vDPA device
>   vdpa: mlx5: support per virtqueue dma device
> 
>  drivers/vdpa/mlx5/net/mlx5_vnet.c |  11 +++
>  drivers/vdpa/vdpa.c               |   5 ++
>  drivers/virtio/virtio_ring.c      | 133 +++++++++++++++++++++---------
>  drivers/virtio/virtio_vdpa.c      |  13 ++-
>  include/linux/vdpa.h              |   6 ++
>  include/linux/virtio_ring.h       |  16 ++++
>  6 files changed, 141 insertions(+), 43 deletions(-)
> 
> --
> 2.25.1

On Wed, Jan 11, 2023 at 9:33 PM Eli Cohen <elic@nvidia.com> wrote:
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Wednesday, 11 January 2023 8:28
> > To: mst@redhat.com; jasowang@redhat.com
> > Cc: Eli Cohen <elic@nvidia.com>; gdawar@amd.com;
> > virtualization@lists.linux-foundation.org; linux-kernel@vger.kernel.org;
> > tanuj.kamde@amd.com
> > Subject: [PATCH 0/5] virtio_ring: per virtqueue DMA device
> >
> > Hi All:
> >
> > In some cases, the virtqueue could be backed by different devices. One
> > example is that in the case of vDPA some parent may emualte virtqueue
> > via vringh. In this case, it would be wrong if we stick with the
> > physical DMA device for software emulated device, since there's no
> > easy way for vringh to know about the hardware IOMMU mappings.
> >
> > So this series tries to introduce per virtqueue DMA device, then
> > software virtqueues can utilize the transport specific method to
> > assign appropirate DMA device.
> >
> > This fixes the crash of mlx5_vdpa + virtio_vdpa when platform IOMMU is
> > enabled but not in the passthrough mode. The reason for the crash is
> > that the virito_ring tries to map the control virtqueue into platform
> > IOMMU but the vringh assumes a direct mapping (PA as IOVA). This is
> > fixed by advetise the vDPA device that doesnt do DMA (without a DMA
> > ops). So DMA API can go with the direct mapping then the vringh will
> > be happy since mlx5_vdpa assuems a direct/identical mapping by
> > default.
> >
>
> Could you provide instructions how to reproduce the crash you were seeing?

It should be something like:

1) boot host kernel with iommu enabled but not in passthrough mode: I
use intel_iommu=on
2) create vdpa device on top of mlx5_vdpa VF
3) bind the vdpa device to virtio_vdpa

Then I can see the crash.

Thanks

>
> > Please review.
> >
> > Thanks
> >
> > Jason Wang (5):
> >   virtio_ring: per virtqueue dma device
> >   vdpa: introduce get_vq_dma_device()
> >   virtio-vdpa: support per vq dma device
> >   vdpa: set dma mask for vDPA device
> >   vdpa: mlx5: support per virtqueue dma device
> >
> >  drivers/vdpa/mlx5/net/mlx5_vnet.c |  11 +++
> >  drivers/vdpa/vdpa.c               |   5 ++
> >  drivers/virtio/virtio_ring.c      | 133 +++++++++++++++++++++---------
> >  drivers/virtio/virtio_vdpa.c      |  13 ++-
> >  include/linux/vdpa.h              |   6 ++
> >  include/linux/virtio_ring.h       |  16 ++++
> >  6 files changed, 141 insertions(+), 43 deletions(-)
> >
> > --
> > 2.25.1
>