[v1] virtio-scsi: add iothread-vq-mapping parameter

[PATCH 00/12] virtio-scsi: add iothread-vq-mapping parameter

Posted by Stefan Hajnoczi 1 month, 2 weeks ago

Implement --device virtio-scsi-pci,iothread-vq-mapping= support so that
virtqueues can be assigned to different IOThreads. This improves SMP guest
scalability where I/O-intensive applications can become bottlenecked on a
single IOThread.

The following benchmark results show the effect of iothread-vq-mapping. fio
randread 4k iodepth=64 results from a 4 vCPU guest with an Intel P4800X SSD:
iothreads IOPS
------------------------------
1         189576
2         312698
4         346744

The virtio-scsi device model and core SCSI emulation currently assume that
requests are processed in a single AioContext. This patch series goes about
modifying this as follows:

scsi-disk: drop unused SCSIDiskState->bh field
dma: use current AioContext for dma_blk_io()

Make dma-helpers.c support the QEMU multi-queue block layer by using
qemu_get_current_aio_context().

scsi: track per-SCSIRequest AioContext
scsi: introduce requests_lock

Make the core SCSI emulation code support processing requests in multiple
AioContexts by protecting the per-SCSIDevice requests list.

virtio-scsi: introduce event and ctrl virtqueue locks
virtio-scsi: protect events_dropped field
virtio-scsi: perform TMFs in appropriate AioContexts

Make the virtio-scsi emulation code support processing requests in multiple
AioContexts. The event and ctrl virtqueues can interact with multiple
AioContexts. Especially the SCSI Task Management Functions (TMFs) handled by
the ctrl virtqueue need to be made thread-safe.

virtio-blk: extract cleanup_iothread_vq_mapping() function
virtio-blk: tidy up iothread_vq_mapping functions
virtio: extract iothread-vq-mapping.h API
virtio-scsi: add iothread-vq-mapping parameter

Port over the iothread-vq-mapping qdev property from virtio-blk to virtio-scsi.

virtio-scsi: handle ctrl virtqueue in main loop

Simplify TMF handling now that there is no longer a single AioContext where all
requests are processed.

Stefan Hajnoczi (12):
  scsi-disk: drop unused SCSIDiskState->bh field
  dma: use current AioContext for dma_blk_io()
  scsi: track per-SCSIRequest AioContext
  scsi: introduce requests_lock
  virtio-scsi: introduce event and ctrl virtqueue locks
  virtio-scsi: protect events_dropped field
  virtio-scsi: perform TMFs in appropriate AioContexts
  virtio-blk: extract cleanup_iothread_vq_mapping() function
  virtio-blk: tidy up iothread_vq_mapping functions
  virtio: extract iothread-vq-mapping.h API
  virtio-scsi: add iothread-vq-mapping parameter
  virtio-scsi: handle ctrl virtqueue in main loop

 include/hw/scsi/scsi.h                  |   8 +-
 include/hw/virtio/iothread-vq-mapping.h |  45 ++
 include/hw/virtio/virtio-scsi.h         |  15 +-
 include/system/dma.h                    |   3 +-
 hw/block/virtio-blk.c                   | 132 +-----
 hw/ide/core.c                           |   3 +-
 hw/ide/macio.c                          |   3 +-
 hw/scsi/scsi-bus.c                      | 121 ++++--
 hw/scsi/scsi-disk.c                     |  24 +-
 hw/scsi/virtio-scsi-dataplane.c         |  96 +++--
 hw/scsi/virtio-scsi.c                   | 534 ++++++++++++++----------
 hw/virtio/iothread-vq-mapping.c         | 131 ++++++
 system/dma-helpers.c                    |   8 +-
 hw/virtio/meson.build                   |   1 +
 14 files changed, 672 insertions(+), 452 deletions(-)
 create mode 100644 include/hw/virtio/iothread-vq-mapping.h
 create mode 100644 hw/virtio/iothread-vq-mapping.c

-- 
2.48.1

Re: [PATCH 00/12] virtio-scsi: add iothread-vq-mapping parameter

Posted by Kevin Wolf 3 weeks, 3 days ago

Am 13.02.2025 um 19:00 hat Stefan Hajnoczi geschrieben:
> Implement --device virtio-scsi-pci,iothread-vq-mapping= support so that
> virtqueues can be assigned to different IOThreads. This improves SMP guest
> scalability where I/O-intensive applications can become bottlenecked on a
> single IOThread.
> 
> The following benchmark results show the effect of iothread-vq-mapping. fio
> randread 4k iodepth=64 results from a 4 vCPU guest with an Intel P4800X SSD:
> iothreads IOPS
> ------------------------------
> 1         189576
> 2         312698
> 4         346744
> 
> The virtio-scsi device model and core SCSI emulation currently assume that
> requests are processed in a single AioContext. This patch series goes about
> modifying this as follows:
> 
> scsi-disk: drop unused SCSIDiskState->bh field
> dma: use current AioContext for dma_blk_io()
> 
> Make dma-helpers.c support the QEMU multi-queue block layer by using
> qemu_get_current_aio_context().
> 
> scsi: track per-SCSIRequest AioContext
> scsi: introduce requests_lock
> 
> Make the core SCSI emulation code support processing requests in multiple
> AioContexts by protecting the per-SCSIDevice requests list.
> 
> virtio-scsi: introduce event and ctrl virtqueue locks
> virtio-scsi: protect events_dropped field
> virtio-scsi: perform TMFs in appropriate AioContexts
> 
> Make the virtio-scsi emulation code support processing requests in multiple
> AioContexts. The event and ctrl virtqueues can interact with multiple
> AioContexts. Especially the SCSI Task Management Functions (TMFs) handled by
> the ctrl virtqueue need to be made thread-safe.
> 
> virtio-blk: extract cleanup_iothread_vq_mapping() function
> virtio-blk: tidy up iothread_vq_mapping functions
> virtio: extract iothread-vq-mapping.h API
> virtio-scsi: add iothread-vq-mapping parameter
> 
> Port over the iothread-vq-mapping qdev property from virtio-blk to virtio-scsi.
> 
> virtio-scsi: handle ctrl virtqueue in main loop
> 
> Simplify TMF handling now that there is no longer a single AioContext where all
> requests are processed.
> 
> Stefan Hajnoczi (12):
>   scsi-disk: drop unused SCSIDiskState->bh field
>   dma: use current AioContext for dma_blk_io()
>   scsi: track per-SCSIRequest AioContext
>   scsi: introduce requests_lock
>   virtio-scsi: introduce event and ctrl virtqueue locks
>   virtio-scsi: protect events_dropped field
>   virtio-scsi: perform TMFs in appropriate AioContexts
>   virtio-blk: extract cleanup_iothread_vq_mapping() function
>   virtio-blk: tidy up iothread_vq_mapping functions
>   virtio: extract iothread-vq-mapping.h API
>   virtio-scsi: add iothread-vq-mapping parameter
>   virtio-scsi: handle ctrl virtqueue in main loop

Patches 1-10: Reviewed-by: Kevin Wolf <kwolf@redhat.com>

Re: [PATCH 00/12] virtio-scsi: add iothread-vq-mapping parameter

Posted by Stefan Hajnoczi 3 weeks, 2 days ago

On Mon, Mar 10, 2025 at 03:43:57PM +0100, Kevin Wolf wrote:
> Am 13.02.2025 um 19:00 hat Stefan Hajnoczi geschrieben:
> > Implement --device virtio-scsi-pci,iothread-vq-mapping= support so that
> > virtqueues can be assigned to different IOThreads. This improves SMP guest
> > scalability where I/O-intensive applications can become bottlenecked on a
> > single IOThread.
> > 
> > The following benchmark results show the effect of iothread-vq-mapping. fio
> > randread 4k iodepth=64 results from a 4 vCPU guest with an Intel P4800X SSD:
> > iothreads IOPS
> > ------------------------------
> > 1         189576
> > 2         312698
> > 4         346744
> > 
> > The virtio-scsi device model and core SCSI emulation currently assume that
> > requests are processed in a single AioContext. This patch series goes about
> > modifying this as follows:
> > 
> > scsi-disk: drop unused SCSIDiskState->bh field
> > dma: use current AioContext for dma_blk_io()
> > 
> > Make dma-helpers.c support the QEMU multi-queue block layer by using
> > qemu_get_current_aio_context().
> > 
> > scsi: track per-SCSIRequest AioContext
> > scsi: introduce requests_lock
> > 
> > Make the core SCSI emulation code support processing requests in multiple
> > AioContexts by protecting the per-SCSIDevice requests list.
> > 
> > virtio-scsi: introduce event and ctrl virtqueue locks
> > virtio-scsi: protect events_dropped field
> > virtio-scsi: perform TMFs in appropriate AioContexts
> > 
> > Make the virtio-scsi emulation code support processing requests in multiple
> > AioContexts. The event and ctrl virtqueues can interact with multiple
> > AioContexts. Especially the SCSI Task Management Functions (TMFs) handled by
> > the ctrl virtqueue need to be made thread-safe.
> > 
> > virtio-blk: extract cleanup_iothread_vq_mapping() function
> > virtio-blk: tidy up iothread_vq_mapping functions
> > virtio: extract iothread-vq-mapping.h API
> > virtio-scsi: add iothread-vq-mapping parameter
> > 
> > Port over the iothread-vq-mapping qdev property from virtio-blk to virtio-scsi.
> > 
> > virtio-scsi: handle ctrl virtqueue in main loop
> > 
> > Simplify TMF handling now that there is no longer a single AioContext where all
> > requests are processed.
> > 
> > Stefan Hajnoczi (12):
> >   scsi-disk: drop unused SCSIDiskState->bh field
> >   dma: use current AioContext for dma_blk_io()
> >   scsi: track per-SCSIRequest AioContext
> >   scsi: introduce requests_lock
> >   virtio-scsi: introduce event and ctrl virtqueue locks
> >   virtio-scsi: protect events_dropped field
> >   virtio-scsi: perform TMFs in appropriate AioContexts
> >   virtio-blk: extract cleanup_iothread_vq_mapping() function
> >   virtio-blk: tidy up iothread_vq_mapping functions
> >   virtio: extract iothread-vq-mapping.h API
> >   virtio-scsi: add iothread-vq-mapping parameter
> >   virtio-scsi: handle ctrl virtqueue in main loop
> 
> Patches 1-10: Reviewed-by: Kevin Wolf <kwolf@redhat.com>

Thank you. I'm preparing another revision that applies a fixed mapping
to the ctrl and event vqs and only exposes the command vqs through the
iothread-vq-mapping property.

Stefan

Re: [PATCH 00/12] virtio-scsi: add iothread-vq-mapping parameter

Posted by Peter Krempa 1 month, 1 week ago

On Thu, Feb 13, 2025 at 13:00:31 -0500, Stefan Hajnoczi wrote:
> Implement --device virtio-scsi-pci,iothread-vq-mapping= support so that
> virtqueues can be assigned to different IOThreads. This improves SMP guest
> scalability where I/O-intensive applications can become bottlenecked on a
> single IOThread.

Hi I was playing around with this along with the patches and I can see
that multiple iothreads do get loaded when I configure the mapping. It
was a bit tricky to spot it though among the 200-odd 'worker' threads
qemu spawned :D (but spawns also without this feature being used).

I didn't do any benchmarks ...

> The following benchmark results show the effect of iothread-vq-mapping. fio
> randread 4k iodepth=64 results from a 4 vCPU guest with an Intel P4800X SSD:
> iothreads IOPS
> ------------------------------
> 1         189576
> 2         312698
> 4         346744

... so I'll trust you here, but at least configuration wise this seems
to work.

As discussed on the libvirt patchset it might be a good idea to
docuement that the ctrl and event queues need to be mapped as well if
you'll be dealing with docs.

Tested-by: Peter Krempa <pkrempa@redhat.com>