On Thu, Mar 05, 2020 at 05:07:59PM +0000, Stefan Hajnoczi wrote:
> A guest with 100 virtio-blk-pci,num-queues=32 devices only reaches 10k IOPS
> while a guest with a single device reaches 105k IOPS
> (rw=randread,bs=4k,iodepth=1,ioengine=libaio).
>
> The bottleneck is that aio_poll() userspace polling iterates over all
> AioHandlers to invoke their ->io_poll() callbacks. All AioHandlers are polled
> even if only one of them was recently active. Therefore a guest with many
> disks is slower than a guest with a single disk even when the workload only
> accesses a single disk.
>
> This patch series solves this scalability problem so that IOPS is unaffected by
> the number of devices. The trick is to poll only AioHandlers that were
> recently active so that userspace polling scales well.
>
> Unfortunately it's not possible to accomplish this with the existing epoll(7)
> fd monitoring implementation. This patch series adds a Linux io_uring fd
> monitoring implementation. The critical feature is that io_uring can check the
> readiness of file descriptors through userspace polling. This makes it
> possible to safely poll a subset of AioHandlers from userspace without risk of
> starving the other AioHandlers.
>
> Stefan Hajnoczi (7):
> aio-posix: completely stop polling when disabled
> aio-posix: move RCU_READ_LOCK() into run_poll_handlers()
> aio-posix: extract ppoll(2) and epoll(7) fd monitoring
> aio-posix: simplify FDMonOps->update() prototype
> aio-posix: add io_uring fd monitoring implementation
> aio-posix: support userspace polling of fd monitoring
> aio-posix: remove idle poll handlers to improve scalability
>
> MAINTAINERS | 2 +
> configure | 5 +
> include/block/aio.h | 70 ++++++-
> util/Makefile.objs | 3 +
> util/aio-posix.c | 449 ++++++++++++++----------------------------
> util/aio-posix.h | 81 ++++++++
> util/fdmon-epoll.c | 155 +++++++++++++++
> util/fdmon-io_uring.c | 332 +++++++++++++++++++++++++++++++
> util/fdmon-poll.c | 107 ++++++++++
> util/trace-events | 2 +
> 10 files changed, 898 insertions(+), 308 deletions(-)
> create mode 100644 util/aio-posix.h
> create mode 100644 util/fdmon-epoll.c
> create mode 100644 util/fdmon-io_uring.c
> create mode 100644 util/fdmon-poll.c
>
> --
> 2.24.1
>
Thanks, applied to my block tree:
https://github.com/stefanha/qemu/commits/block
Stefan