aio-posix: towards an O(1) event loop

[PATCH 0/5] aio-posix: towards an O(1) event loop

Posted by Stefan Hajnoczi 4 years, 2 months ago

This patch series makes AioHandler deletion and dispatch O(1) with respect to
the total number of registered handlers.  The event loop has scalability
problems when many AioHandlers are registered because it is O(n).  Linux
epoll(7) is used to avoid scanning over all pollfds but parts of the code still
scan all AioHandlers.

This series reduces QEMU CPU utilization and therefore increases IOPS,
especially for guests that have many devices.  It was tested with 32 vCPUs, 2
virtio-blk,num-queues=1,iothread=iothread1, and 99
virtio-blk,num-queues=32,iothread=iothread1 devices.  Using an IOThread is
necessary because this series does not improve the glib main loop, a non-goal
since the glib API is inherently O(n).

AioContext polling remains O(n) and will be addressed in a separate patch
series.  This patch series increases IOPS from 260k to 300k when AioContext
polling is commented out
(rw=randread,bs=4k,iodepth=32,ioengine=libaio,direct=1).

Stefan Hajnoczi (5):
  aio-posix: fix use after leaving scope in aio_poll()
  aio-posix: don't pass ns timeout to epoll_wait()
  qemu/queue.h: add QLIST_SAFE_REMOVE()
  aio-posix: make AioHandler deletion O(1)
  aio-posix: make AioHandler dispatch O(1) with epoll

 block.c              |   5 +-
 chardev/spice.c      |   4 +-
 include/block/aio.h  |   6 +-
 include/qemu/queue.h |  17 +++++
 util/aio-posix.c     | 172 +++++++++++++++++++++++++++++--------------
 5 files changed, 141 insertions(+), 63 deletions(-)

-- 
2.24.1

Re: [PATCH 0/5] aio-posix: towards an O(1) event loop

Posted by Stefan Hajnoczi 4 years, 2 months ago

On Fri, Feb 14, 2020 at 05:17:07PM +0000, Stefan Hajnoczi wrote:
> This patch series makes AioHandler deletion and dispatch O(1) with respect to
> the total number of registered handlers.  The event loop has scalability
> problems when many AioHandlers are registered because it is O(n).  Linux
> epoll(7) is used to avoid scanning over all pollfds but parts of the code still
> scan all AioHandlers.
> 
> This series reduces QEMU CPU utilization and therefore increases IOPS,
> especially for guests that have many devices.  It was tested with 32 vCPUs, 2
> virtio-blk,num-queues=1,iothread=iothread1, and 99
> virtio-blk,num-queues=32,iothread=iothread1 devices.  Using an IOThread is
> necessary because this series does not improve the glib main loop, a non-goal
> since the glib API is inherently O(n).
> 
> AioContext polling remains O(n) and will be addressed in a separate patch
> series.  This patch series increases IOPS from 260k to 300k when AioContext
> polling is commented out
> (rw=randread,bs=4k,iodepth=32,ioengine=libaio,direct=1).
> 
> Stefan Hajnoczi (5):
>   aio-posix: fix use after leaving scope in aio_poll()
>   aio-posix: don't pass ns timeout to epoll_wait()
>   qemu/queue.h: add QLIST_SAFE_REMOVE()
>   aio-posix: make AioHandler deletion O(1)
>   aio-posix: make AioHandler dispatch O(1) with epoll
> 
>  block.c              |   5 +-
>  chardev/spice.c      |   4 +-
>  include/block/aio.h  |   6 +-
>  include/qemu/queue.h |  17 +++++
>  util/aio-posix.c     | 172 +++++++++++++++++++++++++++++--------------
>  5 files changed, 141 insertions(+), 63 deletions(-)
> 
> -- 
> 2.24.1
> 

Thanks, applied to my block tree:
https://github.com/stefanha/qemu/commits/block

Stefan