This series of performance enhancements brings iothread and polling
capabilities to qemu-nvme. The first two patches implements support
for irqfd, which solves thread safety problems for interrupt emulation
outside the main loop thread. The third patch adds support for
emulating NVMe IO queues in a dedicated iothread, which avoid
interference from other devices running on the main loop thread.
The fourth patch implements SQ and CQ pollers, thus enabling polling
on SQ and CQ.

After all these optimizations, our performance becomes similar to
virtio-blk.

Comparison (KIOPS):

QD                    1   4  16  64
virtio-blk           59 185 260 256
nvme                 53 155 245 309
virtio-blk-polling   88 212 210 213
nvme-polling        123 165 189 191

Changes since v2:
 - Add polling support
 - Do not set up MSI-X masking handlers when irq-eventfd is off

Changes since v1:
 - Avoid duplicate initilization of cq timer

Jinhao Fan (4):
  hw/nvme: support irq(de)assertion with eventfd
  hw/nvme: use KVM irqfd when available
  hw/nvme: add iothread support
  hw/nvme: add polling support

 hw/nvme/ctrl.c       | 394 ++++++++++++++++++++++++++++++++++++++++---
 hw/nvme/ns.c         |  21 ++-
 hw/nvme/nvme.h       |  13 +-
 hw/nvme/trace-events |   3 +
 4 files changed, 401 insertions(+), 30 deletions(-)

-- 
2.25.1

On Aug 27 17:12, Jinhao Fan wrote:
> This series of performance enhancements brings iothread and polling
> capabilities to qemu-nvme. The first two patches implements support
> for irqfd, which solves thread safety problems for interrupt emulation
> outside the main loop thread. The third patch adds support for
> emulating NVMe IO queues in a dedicated iothread, which avoid
> interference from other devices running on the main loop thread.
> The fourth patch implements SQ and CQ pollers, thus enabling polling
> on SQ and CQ.
> 
> After all these optimizations, our performance becomes similar to
> virtio-blk.
> 
> Comparison (KIOPS):
> 
> QD                    1   4  16  64
> virtio-blk           59 185 260 256
> nvme                 53 155 245 309
> virtio-blk-polling   88 212 210 213
> nvme-polling        123 165 189 191
> 

While reviewing this I noticed that the commit that introduced the
ioeventfd, we end up disabling batching of cqes (that is, we post and
interrupt for *each* cqe).

I'll post a patch to rectify that. With that, I am seeing progressive
performance improvements with the patches in this series.