[v3] virtiofs dax patches

[PATCH v3 00/26] virtiofs dax patches

Posted by Dr. David Alan Gilbert (git) 2 years, 12 months ago

From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>

  This series adds support for acceleration of virtiofs via DAX
mapping, using features added in the 5.11 Linux kernel.

  DAX originally existed in the kernel for mapping real storage
devices directly into memory, so that reads/writes turn into
reads/writes directly mapped into the storage device.

  virtiofs's DAX support is similar; a PCI BAR is exposed on the
virtiofs device corresponding to a DAX 'cache' of a user defined size.
The guest daemon then requests files to be mapped into that cache;
when that happens the virtiofsd sends filedescriptors and commands back
to the QEMU that mmap's those files directly into the memory slot
exposed to kvm.  The guest can then directly read/write to the files
exposed by virtiofs by reading/writing into the BAR.

  A typical invocation would be:
     -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs,cache-size=4G

and then the guest must mount with -o dax

  Note that the cache doesn't really take VM up on the host, because
everything placed there is just an mmap of a file, so you can afford
to use quite a large cache size.

  Unlike a real DAX device, the cache is a finite size that's
potentially smaller than the underlying filesystem (especially when
mapping granuality is taken into account).  Mapping, unmapping and
remapping must take place to juggle files into the cache if it's too
small.  Some workloads benefit more than others.

Gotchas:
  a) If something else on the host truncates an mmap'd file,
kvm gets rather upset;  for this reason it's advised that DAX is
currently only suitable for use on non-shared filesystems.

(This series, with a couple of other patches, is at:
https://gitlab.com/virtio-fs/qemu/-/tree/dgilbert-dax-2021-04-28 )

Dave

v3
  Review cleanups
  Fix some printf formating issues

Dr. David Alan Gilbert (21):
  virtiofs: Fixup printf args
  virtiofsd: Don't assume header layout
  DAX: vhost-user: Rework slave return values
  DAX: libvhost-user: Route slave message payload
  DAX: libvhost-user: Allow popping a queue element with bad pointers
  DAX subprojects/libvhost-user: Add virtio-fs slave types
  DAX: virtio: Add shared memory capability
  DAX: virtio-fs: Add cache BAR
  DAX: virtio-fs: Add vhost-user slave commands for mapping
  DAX: virtio-fs: Fill in slave commands for mapping
  DAX: virtiofsd Add cache accessor functions
  DAX: virtiofsd: Add setup/remove mappings fuse commands
  DAX: virtiofsd: Add setup/remove mapping handlers to passthrough_ll
  DAX: virtiofsd: Wire up passthrough_ll's lo_setupmapping
  DAX: virtiofsd: route se down to destroy method
  DAX: virtiofsd: Perform an unmap on destroy
  DAX/unmap: virtiofsd: Add VHOST_USER_SLAVE_FS_IO
  DAX/unmap virtiofsd: Add wrappers for VHOST_USER_SLAVE_FS_IO
  DAX/unmap virtiofsd: Parse unmappable elements
  DAX/unmap virtiofsd: Route unmappable reads
  DAX/unmap virtiofsd: route unmappable write to slave command

Stefan Hajnoczi (1):
  DAX:virtiofsd: implement FUSE_INIT map_alignment field

Vivek Goyal (4):
  DAX: virtiofsd: Make lo_removemapping() work
  vhost-user-fs: Extend VhostUserFSSlaveMsg to pass additional info
  vhost-user-fs: Implement drop CAP_FSETID functionality
  virtiofsd: Ask qemu to drop CAP_FSETID if client asked for it

 block/export/vhost-user-blk-server.c      |   2 +-
 contrib/vhost-user-blk/vhost-user-blk.c   |   3 +-
 contrib/vhost-user-gpu/vhost-user-gpu.c   |   5 +-
 contrib/vhost-user-input/main.c           |   4 +-
 contrib/vhost-user-scsi/vhost-user-scsi.c |   2 +-
 docs/interop/vhost-user.rst               |  37 ++
 hw/virtio/meson.build                     |   1 +
 hw/virtio/trace-events                    |   6 +
 hw/virtio/vhost-backend.c                 |   6 +-
 hw/virtio/vhost-user-fs-pci.c             |  32 ++
 hw/virtio/vhost-user-fs.c                 | 395 ++++++++++++++++++++++
 hw/virtio/vhost-user.c                    |  62 +++-
 hw/virtio/virtio-pci.c                    |  20 ++
 hw/virtio/virtio-pci.h                    |   4 +
 include/hw/virtio/vhost-backend.h         |   2 +-
 include/hw/virtio/vhost-user-fs.h         |  43 +++
 meson.build                               |   6 +
 subprojects/libvhost-user/libvhost-user.c | 113 ++++++-
 subprojects/libvhost-user/libvhost-user.h |  57 +++-
 tests/vhost-user-bridge.c                 |   4 +-
 tools/virtiofsd/buffer.c                  |  22 +-
 tools/virtiofsd/fuse_common.h             |  17 +-
 tools/virtiofsd/fuse_lowlevel.c           |  92 ++++-
 tools/virtiofsd/fuse_lowlevel.h           |  78 ++++-
 tools/virtiofsd/fuse_virtio.c             | 372 ++++++++++++++++----
 tools/virtiofsd/passthrough_ll.c          | 138 +++++++-
 26 files changed, 1393 insertions(+), 130 deletions(-)

-- 
2.31.1

Re: [PATCH v3 00/26] virtiofs dax patches

Posted by no-reply@patchew.org 2 years, 12 months ago

Patchew URL: https://patchew.org/QEMU/20210428110100.27757-1-dgilbert@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20210428110100.27757-1-dgilbert@redhat.com
Subject: [PATCH v3 00/26] virtiofs dax patches

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 - [tag update]      patchew/20210415102321.3987935-1-philmd@redhat.com -> patchew/20210415102321.3987935-1-philmd@redhat.com
 - [tag update]      patchew/20210422145335.65814-1-mreitz@redhat.com -> patchew/20210422145335.65814-1-mreitz@redhat.com
 - [tag update]      patchew/20210427135147.111218-1-lvivier@redhat.com -> patchew/20210427135147.111218-1-lvivier@redhat.com
 - [tag update]      patchew/20210427192658.266933-1-f4bug@amsat.org -> patchew/20210427192658.266933-1-f4bug@amsat.org
 * [new tag]         patchew/20210428110100.27757-1-dgilbert@redhat.com -> patchew/20210428110100.27757-1-dgilbert@redhat.com
Switched to a new branch 'test'
ccf0714 virtiofsd: Ask qemu to drop CAP_FSETID if client asked for it
dae0067 vhost-user-fs: Implement drop CAP_FSETID functionality
addc004 vhost-user-fs: Extend VhostUserFSSlaveMsg to pass additional info
bf4ce2a DAX:virtiofsd: implement FUSE_INIT map_alignment field
188f074 DAX/unmap virtiofsd: route unmappable write to slave command
9bcd730 DAX/unmap virtiofsd: Route unmappable reads
b95bb5c DAX/unmap virtiofsd: Parse unmappable elements
e98bda0 DAX/unmap virtiofsd: Add wrappers for VHOST_USER_SLAVE_FS_IO
f8bc115 DAX/unmap: virtiofsd: Add VHOST_USER_SLAVE_FS_IO
e563a49 DAX: virtiofsd: Perform an unmap on destroy
4977fed DAX: virtiofsd: route se down to destroy method
e86d999 DAX: virtiofsd: Make lo_removemapping() work
b43ed18 DAX: virtiofsd: Wire up passthrough_ll's lo_setupmapping
80b6613 DAX: virtiofsd: Add setup/remove mapping handlers to passthrough_ll
497a551 DAX: virtiofsd: Add setup/remove mappings fuse commands
a68ac68 DAX: virtiofsd Add cache accessor functions
47fd841 DAX: virtio-fs: Fill in slave commands for mapping
e22b4f2 DAX: virtio-fs: Add vhost-user slave commands for mapping
43cefc1 DAX: virtio-fs: Add cache BAR
ee3d0da DAX: virtio: Add shared memory capability
9532b91 DAX subprojects/libvhost-user: Add virtio-fs slave types
7a510ed DAX: libvhost-user: Allow popping a queue element with bad pointers
79824d5 DAX: libvhost-user: Route slave message payload
4d503d0 DAX: vhost-user: Rework slave return values
d18fd97 virtiofsd: Don't assume header layout
301ba24 virtiofs: Fixup printf args

=== OUTPUT BEGIN ===
1/26 Checking commit 301ba247ecca (virtiofs: Fixup printf args)
2/26 Checking commit d18fd977f0c5 (virtiofsd: Don't assume header layout)
3/26 Checking commit 4d503d03bfb6 (DAX: vhost-user: Rework slave return values)
4/26 Checking commit 79824d57ac9c (DAX: libvhost-user: Route slave message payload)
5/26 Checking commit 7a510edf1169 (DAX: libvhost-user: Allow popping a queue element with bad pointers)
6/26 Checking commit 9532b917fec4 (DAX subprojects/libvhost-user: Add virtio-fs slave types)
7/26 Checking commit ee3d0daebc76 (DAX: virtio: Add shared memory capability)
8/26 Checking commit 43cefc18bd9b (DAX: virtio-fs: Add cache BAR)
9/26 Checking commit e22b4f256a22 (DAX: virtio-fs: Add vhost-user slave commands for mapping)
10/26 Checking commit 47fd84136d6f (DAX: virtio-fs: Fill in slave commands for mapping)
11/26 Checking commit a68ac68dcf41 (DAX: virtiofsd Add cache accessor functions)
12/26 Checking commit 497a5518a931 (DAX: virtiofsd: Add setup/remove mappings fuse commands)
13/26 Checking commit 80b661353cd4 (DAX: virtiofsd: Add setup/remove mapping handlers to passthrough_ll)
14/26 Checking commit b43ed18535c4 (DAX: virtiofsd: Wire up passthrough_ll's lo_setupmapping)
15/26 Checking commit e86d999fc18c (DAX: virtiofsd: Make lo_removemapping() work)
16/26 Checking commit 4977fed192e3 (DAX: virtiofsd: route se down to destroy method)
17/26 Checking commit e563a49d4b2a (DAX: virtiofsd: Perform an unmap on destroy)
18/26 Checking commit f8bc115092d8 (DAX/unmap: virtiofsd: Add VHOST_USER_SLAVE_FS_IO)
19/26 Checking commit e98bda0c581d (DAX/unmap virtiofsd: Add wrappers for VHOST_USER_SLAVE_FS_IO)
ERROR: unnecessary whitespace before a quoted newline
#100: FILE: tools/virtiofsd/fuse_virtio.c:1183:
+    fuse_log(FUSE_LOG_DEBUG, "%s: result=%" PRId64 " \n", __func__, result);

total: 1 errors, 0 warnings, 75 lines checked

Patch 19/26 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

20/26 Checking commit b95bb5c22789 (DAX/unmap virtiofsd: Parse unmappable elements)
21/26 Checking commit 9bcd730dfe66 (DAX/unmap virtiofsd: Route unmappable reads)
22/26 Checking commit 188f074554d3 (DAX/unmap virtiofsd: route unmappable write to slave command)
23/26 Checking commit bf4ce2af375c (DAX:virtiofsd: implement FUSE_INIT map_alignment field)
24/26 Checking commit addc0047af4a (vhost-user-fs: Extend VhostUserFSSlaveMsg to pass additional info)
25/26 Checking commit dae00676a878 (vhost-user-fs: Implement drop CAP_FSETID functionality)
26/26 Checking commit ccf0714484af (virtiofsd: Ask qemu to drop CAP_FSETID if client asked for it)
ERROR: unnecessary whitespace before a quoted newline
#125: FILE: tools/virtiofsd/fuse_virtio.c:1319:
     fuse_log(FUSE_LOG_DEBUG, "%s: result=%" PRId64 " \n", __func__, result);

total: 1 errors, 0 warnings, 88 lines checked

Patch 26/26 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20210428110100.27757-1-dgilbert@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [PATCH v3 00/26] virtiofs dax patches

Posted by Stefan Hajnoczi 2 years, 11 months ago

On Wed, Apr 28, 2021 at 12:00:34PM +0100, Dr. David Alan Gilbert (git) wrote:
> From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> 
>   This series adds support for acceleration of virtiofs via DAX
> mapping, using features added in the 5.11 Linux kernel.
> 
>   DAX originally existed in the kernel for mapping real storage
> devices directly into memory, so that reads/writes turn into
> reads/writes directly mapped into the storage device.
> 
>   virtiofs's DAX support is similar; a PCI BAR is exposed on the
> virtiofs device corresponding to a DAX 'cache' of a user defined size.
> The guest daemon then requests files to be mapped into that cache;
> when that happens the virtiofsd sends filedescriptors and commands back
> to the QEMU that mmap's those files directly into the memory slot
> exposed to kvm.  The guest can then directly read/write to the files
> exposed by virtiofs by reading/writing into the BAR.
> 
>   A typical invocation would be:
>      -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs,cache-size=4G
> 
> and then the guest must mount with -o dax
> 
>   Note that the cache doesn't really take VM up on the host, because
> everything placed there is just an mmap of a file, so you can afford
> to use quite a large cache size.
> 
>   Unlike a real DAX device, the cache is a finite size that's
> potentially smaller than the underlying filesystem (especially when
> mapping granuality is taken into account).  Mapping, unmapping and
> remapping must take place to juggle files into the cache if it's too
> small.  Some workloads benefit more than others.
> 
> Gotchas:
>   a) If something else on the host truncates an mmap'd file,
> kvm gets rather upset;  for this reason it's advised that DAX is
> currently only suitable for use on non-shared filesystems.
> 
> (This series, with a couple of other patches, is at:
> https://gitlab.com/virtio-fs/qemu/-/tree/dgilbert-dax-2021-04-28 )

Overall this looks close but I don't think the FSETID support should be
added to QEMU. Please see my comment on the final patch.

Stefan