[PATCH 000/104] virtiofs daemon [all]

Dr. David Alan Gilbert (git) posted 104 patches 4 years, 3 months ago
Failed in applying to current master (apply log)
Test asan failed
Test checkpatch failed
Test FreeBSD failed
Test docker-mingw@fedora failed
Test docker-clang@ubuntu failed
Test docker-quick@centos7 failed
There is a newer version of this series
.gitignore                                |    1 +
MAINTAINERS                               |    8 +
Makefile                                  |   20 +
Makefile.objs                             |    1 +
configure                                 |   16 +
contrib/libvhost-user/libvhost-user.c     |   57 +-
contrib/libvhost-user/libvhost-user.h     |    6 +
docs/interop/vhost-user.json              |    4 +-
hw/virtio/vhost-user.c                    |    2 +-
include/standard-headers/linux/fuse.h     |  891 ++++++
scripts/update-linux-headers.sh           |    1 +
tools/virtiofsd/50-qemu-virtiofsd.json.in |    5 +
tools/virtiofsd/Makefile.objs             |   13 +
tools/virtiofsd/buffer.c                  |  422 +++
tools/virtiofsd/fuse.h                    | 1287 +++++++++
tools/virtiofsd/fuse_common.h             |  884 ++++++
tools/virtiofsd/fuse_i.h                  |  134 +
tools/virtiofsd/fuse_log.c                |   44 +
tools/virtiofsd/fuse_log.h                |   74 +
tools/virtiofsd/fuse_loop_mt.c            |   56 +
tools/virtiofsd/fuse_lowlevel.c           | 2782 +++++++++++++++++++
tools/virtiofsd/fuse_lowlevel.h           | 2043 ++++++++++++++
tools/virtiofsd/fuse_misc.h               |   60 +
tools/virtiofsd/fuse_opt.c                |  449 +++
tools/virtiofsd/fuse_opt.h                |  272 ++
tools/virtiofsd/fuse_signals.c            |   98 +
tools/virtiofsd/fuse_virtio.c             |  968 +++++++
tools/virtiofsd/fuse_virtio.h             |   33 +
tools/virtiofsd/helper.c                  |  333 +++
tools/virtiofsd/passthrough_helpers.h     |   51 +
tools/virtiofsd/passthrough_ll.c          | 2998 +++++++++++++++++++++
tools/virtiofsd/seccomp.c                 |  155 ++
tools/virtiofsd/seccomp.h                 |   16 +
tools/virtiofsd/security.rst              |  118 +
tools/virtiofsd/virtiofsd.texi            |   85 +
35 files changed, 14373 insertions(+), 14 deletions(-)
create mode 100644 include/standard-headers/linux/fuse.h
create mode 100644 tools/virtiofsd/50-qemu-virtiofsd.json.in
create mode 100644 tools/virtiofsd/Makefile.objs
create mode 100644 tools/virtiofsd/buffer.c
create mode 100644 tools/virtiofsd/fuse.h
create mode 100644 tools/virtiofsd/fuse_common.h
create mode 100644 tools/virtiofsd/fuse_i.h
create mode 100644 tools/virtiofsd/fuse_log.c
create mode 100644 tools/virtiofsd/fuse_log.h
create mode 100644 tools/virtiofsd/fuse_loop_mt.c
create mode 100644 tools/virtiofsd/fuse_lowlevel.c
create mode 100644 tools/virtiofsd/fuse_lowlevel.h
create mode 100644 tools/virtiofsd/fuse_misc.h
create mode 100644 tools/virtiofsd/fuse_opt.c
create mode 100644 tools/virtiofsd/fuse_opt.h
create mode 100644 tools/virtiofsd/fuse_signals.c
create mode 100644 tools/virtiofsd/fuse_virtio.c
create mode 100644 tools/virtiofsd/fuse_virtio.h
create mode 100644 tools/virtiofsd/helper.c
create mode 100644 tools/virtiofsd/passthrough_helpers.h
create mode 100644 tools/virtiofsd/passthrough_ll.c
create mode 100644 tools/virtiofsd/seccomp.c
create mode 100644 tools/virtiofsd/seccomp.h
create mode 100644 tools/virtiofsd/security.rst
create mode 100644 tools/virtiofsd/virtiofsd.texi
[PATCH 000/104] virtiofs daemon [all]
Posted by Dr. David Alan Gilbert (git) 4 years, 3 months ago
From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>

Hi,
  This is a full set for virtiofsd - a daemon
that implements the user space side of virtiofs.

Unlike my previous post, this is a full set rather
than split up into base/security etc.

The set pulls in a big chunk of the upstream libfuse library
(unmodified so that it's easy to check it really is upstream),
chops all the stuff out we don't need and then adds the
new transport we need.

I've formatted everything into qemu's code style - using indent
and the clang tools for the files I've imported.  I've cleared all
reviewed-by's given how much I've rebased and tweeked it.

We can't just link with libfuse, since we have to make ABI incompatible
changes for the new transport and it's quite invasive; the library is
designed to be the basis for multiple filesystems, but all on the same
transport.

Running this daemon is typically done with:

   ./virtiofsd -o vhost_user_socket=/path/socket -o source=/path/to/fs

connected to a qemu that's then started with:
   -chardev socket,id=char0,path=/path/socket -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs

and then in the guest mount with:
   mount -t virtiofs myfs /mnt

Our development branch is: https://gitlab.com/virtio-fs/qemu/tree/virtio-fs-dev

This code is going into tools/virtiofsd  based on the previous long
discussion.
It relies on Paolo's 'build: rename CONFIG_LIBCAP to
CONFIG_LIBCAP_NG' patch.

Breakdown:
  Patches 1..12: Importing upstream libfuse
         13..31: Basic wiring to get vhost transport
         32..56: Security changes
         57..  : Individual fixes

Thank you to all those who have contributed code.

Dave

Dr. David Alan Gilbert (34):
  virtiofsd: Pull in upstream headers
  virtiofsd: Pull in kernel's fuse.h
  virtiofsd: Add auxiliary .c's
  virtiofsd: Add fuse_lowlevel.c
  virtiofsd: Add passthrough_ll
  virtiofsd: Trim down imported files
  virtiofsd: Format imported files to qemu style
  virtiofsd: Fix fuse_daemonize ignored return values
  virtiofsd: Fix common header and define for QEMU builds
  virtiofsd: Trim out compatibility code
  virtiofsd: Add options for virtio
  virtiofsd: Open vhost connection instead of mounting
  virtiofsd: Start wiring up vhost-user
  virtiofsd: Add main virtio loop
  virtiofsd: get/set features callbacks
  virtiofsd: Start queue threads
  virtiofsd: Poll kick_fd for queue
  virtiofsd: Start reading commands from queue
  virtiofsd: Send replies to messages
  virtiofsd: Keep track of replies
  virtiofsd: Add Makefile wiring for virtiofsd contrib
  virtiofsd: Fast path for virtio read
  virtiofs: Add maintainers entry
  virtiofsd: Plumb fuse_bufvec through to do_write_buf
  virtiofsd: Pass write iov's all the way through
  virtiofsd: cap-ng helpers
  virtiofsd: Handle reinit
  virtiofsd: Handle hard reboot
  virtiofsd: Kill threads when queues are stopped
  vhost-user: Print unexpected slave message types
  contrib/libvhost-user: Protect slave fd with mutex
  virtiofsd: Clean up inodes on destroy
  libvhost-user: Fix some memtable remap cases
  virtiofsd: Convert lo_destroy to take the lo->mutex lock itself

Eric Ren (1):
  virtiofsd: fix incorrect error handling in lo_do_lookup

Eryu Guan (2):
  virtiofsd: print log only when priority is high enough
  virtiofsd: convert more fprintf and perror to use fuse log infra

Jiufei Xue (1):
  virtiofsd: support nanosecond resolution for file timestamp

Liu Bo (6):
  virtiofsd: fix error handling in main()
  virtiofsd: cleanup allocated resource in se
  virtiofsd: fix memory leak on lo.source
  virtiofsd: add helper for lo_data cleanup
  virtiofsd: enable PARALLEL_DIROPS during INIT
  Virtiofsd: fix memory leak on fuse queueinfo

Masayoshi Mizuma (3):
  virtiofsd: Add ID to the log with FUSE_LOG_DEBUG level
  virtiofsd: Add timestamp to the log with FUSE_LOG_DEBUG level
  virtiofsd: Prevent multiply running with same vhost_user_socket

Miklos Szeredi (10):
  virtiofsd: passthrough_ll: add fallback for racy ops
  virtiofsd: passthrough_ll: add renameat2 support
  virtiofsd: passthrough_ll: disable readdirplus on cache=never
  virtiofsd: passthrough_ll: control readdirplus
  virtiofsd: rename unref_inode() to unref_inode_lolocked()
  virtiofsd: fail when parent inode isn't known in lo_do_lookup()
  virtiofsd: extract root inode init into setup_root()
  virtiofsd: passthrough_ll: fix refcounting on remove/rename
  virtiofsd: passthrough_ll: clean up cache related options
  virtiofsd: passthrough_ll: use hashtable

Misono Tomohiro (1):
  virtiofsd: Fix data corruption with O_APPEND wirte in writeback mode

Peng Tao (1):
  virtiofsd: do not always set FUSE_FLOCK_LOCKS

Stefan Hajnoczi (37):
  virtiofsd: remove mountpoint dummy argument
  virtiofsd: remove unused notify reply support
  virtiofsd: add -o source=PATH to help output
  virtiofsd: add --fd=FDNUM fd passing option
  virtiofsd: make -f (foreground) the default
  virtiofsd: add vhost-user.json file
  virtiofsd: add --print-capabilities option
  virtiofsd: passthrough_ll: add lo_map for ino/fh indirection
  virtiofsd: passthrough_ll: add ino_map to hide lo_inode pointers
  virtiofsd: passthrough_ll: add dirp_map to hide lo_dirp pointers
  virtiofsd: passthrough_ll: add fd_map to hide file descriptors
  virtiofsd: validate path components
  virtiofsd: add fuse_mbuf_iter API
  virtiofsd: validate input buffer sizes in do_write_buf()
  virtiofsd: check input buffer size in fuse_lowlevel.c ops
  virtiofsd: prevent ".." escape in lo_do_lookup()
  virtiofsd: prevent ".." escape in lo_do_readdir()
  virtiofsd: use /proc/self/fd/ O_PATH file descriptor
  virtiofsd: sandbox mount namespace
  virtiofsd: move to an empty network namespace
  virtiofsd: move to a new pid namespace
  virtiofsd: add seccomp whitelist
  virtiofsd: set maximum RLIMIT_NOFILE limit
  virtiofsd: fix libfuse information leaks
  virtiofsd: add security guide document
  virtiofsd: add --syslog command-line option
  virtiofsd: use fuse_lowlevel_is_virtio() in fuse_session_destroy()
  virtiofsd: prevent fv_queue_thread() vs virtio_loop() races
  virtiofsd: make lo_release() atomic
  virtiofsd: prevent races with lo_dirp_put()
  virtiofsd: rename inode->refcount to inode->nlookup
  virtiofsd: add man page
  virtiofsd: introduce inode refcount to prevent use-after-free
  virtiofsd: process requests in a thread pool
  virtiofsd: prevent FUSE_INIT/FUSE_DESTROY races
  virtiofsd: fix lo_destroy() resource leaks
  virtiofsd: add --thread-pool-size=NUM option

Vivek Goyal (6):
  virtiofsd: Make fsync work even if only inode is passed in
  virtiofsd: passthrough_ll: create new files in caller's context
  virtiofsd: Parse flag FUSE_WRITE_KILL_PRIV
  virtiofsd: Drop CAP_FSETID if client asked for it
  virtiofsd: Support remote posix locks
  virtiofsd: Reset O_DIRECT flag during file open

piaojun (2):
  virtiofsd: add definition of fuse_buf_writev()
  virtiofsd: use fuse_buf_writev to replace fuse_buf_write for better
    performance

 .gitignore                                |    1 +
 MAINTAINERS                               |    8 +
 Makefile                                  |   20 +
 Makefile.objs                             |    1 +
 configure                                 |   16 +
 contrib/libvhost-user/libvhost-user.c     |   57 +-
 contrib/libvhost-user/libvhost-user.h     |    6 +
 docs/interop/vhost-user.json              |    4 +-
 hw/virtio/vhost-user.c                    |    2 +-
 include/standard-headers/linux/fuse.h     |  891 ++++++
 scripts/update-linux-headers.sh           |    1 +
 tools/virtiofsd/50-qemu-virtiofsd.json.in |    5 +
 tools/virtiofsd/Makefile.objs             |   13 +
 tools/virtiofsd/buffer.c                  |  422 +++
 tools/virtiofsd/fuse.h                    | 1287 +++++++++
 tools/virtiofsd/fuse_common.h             |  884 ++++++
 tools/virtiofsd/fuse_i.h                  |  134 +
 tools/virtiofsd/fuse_log.c                |   44 +
 tools/virtiofsd/fuse_log.h                |   74 +
 tools/virtiofsd/fuse_loop_mt.c            |   56 +
 tools/virtiofsd/fuse_lowlevel.c           | 2782 +++++++++++++++++++
 tools/virtiofsd/fuse_lowlevel.h           | 2043 ++++++++++++++
 tools/virtiofsd/fuse_misc.h               |   60 +
 tools/virtiofsd/fuse_opt.c                |  449 +++
 tools/virtiofsd/fuse_opt.h                |  272 ++
 tools/virtiofsd/fuse_signals.c            |   98 +
 tools/virtiofsd/fuse_virtio.c             |  968 +++++++
 tools/virtiofsd/fuse_virtio.h             |   33 +
 tools/virtiofsd/helper.c                  |  333 +++
 tools/virtiofsd/passthrough_helpers.h     |   51 +
 tools/virtiofsd/passthrough_ll.c          | 2998 +++++++++++++++++++++
 tools/virtiofsd/seccomp.c                 |  155 ++
 tools/virtiofsd/seccomp.h                 |   16 +
 tools/virtiofsd/security.rst              |  118 +
 tools/virtiofsd/virtiofsd.texi            |   85 +
 35 files changed, 14373 insertions(+), 14 deletions(-)
 create mode 100644 include/standard-headers/linux/fuse.h
 create mode 100644 tools/virtiofsd/50-qemu-virtiofsd.json.in
 create mode 100644 tools/virtiofsd/Makefile.objs
 create mode 100644 tools/virtiofsd/buffer.c
 create mode 100644 tools/virtiofsd/fuse.h
 create mode 100644 tools/virtiofsd/fuse_common.h
 create mode 100644 tools/virtiofsd/fuse_i.h
 create mode 100644 tools/virtiofsd/fuse_log.c
 create mode 100644 tools/virtiofsd/fuse_log.h
 create mode 100644 tools/virtiofsd/fuse_loop_mt.c
 create mode 100644 tools/virtiofsd/fuse_lowlevel.c
 create mode 100644 tools/virtiofsd/fuse_lowlevel.h
 create mode 100644 tools/virtiofsd/fuse_misc.h
 create mode 100644 tools/virtiofsd/fuse_opt.c
 create mode 100644 tools/virtiofsd/fuse_opt.h
 create mode 100644 tools/virtiofsd/fuse_signals.c
 create mode 100644 tools/virtiofsd/fuse_virtio.c
 create mode 100644 tools/virtiofsd/fuse_virtio.h
 create mode 100644 tools/virtiofsd/helper.c
 create mode 100644 tools/virtiofsd/passthrough_helpers.h
 create mode 100644 tools/virtiofsd/passthrough_ll.c
 create mode 100644 tools/virtiofsd/seccomp.c
 create mode 100644 tools/virtiofsd/seccomp.h
 create mode 100644 tools/virtiofsd/security.rst
 create mode 100644 tools/virtiofsd/virtiofsd.texi

-- 
2.23.0


Re: [PATCH 000/104] virtiofs daemon [all]
Posted by no-reply@patchew.org 4 years, 3 months ago
Patchew URL: https://patchew.org/QEMU/20191212163904.159893-1-dgilbert@redhat.com/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===




The full log is available at
http://patchew.org/logs/20191212163904.159893-1-dgilbert@redhat.com/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Re: [PATCH 000/104] virtiofs daemon [all]
Posted by Dr. David Alan Gilbert 4 years, 2 months ago
Hi,
  Here's a summary of the review status of this set,

Total: 109 Reviewed: 81 New: 5  changed/rr: 8

The first column is:
  'blank' - no change
  'D' - some diff from the original patch
        - from a simple diff script
  'N' - a new patch

The second column is:
  'R' - has a Reviewed-by
  'c' - Someone reviewed it and asked for changes
  ' ' - no one reviewed it

That leaves ~15 patches from this post that haven't
had reviews (and aren't new).

As soon as most of those are reviewed I'll rebase and repost.

Dave

  R : virtiofsd-Pull-in-upstream-headers
  R : virtiofsd-Pull-in-kernel-s-fuse.h
  R : virtiofsd-Add-auxiliary-.c-s
  R : virtiofsd-Add-fuse_lowlevel.c
D R : virtiofsd-Add-passthrough_ll
D R : virtiofsd-Trim-down-imported-files
D R : virtiofsd-Format-imported-files-to-qemu-style
  R : virtiofsd-remove-mountpoint-dummy-argument
  R : virtiofsd-remove-unused-notify-reply-support
  R : virtiofsd-Fix-fuse_daemonize-ignored-return-value
D c : virtiofsd-Fix-common-header-and-define-for-QEMU-b
  R : virtiofsd-Trim-out-compatibility-code
N   : vitriofsd-passthrough_ll-fix-fallocate-ifdefs
  R : virtiofsd-Make-fsync-work-even-if-only-inode-is-p
D c : virtiofsd-Add-options-for-virtio
  R : virtiofsd-add-o-source-PATH-to-help-output
  R : virtiofsd-Open-vhost-connection-instead-of-mounti
  R : virtiofsd-Start-wiring-up-vhost-user
  R : virtiofsd-Add-main-virtio-loop
  R : virtiofsd-get-set-features-callbacks
  R : virtiofsd-Start-queue-threads
  R : virtiofsd-Poll-kick_fd-for-queue
  R : virtiofsd-Start-reading-commands-from-queue
  R : virtiofsd-Send-replies-to-messages
  R : virtiofsd-Keep-track-of-replies
  R : virtiofsd-Add-Makefile-wiring-for-virtiofsd-contr
    : virtiofsd-Fast-path-for-virtio-read
  R : virtiofsd-add-fd-FDNUM-fd-passing-option
  R : virtiofsd-make-f-foreground-the-default
  R : virtiofsd-add-vhost-user.json-file
  R : virtiofsd-add-print-capabilities-option
  R : virtiofs-Add-maintainers-entry
D c : virtiofsd-passthrough_ll-create-new-files-in-call
    : virtiofsd-passthrough_ll-add-lo_map-for-ino-fh-in
    : virtiofsd-passthrough_ll-add-ino_map-to-hide-lo_i
    : virtiofsd-passthrough_ll-add-dirp_map-to-hide-lo_
D   : virtiofsd-passthrough_ll-add-fd_map-to-hide-file-
D   : virtiofsd-passthrough_ll-add-fallback-for-racy-op
  R : virtiofsd-validate-path-components
    : virtiofsd-Plumb-fuse_bufvec-through-to-do_write_b
    : virtiofsd-Pass-write-iov-s-all-the-way-through
    : virtiofsd-add-fuse_mbuf_iter-API
  R : virtiofsd-validate-input-buffer-sizes-in-do_write
  R : virtiofsd-check-input-buffer-size-in-fuse_lowleve
  R : virtiofsd-prevent-.-escape-in-lo_do_lookup
  R : virtiofsd-prevent-.-escape-in-lo_do_readdir
D R : virtiofsd-use-proc-self-fd-O_PATH-file-descriptor
  R : virtiofsd-sandbox-mount-namespace
  R : virtiofsd-move-to-an-empty-network-namespace
  R : virtiofsd-move-to-a-new-pid-namespace
D c : virtiofsd-add-seccomp-whitelist
  R : virtiofsd-Parse-flag-FUSE_WRITE_KILL_PRIV
  R : virtiofsd-cap-ng-helpers
  R : virtiofsd-Drop-CAP_FSETID-if-client-asked-for-it
  R : virtiofsd-set-maximum-RLIMIT_NOFILE-limit
  R : virtiofsd-fix-libfuse-information-leaks
N   : docs-Add-docs-tools
D R : virtiofsd-add-security-guide-document
  R : virtiofsd-add-syslog-command-line-option
D R : virtiofsd-print-log-only-when-priority-is-high-en
D c : virtiofsd-Add-ID-to-the-log-with-FUSE_LOG_DEBUG-l
D   : virtiofsd-Add-timestamp-to-the-log-with-FUSE_LOG_
  R : virtiofsd-Handle-reinit
  R : virtiofsd-Handle-hard-reboot
D R : virtiofsd-Kill-threads-when-queues-are-stopped
  R : vhost-user-Print-unexpected-slave-message-types
  R : contrib-libvhost-user-Protect-slave-fd-with-mutex
D c : virtiofsd-passthrough_ll-add-renameat2-support
  R : virtiofsd-passthrough_ll-disable-readdirplus-on-c
D   : virtiofsd-passthrough_ll-control-readdirplus
  R : virtiofsd-rename-unref_inode-to-unref_inode_loloc
  R : virtiofsd-fail-when-parent-inode-isn-t-known-in-l
D c : virtiofsd-extract-root-inode-init-into-setup_root
  R : virtiofsd-passthrough_ll-fix-refcounting-on-remov
D R : virtiofsd-passthrough_ll-clean-up-cache-related-o
  R : virtiofsd-passthrough_ll-use-hashtable
  R : virtiofsd-Clean-up-inodes-on-destroy
  R : virtiofsd-support-nanosecond-resolution-for-file-
  R : virtiofsd-fix-error-handling-in-main
  R : virtiofsd-cleanup-allocated-resource-in-se
D c : virtiofsd-fix-memory-leak-on-lo.source
D R : virtiofsd-add-helper-for-lo_data-cleanup
  R : virtiofsd-Prevent-multiply-running-with-same-vhos
  R : virtiofsd-enable-PARALLEL_DIROPS-during-INIT
  R : virtiofsd-fix-incorrect-error-handling-in-lo_do_l
D R : Virtiofsd-fix-memory-leak-on-fuse-queueinfo
D R : virtiofsd-Support-remote-posix-locks
  R : virtiofsd-use-fuse_lowlevel_is_virtio-in-fuse_ses
  R : virtiofsd-prevent-fv_queue_thread-vs-virtio_loop-
  R : virtiofsd-make-lo_release-atomic
    : virtiofsd-prevent-races-with-lo_dirp_put
    : virtiofsd-rename-inode-refcount-to-inode-nlookup
    : libvhost-user-Fix-some-memtable-remap-cases
D R : virtiofsd-add-man-page
D R : virtiofsd-introduce-inode-refcount-to-prevent-use
  R : virtiofsd-do-not-always-set-FUSE_FLOCK_LOCKS
D R : virtiofsd-convert-more-fprintf-and-perror-to-use-
  R : virtiofsd-Reset-O_DIRECT-flag-during-file-open
  R : virtiofsd-Fix-data-corruption-with-O_APPEND-wirte
D R : virtiofsd-add-definition-of-fuse_buf_writev
D R : virtiofsd-use-fuse_buf_writev-to-replace-fuse_buf
D   : virtiofsd-process-requests-in-a-thread-pool
  R : virtiofsd-prevent-FUSE_INIT-FUSE_DESTROY-races
    : virtiofsd-fix-lo_destroy-resource-leaks
  R : virtiofsd-add-thread-pool-size-NUM-option
    : virtiofsd-Convert-lo_destroy-to-take-the-lo-mutex
N R : virtiofsd-passthrough_ll-Pass-errno-to-fuse_reply
N R : virtiofsd-stop-all-queue-threads-on-exit-in-virti
N   : virtiofsd-add-some-options-to-the-help-message
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK