v9
 - move logical block size check function to a utility function
 - fix issues regarding license, coding style, memory deallocation, etc.

v8
 - re-try connecting to socket server to fix asan error
 - fix license naming issue

v7
 - fix docker-test-debug@fedora errors by freeing malloced memory

v6
 - add missing license header and include guard
 - vhost-user server only serve one client one time
 - fix a bug in custom vu_message_read
 - using qemu-storage-daemon to start vhost-user-blk-server
 - a bug fix to pass docker-test-clang@ubuntu

v5:
 * re-use vu_kick_cb in libvhost-user
 * keeping processing VhostUserMsg in the same coroutine until there is
   detachment/attachment of AIOContext
 * Spawn separate coroutine for each VuVirtqElement
 * Other changes including relocating vhost-user-blk-server.c, coding
   style etc.

v4:
 * add object properties in class_init
 * relocate vhost-user-blk-test
 * other changes including using SocketAddress, coding style, etc.

v3:
 * separate generic vhost-user-server code from vhost-user-blk-server
   code
 * re-write vu_message_read and kick hander function as coroutines to
   directly call blk_co_preadv, blk_co_pwritev, etc.
 * add aio_context notifier functions to support multi-threading model
 * other fixes regarding coding style, warning report, etc.

v2:
 * Only enable this feature for Linux because eventfd is a Linux-specific
   feature


This patch series is an implementation of vhost-user block device
backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be
started using object_add,

 (qemu) object_add vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512
 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512

Then vhost-user client can connect to the server backend.
For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket -device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mounting it.


Coiby Xu (5):
  Allow vu_message_read to be replaced
  generic vhost user server
  move logical block size check function to a common utility function
  vhost-user block device backend server
  new qTest case to test the vhost-user-blk-server

 block/Makefile.objs                        |   1 +
 block/export/vhost-user-blk-server.c       | 669 +++++++++++++++++++
 block/export/vhost-user-blk-server.h       |  35 +
 contrib/libvhost-user/libvhost-user-glib.c |   2 +-
 contrib/libvhost-user/libvhost-user.c      |  11 +-
 contrib/libvhost-user/libvhost-user.h      |  21 +
 hw/core/qdev-properties.c                  |  18 +-
 softmmu/vl.c                               |   4 +
 tests/Makefile.include                     |   3 +-
 tests/qtest/Makefile.include               |   2 +
 tests/qtest/libqos/vhost-user-blk.c        | 130 ++++
 tests/qtest/libqos/vhost-user-blk.h        |  48 ++
 tests/qtest/libqtest.c                     |  35 +-
 tests/qtest/libqtest.h                     |  17 +
 tests/qtest/vhost-user-blk-test.c          | 739 +++++++++++++++++++++
 tests/vhost-user-bridge.c                  |   2 +
 tools/virtiofsd/fuse_virtio.c              |   4 +-
 util/Makefile.objs                         |   2 +
 util/block-helpers.c                       |  46 ++
 util/block-helpers.h                       |   7 +
 util/vhost-user-server.c                   | 400 +++++++++++
 util/vhost-user-server.h                   |  61 ++
 22 files changed, 2231 insertions(+), 26 deletions(-)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
 create mode 100644 util/block-helpers.c
 create mode 100644 util/block-helpers.h
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

--
2.27.0

On Mon, Jun 15, 2020 at 02:39:02AM +0800, Coiby Xu wrote:
> v9
>  - move logical block size check function to a utility function
>  - fix issues regarding license, coding style, memory deallocation, etc.

I have replied with patches that you can consider squashing into your
series. I was testing this patch series and decided it was easier to
send code than to go back and write review comments since I was already
on a git branch.

My patches can be combined into your original patches using "git rebase
-i" and the "fixup" or "squash" directive.

Please add my Signed-off-by: line to affected patches when squashing
patches so that the git log records that I have confirmed that I have
permission to contribute this code.

If you have questions about any of the patches, please let me know.

Stefan

On Fri, Jun 19, 2020 at 01:07:46PM +0100, Stefan Hajnoczi wrote:
>On Mon, Jun 15, 2020 at 02:39:02AM +0800, Coiby Xu wrote:
>> v9
>>  - move logical block size check function to a utility function
>>  - fix issues regarding license, coding style, memory deallocation, etc.
>
>I have replied with patches that you can consider squashing into your
>series. I was testing this patch series and decided it was easier to
>send code than to go back and write review comments since I was already
>on a git branch.
>
>My patches can be combined into your original patches using "git rebase
>-i" and the "fixup" or "squash" directive.
>
>Please add my Signed-off-by: line to affected patches when squashing
>patches so that the git log records that I have confirmed that I have
>permission to contribute this code.
>
>If you have questions about any of the patches, please let me know.

Besides your Signed-off-by: line, shouldn't I also add copyright info to
the affected files as follows?

  * Copyright (c) 2020 Coiby Xu <coiby.xu@gmail.com>
  *
  * Copyright (c) 2020 Red Hat, Inc., Stefan Hajnoczi <stefanha@redhat.com>
  *


--
Best regards,
Coiby

On Thu, Jun 25, 2020 at 08:46:56PM +0800, Coiby Xu wrote:
> On Fri, Jun 19, 2020 at 01:07:46PM +0100, Stefan Hajnoczi wrote:
> > On Mon, Jun 15, 2020 at 02:39:02AM +0800, Coiby Xu wrote:
> > > v9
> > >  - move logical block size check function to a utility function
> > >  - fix issues regarding license, coding style, memory deallocation, etc.
> > 
> > I have replied with patches that you can consider squashing into your
> > series. I was testing this patch series and decided it was easier to
> > send code than to go back and write review comments since I was already
> > on a git branch.
> > 
> > My patches can be combined into your original patches using "git rebase
> > -i" and the "fixup" or "squash" directive.
> > 
> > Please add my Signed-off-by: line to affected patches when squashing
> > patches so that the git log records that I have confirmed that I have
> > permission to contribute this code.
> > 
> > If you have questions about any of the patches, please let me know.
> 
> Besides your Signed-off-by: line, shouldn't I also add copyright info to
> the affected files as follows?
> 
>  * Copyright (c) 2020 Coiby Xu <coiby.xu@gmail.com>
>  *
>  * Copyright (c) 2020 Red Hat, Inc., Stefan Hajnoczi <stefanha@redhat.com>

The following would be good:

 * Copyright (c) 2020 Red Hat, Inc.

Thanks,
Stefan

On Fri, Jun 19, 2020 at 01:07:46PM +0100, Stefan Hajnoczi wrote:
>On Mon, Jun 15, 2020 at 02:39:02AM +0800, Coiby Xu wrote:
>> v9
>>  - move logical block size check function to a utility function
>>  - fix issues regarding license, coding style, memory deallocation, etc.
>
>I have replied with patches that you can consider squashing into your
>series. I was testing this patch series and decided it was easier to
>send code than to go back and write review comments since I was already
>on a git branch.
>
>My patches can be combined into your original patches using "git rebase
>-i" and the "fixup" or "squash" directive.
>
>Please add my Signed-off-by: line to affected patches when squashing
>patches so that the git log records that I have confirmed that I have
>permission to contribute this code.

I was thinking on how to incorporate your work while reading the emails.
You just provide the instruction! Thank you!


--
Best regards,
Coiby

Patchew URL: https://patchew.org/QEMU/20200614183907.514282-1-coiby.xu@gmail.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      hw/gpio/trace.o
  CC      hw/riscv/trace.o
/tmp/qemu-test/src/util/vhost-user-server.c: In function 'vu_message_read':
/tmp/qemu-test/src/util/vhost-user-server.c:142:30: error: 'VHOST_MEMORY_MAX_NREGIONS' undeclared (first use in this function)
                              VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
                              ^
/tmp/qemu-test/src/util/vhost-user-server.c:142:30: note: each undeclared identifier is reported only once for each function it appears in
make: *** [util/vhost-user-server.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 669, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=408a16efe57d44399707cea73c0c46cb', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-hyuz9qap/src/docker-src.2020-06-14-15.10.05.3902:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=408a16efe57d44399707cea73c0c46cb
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-hyuz9qap/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    2m3.838s
user    0m8.880s


The full log is available at
http://patchew.org/logs/20200614183907.514282-1-coiby.xu@gmail.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Patchew URL: https://patchew.org/QEMU/20200614183907.514282-1-coiby.xu@gmail.com/



Hi,

This series failed the asan build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      stubs/vm-stop.o
  CC      ui/input-keymap.o
  CC      qemu-keymap.o
/tmp/qemu-test/src/util/vhost-user-server.c:142:30: error: use of undeclared identifier 'VHOST_MEMORY_MAX_NREGIONS'
                             VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
                             ^
1 error generated.
make: *** [/tmp/qemu-test/src/rules.mak:69: util/vhost-user-server.o] Error 1
make: *** Waiting for unfinished jobs....
  CC      contrib/elf2dmp/main.o
Traceback (most recent call last):
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=99a7788ed7f54df1b0ebb7f91beb522e', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=x86_64-softmmu', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-9goaol7z/src/docker-src.2020-06-14-15.12.59.10439:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-debug']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=99a7788ed7f54df1b0ebb7f91beb522e
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-9goaol7z/src'
make: *** [docker-run-test-debug@fedora] Error 2

real    3m29.335s
user    0m8.313s


The full log is available at
http://patchew.org/logs/20200614183907.514282-1-coiby.xu@gmail.com/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

On Sun, Jun 14, 2020 at 12:16:28PM -0700, no-reply@patchew.org wrote:
>Patchew URL: https://patchew.org/QEMU/20200614183907.514282-1-coiby.xu@gmail.com/
>
>
>
>Hi,
>
>This series failed the asan build test. Please find the testing commands and
>their output below. If you have Docker installed, you can probably reproduce it
>locally.
>
>=== TEST SCRIPT BEGIN ===
>#!/bin/bash
>export ARCH=x86_64
>make docker-image-fedora V=1 NETWORK=1
>time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
>=== TEST SCRIPT END ===
>
>  CC      stubs/vm-stop.o
>  CC      ui/input-keymap.o
>  CC      qemu-keymap.o
>/tmp/qemu-test/src/util/vhost-user-server.c:142:30: error: use of undeclared identifier 'VHOST_MEMORY_MAX_NREGIONS'
>                             VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
>                             ^
>
>The full log is available at
>http://patchew.org/logs/20200614183907.514282-1-coiby.xu@gmail.com/testing.asan/?type=message.

I couldn't re-produce this error locally for both docker-test-quick@centos7
and this docker test. And I can't see any reason for this error to occur since
VHOST_MEMORY_MAX_NREGIONS is defined in contrib/libvhost-user/libvhost-user.h
which has been included by util/vhost-user-server.h.

--
Best regards,
Coiby

On Tue, Jun 16, 2020 at 02:52:16PM +0800, Coiby Xu wrote:
> On Sun, Jun 14, 2020 at 12:16:28PM -0700, no-reply@patchew.org wrote:
> > Patchew URL: https://patchew.org/QEMU/20200614183907.514282-1-coiby.xu@gmail.com/
> > 
> > 
> > 
> > Hi,
> > 
> > This series failed the asan build test. Please find the testing commands and
> > their output below. If you have Docker installed, you can probably reproduce it
> > locally.
> > 
> > === TEST SCRIPT BEGIN ===
> > #!/bin/bash
> > export ARCH=x86_64
> > make docker-image-fedora V=1 NETWORK=1
> > time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
> > === TEST SCRIPT END ===
> > 
> >  CC      stubs/vm-stop.o
> >  CC      ui/input-keymap.o
> >  CC      qemu-keymap.o
> > /tmp/qemu-test/src/util/vhost-user-server.c:142:30: error: use of undeclared identifier 'VHOST_MEMORY_MAX_NREGIONS'
> >                             VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
> >                             ^
> > 
> > The full log is available at
> > http://patchew.org/logs/20200614183907.514282-1-coiby.xu@gmail.com/testing.asan/?type=message.
> 
> I couldn't re-produce this error locally for both docker-test-quick@centos7
> and this docker test. And I can't see any reason for this error to occur since
> VHOST_MEMORY_MAX_NREGIONS is defined in contrib/libvhost-user/libvhost-user.h
> which has been included by util/vhost-user-server.h.

Please see the recent change in commit
b650d5f4b1cd3f9f8c4fdb319838c5c1e0695e41 ("Lift max ram slots limit in
libvhost-user").

The error can be solved by replacing VHOST_MEMORY_MAX_NREGIONS with
VHOST_MEMORY_BASELINE_NREGIONS in util/vhost-user-server.c.

Stefan

On Thu, Jun 18, 2020 at 09:27:48AM +0100, Stefan Hajnoczi wrote:
>On Tue, Jun 16, 2020 at 02:52:16PM +0800, Coiby Xu wrote:
>> On Sun, Jun 14, 2020 at 12:16:28PM -0700, no-reply@patchew.org wrote:
>> > Patchew URL: https://patchew.org/QEMU/20200614183907.514282-1-coiby.xu@gmail.com/
>> >
>> >
>> >
>> > Hi,
>> >
>> > This series failed the asan build test. Please find the testing commands and
>> > their output below. If you have Docker installed, you can probably reproduce it
>> > locally.
>> >
>> > === TEST SCRIPT BEGIN ===
>> > #!/bin/bash
>> > export ARCH=x86_64
>> > make docker-image-fedora V=1 NETWORK=1
>> > time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
>> > === TEST SCRIPT END ===
>> >
>> >  CC      stubs/vm-stop.o
>> >  CC      ui/input-keymap.o
>> >  CC      qemu-keymap.o
>> > /tmp/qemu-test/src/util/vhost-user-server.c:142:30: error: use of undeclared identifier 'VHOST_MEMORY_MAX_NREGIONS'
>> >                             VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
>> >                             ^
>> >
>> > The full log is available at
>> > http://patchew.org/logs/20200614183907.514282-1-coiby.xu@gmail.com/testing.asan/?type=message.
>>
>> I couldn't re-produce this error locally for both docker-test-quick@centos7
>> and this docker test. And I can't see any reason for this error to occur since
>> VHOST_MEMORY_MAX_NREGIONS is defined in contrib/libvhost-user/libvhost-user.h
>> which has been included by util/vhost-user-server.h.
>
>Please see the recent change in commit
>b650d5f4b1cd3f9f8c4fdb319838c5c1e0695e41 ("Lift max ram slots limit in
>libvhost-user").
>
>The error can be solved by replacing VHOST_MEMORY_MAX_NREGIONS with
>VHOST_MEMORY_BASELINE_NREGIONS in util/vhost-user-server.c.

Thank you for the clarification! I did run "git pull" when checking this error.
It seems there is a delay when pulling updates from git://git.qemu.org/qemu.git.


--
Best regards,
Coiby

On Tue, Jun 16, 2020 at 02:52:16PM +0800, Coiby Xu wrote:
> On Sun, Jun 14, 2020 at 12:16:28PM -0700, no-reply@patchew.org wrote:
> > Patchew URL: https://patchew.org/QEMU/20200614183907.514282-1-coiby.xu@gmail.com/
> > 
> > 
> > 
> > Hi,
> > 
> > This series failed the asan build test. Please find the testing commands and
> > their output below. If you have Docker installed, you can probably reproduce it
> > locally.
> > 
> > === TEST SCRIPT BEGIN ===
> > #!/bin/bash
> > export ARCH=x86_64
> > make docker-image-fedora V=1 NETWORK=1
> > time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
> > === TEST SCRIPT END ===
> > 
> >  CC      stubs/vm-stop.o
> >  CC      ui/input-keymap.o
> >  CC      qemu-keymap.o
> > /tmp/qemu-test/src/util/vhost-user-server.c:142:30: error: use of undeclared identifier 'VHOST_MEMORY_MAX_NREGIONS'
> >                             VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
> >                             ^
> > 
> > The full log is available at
> > http://patchew.org/logs/20200614183907.514282-1-coiby.xu@gmail.com/testing.asan/?type=message.
> 
> I couldn't re-produce this error locally for both docker-test-quick@centos7
> and this docker test. And I can't see any reason for this error to occur since
> VHOST_MEMORY_MAX_NREGIONS is defined in contrib/libvhost-user/libvhost-user.h
> which has been included by util/vhost-user-server.h.

Using G_N_ELEMENTS(vmsg->fds) instead of VHOST_MEMORY_MAX_NREGIONS is an
even cleaner fix.

Stefan

On Thu, Jun 18, 2020 at 09:28:44AM +0100, Stefan Hajnoczi wrote:
>On Tue, Jun 16, 2020 at 02:52:16PM +0800, Coiby Xu wrote:
>> On Sun, Jun 14, 2020 at 12:16:28PM -0700, no-reply@patchew.org wrote:
>> > Patchew URL: https://patchew.org/QEMU/20200614183907.514282-1-coiby.xu@gmail.com/
>> >
>> >
>> >
>> > Hi,
>> >
>> > This series failed the asan build test. Please find the testing commands and
>> > their output below. If you have Docker installed, you can probably reproduce it
>> > locally.
>> >
>> > === TEST SCRIPT BEGIN ===
>> > #!/bin/bash
>> > export ARCH=x86_64
>> > make docker-image-fedora V=1 NETWORK=1
>> > time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
>> > === TEST SCRIPT END ===
>> >
>> >  CC      stubs/vm-stop.o
>> >  CC      ui/input-keymap.o
>> >  CC      qemu-keymap.o
>> > /tmp/qemu-test/src/util/vhost-user-server.c:142:30: error: use of undeclared identifier 'VHOST_MEMORY_MAX_NREGIONS'
>> >                             VHOST_MEMORY_MAX_NREGIONS, nfds + nfds_t);
>> >                             ^
>> >
>> > The full log is available at
>> > http://patchew.org/logs/20200614183907.514282-1-coiby.xu@gmail.com/testing.asan/?type=message.
>>
>> I couldn't re-produce this error locally for both docker-test-quick@centos7
>> and this docker test. And I can't see any reason for this error to occur since
>> VHOST_MEMORY_MAX_NREGIONS is defined in contrib/libvhost-user/libvhost-user.h
>> which has been included by util/vhost-user-server.h.
>
>Using G_N_ELEMENTS(vmsg->fds) instead of VHOST_MEMORY_MAX_NREGIONS is an
>even cleaner fix.
>
>Stefan

Thank you for this cleaner fix!

--
Best regards,
Coiby

On Mon, Jun 15, 2020 at 02:39:02AM +0800, Coiby Xu wrote:
> v9
>  - move logical block size check function to a utility function
>  - fix issues regarding license, coding style, memory deallocation, etc.

Great to see you are back, Coiby! Looking forward to reviewing v10.

Stefan

On Mon, Jun 15, 2020 at 02:39:02AM +0800, Coiby Xu wrote:
> v9
>  - move logical block size check function to a utility function
>  - fix issues regarding license, coding style, memory deallocation, etc.

Hi,
Any update on v10?

Please let me know if there's anything I can do to help.

Stefan

On Tue, Sep 15, 2020 at 04:35:57PM +0100, Stefan Hajnoczi wrote:
>On Mon, Jun 15, 2020 at 02:39:02AM +0800, Coiby Xu wrote:
>> v9
>>  - move logical block size check function to a utility function
>>  - fix issues regarding license, coding style, memory deallocation, etc.
>
>Hi,
>Any update on v10?
>
>Please let me know if there's anything I can do to help.
>
>Stefan

Thank you for ping me! v10 has been submitted.


--
Best regards,
Coiby