[PATCH v10 0/7] vhost-user block device backend implementation

Coiby Xu posted 7 patches 3 years, 6 months ago
Test docker-quick@centos7 failed
Test docker-mingw@fedora failed
Test checkpatch failed
Test FreeBSD failed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20200918080912.321299-1-coiby.xu@gmail.com
Maintainers: Kevin Wolf <kwolf@redhat.com>, Paolo Bonzini <pbonzini@redhat.com>, Max Reitz <mreitz@redhat.com>, "Daniel P. Berrangé" <berrange@redhat.com>, Eduardo Habkost <ehabkost@redhat.com>
MAINTAINERS                                |   8 +
block/export/vhost-user-blk-server.c       | 661 ++++++++++++++++++
block/export/vhost-user-blk-server.h       |  36 +
block/meson.build                          |   1 +
contrib/libvhost-user/libvhost-user-glib.c |   2 +-
contrib/libvhost-user/libvhost-user.c      |  15 +-
contrib/libvhost-user/libvhost-user.h      |  21 +
hw/core/qdev-properties.c                  |  31 +-
softmmu/vl.c                               |   4 +
tests/qtest/libqos/libqtest.h              |  17 +
tests/qtest/libqos/meson.build             |   1 +
tests/qtest/libqos/vhost-user-blk.c        | 129 ++++
tests/qtest/libqos/vhost-user-blk.h        |  48 ++
tests/qtest/libqtest.c                     |  36 +-
tests/qtest/meson.build                    |   4 +-
tests/qtest/vhost-user-blk-test.c          | 751 +++++++++++++++++++++
tests/vhost-user-bridge.c                  |   2 +
tools/virtiofsd/fuse_virtio.c              |   4 +-
util/block-helpers.c                       |  46 ++
util/block-helpers.h                       |  19 +
util/meson.build                           |   2 +
util/vhost-user-server.c                   | 428 ++++++++++++
util/vhost-user-server.h                   |  65 ++
23 files changed, 2292 insertions(+), 39 deletions(-)
create mode 100644 block/export/vhost-user-blk-server.c
create mode 100644 block/export/vhost-user-blk-server.h
create mode 100644 tests/qtest/libqos/vhost-user-blk.c
create mode 100644 tests/qtest/libqos/vhost-user-blk.h
create mode 100644 tests/qtest/vhost-user-blk-test.c
create mode 100644 util/block-helpers.c
create mode 100644 util/block-helpers.h
create mode 100644 util/vhost-user-server.c
create mode 100644 util/vhost-user-server.h
[PATCH v10 0/7] vhost-user block device backend implementation
Posted by Coiby Xu 3 years, 6 months ago
v10
 - Use a linked list of VuFdWatch objects to keep kick info [Stefan]
 - Merge improvements and fixes from Stefan
 - Rename libvhost-user's vu_message_read to vu_message_read_default [Kevin]
 - When shutting down the client, wait for the coroutine of processing
   vhost-user messages to be finished [Kevin]
 - Allocate struct req_data on the heap [Kevin]
 - Improve coding of qtest case [Thomas]
 - Fix several memory leaks detected by ASAN

v9
 - move logical block size check function to a utility function
 - fix issues regarding license, coding style, memory deallocation, etc.

v8
 - re-try connecting to socket server to fix asan error
 - fix license naming issue

v7
 - fix docker-test-debug@fedora errors by freeing malloced memory

v6
 - add missing license header and include guard
 - vhost-user server only serve one client one time
 - fix a bug in custom vu_message_read
 - using qemu-storage-daemon to start vhost-user-blk-server
 - a bug fix to pass docker-test-clang@ubuntu

v5:
 * re-use vu_kick_cb in libvhost-user
 * keeping processing VhostUserMsg in the same coroutine until there is
   detachment/attachment of AIOContext
 * Spawn separate coroutine for each VuVirtqElement
 * Other changes including relocating vhost-user-blk-server.c, coding
   style etc.

v4:
 * add object properties in class_init
 * relocate vhost-user-blk-test
 * other changes including using SocketAddress, coding style, etc.

v3:
 * separate generic vhost-user-server code from vhost-user-blk-server
   code
 * re-write vu_message_read and kick hander function as coroutines to
   directly call blk_co_preadv, blk_co_pwritev, etc.
 * add aio_context notifier functions to support multi-threading model
 * other fixes regarding coding style, warning report, etc.

v2:
 * Only enable this feature for Linux because eventfd is a Linux-specific
   feature


This patch series is an implementation of vhost-user block device
backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be
started using object_add,

 (qemu) object_add vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512
 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512

Then vhost-user client can connect to the server backend.
For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket -device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mounting it.

Coiby Xu (7):
  libvhost-user: Allow vu_message_read to be replaced
  libvhost-user: remove watch for kick_fd when de-initialize vu-dev
  util/vhost-user-server: generic vhost user server
  block: move logical block size check function to a common utility
    function
  block/export: vhost-user block device backend server
  test: new qTest case to test the vhost-user-blk-server
  MAINTAINERS: Add vhost-user block device backend server maintainer

 MAINTAINERS                                |   8 +
 block/export/vhost-user-blk-server.c       | 661 ++++++++++++++++++
 block/export/vhost-user-blk-server.h       |  36 +
 block/meson.build                          |   1 +
 contrib/libvhost-user/libvhost-user-glib.c |   2 +-
 contrib/libvhost-user/libvhost-user.c      |  15 +-
 contrib/libvhost-user/libvhost-user.h      |  21 +
 hw/core/qdev-properties.c                  |  31 +-
 softmmu/vl.c                               |   4 +
 tests/qtest/libqos/libqtest.h              |  17 +
 tests/qtest/libqos/meson.build             |   1 +
 tests/qtest/libqos/vhost-user-blk.c        | 129 ++++
 tests/qtest/libqos/vhost-user-blk.h        |  48 ++
 tests/qtest/libqtest.c                     |  36 +-
 tests/qtest/meson.build                    |   4 +-
 tests/qtest/vhost-user-blk-test.c          | 751 +++++++++++++++++++++
 tests/vhost-user-bridge.c                  |   2 +
 tools/virtiofsd/fuse_virtio.c              |   4 +-
 util/block-helpers.c                       |  46 ++
 util/block-helpers.h                       |  19 +
 util/meson.build                           |   2 +
 util/vhost-user-server.c                   | 428 ++++++++++++
 util/vhost-user-server.h                   |  65 ++
 23 files changed, 2292 insertions(+), 39 deletions(-)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c
 create mode 100644 util/block-helpers.c
 create mode 100644 util/block-helpers.h
 create mode 100644 util/vhost-user-server.c
 create mode 100644 util/vhost-user-server.h

--
2.28.0


Re: [PATCH v10 0/7] vhost-user block device backend implementation
Posted by Stefan Hajnoczi 3 years, 6 months ago
On Fri, Sep 18, 2020 at 04:09:05PM +0800, Coiby Xu wrote:
> v10
>  - Use a linked list of VuFdWatch objects to keep kick info [Stefan]
>  - Merge improvements and fixes from Stefan
>  - Rename libvhost-user's vu_message_read to vu_message_read_default [Kevin]
>  - When shutting down the client, wait for the coroutine of processing
>    vhost-user messages to be finished [Kevin]
>  - Allocate struct req_data on the heap [Kevin]
>  - Improve coding of qtest case [Thomas]
>  - Fix several memory leaks detected by ASAN

Great, thank you for sending this. I spent Friday looking at the first
few patches and running qemu-storage-daemon with this.

I will post a follow-up series with cleanups and port it to the new QAPI
block exports API (NBD, FUSE, vhost-user-blk, etc).

Kevin is currently on vacation but will be back soon. If he has no
further comments we could merge this.

Stefan
Re: [PATCH v10 0/7] vhost-user block device backend implementation
Posted by Stefan Hajnoczi 3 years, 5 months ago
On Fri, Sep 18, 2020 at 04:09:05PM +0800, Coiby Xu wrote:
> v10
>  - Use a linked list of VuFdWatch objects to keep kick info [Stefan]
>  - Merge improvements and fixes from Stefan
>  - Rename libvhost-user's vu_message_read to vu_message_read_default [Kevin]
>  - When shutting down the client, wait for the coroutine of processing
>    vhost-user messages to be finished [Kevin]
>  - Allocate struct req_data on the heap [Kevin]
>  - Improve coding of qtest case [Thomas]
>  - Fix several memory leaks detected by ASAN
> 
> v9
>  - move logical block size check function to a utility function
>  - fix issues regarding license, coding style, memory deallocation, etc.
> 
> v8
>  - re-try connecting to socket server to fix asan error
>  - fix license naming issue
> 
> v7
>  - fix docker-test-debug@fedora errors by freeing malloced memory
> 
> v6
>  - add missing license header and include guard
>  - vhost-user server only serve one client one time
>  - fix a bug in custom vu_message_read
>  - using qemu-storage-daemon to start vhost-user-blk-server
>  - a bug fix to pass docker-test-clang@ubuntu
> 
> v5:
>  * re-use vu_kick_cb in libvhost-user
>  * keeping processing VhostUserMsg in the same coroutine until there is
>    detachment/attachment of AIOContext
>  * Spawn separate coroutine for each VuVirtqElement
>  * Other changes including relocating vhost-user-blk-server.c, coding
>    style etc.
> 
> v4:
>  * add object properties in class_init
>  * relocate vhost-user-blk-test
>  * other changes including using SocketAddress, coding style, etc.
> 
> v3:
>  * separate generic vhost-user-server code from vhost-user-blk-server
>    code
>  * re-write vu_message_read and kick hander function as coroutines to
>    directly call blk_co_preadv, blk_co_pwritev, etc.
>  * add aio_context notifier functions to support multi-threading model
>  * other fixes regarding coding style, warning report, etc.
> 
> v2:
>  * Only enable this feature for Linux because eventfd is a Linux-specific
>    feature
> 
> 
> This patch series is an implementation of vhost-user block device
> backend server, thanks to Stefan and Kevin's guidance.
> 
> Vhost-user block device backend server is a UserCreatable object and can be
> started using object_add,
> 
>  (qemu) object_add vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512
>  (qemu) object_del ID
> 
> or appending the "-object" option when starting QEMU,
> 
>   $ -object vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512
> 
> Then vhost-user client can connect to the server backend.
> For example, QEMU could act as a client,
> 
>   $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket -device vhost-user-blk-pci,id=blk0,chardev=char1
> 
> And guest OS could access this vhost-user block device after mounting it.
> 
> Coiby Xu (7):
>   libvhost-user: Allow vu_message_read to be replaced
>   libvhost-user: remove watch for kick_fd when de-initialize vu-dev
>   util/vhost-user-server: generic vhost user server
>   block: move logical block size check function to a common utility
>     function
>   block/export: vhost-user block device backend server
>   test: new qTest case to test the vhost-user-blk-server
>   MAINTAINERS: Add vhost-user block device backend server maintainer
> 
>  MAINTAINERS                                |   8 +
>  block/export/vhost-user-blk-server.c       | 661 ++++++++++++++++++
>  block/export/vhost-user-blk-server.h       |  36 +
>  block/meson.build                          |   1 +
>  contrib/libvhost-user/libvhost-user-glib.c |   2 +-
>  contrib/libvhost-user/libvhost-user.c      |  15 +-
>  contrib/libvhost-user/libvhost-user.h      |  21 +
>  hw/core/qdev-properties.c                  |  31 +-
>  softmmu/vl.c                               |   4 +
>  tests/qtest/libqos/libqtest.h              |  17 +
>  tests/qtest/libqos/meson.build             |   1 +
>  tests/qtest/libqos/vhost-user-blk.c        | 129 ++++
>  tests/qtest/libqos/vhost-user-blk.h        |  48 ++
>  tests/qtest/libqtest.c                     |  36 +-
>  tests/qtest/meson.build                    |   4 +-
>  tests/qtest/vhost-user-blk-test.c          | 751 +++++++++++++++++++++
>  tests/vhost-user-bridge.c                  |   2 +
>  tools/virtiofsd/fuse_virtio.c              |   4 +-
>  util/block-helpers.c                       |  46 ++
>  util/block-helpers.h                       |  19 +
>  util/meson.build                           |   2 +
>  util/vhost-user-server.c                   | 428 ++++++++++++
>  util/vhost-user-server.h                   |  65 ++
>  23 files changed, 2292 insertions(+), 39 deletions(-)
>  create mode 100644 block/export/vhost-user-blk-server.c
>  create mode 100644 block/export/vhost-user-blk-server.h
>  create mode 100644 tests/qtest/libqos/vhost-user-blk.c
>  create mode 100644 tests/qtest/libqos/vhost-user-blk.h
>  create mode 100644 tests/qtest/vhost-user-blk-test.c
>  create mode 100644 util/block-helpers.c
>  create mode 100644 util/block-helpers.h
>  create mode 100644 util/vhost-user-server.c
>  create mode 100644 util/vhost-user-server.h
> 
> --
> 2.28.0
> 

Thanks, applied to my block tree:
https://github.com/stefanha/qemu/commits/block

Stefan
Re: [PATCH v10 0/7] vhost-user block device backend implementation
Posted by Coiby Xu 3 years, 5 months ago
On Fri, Oct 09, 2020 at 11:18:43AM +0100, Stefan Hajnoczi wrote:
>On Fri, Sep 18, 2020 at 04:09:05PM +0800, Coiby Xu wrote:
>> v10
>>  - Use a linked list of VuFdWatch objects to keep kick info [Stefan]
>>  - Merge improvements and fixes from Stefan
>>  - Rename libvhost-user's vu_message_read to vu_message_read_default [Kevin]
>>  - When shutting down the client, wait for the coroutine of processing
>>    vhost-user messages to be finished [Kevin]
>>  - Allocate struct req_data on the heap [Kevin]
>>  - Improve coding of qtest case [Thomas]
>>  - Fix several memory leaks detected by ASAN
>>
>> v9
>>  - move logical block size check function to a utility function
>>  - fix issues regarding license, coding style, memory deallocation, etc.
>>
>> v8
>>  - re-try connecting to socket server to fix asan error
>>  - fix license naming issue
>>
>> v7
>>  - fix docker-test-debug@fedora errors by freeing malloced memory
>>
>> v6
>>  - add missing license header and include guard
>>  - vhost-user server only serve one client one time
>>  - fix a bug in custom vu_message_read
>>  - using qemu-storage-daemon to start vhost-user-blk-server
>>  - a bug fix to pass docker-test-clang@ubuntu
>>
>> v5:
>>  * re-use vu_kick_cb in libvhost-user
>>  * keeping processing VhostUserMsg in the same coroutine until there is
>>    detachment/attachment of AIOContext
>>  * Spawn separate coroutine for each VuVirtqElement
>>  * Other changes including relocating vhost-user-blk-server.c, coding
>>    style etc.
>>
>> v4:
>>  * add object properties in class_init
>>  * relocate vhost-user-blk-test
>>  * other changes including using SocketAddress, coding style, etc.
>>
>> v3:
>>  * separate generic vhost-user-server code from vhost-user-blk-server
>>    code
>>  * re-write vu_message_read and kick hander function as coroutines to
>>    directly call blk_co_preadv, blk_co_pwritev, etc.
>>  * add aio_context notifier functions to support multi-threading model
>>  * other fixes regarding coding style, warning report, etc.
>>
>> v2:
>>  * Only enable this feature for Linux because eventfd is a Linux-specific
>>    feature
>>
>>
>> This patch series is an implementation of vhost-user block device
>> backend server, thanks to Stefan and Kevin's guidance.
>>
>> Vhost-user block device backend server is a UserCreatable object and can be
>> started using object_add,
>>
>>  (qemu) object_add vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512
>>  (qemu) object_del ID
>>
>> or appending the "-object" option when starting QEMU,
>>
>>   $ -object vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512
>>
>> Then vhost-user client can connect to the server backend.
>> For example, QEMU could act as a client,
>>
>>   $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket -device vhost-user-blk-pci,id=blk0,chardev=char1
>>
>> And guest OS could access this vhost-user block device after mounting it.
>>
>> Coiby Xu (7):
>>   libvhost-user: Allow vu_message_read to be replaced
>>   libvhost-user: remove watch for kick_fd when de-initialize vu-dev
>>   util/vhost-user-server: generic vhost user server
>>   block: move logical block size check function to a common utility
>>     function
>>   block/export: vhost-user block device backend server
>>   test: new qTest case to test the vhost-user-blk-server
>>   MAINTAINERS: Add vhost-user block device backend server maintainer
>>
>>  MAINTAINERS                                |   8 +
>>  block/export/vhost-user-blk-server.c       | 661 ++++++++++++++++++
>>  block/export/vhost-user-blk-server.h       |  36 +
>>  block/meson.build                          |   1 +
>>  contrib/libvhost-user/libvhost-user-glib.c |   2 +-
>>  contrib/libvhost-user/libvhost-user.c      |  15 +-
>>  contrib/libvhost-user/libvhost-user.h      |  21 +
>>  hw/core/qdev-properties.c                  |  31 +-
>>  softmmu/vl.c                               |   4 +
>>  tests/qtest/libqos/libqtest.h              |  17 +
>>  tests/qtest/libqos/meson.build             |   1 +
>>  tests/qtest/libqos/vhost-user-blk.c        | 129 ++++
>>  tests/qtest/libqos/vhost-user-blk.h        |  48 ++
>>  tests/qtest/libqtest.c                     |  36 +-
>>  tests/qtest/meson.build                    |   4 +-
>>  tests/qtest/vhost-user-blk-test.c          | 751 +++++++++++++++++++++
>>  tests/vhost-user-bridge.c                  |   2 +
>>  tools/virtiofsd/fuse_virtio.c              |   4 +-
>>  util/block-helpers.c                       |  46 ++
>>  util/block-helpers.h                       |  19 +
>>  util/meson.build                           |   2 +
>>  util/vhost-user-server.c                   | 428 ++++++++++++
>>  util/vhost-user-server.h                   |  65 ++
>>  23 files changed, 2292 insertions(+), 39 deletions(-)
>>  create mode 100644 block/export/vhost-user-blk-server.c
>>  create mode 100644 block/export/vhost-user-blk-server.h
>>  create mode 100644 tests/qtest/libqos/vhost-user-blk.c
>>  create mode 100644 tests/qtest/libqos/vhost-user-blk.h
>>  create mode 100644 tests/qtest/vhost-user-blk-test.c
>>  create mode 100644 util/block-helpers.c
>>  create mode 100644 util/block-helpers.h
>>  create mode 100644 util/vhost-user-server.c
>>  create mode 100644 util/vhost-user-server.h
>>
>> --
>> 2.28.0
>>
>
>Thanks, applied to my block tree:
>https://github.com/stefanha/qemu/commits/block
>

Thank you for improving the code! I'm going to study the changes to
understand how the improvements are made.

>Stefan



--
Best regards,
Coiby

Re: [PATCH v10 0/7] vhost-user block device backend implementation
Posted by Stefan Hajnoczi 3 years, 6 months ago
Thanks for contributing this feature! I have replied with comments about
additional steps I'm taking (fixes, cleanups, etc).

This series looks good:

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Re: [PATCH v10 0/7] vhost-user block device backend implementation
Posted by Stefan Hajnoczi 3 years, 6 months ago
On Fri, Sep 18, 2020 at 04:09:05PM +0800, Coiby Xu wrote:
> v10
>  - Use a linked list of VuFdWatch objects to keep kick info [Stefan]
>  - Merge improvements and fixes from Stefan
>  - Rename libvhost-user's vu_message_read to vu_message_read_default [Kevin]
>  - When shutting down the client, wait for the coroutine of processing
>    vhost-user messages to be finished [Kevin]
>  - Allocate struct req_data on the heap [Kevin]
>  - Improve coding of qtest case [Thomas]
>  - Fix several memory leaks detected by ASAN

Hi,
I have sent a patch series with cleanups and a conversion to the new
block exports API:
https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg07946.html

I will do a final review of your series tomorrow. Thanks!

Stefan