[Qemu-devel] [PATCH v3 00/19] nbd: add qemu-nbd --list

Eric Blake posted 19 patches 5 years, 3 months ago
Test docker-mingw@fedora failed
Test docker-clang@ubuntu passed
Test asan passed
Test checkpatch passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20190112175812.27068-1-eblake@redhat.com
Maintainers: Kevin Wolf <kwolf@redhat.com>, Max Reitz <mreitz@redhat.com>, Eric Blake <eblake@redhat.com>
There is a newer version of this series
qemu-nbd.texi              | 114 ++++--
Makefile                   |   2 +
include/block/nbd.h        |  31 +-
block/nbd-client.c         |   9 +-
blockdev-nbd.c             |  10 +-
nbd/client.c               | 756 ++++++++++++++++++++++++++-----------
nbd/server.c               |  23 +-
qemu-nbd.c                 | 221 ++++++++---
nbd/trace-events           |  11 +-
scripts/texi2pod.pl        |   2 +-
tests/qemu-iotests/223     |   2 +
tests/qemu-iotests/223.out |  20 +
tests/qemu-iotests/233     |  19 +-
tests/qemu-iotests/233.out |  15 +
14 files changed, 910 insertions(+), 325 deletions(-)
[Qemu-devel] [PATCH v3 00/19] nbd: add qemu-nbd --list
Posted by Eric Blake 5 years, 3 months ago
I got tired of debugging whether a server was advertising the
correct things during negotiation by inspecting the trace
logs of qemu-io as client - not to mention that without SOME
sort of client tracing particular commands, we can't easily
regression test the server for correct behavior.  The final
straw was at KVM Forum, when Nir asked me to make sure there
was a way to easily determine if an NBD server is exposing what
we really want (and fixing x-dirty-bitmap to behave saner fell
out as a result of answering that question).

I note that upstream NBD has 'nbd-client -l $host' for querying
just export names (with no quoting, so you have to know that
a blank line means the default export), but it wasn't powerful
enough, so I implemented 'qemu-nbd -L' to document everything.
Upstream NBD has separate 'nbd-client' and 'nbd-server' binaries,
while we only have 'qemu-nbd' (which is normally just a server,
but 'qemu-nbd -c' also operates a second thread as a client).
Our other uses of qemu as NBD client are for consuming a block
device (as in qemu-io, qemu-img, or a drive to qemu) - but those
binaries are less suited to something so specific to the NBD
protocol.

Bonus: As a result of my work on this series, nbdkit now supports
NBD_OPT_INFO (my interoperability testing between server
implementations has been paying off, both at fixing server bugs,
and at making this code more reliable across difference in valid
servers).

Also available at:
https://repo.or.cz/qemu/ericb.git qemu-nbd-list-v2

Based-on: <20181221093529.23855-1-jsnow@redhat.com>
[jsnow: 0/11 bitmaps: remove x- prefix from QMP api]
Based-on: <20190111163519.11457-1-philmd@redhat.com>
[philmd: qemu-nbd: Rename 'exp' variable clashing with math::exp() symbol]
Based-on: <20190111194720.15671-1-eblake@redhat.com>
[eblake: 0/8 Promote x-nbd-server-add-bitmap to stable]

Since v2:
- Several patches merged already
- 3 new patches based on audit of off_t vs. strtol
- rebase patches on top of other changes, such as qemu-nbd --bitmap
- address various review comments [Vladimir, Rich]
- drop patch 12/22

001/19:[0020] [FC] 'maint: Allow for EXAMPLES in texi2pod'
002/19:[0030] [FC] 'qemu-nbd: Enhance man page'
003/19:[down] 'qemu-nbd: Sanity check partition bounds'
004/19:[down] 'nbd/server: Hoist length check to qemp_nbd_server_add'
005/19:[down] 'nbd/server: Favor [u]int64_t over off_t'
006/19:[0007] [FC] 'qemu-nbd: Avoid strtol open-coding'
007/19:[0016] [FC] 'nbd/client: Refactor nbd_receive_list()'
008/19:[----] [--] 'nbd/client: Move export name into NBDExportInfo'
009/19:[----] [--] 'nbd/client: Change signature of nbd_negotiate_simple_meta_context()'
010/19:[0007] [FC] 'nbd/client: Split out nbd_send_one_meta_context()'
011/19:[0048] [FC] 'nbd/client: Split out nbd_receive_one_meta_context()'
012/19:[----] [--] 'nbd/client: Refactor return of nbd_receive_negotiate()'
013/19:[----] [-C] 'nbd/client: Split handshake into two functions'
014/19:[----] [--] 'nbd/client: Pull out oldstyle size determination'
015/19:[0008] [FC] 'nbd/client: Add nbd_receive_export_list()'
016/19:[----] [-C] 'nbd/client: Add meta contexts to nbd_receive_export_list()'
017/19:[0015] [FC] 'qemu-nbd: Add --list option'
018/19:[----] [--] 'nbd/client: Work around 3.0 bug for listing meta contexts'
019/19:[0002] [FC] 'iotests: Enhance 223, 233 to cover 'qemu-nbd --list''

Eric Blake (19):
  maint: Allow for EXAMPLES in texi2pod
  qemu-nbd: Enhance man page
  qemu-nbd: Sanity check partition bounds
  nbd/server: Hoist length check to qemp_nbd_server_add
  nbd/server: Favor [u]int64_t over off_t
  qemu-nbd: Avoid strtol open-coding
  nbd/client: Refactor nbd_receive_list()
  nbd/client: Move export name into NBDExportInfo
  nbd/client: Change signature of nbd_negotiate_simple_meta_context()
  nbd/client: Split out nbd_send_one_meta_context()
  nbd/client: Split out nbd_receive_one_meta_context()
  nbd/client: Refactor return of nbd_receive_negotiate()
  nbd/client: Split handshake into two functions
  nbd/client: Pull out oldstyle size determination
  nbd/client: Add nbd_receive_export_list()
  nbd/client: Add meta contexts to nbd_receive_export_list()
  qemu-nbd: Add --list option
  nbd/client: Work around 3.0 bug for listing meta contexts
  iotests: Enhance 223, 233 to cover 'qemu-nbd --list'

 qemu-nbd.texi              | 114 ++++--
 Makefile                   |   2 +
 include/block/nbd.h        |  31 +-
 block/nbd-client.c         |   9 +-
 blockdev-nbd.c             |  10 +-
 nbd/client.c               | 756 ++++++++++++++++++++++++++-----------
 nbd/server.c               |  23 +-
 qemu-nbd.c                 | 221 ++++++++---
 nbd/trace-events           |  11 +-
 scripts/texi2pod.pl        |   2 +-
 tests/qemu-iotests/223     |   2 +
 tests/qemu-iotests/223.out |  20 +
 tests/qemu-iotests/233     |  19 +-
 tests/qemu-iotests/233.out |  15 +
 14 files changed, 910 insertions(+), 325 deletions(-)

-- 
2.20.1


Re: [Qemu-devel] [PATCH v3 00/19] nbd: add qemu-nbd --list
Posted by Vladimir Sementsov-Ogievskiy 5 years, 3 months ago
12.01.2019 20:57, Eric Blake wrote:
> I got tired of debugging whether a server was advertising the
> correct things during negotiation by inspecting the trace
> logs of qemu-io as client - not to mention that without SOME
> sort of client tracing particular commands, we can't easily
> regression test the server for correct behavior.  The final
> straw was at KVM Forum, when Nir asked me to make sure there
> was a way to easily determine if an NBD server is exposing what
> we really want (and fixing x-dirty-bitmap to behave saner fell
> out as a result of answering that question).
> 
> I note that upstream NBD has 'nbd-client -l $host' for querying
> just export names (with no quoting, so you have to know that
> a blank line means the default export), but it wasn't powerful
> enough, so I implemented 'qemu-nbd -L' to document everything.
> Upstream NBD has separate 'nbd-client' and 'nbd-server' binaries,
> while we only have 'qemu-nbd' (which is normally just a server,
> but 'qemu-nbd -c' also operates a second thread as a client).
> Our other uses of qemu as NBD client are for consuming a block
> device (as in qemu-io, qemu-img, or a drive to qemu) - but those
> binaries are less suited to something so specific to the NBD
> protocol.
> 
> Bonus: As a result of my work on this series, nbdkit now supports
> NBD_OPT_INFO (my interoperability testing between server
> implementations has been paying off, both at fixing server bugs,
> and at making this code more reliable across difference in valid
> servers).
> 
> Also available at:
> https://repo.or.cz/qemu/ericb.git qemu-nbd-list-v2
> 
> Based-on: <20181221093529.23855-1-jsnow@redhat.com>
> [jsnow: 0/11 bitmaps: remove x- prefix from QMP api]
> Based-on: <20190111163519.11457-1-philmd@redhat.com>
> [philmd: qemu-nbd: Rename 'exp' variable clashing with math::exp() symbol]
> Based-on: <20190111194720.15671-1-eblake@redhat.com>
> [eblake: 0/8 Promote x-nbd-server-add-bitmap to stable]
> 
> Since v2:
> - Several patches merged already
> - 3 new patches based on audit of off_t vs. strtol
> - rebase patches on top of other changes, such as qemu-nbd --bitmap
> - address various review comments [Vladimir, Rich]
> - drop patch 12/22
> 
> 001/19:[0020] [FC] 'maint: Allow for EXAMPLES in texi2pod'
> 002/19:[0030] [FC] 'qemu-nbd: Enhance man page'

Strange, but I don't have 02 in my mailbox..


-- 
Best regards,
Vladimir
Re: [Qemu-devel] [PATCH v3 00/19] nbd: add qemu-nbd --list
Posted by Eric Blake 5 years, 3 months ago
On 1/17/19 5:38 AM, Vladimir Sementsov-Ogievskiy wrote:

>> Since v2:
>> - Several patches merged already
>> - 3 new patches based on audit of off_t vs. strtol
>> - rebase patches on top of other changes, such as qemu-nbd --bitmap
>> - address various review comments [Vladimir, Rich]
>> - drop patch 12/22
>>
>> 001/19:[0020] [FC] 'maint: Allow for EXAMPLES in texi2pod'
>> 002/19:[0030] [FC] 'qemu-nbd: Enhance man page'
> 
> Strange, but I don't have 02 in my mailbox..

I'll bounce a copy to you...

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v3 00/19] nbd: add qemu-nbd --list
Posted by Vladimir Sementsov-Ogievskiy 5 years, 3 months ago
12.01.2019 20:57, Eric Blake wrote:
> I got tired of debugging whether a server was advertising the
> correct things during negotiation by inspecting the trace
> logs of qemu-io as client - not to mention that without SOME
> sort of client tracing particular commands, we can't easily
> regression test the server for correct behavior.  The final
> straw was at KVM Forum, when Nir asked me to make sure there
> was a way to easily determine if an NBD server is exposing what
> we really want (and fixing x-dirty-bitmap to behave saner fell
> out as a result of answering that question).
> 
> I note that upstream NBD has 'nbd-client -l $host' for querying
> just export names (with no quoting, so you have to know that
> a blank line means the default export), but it wasn't powerful
> enough, so I implemented 'qemu-nbd -L' to document everything.
> Upstream NBD has separate 'nbd-client' and 'nbd-server' binaries,
> while we only have 'qemu-nbd' (which is normally just a server,
> but 'qemu-nbd -c' also operates a second thread as a client).
> Our other uses of qemu as NBD client are for consuming a block
> device (as in qemu-io, qemu-img, or a drive to qemu) - but those
> binaries are less suited to something so specific to the NBD
> protocol.
> 
> Bonus: As a result of my work on this series, nbdkit now supports
> NBD_OPT_INFO (my interoperability testing between server
> implementations has been paying off, both at fixing server bugs,
> and at making this code more reliable across difference in valid
> servers).
> 
> Also available at:
> https://repo.or.cz/qemu/ericb.git qemu-nbd-list-v2

hmm, not -v2. But not -v3 too, as  I've tried to apply these patches on
-v3 base (commit under "maint: Allow for EXAMPLES in texi2pod") and it
failed.

Could you please fix tag?

> 
> Based-on: <20181221093529.23855-1-jsnow@redhat.com>
> [jsnow: 0/11 bitmaps: remove x- prefix from QMP api]
> Based-on: <20190111163519.11457-1-philmd@redhat.com>
> [philmd: qemu-nbd: Rename 'exp' variable clashing with math::exp() symbol]
> Based-on: <20190111194720.15671-1-eblake@redhat.com>
> [eblake: 0/8 Promote x-nbd-server-add-bitmap to stable]
> 
> Since v2:
> - Several patches merged already
> - 3 new patches based on audit of off_t vs. strtol
> - rebase patches on top of other changes, such as qemu-nbd --bitmap
> - address various review comments [Vladimir, Rich]
> - drop patch 12/22
> 
> 001/19:[0020] [FC] 'maint: Allow for EXAMPLES in texi2pod'
> 002/19:[0030] [FC] 'qemu-nbd: Enhance man page'
> 003/19:[down] 'qemu-nbd: Sanity check partition bounds'
> 004/19:[down] 'nbd/server: Hoist length check to qemp_nbd_server_add'
> 005/19:[down] 'nbd/server: Favor [u]int64_t over off_t'
> 006/19:[0007] [FC] 'qemu-nbd: Avoid strtol open-coding'
> 007/19:[0016] [FC] 'nbd/client: Refactor nbd_receive_list()'
> 008/19:[----] [--] 'nbd/client: Move export name into NBDExportInfo'
> 009/19:[----] [--] 'nbd/client: Change signature of nbd_negotiate_simple_meta_context()'
> 010/19:[0007] [FC] 'nbd/client: Split out nbd_send_one_meta_context()'
> 011/19:[0048] [FC] 'nbd/client: Split out nbd_receive_one_meta_context()'
> 012/19:[----] [--] 'nbd/client: Refactor return of nbd_receive_negotiate()'
> 013/19:[----] [-C] 'nbd/client: Split handshake into two functions'
> 014/19:[----] [--] 'nbd/client: Pull out oldstyle size determination'
> 015/19:[0008] [FC] 'nbd/client: Add nbd_receive_export_list()'
> 016/19:[----] [-C] 'nbd/client: Add meta contexts to nbd_receive_export_list()'
> 017/19:[0015] [FC] 'qemu-nbd: Add --list option'
> 018/19:[----] [--] 'nbd/client: Work around 3.0 bug for listing meta contexts'
> 019/19:[0002] [FC] 'iotests: Enhance 223, 233 to cover 'qemu-nbd --list''
> 
> Eric Blake (19):
>    maint: Allow for EXAMPLES in texi2pod
>    qemu-nbd: Enhance man page
>    qemu-nbd: Sanity check partition bounds
>    nbd/server: Hoist length check to qemp_nbd_server_add
>    nbd/server: Favor [u]int64_t over off_t
>    qemu-nbd: Avoid strtol open-coding
>    nbd/client: Refactor nbd_receive_list()
>    nbd/client: Move export name into NBDExportInfo
>    nbd/client: Change signature of nbd_negotiate_simple_meta_context()
>    nbd/client: Split out nbd_send_one_meta_context()
>    nbd/client: Split out nbd_receive_one_meta_context()
>    nbd/client: Refactor return of nbd_receive_negotiate()
>    nbd/client: Split handshake into two functions
>    nbd/client: Pull out oldstyle size determination
>    nbd/client: Add nbd_receive_export_list()
>    nbd/client: Add meta contexts to nbd_receive_export_list()
>    qemu-nbd: Add --list option
>    nbd/client: Work around 3.0 bug for listing meta contexts
>    iotests: Enhance 223, 233 to cover 'qemu-nbd --list'
> 
>   qemu-nbd.texi              | 114 ++++--
>   Makefile                   |   2 +
>   include/block/nbd.h        |  31 +-
>   block/nbd-client.c         |   9 +-
>   blockdev-nbd.c             |  10 +-
>   nbd/client.c               | 756 ++++++++++++++++++++++++++-----------
>   nbd/server.c               |  23 +-
>   qemu-nbd.c                 | 221 ++++++++---
>   nbd/trace-events           |  11 +-
>   scripts/texi2pod.pl        |   2 +-
>   tests/qemu-iotests/223     |   2 +
>   tests/qemu-iotests/223.out |  20 +
>   tests/qemu-iotests/233     |  19 +-
>   tests/qemu-iotests/233.out |  15 +
>   14 files changed, 910 insertions(+), 325 deletions(-)
> 


-- 
Best regards,
Vladimir
Re: [Qemu-devel] [PATCH v3 00/19] nbd: add qemu-nbd --list
Posted by Eric Blake 5 years, 3 months ago
On 1/14/19 6:22 AM, Vladimir Sementsov-Ogievskiy wrote:

>>
>> Also available at:
>> https://repo.or.cz/qemu/ericb.git qemu-nbd-list-v2
> 
> hmm, not -v2. But not -v3 too, as  I've tried to apply these patches on
> -v3 base (commit under "maint: Allow for EXAMPLES in texi2pod") and it
> failed.
> 
> Could you please fix tag?

Yes, I typo'd my intent to advertise a v3 tag.  To avoid confusion with
anyone that may have already pulled the old tag, I've now pushed a
qemu-nbd-list-v3a tag, on top of my most recent pull request.

Based-on:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg02960.html

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v3 00/19] nbd: add qemu-nbd --list
Posted by no-reply@patchew.org 5 years, 3 months ago
Patchew URL: https://patchew.org/QEMU/20190112175812.27068-1-eblake@redhat.com/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
time make docker-test-mingw@fedora SHOW_ENV=1 J=14
=== TEST SCRIPT END ===

  CC      qom/container.o
  CC      qom/qom-qobject.o
/tmp/qemu-test/src/block/sheepdog.c: In function 'find_vdi_name':
/tmp/qemu-test/src/block/sheepdog.c:1239:5: error: 'strncpy' specified bound 256 equals destination size [-Werror=stringop-truncation]
     strncpy(buf + SD_MAX_VDI_LEN, tag, SD_MAX_VDI_TAG_LEN);
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors


The full log is available at
http://patchew.org/logs/20190112175812.27068-1-eblake@redhat.com/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com