I got tired of debugging whether a server was advertising the
correct things during negotiation by inspecting the trace
logs of qemu-io as client - not to mention that without SOME
sort of client tracing particular commands, we can't easily
regression test the server for correct behavior.  The final
straw was at KVM Forum, when Nir asked me to make sure there
was a way to easily determine if an NBD server is exposing what
we really want (and fixing x-dirty-bitmap to behave saner fell
out as a result of answering that question).

I note that upstream NBD has 'nbd-client -l $host' for querying
just export names (with no quoting, so you have to know that
a blank line means the default export), but it wasn't powerful
enough, so I implemented 'qemu-nbd -L' to document everything.
Upstream NBD has separate 'nbd-client' and 'nbd-server' binaries,
while we only have 'qemu-nbd' (which is normally just a server,
but 'qemu-nbd -c' also operates a second thread as a client).
Our other uses of qemu as NBD client are for consuming a block
device (as in qemu-io, qemu-img, or a drive to qemu) - but those
binaries are less suited to something so specific to the NBD
protocol.

Bonus: As a result of my work on this series, nbdkit now supports
NBD_OPT_INFO (my interoperability testing between server
implementations has been paying off, both at fixing server bugs,
and at making this code more reliable across difference in valid
servers).

Based-on: <20181130023232.3079982-1-eblake@redhat.com>
[0/3] NBD dirty bitmap cleanups

(well, only weakly based on it - the two series can be applied
in either order, where if you apply this series but revert 12/14,
then iotest 223 in 14/14 will expose the bug fixed by patch 1/3
in the other series)

Also available at:
https://repo.or.cz/qemu/ericb.git qemu-nbd-list-v1

Eric Blake (14):
  qemu-nbd: Use program name in error messages
  nbd/client: More consistent error messages
  qemu-nbd: Fail earlier for -c/-d on non-linux
  qemu-nbd: Simplify --partition handling
  nbd/client: Drop pointless buf variable
  nbd/client: Move export name into NBDExportInfo
  nbd/client: Refactor nbd_negotiate_simple_meta_context()
  nbd/client: Refactor nbd_receive_list()
  nbd/client: Refactor return of nbd_receive_negotiate()
  nbd/client: Split handshake into two functions
  nbd/client: Add nbd_receive_export_list()
  nbd/client: Work around 3.0 bug for listing meta contexts
  qemu-nbd: Add --list option
  iotests: Enhance 223, 233 to cover 'qemu-nbd --list'

 include/block/nbd.h        |  23 +-
 nbd/nbd-internal.h         |   1 +
 block/nbd-client.c         |   5 +-
 nbd/client.c               | 595 +++++++++++++++++++++++++------------
 qemu-nbd.c                 | 196 ++++++++++--
 nbd/trace-events           |   8 +-
 tests/qemu-iotests/223     |   2 +
 tests/qemu-iotests/223.out |  20 ++
 tests/qemu-iotests/233     |  10 +
 tests/qemu-iotests/233.out |  17 +-
 10 files changed, 650 insertions(+), 227 deletions(-)

-- 
2.17.2

On Fri, Nov 30, 2018 at 04:03:29PM -0600, Eric Blake wrote:
> I note that upstream NBD has 'nbd-client -l $host' for querying
> just export names (with no quoting, so you have to know that
> a blank line means the default export), but it wasn't powerful
> enough, so I implemented 'qemu-nbd -L' to document everything.
> Upstream NBD has separate 'nbd-client' and 'nbd-server' binaries,
> while we only have 'qemu-nbd' (which is normally just a server,
> but 'qemu-nbd -c' also operates a second thread as a client).
> Our other uses of qemu as NBD client are for consuming a block
> device (as in qemu-io, qemu-img, or a drive to qemu) - but those
> binaries are less suited to something so specific to the NBD
> protocol.

I tried it against nbdkit and it works (obviously):

$ ./qemu-nbd -L 
exports available: 1
 export: ''
  size:  67108864
  flags: 0x61 ( trim zeroes )

What I couldn't work out is how to connect to other hosts.  It's
possible to add -p or -k to change the localhost port number or to use
a Unix domain socket (and I checked both work).  However can we
connect to remote hosts?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

On 12/1/18 1:42 AM, Richard W.M. Jones wrote:
> On Fri, Nov 30, 2018 at 04:03:29PM -0600, Eric Blake wrote:
>> I note that upstream NBD has 'nbd-client -l $host' for querying
>> just export names (with no quoting, so you have to know that
>> a blank line means the default export), but it wasn't powerful
>> enough, so I implemented 'qemu-nbd -L' to document everything.
>> Upstream NBD has separate 'nbd-client' and 'nbd-server' binaries,
>> while we only have 'qemu-nbd' (which is normally just a server,
>> but 'qemu-nbd -c' also operates a second thread as a client).
>> Our other uses of qemu as NBD client are for consuming a block
>> device (as in qemu-io, qemu-img, or a drive to qemu) - but those
>> binaries are less suited to something so specific to the NBD
>> protocol.
> 
> I tried it against nbdkit and it works (obviously):
> 
> $ ./qemu-nbd -L
> exports available: 1
>   export: ''
>    size:  67108864
>    flags: 0x61 ( trim zeroes )
> 
> What I couldn't work out is how to connect to other hosts.  It's
> possible to add -p or -k to change the localhost port number or to use
> a Unix domain socket (and I checked both work).  However can we
> connect to remote hosts?

Should work by adding '-b host' (if -b is omitted, it defaults to 
0.0.0.0).  However, I admit that so far I have only tested '-b 
localhost' or '-b 127.0.0.1' (per the iotest additions at the end of the 
series), so it could well be something that I missed in setting up the 
client socket in nbd_build_socket_address().

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

On Sat, Dec 01, 2018 at 07:57:51AM -0600, Eric Blake wrote:
> On 12/1/18 1:42 AM, Richard W.M. Jones wrote:
> >On Fri, Nov 30, 2018 at 04:03:29PM -0600, Eric Blake wrote:
> >>I note that upstream NBD has 'nbd-client -l $host' for querying
> >>just export names (with no quoting, so you have to know that
> >>a blank line means the default export), but it wasn't powerful
> >>enough, so I implemented 'qemu-nbd -L' to document everything.
> >>Upstream NBD has separate 'nbd-client' and 'nbd-server' binaries,
> >>while we only have 'qemu-nbd' (which is normally just a server,
> >>but 'qemu-nbd -c' also operates a second thread as a client).
> >>Our other uses of qemu as NBD client are for consuming a block
> >>device (as in qemu-io, qemu-img, or a drive to qemu) - but those
> >>binaries are less suited to something so specific to the NBD
> >>protocol.
> >
> >I tried it against nbdkit and it works (obviously):
> >
> >$ ./qemu-nbd -L
> >exports available: 1
> >  export: ''
> >   size:  67108864
> >   flags: 0x61 ( trim zeroes )
> >
> >What I couldn't work out is how to connect to other hosts.  It's
> >possible to add -p or -k to change the localhost port number or to use
> >a Unix domain socket (and I checked both work).  However can we
> >connect to remote hosts?
> 
> Should work by adding '-b host' (if -b is omitted, it defaults to
> 0.0.0.0).  However, I admit that so far I have only tested '-b
> localhost' or '-b 127.0.0.1' (per the iotest additions at the end of
> the series), so it could well be something that I missed in setting
> up the client socket in nbd_build_socket_address().

Must have missed that option.  It does work, thanks.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html