On both macOS and Linux, the default send buffer size is too small causing poor
performance when reading and writing to qemu-nbd. A simple way to experience
this is to compare TCP and unix sockets, showing that TCP socket is much
faster. Programs like nbdcopy partly mitigate this by using multiple NBD
connections.

On macOS the default send buffer size is 8192 bytes. Increasing the send buffer
size to 2 MiB shows up to *12.6 times higher throughput* and lower cpu usage.

On Linux the default and maximum buffer size is 212992 bytes. Increasing the
send buffer size to 2 MiB shows up to *2.7 times higher throughput* and lower
cpu usage. On older machine we see very little improvement, up to 1.03 times
higher throughput.

We likely have the same issue on other platforms. It should be easy to enable
this change for more platform by defining UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE.

Changes since v3:
- Modify only NBD sockets, since not all sockets need a larger buffer size, and
  the fixed memory overhead per socker may be a significant amount of extra
  memory overhead on a host. (Daniel)
- Add qio_channel_socket_set_send_buffer() helper to set buffer size from nbd
  code.
- Remove the check for SOCK_STREAM since NBD does not support SOCK_DGRAM.
- Warn if we fail to set the socket send buffer size.

v3 was here:
https://lists.gnu.org/archive/html/qemu-devel/2025-04/msg05096.html

Nir Soffer (3):
  io: Add helper for setting socket send buffer size
  nbd: Set unix socket send buffer on macOS
  nbd: Set unix socket send buffer on Linux

 include/io/channel-socket.h | 13 +++++++++++++
 io/channel-socket.c         | 11 +++++++++++
 nbd/client-connection.c     |  3 +++
 nbd/common.c                | 26 ++++++++++++++++++++++++++
 nbd/nbd-internal.h          |  5 +++++
 nbd/server.c                |  2 ++
 6 files changed, 60 insertions(+)

-- 
2.39.5 (Apple Git-154)

On Sat, May 17, 2025 at 11:11:51PM +0300, Nir Soffer wrote:
> On both macOS and Linux, the default send buffer size is too small causing poor
> performance when reading and writing to qemu-nbd. A simple way to experience
> this is to compare TCP and unix sockets, showing that TCP socket is much
> faster. Programs like nbdcopy partly mitigate this by using multiple NBD
> connections.
> 
> On macOS the default send buffer size is 8192 bytes. Increasing the send buffer
> size to 2 MiB shows up to *12.6 times higher throughput* and lower cpu usage.
> 
> On Linux the default and maximum buffer size is 212992 bytes. Increasing the
> send buffer size to 2 MiB shows up to *2.7 times higher throughput* and lower
> cpu usage. On older machine we see very little improvement, up to 1.03 times
> higher throughput.
> 
> We likely have the same issue on other platforms. It should be easy to enable
> this change for more platform by defining UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE.

Thanks; I'm staging this through my NBD tree.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org