From nobody Sat Nov 15 23:34:44 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1745772691; cv=none; d=zohomail.com; s=zohoarc; b=H1Pss/slclM02ql3W/QYNvXuQJIDm5Abb2BsAzUtDDpT8QxK8PWq3YNYFBxwlkLpL0hmvpAwkQRw0Otr2N56xR9hJiq+mj62IbeRJon0PyOcnB0RKjt91qdR9XKD5deT8i1q2fVyZ8z5G4aURWt4VjxclI5U9e7aGP9BS/XNP9o= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1745772691; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=s1t7O7lKEtcAOseDx12/Rm6S4l1mtQh+wRnuTPzrVdA=; b=IGpwQnk2gvM+YnfyrX32SQd2NxHXzz6G0iPg1iu2FP62XmRf4X99uXnyu7cxYsQeIE5Bl8iv/xbFr5TN7fXzDb61/v5BjCPXaZ4v5krxsMe4MS8i5pNIk0uhcR+fF0lw6TtEdilRzSvuISc6mjF68vEC3f4MvcAR6Wtn1NAH6pI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1745772691023917.2454735258717; Sun, 27 Apr 2025 09:51:31 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1u95DJ-0000zK-Ky; Sun, 27 Apr 2025 12:50:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1u95DI-0000z4-8u for qemu-devel@nongnu.org; Sun, 27 Apr 2025 12:50:44 -0400 Received: from mail-wr1-x433.google.com ([2a00:1450:4864:20::433]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1u95DB-0005pT-AB for qemu-devel@nongnu.org; Sun, 27 Apr 2025 12:50:39 -0400 Received: by mail-wr1-x433.google.com with SMTP id ffacd0b85a97d-3a07a7b4ac7so733716f8f.2 for ; Sun, 27 Apr 2025 09:50:36 -0700 (PDT) Received: from localhost.localdomain (46-116-102-127.bb.netvision.net.il. [46.116.102.127]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3a073c8cd7fsm8783139f8f.1.2025.04.27.09.50.33 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sun, 27 Apr 2025 09:50:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1745772635; x=1746377435; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=s1t7O7lKEtcAOseDx12/Rm6S4l1mtQh+wRnuTPzrVdA=; b=a1/4qPodf8yd7GpAyFmsWgUi+X/C/kjOvFWAJ/XgBdC6UZnmLgOl21BvUhLmlNO3vp 2D4Ysbcoyq/SAKmzhpBlr8N/cpzpyhsbo6KlPno9b9h/Whpk62S7EpaAR2j94cOBGzAx 7fiPd+WUNrarf38PNAho2m5eXqyQlrQiTvROC7RSOhjQpCtymSowUdTqJ2/YLuiOwVs2 OL8YtHbqGinylsnUwMw/TlZJDrQkqPEHcU8i7/ms8KScKfG084w+ZZo+z9qcAk2YGOJJ T8OeICSSBU8rbVO/RZLmjrXziJmn8wG8K4MqHrRYTrHl+G3ZFGBf0r6TxShvc6soGLzl viEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745772635; x=1746377435; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=s1t7O7lKEtcAOseDx12/Rm6S4l1mtQh+wRnuTPzrVdA=; b=kr/jqM/KWz2E1rRgm9yihb4wqlKC+8qFTkjBtaMlgRrnwWLlZ+6xxduqhbGKoSGIBh +WW+l1kbCxJJmBui2UOfv6lm5p5cSqVJ7JIiIdElOnSAhZhiN1PfVwYkecPNG2EkVKlX 3BoVL4V7aIH2TAr+u7Ksb/VUFHpA/4gYCmMukTZkU9RyLxKybvwR+mG7Zm176fRoO2yZ u6LRoqKjd5qjbwFV228LbuSW2hal6G8AopGLU736R3VOppVRyT9jfZ4RnYSE2eILmXHQ jeCCnCIZwf5NADSVOx15XI73iLDnmeXVZfx/1qR0mvslTcHQ4tNahs3eoQUVRrj2t/Vf wSZg== X-Gm-Message-State: AOJu0Yxd+0TzGAYezZrZm4Z798kP5zi0NgzZL8/uNtDDNHD0l70wSxgn GChfxCPCbVg+apE6m3E6rZK7m701WPO47pwlGrF96JjKLaXlcCStc/po1w== X-Gm-Gg: ASbGnctmrn1MF8T3hOfxUvX3+rhJ3vauQTSMh+cJ4xF2owSkyoT734qkqTdR2PS2w97 49oUdbYNaNCS5CdUOocLQc1q5RhcwtkwifiPdYrsEWvDCgSa0Knms7jaKwyPwKc/FqjnwKwx2JP T/F3c6xE0y2c9c6Mc5vimkIueVDQCDzg/svDPidT5hYcLeVcqMrI8ucAkArnNdxZ8dnFiDD90m2 o1UlnZBA4+0xNKn0UHC2gbffHr9goG9N3X6fODCovwcf+wtxQ8yZHmKBTVnhveHFVTmrXeKCtYk 35yjL9qa+PBDDnfulgPphpQCc67lly4neKPQ7cJ46ospoi1R7sq+HpluRPsDSkV5jI42KkidOP4 yaMLcDDx0H4+syc1OnJm6FMFvSVPE X-Google-Smtp-Source: AGHT+IFYFc93qIQO2rsMtlUEDAaPXnznx1ydvvWPA4g+rsIZ/q5GlgQs9abU4lDp1+fY2OlSNjKjvg== X-Received: by 2002:a5d:5f4d:0:b0:38f:503a:d93f with SMTP id ffacd0b85a97d-3a074f15168mr6849577f8f.40.1745772635022; Sun, 27 Apr 2025 09:50:35 -0700 (PDT) From: Nir Soffer To: qemu-devel@nongnu.org Cc: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , "Richard Jones" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , "Eric Blake" , Nir Soffer Subject: [PATCH v3 1/2] io: Increase unix socket buffers size on macOS Date: Sun, 27 Apr 2025 19:50:28 +0300 Message-Id: <20250427165029.9072-2-nirsof@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250427165029.9072-1-nirsof@gmail.com> References: <20250427165029.9072-1-nirsof@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::433; envelope-from=nirsof@gmail.com; helo=mail-wr1-x433.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1745772692524019100 Content-Type: text/plain; charset="utf-8" On macOS we need to increase unix stream socket buffers size on the client and server to get good performance. We set socket buffers on macOS after connecting or accepting a client connection. For unix datagram socket we need different configuration that can be done later. Testing shows that setting socket receive buffer size (SO_RCVBUF) has no effect on performance, so we set only the send buffer size (SO_SNDBUF). It seems to work like Linux but not documented. Testing shows that optimal buffer size is 512k to 4 MiB, depending on the test case. The difference is very small, so I chose 2 MiB. I tested reading from qemu-nbd and writing to qemu-nbd with qemu-img and computing a blkhash with nbdcopy and blksum. To focus on NBD communication and get less noisy results, I tested reading and writing to null-co driver. I added a read-pattern option to the null-co driver to return data full of 0xff: NULL=3D"json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '1= 0g', 'read-pattern': -1}}" For testing buffer size I added an environment variable for setting the socket buffer size. Read from qemu-nbd via qemu-img convert. In this test buffer size of 2m is optimal (12.6 times faster). qemu-nbd -r -t -e 0 -f raw -k /tmp/nbd.sock "$NULL" & qemu-img convert -f raw -O raw -W -n "nbd+unix:///?socket=3D/tmp/nbd.so= ck" "$NULL" | buffer size | time | user | system | |-------------|---------|---------|---------| | default | 13.361 | 2.653 | 5.702 | | 65536 | 2.283 | 0.204 | 1.318 | | 131072 | 1.673 | 0.062 | 1.008 | | 262144 | 1.592 | 0.053 | 0.952 | | 524288 | 1.496 | 0.049 | 0.887 | | 1048576 | 1.234 | 0.047 | 0.738 | | 2097152 | 1.060 | 0.080 | 0.602 | | 4194304 | 1.061 | 0.076 | 0.604 | Write to qemu-nbd with qemu-img convert. In this test buffer size of 2m is optimal (9.2 times faster). qemu-nbd -t -e 0 -f raw -k /tmp/nbd.sock "$NULL" & qemu-img convert -f raw -O raw -W -n "$NULL" "nbd+unix:///?socket=3D/tm= p/nbd.sock" | buffer size | time | user | system | |-------------|---------|---------|---------| | default | 8.063 | 2.522 | 4.184 | | 65536 | 1.472 | 0.430 | 0.867 | | 131072 | 1.071 | 0.297 | 0.654 | | 262144 | 1.012 | 0.239 | 0.587 | | 524288 | 0.970 | 0.201 | 0.514 | | 1048576 | 0.895 | 0.184 | 0.454 | | 2097152 | 0.877 | 0.174 | 0.440 | | 4194304 | 0.944 | 0.231 | 0.535 | Compute a blkhash with nbdcopy, using 4 NBD connections and 256k request size. In this test buffer size of 4m is optimal (5.1 times faster). qemu-nbd -r -t -e 0 -f raw -k /tmp/nbd.sock "$NULL" & nbdcopy --blkhash "nbd+unix:///?socket=3D/tmp/nbd.sock" null: | buffer size | time | user | system | |-------------|---------|---------|---------| | default | 8.624 | 5.727 | 6.507 | | 65536 | 2.563 | 4.760 | 2.498 | | 131072 | 1.903 | 4.559 | 2.093 | | 262144 | 1.759 | 4.513 | 1.935 | | 524288 | 1.729 | 4.489 | 1.924 | | 1048576 | 1.696 | 4.479 | 1.884 | | 2097152 | 1.710 | 4.480 | 1.763 | | 4194304 | 1.687 | 4.479 | 1.712 | Compute a blkhash with blksum, using 1 NBD connection and 256k read size. In this test buffer size of 512k is optimal (10.3 times faster). qemu-nbd -r -t -e 0 -f raw -k /tmp/nbd.sock "$NULL" & blksum "nbd+unix:///?socket=3D/tmp/nbd.sock" | buffer size | time | user | system | |-------------|---------|---------|---------| | default | 13.085 | 5.664 | 6.461 | | 65536 | 3.299 | 5.106 | 2.515 | | 131072 | 2.396 | 4.989 | 2.069 | | 262144 | 1.607 | 4.724 | 1.555 | | 524288 | 1.271 | 4.528 | 1.224 | | 1048576 | 1.294 | 4.565 | 1.333 | | 2097152 | 1.299 | 4.569 | 1.344 | | 4194304 | 1.291 | 4.559 | 1.327 | Signed-off-by: Nir Soffer --- io/channel-socket.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/io/channel-socket.c b/io/channel-socket.c index 608bcf066e..06901ab694 100644 --- a/io/channel-socket.c +++ b/io/channel-socket.c @@ -21,6 +21,7 @@ #include "qapi/error.h" #include "qapi/qapi-visit-sockets.h" #include "qemu/module.h" +#include "qemu/units.h" #include "io/channel-socket.h" #include "io/channel-util.h" #include "io/channel-watch.h" @@ -37,6 +38,33 @@ =20 #define SOCKET_MAX_FDS 16 =20 +/* + * Testing shows that 2m send buffer gives best throuput and lowest cpu us= age. + * Changing the receive buffer size has no effect on performance. + */ +#ifdef __APPLE__ +#define UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE (2 * MiB) +#endif /* __APPLE__ */ + +static void qio_channel_socket_set_buffers(QIOChannelSocket *ioc) +{ + if (ioc->localAddr.ss_family =3D=3D AF_UNIX) { + int type; + socklen_t type_len =3D sizeof(type); + + if (getsockopt(ioc->fd, SOL_SOCKET, SO_TYPE, &type, &type_len) =3D= =3D -1) { + return; + } + +#ifdef UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE + if (type =3D=3D SOCK_STREAM) { + const int value =3D UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE; + setsockopt(ioc->fd, SOL_SOCKET, SO_SNDBUF, &value, sizeof(valu= e)); + } +#endif /* UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE */ + } +} + SocketAddress * qio_channel_socket_get_local_address(QIOChannelSocket *ioc, Error **errp) @@ -174,6 +202,8 @@ int qio_channel_socket_connect_sync(QIOChannelSocket *i= oc, } #endif =20 + qio_channel_socket_set_buffers(ioc); + qio_channel_set_feature(QIO_CHANNEL(ioc), QIO_CHANNEL_FEATURE_READ_MSG_PEEK); =20 @@ -410,6 +440,8 @@ qio_channel_socket_accept(QIOChannelSocket *ioc, } #endif /* WIN32 */ =20 + qio_channel_socket_set_buffers(cioc); + qio_channel_set_feature(QIO_CHANNEL(cioc), QIO_CHANNEL_FEATURE_READ_MSG_PEEK); =20 --=20 2.39.5 (Apple Git-154) From nobody Sat Nov 15 23:34:44 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1745772690; cv=none; d=zohomail.com; s=zohoarc; b=Z9bLvSKnxCdBHBubeFHkJn5Oy8BD83kHkvxaAvLGJ8cr9lqH47fYrBPPnfJk1Q9IadMpBdnMd6dWSrvWYN6n5c0SFPR/A2da2Fd84Zqiwc0RgSEE/Si2XkKI0e7k7uHd2PqMG7aQZpYY3xB4Ee7P8tzNu9V8m4ClylNWfruP3FU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1745772690; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=7BQ16vIw1EHB1x27l7KHahTZhvMciox87CMxJC1bBgM=; b=EsKNf1ovlpWOksZ38r1Q0oXsw0A8+qss9Etquek59qMhIhD6A4GuoZx24Xm3DB/Cg3LVzh6TTYbBpBfckTZAgJVpUB7KpM4rZ4I4Pvg1AFFzDu5Xl6/AJueN03SeiaBnxyCbJiwJMmeWbgu37xZqtbVVBu+ARvrrzgR5szAbB0k= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1745772690941408.7989118293078; Sun, 27 Apr 2025 09:51:30 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1u95DL-00010X-1C; Sun, 27 Apr 2025 12:50:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1u95DJ-0000zN-NL for qemu-devel@nongnu.org; Sun, 27 Apr 2025 12:50:45 -0400 Received: from mail-wm1-x333.google.com ([2a00:1450:4864:20::333]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1u95DE-0005pj-RC for qemu-devel@nongnu.org; Sun, 27 Apr 2025 12:50:45 -0400 Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-43cf680d351so28769555e9.0 for ; Sun, 27 Apr 2025 09:50:37 -0700 (PDT) Received: from localhost.localdomain (46-116-102-127.bb.netvision.net.il. [46.116.102.127]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3a073c8cd7fsm8783139f8f.1.2025.04.27.09.50.35 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sun, 27 Apr 2025 09:50:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1745772636; x=1746377436; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7BQ16vIw1EHB1x27l7KHahTZhvMciox87CMxJC1bBgM=; b=ltuhItOGsvxUJ6yOwPrUYdTaTlhKEV9IBc6EYr1bEAghnjIrKDYs+H7SmygndyVURJ AHj9Jgo2EdfX5PzMtQM5vC6Ef5VCi+nTEaAKS/UOl2l93jBMlJ2dpnbZYQgUP92tco/N L7StHpwZFHh7e9WdBR6IT7ecB/qUDlzwJNGz5eqZQ1Qt7NpDwxI31h04K1gXnGGvyCPI ty92J3oW2WGNFx5Cx4YpS2+9rpmjv87UR6XCO02y/Bj1ntS/Rs/NH6LX52jA3cjQ2hO1 EMBxOuOhZj56AvJn8bHw7hg9d/R74/b6gWip+Fy0EL5ea1JEq2k8z+zN912wjqAhXytj ajmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745772636; x=1746377436; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7BQ16vIw1EHB1x27l7KHahTZhvMciox87CMxJC1bBgM=; b=nRpKG2jQeT4cs4jWA1B6votZ6M6M2YMPIITYXZ3wJ1y4vKIxgU3TvcV/6I8VRiwIqz yaGw3YvU3lYD8jiQ+AMJ/6wRBMC8pE9kxeuoi4pblIQoXRFtTMDTwZ7EPjOI2sNqr6I3 XLg7TfrWQwwNO7v5X+rZFXSnHbJb0JmDrmRdMZqpXPQCmgmSjrRiGzQ/W3VskLVYv0VC qsNrJL/V2bHz+KVqnTBd1rg3RSUZ4A3x9KAsmp256q/JqW+LFR2m1HMbH1cRe5cZEwTs YNTvNui9MCNf3YYjpULojfWhxdB2wut+hpVxHTznxMH8fy+ADP1JW5NkV5lJ+qbRasWK 9LgA== X-Gm-Message-State: AOJu0Yw2CovvvnKIsFpuu0xx4H6xto432jSruOG6mmJKfovieBep68Uy duhlSytHIKI8TqlDQ43b9J6itSjzC0/98QK/th/j0fMH3hY85Bu29wFrWw== X-Gm-Gg: ASbGncuDHz/YEeH8OhnKXO0O6Qj033rkiPRL56rfJnq/hHG0h2isZzyO4mdX7MmNat2 hb6cNCZtMFi/XZj7fesN1DfzPKcccRAzsRUf+brL7wPFJy7+w5b18J2Ty+w12OZi29jxg73FirM 2R/HcHIH0fVKLfUsSd2wFPgkzuDggqiBbxwQKs7grTcPy14cvbZ2kRwbg5itEx+K9CbI7CD9mbl UY81obDZ7jmbRzcVHpvnZdoGjOhnfDAhPAFmUOH9gN4svzp8CZ9to0iB7Z5OmtYjhoZrLVchsRg taLyE2tPNKuXX0XDuO5JevkrYdxRmSDP7qODBEdQYxdDQXPv2k1bmBjGXw8ASgoYmTDZjOAwQ0n l+Gb13ZLmG7LQe5eoTjAyGmN0GdD9 X-Google-Smtp-Source: AGHT+IFFwoldIJQiHfNwBoilsP7suJ9qF/0En0sn6t67KOayrXDWvCYI2MH+cLNJBClEn38jmmp9Fw== X-Received: by 2002:a05:600c:6c95:b0:43c:f680:5c2e with SMTP id 5b1f17b1804b1-4409c512c5amr111714175e9.13.1745772636242; Sun, 27 Apr 2025 09:50:36 -0700 (PDT) From: Nir Soffer To: qemu-devel@nongnu.org Cc: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , "Richard Jones" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , "Eric Blake" , Nir Soffer Subject: [PATCH v3 2/2] io: Increase unix socket buffers on Linux Date: Sun, 27 Apr 2025 19:50:29 +0300 Message-Id: <20250427165029.9072-3-nirsof@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250427165029.9072-1-nirsof@gmail.com> References: <20250427165029.9072-1-nirsof@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::333; envelope-from=nirsof@gmail.com; helo=mail-wm1-x333.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1745772692499019100 Content-Type: text/plain; charset="utf-8" Like macOS we have similar issue on Linux. For TCP socket the send buffer size is 2626560 bytes (~2.5 MiB) and we get good performance. However for unix socket the default and maximum buffer size is 212992 bytes (208 KiB) and we see poor performance when using one NBD connection, up to 4 times slower than macOS on the same machine. Tracing shows that for every 2 MiB payload (qemu uses 2 MiB io size), we do 1 recvmsg call with TCP socket, and 10 recvmsg calls with unix socket. Fixing this issue requires changing the maximum send buffer size (the receive buffer size is ignored). This can be done using: $ cat /etc/sysctl.d/net-mem-max.conf net.core.wmem_max =3D 2097152 $ sudo sysctl -p /etc/sysctl.d/net-mem-max.conf With this we can set the socket buffer size to 2 MiB. With the defaults the value requested by qemu is clipped to the maximum size and has no effect. I tested on 2 machines: - Fedora 42 VM on MacBook Pro M2 Max - Dell PowerEdge R640 (Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz) On the older Dell machine we see very little improvement, up to 1.03 higher throughput. On the M2 machine we see up to 2.67 times higher throughput. The following results are from the M2 machine. Reading from qemu-nbd with qemu-img convert. In this test buffer size of 4m is optimal (2.28 times faster). | buffer size | time | user | system | |-------------|---------|---------|---------| | default | 4.292 | 0.243 | 1.604 | | 524288 | 2.167 | 0.058 | 1.288 | | 1048576 | 2.041 | 0.060 | 1.238 | | 2097152 | 1.884 | 0.060 | 1.191 | | 4194304 | 1.881 | 0.054 | 1.196 | Writing to qemu-nbd with qemu-img convert. In this test buffer size of 1m is optimal (2.67 times faster). | buffer size | time | user | system | |-------------|---------|---------|---------| | default | 3.113 | 0.334 | 1.094 | | 524288 | 1.173 | 0.179 | 0.654 | | 1048576 | 1.164 | 0.164 | 0.670 | | 2097152 | 1.227 | 0.197 | 0.663 | | 4194304 | 1.227 | 0.198 | 0.666 | Computing a blkhash with nbdcopy. In this test buffer size of 512k is optimal (1.19 times faster). | buffer size | time | user | system | |-------------|---------|---------|---------| | default | 2.140 | 4.483 | 2.681 | | 524288 | 1.794 | 4.467 | 2.572 | | 1048576 | 1.807 | 4.447 | 2.644 | | 2097152 | 1.822 | 4.461 | 2.698 | | 4194304 | 1.827 | 4.465 | 2.700 | Computing a blkhash with blksum. In this test buffer size of 4m is optimal (2.65 times faster). | buffer size | time | user | system | |-------------|---------|---------|---------| | default | 3.582 | 4.595 | 2.392 | | 524288 | 1.499 | 4.384 | 1.482 | | 1048576 | 1.377 | 4.381 | 1.345 | | 2097152 | 1.388 | 4.389 | 1.354 | | 4194304 | 1.352 | 4.395 | 1.302 | Signed-off-by: Nir Soffer --- io/channel-socket.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/io/channel-socket.c b/io/channel-socket.c index 06901ab694..f2974fab74 100644 --- a/io/channel-socket.c +++ b/io/channel-socket.c @@ -39,12 +39,13 @@ #define SOCKET_MAX_FDS 16 =20 /* - * Testing shows that 2m send buffer gives best throuput and lowest cpu us= age. - * Changing the receive buffer size has no effect on performance. + * Testing shows that 2m send buffer is optimal. Changing the receive buff= er + * size has no effect on performance. + * On Linux we need to increase net.core.wmem_max to make this effective. */ -#ifdef __APPLE__ +#if defined(__APPLE__) || defined(__linux__) #define UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE (2 * MiB) -#endif /* __APPLE__ */ +#endif =20 static void qio_channel_socket_set_buffers(QIOChannelSocket *ioc) { --=20 2.39.5 (Apple Git-154)