From nobody Wed Dec 17 03:58:29 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1747512772; cv=none; d=zohomail.com; s=zohoarc; b=EeRwAeMEATnzBz5jch+mtg3efzNI1nU48RKav7aAEruA9oFcVQUdR3OWsZVQLXO51XX0pmWLbrtfJBxP2V0xWPyAraMXPsCfZGGRF0T9s/VRxmLk+R0dDm5WONRbRKybz3AtdWOM05YBtprd7rlY6h1uOk2pwTNKFJZLL3NStDE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1747512772; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=cIMMYJUwbBBktOXZV3ZqZIlfvVDKWD/JTEdHmGS7mkE=; b=nxnDF4HUnJkDnD5nXg8bcV1qQcayDHzkD7Zy2Q0WOdOCsVlqR40FYfbvtAf37ObN3VKUKbYRAdbz+66h1F0VXT+9f+P0yRBr5pj9LK5y8qwLhH1F5MRrEaayrhxpj+GXwe1wCLmGQCys1Lt9srFEwC50IhiWZtLc6iPsSAhusN8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1747512772922675.4787924288903; Sat, 17 May 2025 13:12:52 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uGNt7-0005S5-6Q; Sat, 17 May 2025 16:12:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uGNt4-0005R9-AY; Sat, 17 May 2025 16:12:02 -0400 Received: from mail-wm1-x32b.google.com ([2a00:1450:4864:20::32b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1uGNt2-0001zz-Pj; Sat, 17 May 2025 16:12:02 -0400 Received: by mail-wm1-x32b.google.com with SMTP id 5b1f17b1804b1-441d437cfaaso18143275e9.1; Sat, 17 May 2025 13:12:00 -0700 (PDT) Received: from localhost.localdomain (46-116-102-127.bb.netvision.net.il. [46.116.102.127]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-442fd59ab21sm78714715e9.38.2025.05.17.13.11.57 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 17 May 2025 13:11:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1747512719; x=1748117519; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cIMMYJUwbBBktOXZV3ZqZIlfvVDKWD/JTEdHmGS7mkE=; b=OinnNcKu4at1hS+vUfX/UKHbA5GavoJf6rWx7sCBcJVfJ4v1EISY/xTjRcp9VFw8Jz tLQAk8FhC2HMb2Ml8cTH007ZF/oRyXl9UDvcX/dlDGHO2EAcZB3PEqjSq4+wvjGwnzZj IowgdXxC6J4KB8Sq0T/tFpHoKrGNv2I1dxKb9CrWY4roHldox2+DNJvBQLh/i3Pl6CKB fp6VBITnX66riLkKuSFHPcWnRsBHGfjsNU0BgoXAx01IKZ8MrCS/0oLvGaIyZMTfVRT/ SrjbF1EeLeUEqGV1p431iWaZ34cjWglI5Bn60g63gO6PYHuP9SqX6wPG6T88GDg/lrwU QlxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747512719; x=1748117519; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cIMMYJUwbBBktOXZV3ZqZIlfvVDKWD/JTEdHmGS7mkE=; b=ti1lmN7CNBqZHyDNk9HHxJ1JO2BdYft1UhNi2w24S/jD5Ad6KLU7MEwuolXr78A0pJ OTYTdPehEJ4VNVFE00QMkWP12EDqVX5lw97T60fBwAVkzzoObkjpI9IrNNvRhUIxXbub ce56ZSou+BjvG2OC0yBgpLY85LR9Zf/rggaxG5MmxiZlhf0LdeavhGXLm0coDwwdIgyu P+oqdKF7rvn/+P6lC5xqc4I6xVtlfQoZNSLxCQGqNJYYq38XaCR3KwJO997dl1PpQpTm /B2kbMJiccVAxS6fOM4t5PX8SP/vBa1tP+xtyj2ZxXQNhRdtdXLcmovCmtPEcR7Bjf/R ZXcw== X-Forwarded-Encrypted: i=1; AJvYcCVFnDi8+njNOKSljwh3U7PwT6UMXZrANrgR+/NTOCUdNUlAXOwSuyIonkpTs8PIDQAuiPZw1Y0RHgzu@nongnu.org X-Gm-Message-State: AOJu0Yy6w1UYrjagALIWGBgQN0kZKDXpO3sKg52iMrAR/rJGdEvvDTX6 yn5xJyJ9b0+kMXy2yrKNzTi1jEfFm7DLv+GBWFC9P9Utsk14crxV/EHXaLqSXSIdt+4= X-Gm-Gg: ASbGncuFF1tEjLi39UVg7kpHNS0s7V/sro0Ppd+cNvDEGcsfygql7qWZnPMUrQkqEtg m2B8Ik3QeU6UFfrNgPulxl6Ii+BQmm7NGtjFIgta/EIsSn4PXhVOxUbbdohYcS4mdPaKNIKmncS 8QT1tkcurAgJwleo7/PLb1IMwo5kp+Ah0IxcqVYW80RKoLhuMOJ5Yv6pD79reynjSI5AGL6kktL TjR9qDipM/KvyiEUpz7omgJ/NDg7jgX2sMhooocK3+k4+O74bXnHT+Cx8lb71BJCSWKUx3Vgb6P 6fAOLQrUHvYbFolrZIrfQ0S7ky8klkreqP9CkwmRESeN0YHyKq56tlCsOjg33+nN0YXnzz0gn0T tUKvVwAiNTAdlVAUa4hQuFxdDQkWLG8obVA== X-Google-Smtp-Source: AGHT+IHFyJ8Qtd2yQNi5HwXukWollkAroIv0NU+b7U3QR90czgquq+UVaHbKLlANjwjeI9Jk+eC1rQ== X-Received: by 2002:a05:600c:528b:b0:43d:4e9:27f3 with SMTP id 5b1f17b1804b1-442fefee1b8mr65824795e9.9.1747512718495; Sat, 17 May 2025 13:11:58 -0700 (PDT) From: Nir Soffer To: qemu-devel@nongnu.org Cc: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , Eric Blake , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , "Richard Jones" , Vladimir Sementsov-Ogievskiy , qemu-block@nongnu.org, Nir Soffer Subject: [PATCH v4 1/3] io: Add helper for setting socket send buffer size Date: Sat, 17 May 2025 23:11:52 +0300 Message-Id: <20250517201154.88456-2-nirsof@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250517201154.88456-1-nirsof@gmail.com> References: <20250517201154.88456-1-nirsof@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::32b; envelope-from=nirsof@gmail.com; helo=mail-wm1-x32b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1747512774817116600 Content-Type: text/plain; charset="utf-8" Testings reading and writing from qemu-nbd using a unix domain socket shows that the platform default send buffer size is too low, leading to poor performance and hight cpu usage. Add a helper for setting socket send buffer size to be used in NBD code. It can also be used in other context. We don't need a helper for receive buffer size since it is not used with unix domain sockets. This is documented for Linux, and not documented for macOS. Failing to set the socket buffer size is not a fatal error, but the caller may want to warn about the failure. Signed-off-by: Nir Soffer Reviewed-by: Daniel P. Berrang=C3=A9 --- include/io/channel-socket.h | 13 +++++++++++++ io/channel-socket.c | 11 +++++++++++ 2 files changed, 24 insertions(+) diff --git a/include/io/channel-socket.h b/include/io/channel-socket.h index ab15577d38..a88cf8b3a9 100644 --- a/include/io/channel-socket.h +++ b/include/io/channel-socket.h @@ -261,5 +261,18 @@ QIOChannelSocket * qio_channel_socket_accept(QIOChannelSocket *ioc, Error **errp); =20 +/** + * qio_channel_socket_set_send_buffer: + * @ioc: the socket channel object + * @size: buffer size + * @errp: pointer to a NULL-initialized error object + * + * Set the underlying socket send buffer size. + * + * Retruns: 0 on success, or -1 on error. + */ +int qio_channel_socket_set_send_buffer(QIOChannelSocket *ioc, + size_t size, + Error **errp); =20 #endif /* QIO_CHANNEL_SOCKET_H */ diff --git a/io/channel-socket.c b/io/channel-socket.c index 608bcf066e..a776cd45f5 100644 --- a/io/channel-socket.c +++ b/io/channel-socket.c @@ -78,6 +78,17 @@ qio_channel_socket_new(void) return sioc; } =20 +int qio_channel_socket_set_send_buffer(QIOChannelSocket *ioc, + size_t size, + Error **errp) +{ + if (setsockopt(ioc->fd, SOL_SOCKET, SO_SNDBUF, &size, sizeof(size)) < = 0) { + error_setg_errno(errp, errno, "Unable to set socket send buffer si= ze"); + return -1; + } + + return 0; +} =20 static int qio_channel_socket_set_fd(QIOChannelSocket *sioc, --=20 2.39.5 (Apple Git-154) From nobody Wed Dec 17 03:58:29 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1747512772; cv=none; d=zohomail.com; s=zohoarc; b=jP0bcYjsxOy0UUEJjEfHS+EMbcJyjZdHQz8XV7eKJnR83QQz4WM39oE4iqcf8isrZMhu/XauLFwlou93GtLs0pTvDgDqhitql9RKYTrqTT9gnqyGF6Gkt0AfhTPBwZSte5dklgwMitVZSYWO7JaHE5X1EDef1v9z5i8G2Rn8Am8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1747512772; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=8Fsc2GCK3vxhvJVRGNGESOqw+ualHxA6ub7eW9EYjAw=; b=RsZVAT91la8927hBXyi1KoMotQi2PYC4T+I5tpD716PyX7p4yckdpMxC3gqDVA3GGpA/EOgI3j3wJyWtClSbyKmPL3ppFLMtPPCkkEn8IAoS4zCYK+x2dUxbqxyKZwtKO5u8pTugDBH8RoOeeUOfCCqugSiiiCqKSf9I6tr9gXM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 174751277269461.003666299416636; Sat, 17 May 2025 13:12:52 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uGNt9-0005Sm-VZ; Sat, 17 May 2025 16:12:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uGNt6-0005S1-Ql; Sat, 17 May 2025 16:12:05 -0400 Received: from mail-wm1-x32b.google.com ([2a00:1450:4864:20::32b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1uGNt4-00020P-SI; Sat, 17 May 2025 16:12:04 -0400 Received: by mail-wm1-x32b.google.com with SMTP id 5b1f17b1804b1-442ec3ce724so25234305e9.0; Sat, 17 May 2025 13:12:02 -0700 (PDT) Received: from localhost.localdomain (46-116-102-127.bb.netvision.net.il. [46.116.102.127]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-442fd59ab21sm78714715e9.38.2025.05.17.13.11.58 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 17 May 2025 13:11:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1747512721; x=1748117521; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8Fsc2GCK3vxhvJVRGNGESOqw+ualHxA6ub7eW9EYjAw=; b=I3sJNpJ0jRlbswTU1M8HFbOOZQXCdFeK2dlvbdBcxpi1nJ87dQKlXdvq0o/HffUxA1 x+RyzUY218P3snlpMQuoQJ8fdq+GeJb4gNUa24GFCM6F7ErFu4vosLAg+qDPY2Sh5WRA mopurAieP+1zKenPKaA2xeyGTplCQEcOXvO9bNVZo6AVFPQAzqZ8c6z1Es6MDERIVUtA /jFJoavmMDDzP5kGTpCXH10f8vqqDvF4LDJmsymX5c3whOEqqBMb0WYHPmuQA80dCLcV 4yUY91hD5/qvpe49hsFIcDGpibqfQrE9CudjHkhZYeTglS4S+rOsPjp23yXS+a/TZizF TR1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747512721; x=1748117521; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8Fsc2GCK3vxhvJVRGNGESOqw+ualHxA6ub7eW9EYjAw=; b=hJ0vVoHiRr5CEPOsuwFdpr248ORKrsrvGyB4PasiQ2fDf8ZgZxJR6zVBEK7IkvqdTB JzjEcurA8KSXN4/m2ZX2pcxedM7pgKGD7fnEvFPIo48IADlUYl5Ml+ZH5rtoZ1RoesWk 7sViAEJqH893DPjkCeSmnrE9BPJArZUW4Ht8Canj5FvAIkxg4mDwdbrH9zXh4VF95Vvz QWUmuy0BNsFXpqIWwYDDHW7Y5Bt2HVjHRAYeyouGr5EqmQlBbw/xq+oszSJDeHAi1blD vOGP1T0vu/o8A9G2l0DlHqgc99ZXzduCnYK1xaTeE7dhqkO+rrxNuoHLauYrTguViLf+ fFsA== X-Forwarded-Encrypted: i=1; AJvYcCWJ/T32rORCOq5vNuw1T20dWk8ZUx6+0p+/SDmC0gWBHYG7/CmqFFdyUB8ETzoCc+A6yJlvArk4d8S5@nongnu.org X-Gm-Message-State: AOJu0YztgYNPLMSJzgUNM8S7bGBQr3pDjcoC0MvGEVzfST+2GH6bNL7a TqaXBChqKR3KoFrs9qg7e63Zi4gupWgzk9tYvQpR3CMuG522D43sOxkk75EPEPfZapk= X-Gm-Gg: ASbGncsXqyDe4w3C/mKS0xEwzdij4K2sw2iDFAFW4XNQewTfb/xRPHtDkRDFRcbv+Cz eL38FDveusf6deJ0TWz62I+ln4rWm6C57+3fRCYigLJLnvSanWpTrR79sC8ETBWfMNpUw+rFMLO hsHwPans8x38pDC9fI6EH6eHAE3duRA5lq0Qao2aF+13zumHuYwYQwChrqCFxOtBIdZcCol1Av1 pMfuvNPdA9MiT3dJ5fUtEE6ancgCbQcOb09nnVibrGQzyskIanxnZT2lYF1V+bq/fVqkAitkhpo s7K3Ol49TCppgiw+kXSj5IWYsMRdRo2nBS+QnouAL00ES9G1Frk6Lg8JHHSn2Y3e6rE4CXgUjWF Kk7nkCNcfQsdCtNja6/d+Og+1ltLRyZknvg== X-Google-Smtp-Source: AGHT+IG5tCXPmaWFtO8t3nA2YfkdgmECuMVvelpAd0pRF7FA2l9MVreKXjUHntEGjqHls84Hz2vyjQ== X-Received: by 2002:a05:600c:3f06:b0:43b:ca39:6c75 with SMTP id 5b1f17b1804b1-442fd63c7aemr91824615e9.16.1747512720615; Sat, 17 May 2025 13:12:00 -0700 (PDT) From: Nir Soffer To: qemu-devel@nongnu.org Cc: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , Eric Blake , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , "Richard Jones" , Vladimir Sementsov-Ogievskiy , qemu-block@nongnu.org, Nir Soffer Subject: [PATCH v4 2/3] nbd: Set unix socket send buffer on macOS Date: Sat, 17 May 2025 23:11:53 +0300 Message-Id: <20250517201154.88456-3-nirsof@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250517201154.88456-1-nirsof@gmail.com> References: <20250517201154.88456-1-nirsof@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::32b; envelope-from=nirsof@gmail.com; helo=mail-wm1-x32b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1747512774908116600 Content-Type: text/plain; charset="utf-8" On macOS we need to increase unix socket buffers size on the client and server to get good performance. We set socket buffers on macOS after connecting or accepting a client connection. Testing shows that setting socket receive buffer size (SO_RCVBUF) has no effect on performance, so we set only the send buffer size (SO_SNDBUF). It seems to work like Linux but not documented. Testing shows that optimal buffer size is 512k to 4 MiB, depending on the test case. The difference is very small, so I chose 2 MiB. I tested reading from qemu-nbd and writing to qemu-nbd with qemu-img and computing a blkhash with nbdcopy and blksum. To focus on NBD communication and get less noisy results, I tested reading and writing to null-co driver. I added a read-pattern option to the null-co driver to return data full of 0xff: NULL=3D"json:{'driver': 'raw', 'file': {'driver': 'null-co', 'size': '1= 0g', 'read-pattern': 255}}" For testing buffer size I added an environment variable for setting the socket buffer size. Read from qemu-nbd via qemu-img convert. In this test buffer size of 2m is optimal (12.6 times faster). qemu-nbd -r -t -e 0 -f raw -k /tmp/nbd.sock "$NULL" & qemu-img convert -f raw -O raw -W -n "nbd+unix:///?socket=3D/tmp/nbd.so= ck" "$NULL" | buffer size | time | user | system | |-------------|---------|---------|---------| | default | 13.361 | 2.653 | 5.702 | | 65536 | 2.283 | 0.204 | 1.318 | | 131072 | 1.673 | 0.062 | 1.008 | | 262144 | 1.592 | 0.053 | 0.952 | | 524288 | 1.496 | 0.049 | 0.887 | | 1048576 | 1.234 | 0.047 | 0.738 | | 2097152 | 1.060 | 0.080 | 0.602 | | 4194304 | 1.061 | 0.076 | 0.604 | Write to qemu-nbd with qemu-img convert. In this test buffer size of 2m is optimal (9.2 times faster). qemu-nbd -t -e 0 -f raw -k /tmp/nbd.sock "$NULL" & qemu-img convert -f raw -O raw -W -n "$NULL" "nbd+unix:///?socket=3D/tm= p/nbd.sock" | buffer size | time | user | system | |-------------|---------|---------|---------| | default | 8.063 | 2.522 | 4.184 | | 65536 | 1.472 | 0.430 | 0.867 | | 131072 | 1.071 | 0.297 | 0.654 | | 262144 | 1.012 | 0.239 | 0.587 | | 524288 | 0.970 | 0.201 | 0.514 | | 1048576 | 0.895 | 0.184 | 0.454 | | 2097152 | 0.877 | 0.174 | 0.440 | | 4194304 | 0.944 | 0.231 | 0.535 | Compute a blkhash with nbdcopy, using 4 NBD connections and 256k request size. In this test buffer size of 4m is optimal (5.1 times faster). qemu-nbd -r -t -e 0 -f raw -k /tmp/nbd.sock "$NULL" & nbdcopy --blkhash "nbd+unix:///?socket=3D/tmp/nbd.sock" null: | buffer size | time | user | system | |-------------|---------|---------|---------| | default | 8.624 | 5.727 | 6.507 | | 65536 | 2.563 | 4.760 | 2.498 | | 131072 | 1.903 | 4.559 | 2.093 | | 262144 | 1.759 | 4.513 | 1.935 | | 524288 | 1.729 | 4.489 | 1.924 | | 1048576 | 1.696 | 4.479 | 1.884 | | 2097152 | 1.710 | 4.480 | 1.763 | | 4194304 | 1.687 | 4.479 | 1.712 | Compute a blkhash with blksum, using 1 NBD connection and 256k read size. In this test buffer size of 512k is optimal (10.3 times faster). qemu-nbd -r -t -e 0 -f raw -k /tmp/nbd.sock "$NULL" & blksum "nbd+unix:///?socket=3D/tmp/nbd.sock" | buffer size | time | user | system | |-------------|---------|---------|---------| | default | 13.085 | 5.664 | 6.461 | | 65536 | 3.299 | 5.106 | 2.515 | | 131072 | 2.396 | 4.989 | 2.069 | | 262144 | 1.607 | 4.724 | 1.555 | | 524288 | 1.271 | 4.528 | 1.224 | | 1048576 | 1.294 | 4.565 | 1.333 | | 2097152 | 1.299 | 4.569 | 1.344 | | 4194304 | 1.291 | 4.559 | 1.327 | Signed-off-by: Nir Soffer Reviewed-by: Daniel P. Berrang=C3=A9 --- nbd/client-connection.c | 3 +++ nbd/common.c | 25 +++++++++++++++++++++++++ nbd/nbd-internal.h | 5 +++++ nbd/server.c | 2 ++ 4 files changed, 35 insertions(+) diff --git a/nbd/client-connection.c b/nbd/client-connection.c index b11e266807..79ea97e4cc 100644 --- a/nbd/client-connection.c +++ b/nbd/client-connection.c @@ -31,6 +31,8 @@ #include "qapi/clone-visitor.h" #include "qemu/coroutine.h" =20 +#include "nbd/nbd-internal.h" + struct NBDClientConnection { /* Initialization constants, never change */ SocketAddress *saddr; /* address to connect to */ @@ -140,6 +142,7 @@ static int nbd_connect(QIOChannelSocket *sioc, SocketAd= dress *addr, return ret; } =20 + nbd_set_socket_send_buffer(sioc); qio_channel_set_delay(QIO_CHANNEL(sioc), false); =20 if (!info) { diff --git a/nbd/common.c b/nbd/common.c index 589a748cfe..9436e9d1d1 100644 --- a/nbd/common.c +++ b/nbd/common.c @@ -18,6 +18,9 @@ =20 #include "qemu/osdep.h" #include "trace.h" +#include "io/channel-socket.h" +#include "qapi/error.h" +#include "qemu/units.h" #include "nbd-internal.h" =20 /* Discard length bytes from channel. Return -errno on failure and 0 on @@ -264,3 +267,25 @@ const char *nbd_mode_lookup(NBDMode mode) return ""; } } + +/* + * Testing shows that 2m send buffer is optimal. Changing the receive buff= er + * size has no effect on performance. + */ +#if defined(__APPLE__) +#define UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE (2 * MiB) +#endif + +void nbd_set_socket_send_buffer(QIOChannelSocket *sioc) +{ +#ifdef UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE + if (sioc->localAddr.ss_family =3D=3D AF_UNIX) { + size_t size =3D UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE; + Error *errp =3D NULL; + + if (qio_channel_socket_set_send_buffer(sioc, size, &errp) < 0) { + warn_report_err(errp); + } + } +#endif /* UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE */ +} diff --git a/nbd/nbd-internal.h b/nbd/nbd-internal.h index 715d92d6ef..6bafeef5dd 100644 --- a/nbd/nbd-internal.h +++ b/nbd/nbd-internal.h @@ -74,4 +74,9 @@ static inline int nbd_write(QIOChannel *ioc, const void *= buffer, size_t size, =20 int nbd_drop(QIOChannel *ioc, size_t size, Error **errp); =20 +/* nbd_set_socket_send_buffer + * Set the socket send buffer size for optimal performance. + */ +void nbd_set_socket_send_buffer(QIOChannelSocket *sioc); + #endif diff --git a/nbd/server.c b/nbd/server.c index 2076fb2666..d242be9811 100644 --- a/nbd/server.c +++ b/nbd/server.c @@ -3291,6 +3291,8 @@ void nbd_client_new(QIOChannelSocket *sioc, client->close_fn =3D close_fn; client->owner =3D owner; =20 + nbd_set_socket_send_buffer(sioc); + co =3D qemu_coroutine_create(nbd_co_client_start, client); qemu_coroutine_enter(co); } --=20 2.39.5 (Apple Git-154) From nobody Wed Dec 17 03:58:29 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1747512794; cv=none; d=zohomail.com; s=zohoarc; b=SMKHqlw+UELn8N7ABaqSPtKGb2Xwx/LU0rmke051Chsg8dK5LzMJuP0snt4n2HSk/5joD6XFZZb0F93ccNfCOVn5sHdN10RaExDvquf5cWzZpsqoKRRx0JIHu+K4URkjg000CIPdFYNOjooeq/ErnX23KGYagRFOcDkPvP9NCkU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1747512794; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=YY9WRuvAYI/I2OQEpJcbR6dtMzBKUajhK446b7lTa94=; b=ftnhQ73REhuryXg6zc7s1ULdzqopfiH7NXwbKSZW6ieZkvx65CvoOiDwUvS286CMcwyJG3/JUXAY599YmlD0LQVWxtcZiN2GNQRb+dzRajDCMmLfdCAmT3qjkXg661kdhxxrKOUiRSdNasHLOg7urZR1XKbyPbUQRUU8+KY9Sbs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1747512794813681.6008934303554; Sat, 17 May 2025 13:13:14 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uGNtA-0005TS-Ii; Sat, 17 May 2025 16:12:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uGNt8-0005Sj-2F; Sat, 17 May 2025 16:12:06 -0400 Received: from mail-wr1-x42d.google.com ([2a00:1450:4864:20::42d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1uGNt6-00020l-Au; Sat, 17 May 2025 16:12:05 -0400 Received: by mail-wr1-x42d.google.com with SMTP id ffacd0b85a97d-3a064a3e143so1604562f8f.3; Sat, 17 May 2025 13:12:03 -0700 (PDT) Received: from localhost.localdomain (46-116-102-127.bb.netvision.net.il. [46.116.102.127]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-442fd59ab21sm78714715e9.38.2025.05.17.13.12.00 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 17 May 2025 13:12:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1747512722; x=1748117522; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YY9WRuvAYI/I2OQEpJcbR6dtMzBKUajhK446b7lTa94=; b=i2IcW32Cpxm6bBEi9Gq+da/71CjxX5BH0XcmoxjRVNxQT889c5v4NU0PCv6ZQvirOB 0K7yt2PmDDe5b/qExCQB9gYbAuaQHFJh//30Cp979QVp7ajgw8Po3zEtsPJfahfIog9M QpftXUdzZjR5Y9dkwP3CxfqrtYo4FhLotpA9ZJtT3Hyn03gyzdek8gXOj4DB3mJ8Gd1n AMbGKjFyTWg1pEdLCU4CcA4/ZnOosTBZ+80J6BCEigoB1aHd5lPOy4fzN4JXXE4p+1qH Pe5X6WMc4EXxFqVdvcH5NN7C7HQ3/xfjzdoLMzE+21Io6AN6nb2auRIGkHA1RGhKEy2/ Rk3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747512722; x=1748117522; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YY9WRuvAYI/I2OQEpJcbR6dtMzBKUajhK446b7lTa94=; b=ZlPRqjc0/S3A/N9QQNQczifO1ExMcZWfAm/tXOMWMtWJLl2RcAjv6pyiqlS6OKOuST JeP3DpLQy0h9rESXTMYoq3fNx0mVLIORefYt9CcKjahX9MoAHBGV79d4uAvHcVlHFJ2a 6XI3aPUjyfhTNOSiG7uSPwQIoafJpk+9feFRU9B01mEZzjA6beO4n8BKgu2dYFMbgGGa S+TRUGP6qj2L+ossql+RSxD203MqpHZa9jRzR/H2nFxYfqHavBvhViw7vTV87L9wojw+ 8LhcaLJv9KACobSsnwb3arIi/X9z6CM/Pv8G+NOYZ0MmqFo4MyMs08ieXDPPLKS7/iiY kYog== X-Forwarded-Encrypted: i=1; AJvYcCXra1WdS7KtmVNCGF6XrH1Z6vbG1m9rhvmc6r5dncD5NVqeehOCCwfjZ/nAUnhuYS5GtH9uzBqSX4DC@nongnu.org X-Gm-Message-State: AOJu0Yx0stuFoFNZjPCK11j1n+IkCgG7CqV5B15d2BS+L/4mFrRs0+yt ZbwUwDs4WShl7lJzj9FUrQXjpyqJxFlizb4DOY4YH5f74Z9cm3uC1vsQh6eL7QMP+dI= X-Gm-Gg: ASbGncsTnmCRtCNZ233l0hAEnYSwTmNF+9WWxPGp/iSyB4Awy9dRixNmS44YB+ItOr+ ndqozhaO3NaplRxe4d4h09r9X1YagUqpeI5moH7U6nHEYUGxAjCOVA66OZsnXv8UPhv/K+NG3hr LvoE/TS0XnaQ5Yl4notZEinxMWoo4iSpJTFltWHFwtei8Ge1z1rv4+UKP4XCzBjBETHNpYJHW7F MNnfcqQ/8kSCnyVM+zkoAZVCkK0QSw+c00p24nbHy1PC08lBAOBeHKORJc5sVh9+KqLkLLV0XAA WJELbGsfyFJGHR709FV4yJfZ3GJtmzI6mFk2HPCwbklvJyZnFBk9TPhhYq87MD0j4ra6WilPlFP oQGDPkS2MfFpAlbtuhjAt/WC5nPydrLkDZw== X-Google-Smtp-Source: AGHT+IF6FvXb5fojyaWS6qu/5ktcaug90aAr3rwYnucpTrR3sZ2pa6hQ4xmTjKxGipZt2b/3J7LmIw== X-Received: by 2002:a05:6000:2410:b0:3a0:9f24:774d with SMTP id ffacd0b85a97d-3a35c853134mr7550237f8f.45.1747512722103; Sat, 17 May 2025 13:12:02 -0700 (PDT) From: Nir Soffer To: qemu-devel@nongnu.org Cc: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , Eric Blake , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , "Richard Jones" , Vladimir Sementsov-Ogievskiy , qemu-block@nongnu.org, Nir Soffer Subject: [PATCH v4 3/3] nbd: Set unix socket send buffer on Linux Date: Sat, 17 May 2025 23:11:54 +0300 Message-Id: <20250517201154.88456-4-nirsof@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250517201154.88456-1-nirsof@gmail.com> References: <20250517201154.88456-1-nirsof@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::42d; envelope-from=nirsof@gmail.com; helo=mail-wr1-x42d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1747512797005116600 Content-Type: text/plain; charset="utf-8" Like macOS we have similar issue on Linux. For TCP socket the send buffer size is 2626560 bytes (~2.5 MiB) and we get good performance. However for unix socket the default and maximum buffer size is 212992 bytes (208 KiB) and we see poor performance when using one NBD connection, up to 4 times slower than macOS on the same machine. Tracing shows that for every 2 MiB payload (qemu uses 2 MiB io size), we do 1 recvmsg call with TCP socket, and 10 recvmsg calls with unix socket. Fixing this issue requires changing the maximum send buffer size (the receive buffer size is ignored). This can be done using: $ cat /etc/sysctl.d/net-mem-max.conf net.core.wmem_max =3D 2097152 $ sudo sysctl -p /etc/sysctl.d/net-mem-max.conf With this we can set the socket buffer size to 2 MiB. With the defaults the value requested by qemu is clipped to the maximum size and has no effect. I tested on 2 machines: - Fedora 42 VM on MacBook Pro M2 Max - Dell PowerEdge R640 (Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz) On the older Dell machine we see very little improvement, up to 1.03 higher throughput. On the M2 machine we see up to 2.67 times higher throughput. The following results are from the M2 machine. Reading from qemu-nbd with qemu-img convert. In this test buffer size of 4m is optimal (2.28 times faster). | buffer size | time | user | system | |-------------|---------|---------|---------| | default | 4.292 | 0.243 | 1.604 | | 524288 | 2.167 | 0.058 | 1.288 | | 1048576 | 2.041 | 0.060 | 1.238 | | 2097152 | 1.884 | 0.060 | 1.191 | | 4194304 | 1.881 | 0.054 | 1.196 | Writing to qemu-nbd with qemu-img convert. In this test buffer size of 1m is optimal (2.67 times faster). | buffer size | time | user | system | |-------------|---------|---------|---------| | default | 3.113 | 0.334 | 1.094 | | 524288 | 1.173 | 0.179 | 0.654 | | 1048576 | 1.164 | 0.164 | 0.670 | | 2097152 | 1.227 | 0.197 | 0.663 | | 4194304 | 1.227 | 0.198 | 0.666 | Computing a blkhash with nbdcopy. In this test buffer size of 512k is optimal (1.19 times faster). | buffer size | time | user | system | |-------------|---------|---------|---------| | default | 2.140 | 4.483 | 2.681 | | 524288 | 1.794 | 4.467 | 2.572 | | 1048576 | 1.807 | 4.447 | 2.644 | | 2097152 | 1.822 | 4.461 | 2.698 | | 4194304 | 1.827 | 4.465 | 2.700 | Computing a blkhash with blksum. In this test buffer size of 4m is optimal (2.65 times faster). | buffer size | time | user | system | |-------------|---------|---------|---------| | default | 3.582 | 4.595 | 2.392 | | 524288 | 1.499 | 4.384 | 1.482 | | 1048576 | 1.377 | 4.381 | 1.345 | | 2097152 | 1.388 | 4.389 | 1.354 | | 4194304 | 1.352 | 4.395 | 1.302 | Signed-off-by: Nir Soffer Reviewed-by: Daniel P. Berrang=C3=A9 --- nbd/common.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/nbd/common.c b/nbd/common.c index 9436e9d1d1..2a133a66c3 100644 --- a/nbd/common.c +++ b/nbd/common.c @@ -271,8 +271,9 @@ const char *nbd_mode_lookup(NBDMode mode) /* * Testing shows that 2m send buffer is optimal. Changing the receive buff= er * size has no effect on performance. + * On Linux we need to increase net.core.wmem_max to make this effective. */ -#if defined(__APPLE__) +#if defined(__APPLE__) || defined(__linux__) #define UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE (2 * MiB) #endif =20 --=20 2.39.5 (Apple Git-154)