From nobody Sat Sep 26 20:00:17 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1788955288; cv=none; d=zohomail.com; s=zohoarc; b=bwptwkQdtF68s4f8WYWX6D4L/xeqNoXGFg9aCr+HGYhq5sZgi94oKd2SwYEFOgf4oCC1IuiGDE9wBN785qNdAI7eSUFL8JHzYgJLH0FKWU8OZ6/pT1C4W6ELMga1RHFg/dXa6NH+Kd0CS7QS8kQEOSWLUbNksFRx72rUcKYPBxY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1788955288; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=W1bXX1vH3nJFE/rc6t4KxJhp5QchV/0nvDLwFkuTeK4=; b=NK+rcggvswmkcojvRgUa0HycK34c8bJr7eKBBfC7Nw8CXs16zEobtgJxHWX1KpTQF3i1q6dEdx+goyunFT/UilvC3PSjhxbNXb5omTle19z/mIAkfvDhXbo6XhMUtPyTCFe9FNlL7l4lMdWfYAVFfidDwqA3IR/lMhbe/q0gGOc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1788955288213264.5383291825683; Wed, 9 Sep 2026 05:01:28 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x4Gyu-0005U0-Of; Wed, 09 Sep 2026 08:00:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x4Gyq-0005TG-SV for qemu-devel@nongnu.org; Wed, 09 Sep 2026 08:00:44 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x4Gyo-00046o-9a for qemu-devel@nongnu.org; Wed, 09 Sep 2026 08:00:44 -0400 Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-36-a9phwIuJMUSZlSYKyFR8ZQ-1; Wed, 09 Sep 2026 08:00:37 -0400 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 3998E19560A2; Wed, 9 Sep 2026 12:00:36 +0000 (UTC) Received: from berrange.csb (headnet03.pony-001.prod.iad2.dc.redhat.com [10.2.32.114]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id C4725195608E; Wed, 9 Sep 2026 12:00:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1788955241; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=W1bXX1vH3nJFE/rc6t4KxJhp5QchV/0nvDLwFkuTeK4=; b=jWS11AsbHuuETEHnipRNbCWsYSDMUxhl/JQoSrBxn9PKkT5QraI1oHraskAfQfdGf2R7vN /jwXLEOQAH6O1k5rm9FdIhjePvnnm9tzl6LsgQFrAL8m2wOW+l0pQOfUqiUc4SaABzKJmo PbIl/+Y4Z8q/dEpXUDqbVjBl4UmglwY= X-MC-Unique: a9phwIuJMUSZlSYKyFR8ZQ-1 X-Mimecast-MFC-AGG-ID: a9phwIuJMUSZlSYKyFR8ZQ_1788955236 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: qemu-devel@nongnu.org Cc: Peter Xu , =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , Fabiano Rosas , Juraj Marcin Subject: [PATCH] io: cork TLS writes to avoid small TLS records Date: Wed, 9 Sep 2026 13:00:33 +0100 Message-ID: <20260909120033.1303511-1-berrange@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=berrange@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: 12 X-Spam_score: 1.2 X-Spam_bar: + X-Spam_report: (1.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_SBL_CSS=3.335, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1788955292734158500 The migration code caches vmstate/ram writes into an iovec and flushes this periodically. The total amount of data to sent may be 100's of KB, but split across many iovec, each of which is potentially quite small. The QIOChannelTLS receives the iovec, but since GNUTLS cannot accept iovec data, it iterates calling send for each element. As a result of the migration data pattern, this results in GNUTLS putting lots of small TLS records on the wire. This has shown writes alternate between about 4k and 30 bytes in some tests. This is triggering the nagle algorithm on migration-test for many of the TLS test cases, resulting in a "go slow" for I/O that eventually hits the migration timeout configured by the test. Not every contributor reports seeing the "go slow" but for those who do see it, it hits >=3D 95% of the time for at least one of the migration TLS test cases run by 'make check'. While we could disable the nagle algorithm (and multifd channels already do this), that would stil result in lots of small TLS records hitting the wire which is not good for throughput. The migration code flushes in batches because it wants large writes for high throughput. To achieve this we must tell GNUTLS to encrypt data but not immediately sent TLS records by "corking" its output. Once the complete iovec has been written to GNUTLS, it can be uncorked allowing it to hit the wire. There is added complexity with uncorking on non-blocking channels as not all encrypted data can be sent at once. qio_channel_write() will return what was sent, but some data might remain pending inside GNUTLS buffers. A later call to qio_channel_write() will provide the same plain text data buffers that have already been cached. Thus the code must attempt to uncork GNUTLS again to clear pending data and then deduct the equivalent amount of plain text. Signed-off-by: Daniel P. Berrang=C3=A9 Reviewed-by: Peter Xu Tested-by: Juraj Marcin --- crypto/tlssession.c | 33 ++++++++++++++ include/crypto/tlssession.h | 25 +++++++++++ include/io/channel-tls.h | 1 + io/channel-tls.c | 87 ++++++++++++++++++++++++++++++++++--- 4 files changed, 140 insertions(+), 6 deletions(-) diff --git a/crypto/tlssession.c b/crypto/tlssession.c index 314e3e96ba..ffb6a27e47 100644 --- a/crypto/tlssession.c +++ b/crypto/tlssession.c @@ -513,6 +513,39 @@ qcrypto_tls_session_read(QCryptoTLSSession *session, } =20 =20 +void qcrypto_tls_session_write_cork(QCryptoTLSSession *sess) +{ + gnutls_record_cork(sess->handle); +} + + +ssize_t qcrypto_tls_session_write_uncork(QCryptoTLSSession *sess, + Error **errp) +{ + int ret; + ret =3D gnutls_record_uncork(sess->handle, 0); + if (ret =3D=3D GNUTLS_E_AGAIN || + ret =3D=3D GNUTLS_E_INTERRUPTED) { + ret =3D gnutls_record_check_corked(sess->handle); + if (ret < 0) { + error_setg(errp, + "Cannot query pending TLS output: %s", + gnutls_strerror(ret)); + return -1; + } + + return ret; + } else if (ret < 0) { + error_setg(errp, + "Cannot uncork TLS output: %s", + gnutls_strerror(ret)); + return -1; + } + + return 0; +} + + size_t qcrypto_tls_session_check_pending(QCryptoTLSSession *session) { diff --git a/include/crypto/tlssession.h b/include/crypto/tlssession.h index 28e419681e..5415503eba 100644 --- a/include/crypto/tlssession.h +++ b/include/crypto/tlssession.h @@ -208,6 +208,31 @@ typedef ssize_t (*QCryptoTLSSessionReadFunc)(void *buf, void *opaque, Error **errp); =20 +/** + * qcrypto_tls_session_write_cork: + * @sess: the TLS session object + * + * Causes future qcrypto_tls_session_write() calls to encrypt + * and queue data, without sending on the wire. + */ +void qcrypto_tls_session_write_cork(QCryptoTLSSession *sess); + +/** + * qcrypto_tls_session_write_uncork: + * @sess: the TLS session object + * @errp: pointer to a NULL-initialized error object + * + * Attempt to send previously queued data. If the underlying + * stream is non-blocking, then only a subset of data (if any) + * may be written. To process outstanding data, this method + * must be called again until it returns 0. + * + * Returns: the number of bytes remaining to be sent, + * or -1 on error. + */ +ssize_t qcrypto_tls_session_write_uncork(QCryptoTLSSession *sess, + Error **errp); + /** * qcrypto_tls_session_set_callbacks: * @sess: the TLS session object diff --git a/include/io/channel-tls.h b/include/io/channel-tls.h index 7e9023570d..9e9b00c034 100644 --- a/include/io/channel-tls.h +++ b/include/io/channel-tls.h @@ -50,6 +50,7 @@ struct QIOChannelTLS { QIOChannelShutdown shutdown; guint hs_ioc_tag; guint bye_ioc_tag; + size_t corked; }; =20 /** diff --git a/io/channel-tls.c b/io/channel-tls.c index 31ec4d236d..05317ed5a3 100644 --- a/io/channel-tls.c +++ b/io/channel-tls.c @@ -21,6 +21,7 @@ #include "qemu/osdep.h" #include "qapi/error.h" #include "qemu/module.h" +#include "qemu/iov.h" #include "io/channel-tls.h" #include "trace.h" #include "qemu/atomic.h" @@ -448,12 +449,24 @@ static ssize_t qio_channel_tls_writev(QIOChannel *ioc, QIOChannelTLS *tioc =3D QIO_CHANNEL_TLS(ioc); size_t i; ssize_t done =3D 0; + ssize_t remain; + g_autofree struct iovec *tmpiov =3D NULL; + size_t ntmpiov =3D 0; =20 - for (i =3D 0 ; i < niov ; i++) { - ssize_t ret =3D qcrypto_tls_session_write(tioc->session, - iov[i].iov_base, - iov[i].iov_len, - errp); + /* + * The previous write encrypted all the data, but some + * was not able to be sent on the wire when uncorked, + * so we returned a short write. The encrypted data + * will still be cached by GNUTLS and the session will + * be in a corked state. + * + * This write call will be trying to write the remaining + * plain text data again, but we must avoid sending that + * into GNUTLS. Instead flush the previously encrypted + * pending data. + */ + while (tioc->corked) { + ssize_t ret =3D qcrypto_tls_session_write_uncork(tioc->session, er= rp); if (ret =3D=3D QCRYPTO_TLS_SESSION_ERR_BLOCK) { if (done) { return done; @@ -463,12 +476,74 @@ static ssize_t qio_channel_tls_writev(QIOChannel *ioc, } else if (ret < 0) { return -1; } + done +=3D (tioc->corked - ret); + tioc->corked =3D ret; + } + + /* + * If we flushed pending data, we must discard an + * equivalent amount of plain text data from this + * write call, and then process what's left over, + * if any. + */ + if (done) { + ssize_t total =3D iov_size(iov, niov); + if (done =3D=3D total) { + return done; + } + + tmpiov =3D g_new0(struct iovec, niov); + ntmpiov =3D iov_copy(tmpiov, niov, iov, niov, + done, total - done); + + iov =3D tmpiov; + niov =3D ntmpiov; + } + + /* + * At this point we should ony be processing "new" + * data, not seen by a previous write call so we + * send to GNUTLS as normal. + */ + qcrypto_tls_session_write_cork(tioc->session); + for (i =3D 0 ; i < niov ; i++) { + ssize_t ret =3D qcrypto_tls_session_write(tioc->session, + iov[i].iov_base, + iov[i].iov_len, + errp); + if (ret =3D=3D QCRYPTO_TLS_SESSION_ERR_BLOCK) { + error_setg(errp, "Unexpected TLS blocking I/O while corked"); + return -1; + } else if (ret < 0) { + return -1; + } done +=3D ret; if (ret < iov[i].iov_len) { break; } } - return done; + + /* + * On non-blocking sockets, uncorking may not succeed + * in sending all encrypted data, so we have to check + * what was actually sent to see if GNUTLS remains in + * the corked state with pending data. + */ + remain =3D qcrypto_tls_session_write_uncork(tioc->session, errp); + if (remain < 0) { + return -1; + } else if (remain) { + tioc->corked =3D remain; + } + /* + * done =3D=3D what we sent to GNUTLS for encryption & sending + * remain =3D=3D subset of 'done' that was encrypted but not sent + */ + if (done && (remain =3D=3D done)) { + return QIO_CHANNEL_ERR_BLOCK; + } else { + return done - remain; + } } =20 static int qio_channel_tls_set_blocking(QIOChannel *ioc, --=20 2.55.0