From nobody Mon Feb 9 16:45:42 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1651160369; cv=none; d=zohomail.com; s=zohoarc; b=Ow/NZts2I8+cYWMjS2xTK4B4A7Z+9v7ilN2hPtQ8/4qe5nO/E53+GemDCFbHEpuz2qDd4T7qysakV68PDnrI+qG/U3SybVgPnfdHD1hZ+qIJz+2gwtRmPujh5dJhSdQWtMeCwfbCsZTY9eYrzh6m7Y7gOO1r7lN026zL15TyEds= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1651160369; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=gnVKMXgOKz3iWsZ5LDrKPMRaRqiVcz2u6i3yf7c7zpY=; b=Ibn+M4xZ/abocvzR3efXCBWUibaHkh7rg2XkWE/XThvBz9Mxz8PFUilfdkGQWkriCc2kVGf16n84giAbBIXMXbOVliT4ZqTVGBq7/rFEyBzcLWjUdA/kxzSFx7tqnxu4xZowo33OLLu0c3qm2myFlndpxlOWKfRgZpWZxaXjEtE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 165116036921635.53091253005482; Thu, 28 Apr 2022 08:39:29 -0700 (PDT) Received: from localhost ([::1]:40826 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nk6Eq-0003ap-65 for importer@patchew.org; Thu, 28 Apr 2022 11:39:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46130) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nk5KY-0006CU-Do for qemu-devel@nongnu.org; Thu, 28 Apr 2022 10:41:18 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:20772) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nk5KW-0006cq-Os for qemu-devel@nongnu.org; Thu, 28 Apr 2022 10:41:18 -0400 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-541-l0DBnZIZN32Y8wXljPFBWQ-1; Thu, 28 Apr 2022 10:41:14 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 67FF03C1EA48 for ; Thu, 28 Apr 2022 14:41:14 +0000 (UTC) Received: from dgilbert-t580.localhost (unknown [10.39.193.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 723FB407DEC3; Thu, 28 Apr 2022 14:41:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1651156876; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gnVKMXgOKz3iWsZ5LDrKPMRaRqiVcz2u6i3yf7c7zpY=; b=gRKJxzY5MnlilKO069wMbY+X1jmKdQTho9H7UqhKORYWX0CS9y2t3U1xUiPmiae/LvRVnR zpS8jBZ5u5BTdNzfB3vAz51Hgr28oBG5qN9oQB0W9/0x3p2b1Myy7fCqjd2WAggouaT1qK r5pXA5rrZGYfXbbiPMcxV7aO0K1PuTQ= X-MC-Unique: l0DBnZIZN32Y8wXljPFBWQ-1 From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, quintela@redhat.com, peterx@redhat.com, leobras@redhat.com, berrange@redhat.com Subject: [PULL 10/11] multifd: Send header packet without flags if zero-copy-send is enabled Date: Thu, 28 Apr 2022 15:40:51 +0100 Message-Id: <20220428144052.263382-11-dgilbert@redhat.com> In-Reply-To: <20220428144052.263382-1-dgilbert@redhat.com> References: <20220428144052.263382-1-dgilbert@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.84 on 10.11.54.1 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=dgilbert@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1651160369765100001 From: Leonardo Bras Since d48c3a0445 ("multifd: Use a single writev on the send side"), sending the header packet and the memory pages happens in the same writev, which can potentially make the migration faster. Using channel-socket as example, this works well with the default copying mechanism of sendmsg(), but with zero-copy-send=3Dtrue, it will cause the migration to often break. This happens because the header packet buffer gets reused quite often, and there is a high chance that by the time the MSG_ZEROCOPY mechanism get to send the buffer, it has already changed, sending the wrong data and causing the migration to abort. It means that, as it is, the buffer for the header packet is not suitable for sending with MSG_ZEROCOPY. In order to enable zero copy for multifd, send the header packet on an individual write(), without any flags, and the remanining pages with a writev(), as it was happening before. This only changes how a migration with zero-copy-send=3Dtrue works, not changing any current behavior for migrations with zero-copy-send=3Dfalse. Signed-off-by: Leonardo Bras Message-Id: <20220426230654.637939-7-leobras@redhat.com> Reviewed-by: Peter Xu Reviewed-by: Daniel P. Berrang=C3=A9 Signed-off-by: Dr. David Alan Gilbert dgilbert: Removed blank line --- migration/multifd.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 15fb668e64..2541cd2322 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -617,6 +617,7 @@ static void *multifd_send_thread(void *opaque) MultiFDSendParams *p =3D opaque; Error *local_err =3D NULL; int ret =3D 0; + bool use_zero_copy_send =3D migrate_use_zero_copy_send(); =20 trace_multifd_send_thread_start(p->id); rcu_register_thread(); @@ -639,9 +640,14 @@ static void *multifd_send_thread(void *opaque) if (p->pending_job) { uint64_t packet_num =3D p->packet_num; uint32_t flags =3D p->flags; - p->iovs_num =3D 1; p->normal_num =3D 0; =20 + if (use_zero_copy_send) { + p->iovs_num =3D 0; + } else { + p->iovs_num =3D 1; + } + for (int i =3D 0; i < p->pages->num; i++) { p->normal[p->normal_num] =3D p->pages->offset[i]; p->normal_num++; @@ -665,8 +671,18 @@ static void *multifd_send_thread(void *opaque) trace_multifd_send(p->id, packet_num, p->normal_num, flags, p->next_packet_size); =20 - p->iov[0].iov_len =3D p->packet_len; - p->iov[0].iov_base =3D p->packet; + if (use_zero_copy_send) { + /* Send header first, without zerocopy */ + ret =3D qio_channel_write_all(p->c, (void *)p->packet, + p->packet_len, &local_err); + if (ret !=3D 0) { + break; + } + } else { + /* Send header using the same writev call */ + p->iov[0].iov_len =3D p->packet_len; + p->iov[0].iov_base =3D p->packet; + } =20 ret =3D qio_channel_writev_all(p->c, p->iov, p->iovs_num, &local_err); --=20 2.35.1