From nobody Tue Nov 26 14:40:25 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1707361759; cv=none; d=zohomail.com; s=zohoarc; b=W+WrJ8cgDqwBUi5eYPl+FpZwEqTNultR/Xko7pg58VQPwulEgeV3cyeYsh0KARbyC2gXTYdxmXirp2figplU89lIfW+onHSbF2e+WE1vLw0RF8lYF1Di+BYg6y8NDYwx8xIW1DOkqxx8eFhCbgGw8UvxuODuSOQ09cfuCbKYLps= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1707361759; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=I3vRV0VDr3Uq1n51oj5HlrvT40qebwi3yFZMQzgX7Og=; b=QdNBxv8NE0h5upB5+Or4mlBKNstu310qYoz/9XcxNFo10d4gnu159VOJwgyl2ovclfJJhZxmN4wB6jXKgNfvIRBAhpuSKvzTgX+WKwWdNfffVuY33D4x/GF4naOTfz2zwpGkDKm81OBlb/lIOMiFWSChRfE0qNu4PJRT3AwyKHg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1707361759898535.8528386418864; Wed, 7 Feb 2024 19:09:19 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rXul7-0007Zj-Aq; Wed, 07 Feb 2024 22:07:29 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rXuk4-0006Pr-Ca for qemu-devel@nongnu.org; Wed, 07 Feb 2024 22:06:25 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rXuk1-0004ND-9j for qemu-devel@nongnu.org; Wed, 07 Feb 2024 22:06:23 -0500 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-324-CgpzQ-C8PI6ZcDBfB8EiPw-1; Wed, 07 Feb 2024 22:06:17 -0500 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9F45C185A780; Thu, 8 Feb 2024 03:06:16 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.45]) by smtp.corp.redhat.com (Postfix) with ESMTP id 759EE492BC7; Thu, 8 Feb 2024 03:06:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1707361580; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I3vRV0VDr3Uq1n51oj5HlrvT40qebwi3yFZMQzgX7Og=; b=CvMnH8mKWFmAWNk6lRnqwV/nm4uNRJ36WEfaefB3/rpcle7khUl5f6Zsu12uGkV96NGR7G 2hKe3WK+PQpj5hq5FbMaHiKjk4iSSaz56kCkzg5/hcCuNyh6SYLddZBLdjYQm/cr22CZes or2M8V39CJWXTaXzHcBgbPeCW4QzaRE= X-MC-Unique: CgpzQ-C8PI6ZcDBfB8EiPw-1 From: peterx@redhat.com To: Peter Maydell , qemu-devel@nongnu.org Cc: peterx@redhat.com, Fabiano Rosas Subject: [PULL 14/34] migration/multifd: Move header prepare/fill into send_prepare() Date: Thu, 8 Feb 2024 11:05:08 +0800 Message-ID: <20240208030528.368214-15-peterx@redhat.com> In-Reply-To: <20240208030528.368214-1-peterx@redhat.com> References: <20240208030528.368214-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.9 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.106, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1707361760172100002 Content-Type: text/plain; charset="utf-8" From: Peter Xu This patch redefines the interfacing of ->send_prepare(). It further simplifies multifd_send_thread() especially on zero copy. Now with the new interface, we require the hook to do all the work for preparing the IOVs to send. After it's completed, the IOVs should be ready to be dumped into the specific multifd QIOChannel later. So now the API looks like: p->pages -----------> send_prepare() -------------> IOVs This also prepares for the case where the input can be extended to even not any p->pages. But that's for later. This patch will achieve similar goal of what Fabiano used to propose here: https://lore.kernel.org/r/20240126221943.26628-1-farosas@suse.de However the send() interface may not be necessary. I'm boldly attaching a "Co-developed-by" for Fabiano. Co-developed-by: Fabiano Rosas Reviewed-by: Fabiano Rosas Link: https://lore.kernel.org/r/20240202102857.110210-14-peterx@redhat.com Signed-off-by: Peter Xu --- migration/multifd.h | 1 + migration/multifd-zlib.c | 4 +++ migration/multifd-zstd.c | 4 +++ migration/multifd.c | 61 ++++++++++++++++++---------------------- 4 files changed, 37 insertions(+), 33 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index 4ec005f53f..34a2ecb9f4 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -208,6 +208,7 @@ typedef struct { } MultiFDMethods; =20 void multifd_register_ops(int method, MultiFDMethods *ops); +void multifd_send_fill_packet(MultiFDSendParams *p); =20 static inline void multifd_send_prepare_header(MultiFDSendParams *p) { diff --git a/migration/multifd-zlib.c b/migration/multifd-zlib.c index 100809abc1..012e3bdea1 100644 --- a/migration/multifd-zlib.c +++ b/migration/multifd-zlib.c @@ -123,6 +123,8 @@ static int zlib_send_prepare(MultiFDSendParams *p, Erro= r **errp) int ret; uint32_t i; =20 + multifd_send_prepare_header(p); + for (i =3D 0; i < pages->num; i++) { uint32_t available =3D z->zbuff_len - out_size; int flush =3D Z_NO_FLUSH; @@ -172,6 +174,8 @@ static int zlib_send_prepare(MultiFDSendParams *p, Erro= r **errp) p->next_packet_size =3D out_size; p->flags |=3D MULTIFD_FLAG_ZLIB; =20 + multifd_send_fill_packet(p); + return 0; } =20 diff --git a/migration/multifd-zstd.c b/migration/multifd-zstd.c index 2023edd8cc..dc8fe43e94 100644 --- a/migration/multifd-zstd.c +++ b/migration/multifd-zstd.c @@ -118,6 +118,8 @@ static int zstd_send_prepare(MultiFDSendParams *p, Erro= r **errp) int ret; uint32_t i; =20 + multifd_send_prepare_header(p); + z->out.dst =3D z->zbuff; z->out.size =3D z->zbuff_len; z->out.pos =3D 0; @@ -161,6 +163,8 @@ static int zstd_send_prepare(MultiFDSendParams *p, Erro= r **errp) p->next_packet_size =3D z->out.pos; p->flags |=3D MULTIFD_FLAG_ZSTD; =20 + multifd_send_fill_packet(p); + return 0; } =20 diff --git a/migration/multifd.c b/migration/multifd.c index cd4467aff4..6aa44340de 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -50,15 +50,15 @@ typedef struct { /** * nocomp_send_setup: setup send side * - * For no compression this function does nothing. - * - * Returns 0 for success or -1 for error - * * @p: Params for the channel that we are using * @errp: pointer to an error */ static int nocomp_send_setup(MultiFDSendParams *p, Error **errp) { + if (migrate_zero_copy_send()) { + p->write_flags |=3D QIO_CHANNEL_WRITE_FLAG_ZERO_COPY; + } + return 0; } =20 @@ -88,7 +88,17 @@ static void nocomp_send_cleanup(MultiFDSendParams *p, Er= ror **errp) */ static int nocomp_send_prepare(MultiFDSendParams *p, Error **errp) { + bool use_zero_copy_send =3D migrate_zero_copy_send(); MultiFDPages_t *pages =3D p->pages; + int ret; + + if (!use_zero_copy_send) { + /* + * Only !zerocopy needs the header in IOV; zerocopy will + * send it separately. + */ + multifd_send_prepare_header(p); + } =20 for (int i =3D 0; i < pages->num; i++) { p->iov[p->iovs_num].iov_base =3D pages->block->host + pages->offse= t[i]; @@ -98,6 +108,18 @@ static int nocomp_send_prepare(MultiFDSendParams *p, Er= ror **errp) =20 p->next_packet_size =3D pages->num * p->page_size; p->flags |=3D MULTIFD_FLAG_NOCOMP; + + multifd_send_fill_packet(p); + + if (use_zero_copy_send) { + /* Send header first, without zerocopy */ + ret =3D qio_channel_write_all(p->c, (void *)p->packet, + p->packet_len, errp); + if (ret !=3D 0) { + return -1; + } + } + return 0; } =20 @@ -266,7 +288,7 @@ static void multifd_pages_clear(MultiFDPages_t *pages) g_free(pages); } =20 -static void multifd_send_fill_packet(MultiFDSendParams *p) +void multifd_send_fill_packet(MultiFDSendParams *p) { MultiFDPacket_t *packet =3D p->packet; MultiFDPages_t *pages =3D p->pages; @@ -688,7 +710,6 @@ static void *multifd_send_thread(void *opaque) MigrationThread *thread =3D NULL; Error *local_err =3D NULL; int ret =3D 0; - bool use_zero_copy_send =3D migrate_zero_copy_send(); =20 thread =3D migration_threads_add(p->name, qemu_get_thread_id()); =20 @@ -713,15 +734,6 @@ static void *multifd_send_thread(void *opaque) MultiFDPages_t *pages =3D p->pages; =20 p->iovs_num =3D 0; - - if (!use_zero_copy_send) { - /* - * Only !zerocopy needs the header in IOV; zerocopy will - * send it separately. - */ - multifd_send_prepare_header(p); - } - assert(pages->num); =20 ret =3D multifd_send_state->ops->send_prepare(p, &local_err); @@ -730,17 +742,6 @@ static void *multifd_send_thread(void *opaque) break; } =20 - multifd_send_fill_packet(p); - - if (use_zero_copy_send) { - /* Send header first, without zerocopy */ - ret =3D qio_channel_write_all(p->c, (void *)p->packet, - p->packet_len, &local_err); - if (ret !=3D 0) { - break; - } - } - ret =3D qio_channel_writev_full_all(p->c, p->iov, p->iovs_num,= NULL, 0, p->write_flags, &local_er= r); if (ret !=3D 0) { @@ -945,13 +946,7 @@ int multifd_save_setup(Error **errp) p->iov =3D g_new0(struct iovec, page_count + 1); p->page_size =3D qemu_target_page_size(); p->page_count =3D page_count; - - if (migrate_zero_copy_send()) { - p->write_flags =3D QIO_CHANNEL_WRITE_FLAG_ZERO_COPY; - } else { - p->write_flags =3D 0; - } - + p->write_flags =3D 0; multifd_new_send_channel_create(p); } =20 --=20 2.43.0