From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869818; cv=none; d=zohomail.com; s=zohoarc; b=kWuHRtzOGHchZURDa74sC+qcivIfdXrcybrMoVFQ86bJcDR5tdXfLEuC3tTm3KxnGRmr9zDFCO01q6eDMImo9gQ0ckK/WcOkbOIoptrzdQIJp9TXIF2JjflTh+50/0ewr/tgB2qDltLM07nEPFC7qyPvMAOlvivEsN5OP8oYn3Q= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869818; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=qzb46DUZSTMJoNeVIqZsw/shF+U8HEgAIFp5xpwtBDY=; b=KyMB1vXGUnaJHaqBOwYIlYSQ5bLVXRBhoBfqQLZUBliYAoezM+BSLChdcbFpVg31AK51nIiH/EUehSnrN7ftFpwz9+usrtESIwn5WYof0ZUo/I04AX8MXFYhPZRALTduIuxnx94zFXoYLE9rMzdLab+yVYKTnRYrrvu0yuThCvo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869818258220.45594267287584; Fri, 2 Feb 2024 02:30:18 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqnf-00087Q-UV; Fri, 02 Feb 2024 05:29:35 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqne-00086z-IW for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:29:34 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqnd-0001Xc-7v for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:29:34 -0500 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-106-WzK2ITtNOGKWR79D4gfjtg-1; Fri, 02 Feb 2024 05:29:28 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 717A8185A784; Fri, 2 Feb 2024 10:29:28 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id A20A240C9444; Fri, 2 Feb 2024 10:29:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869772; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qzb46DUZSTMJoNeVIqZsw/shF+U8HEgAIFp5xpwtBDY=; b=JFm5MIc4iQzffzZWR3d+6997wm3OJ3vDCxoYvg6tQvMrWG/OAGH1eUjA2xm+iYQbFPjIRg uA3jSPQ+sUgSOffVHUvSZD+98nNKe6BFDaE8kVrQmjEdCEKv404c9Fz+LKYqmi4P6Uqa+O 153+OAyYsxCZ+hIUaIN39Pwh0NJzUSw= X-MC-Unique: WzK2ITtNOGKWR79D4gfjtg-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 01/23] migration/multifd: Drop stale comment for multifd zero copy Date: Fri, 2 Feb 2024 18:28:35 +0800 Message-ID: <20240202102857.110210-2-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869819464100001 Content-Type: text/plain; charset="utf-8" From: Peter Xu We've already done that with multifd_flush_after_each_section, for multifd in general. Drop the stale "TODO-like" comment. Reviewed-by: Fabiano Rosas Signed-off-by: Peter Xu --- migration/multifd.c | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 25cbc6dc6b..eee2586770 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -598,17 +598,6 @@ int multifd_send_sync_main(void) } } =20 - /* - * When using zero-copy, it's necessary to flush the pages before any = of - * the pages can be sent again, so we'll make sure the new version of = the - * pages will always arrive _later_ than the old pages. - * - * Currently we achieve this by flushing the zero-page requested writes - * per ram iteration, but in the future we could potentially optimize = it - * to be less frequent, e.g. only after we finished one whole scanning= of - * all the dirty bitmaps. - */ - flush_zero_copy =3D migrate_zero_copy_send(); =20 for (i =3D 0; i < migrate_multifd_channels(); i++) { --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869818; cv=none; d=zohomail.com; s=zohoarc; b=lecPYNM9r7xIj0POwxyV1b36WtRD4VLIttXs3zTvoBRLBs2KYo2rQFx7eEIhp85Tranze98T2Z1J5vsXwElcXolFe03QGUSVPmp5MFt9Liz9JVpE21G223fiB5kDXImMeUxJOflyGMRz61T7QenbkPqWYFY0DgdJ049or+OYHE8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869818; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=SMVA61FG00o5F2sgpzvPDSC3j1syRcvDYKyMGntilGw=; b=TGfOIaqlGDinvTrsCJdaxs9MC9dYutcWUgPZNUtKFpzo5dKNdYqVqwi7dFtRbrBJd2LfybjJnks8YgDGWNMYoV5Cn3HeB3NoD07IORFVvHolsZnAZCDQKnMfluCM0xWTuwQXiiDvT6D4TPOGMwo+Y6NfPNPiRpolV1IXyeqBuGY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869818334482.4810795650511; Fri, 2 Feb 2024 02:30:18 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqnj-00088A-Rr; Fri, 02 Feb 2024 05:29:39 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqnh-00087m-QB for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:29:37 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqng-0001Yh-BA for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:29:37 -0500 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-287-fbHjxloxPQKcoi_qmKVORg-1; Fri, 02 Feb 2024 05:29:33 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D4E24868901; Fri, 2 Feb 2024 10:29:32 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 28CAA400D783; Fri, 2 Feb 2024 10:29:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869775; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SMVA61FG00o5F2sgpzvPDSC3j1syRcvDYKyMGntilGw=; b=BWGfF8Fzabyr0B0smj5qU/vPRwTejDDccrRkI5v9qEJrZWS+qye8Tur/NYUpbxA6xEE/sq GvijLsg0fJQ2dOTloJ4Kmoa1PMwuSGliS5UBvb0D2nKAS1TVI/kaq15/n0YVmeycIm3Ozf DPpNQ7H0Z65XEi2tN6J20I1y0Nhq7qo= X-MC-Unique: fbHjxloxPQKcoi_qmKVORg-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 02/23] migration/multifd: multifd_send_kick_main() Date: Fri, 2 Feb 2024 18:28:36 +0800 Message-ID: <20240202102857.110210-3-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869819469100002 Content-Type: text/plain; charset="utf-8" From: Peter Xu When a multifd sender thread hit errors, it always needs to kick the main thread by kicking all the semaphores that it can be waiting upon. Provide a helper for it and deduplicate the code. Reviewed-by: Fabiano Rosas Signed-off-by: Peter Xu --- migration/multifd.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index eee2586770..b8d2c96533 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -372,6 +372,18 @@ struct { MultiFDMethods *ops; } *multifd_send_state; =20 +/* + * The migration thread can wait on either of the two semaphores. This + * function can be used to kick the main thread out of waiting on either of + * them. Should mostly only be called when something wrong happened with + * the current multifd send thread. + */ +static void multifd_send_kick_main(MultiFDSendParams *p) +{ + qemu_sem_post(&p->sem_sync); + qemu_sem_post(&multifd_send_state->channels_ready); +} + /* * How we use multifd_send_state->pages and channel->pages? * @@ -739,8 +751,7 @@ out: assert(local_err); trace_multifd_send_error(p->id); multifd_send_terminate_threads(local_err); - qemu_sem_post(&p->sem_sync); - qemu_sem_post(&multifd_send_state->channels_ready); + multifd_send_kick_main(p); error_free(local_err); } =20 @@ -781,8 +792,7 @@ static void multifd_tls_outgoing_handshake(QIOTask *tas= k, * is not created, and then tell who pay attention to me. */ p->quit =3D true; - qemu_sem_post(&multifd_send_state->channels_ready); - qemu_sem_post(&p->sem_sync); + multifd_send_kick_main(p); error_free(err); } =20 @@ -852,8 +862,7 @@ static void multifd_new_send_channel_cleanup(MultiFDSen= dParams *p, { migrate_set_error(migrate_get_current(), err); /* Error happen, we need to tell who pay attention to me */ - qemu_sem_post(&multifd_send_state->channels_ready); - qemu_sem_post(&p->sem_sync); + multifd_send_kick_main(p); /* * Although multifd_send_thread is not created, but main migration * thread need to judge whether it is running, so we need to mark --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869844; cv=none; d=zohomail.com; s=zohoarc; b=OWoziHI/glKiO++w6ixbpwWNwk8mIFQ+IRnC5RmpWkvuz/rV4qT6wylQaadJHJxq+rQvsmzhM7c7SVkJxgSwtbW1p49MHl2DrPs5NN7dTEH913bxj4YzC2mvEJpXs/7ESDwNQrUVssccmZxQBCP0MaGtoijlqJp+1CsI3vIQ0zw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869844; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=a9GwF5wr+IkbZjgGSvzhOPqUroVN/4alXq1fUUQ0DrU=; b=S3lKWQgEKiMTeVJu9QfH6qOHnkCRMI3s9XOenCEntHz9CbC6t7KqzhHP36NQBrLtZM8YrtAhLnjhumkLnV4KY20bTywHwbYrP+kQbQckaBYTMbiVWYNUhUirkZREOQcBnNQUYeXpyR4+JIgN/H4rUM3o6QrPlPCje9pqMUzimdc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869844789549.7106975250879; Fri, 2 Feb 2024 02:30:44 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqnm-00088n-R7; Fri, 02 Feb 2024 05:29:42 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqnk-00088D-UW for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:29:40 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqni-0001Z2-UL for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:29:40 -0500 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-114-XaYXVdYaNnOyobl4aROsxQ-1; Fri, 02 Feb 2024 05:29:37 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B28C585A58C; Fri, 2 Feb 2024 10:29:36 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7691640C1231; Fri, 2 Feb 2024 10:29:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869778; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a9GwF5wr+IkbZjgGSvzhOPqUroVN/4alXq1fUUQ0DrU=; b=CO2tB6z/k9knrfFr8anRyHtB5Tnin0MKm2alxJZ2AOsKwblgtGmdEJzr5h3Lpx1fGlQPXC 6Nmao8Llr1JILpzKBgTdtXF8FchTaViopsTk9lGsHlLjq4gDqFNSXWTbNqq9qQy3DHDNFK 5oyvMtOjy3+uPoI6QNhvlCdSlqLlhaY= X-MC-Unique: XaYXVdYaNnOyobl4aROsxQ-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 03/23] migration/multifd: Drop MultiFDSendParams.quit, cleanup error paths Date: Fri, 2 Feb 2024 18:28:37 +0800 Message-ID: <20240202102857.110210-4-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869845112100001 Content-Type: text/plain; charset="utf-8" From: Peter Xu Multifd send side has two fields to indicate error quits: - MultiFDSendParams.quit - &multifd_send_state->exiting Merge them into the global one. The replacement is done by changing all p->quit checks into the global var check. The global check doesn't need any lock. A few more things done on top of this altogether: - multifd_send_terminate_threads() Moving the xchg() of &multifd_send_state->exiting upper, so as to cover the tracepoint, migrate_set_error() and migrate_set_state(). - multifd_send_sync_main() In the 2nd loop, add one more check over the global var to make sure we don't keep the looping if QEMU already decided to quit. - multifd_tls_outgoing_handshake() Use multifd_send_terminate_threads() to set the error state. That has a benefit of updating MigrationState.error to that error too, so we can persist that 1st error we hit in that specific channel. - multifd_new_send_channel_async() Take similar approach like above, drop the migrate_set_error() because multifd_send_terminate_threads() already covers that. Unwrap the helper multifd_new_send_channel_cleanup() along the way; not really needed. Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/multifd.h | 2 -- migration/multifd.c | 85 ++++++++++++++++++--------------------------- 2 files changed, 33 insertions(+), 54 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index 35d11f103c..7c040cb85a 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -95,8 +95,6 @@ typedef struct { QemuMutex mutex; /* is this channel thread running */ bool running; - /* should this thread finish */ - bool quit; /* multifd flags for each packet */ uint32_t flags; /* global number of generated multifd packets */ diff --git a/migration/multifd.c b/migration/multifd.c index b8d2c96533..2c98023d67 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -372,6 +372,11 @@ struct { MultiFDMethods *ops; } *multifd_send_state; =20 +static bool multifd_send_should_exit(void) +{ + return qatomic_read(&multifd_send_state->exiting); +} + /* * The migration thread can wait on either of the two semaphores. This * function can be used to kick the main thread out of waiting on either of @@ -409,7 +414,7 @@ static int multifd_send_pages(void) MultiFDSendParams *p =3D NULL; /* make happy gcc */ MultiFDPages_t *pages =3D multifd_send_state->pages; =20 - if (qatomic_read(&multifd_send_state->exiting)) { + if (multifd_send_should_exit()) { return -1; } =20 @@ -421,14 +426,11 @@ static int multifd_send_pages(void) */ next_channel %=3D migrate_multifd_channels(); for (i =3D next_channel;; i =3D (i + 1) % migrate_multifd_channels()) { - p =3D &multifd_send_state->params[i]; - - qemu_mutex_lock(&p->mutex); - if (p->quit) { - error_report("%s: channel %d has already quit!", __func__, i); - qemu_mutex_unlock(&p->mutex); + if (multifd_send_should_exit()) { return -1; } + p =3D &multifd_send_state->params[i]; + qemu_mutex_lock(&p->mutex); if (!p->pending_job) { p->pending_job++; next_channel =3D (i + 1) % migrate_multifd_channels(); @@ -483,6 +485,16 @@ static void multifd_send_terminate_threads(Error *err) { int i; =20 + /* + * We don't want to exit each threads twice. Depending on where + * we get the error, or if there are two independent errors in two + * threads at the same time, we can end calling this function + * twice. + */ + if (qatomic_xchg(&multifd_send_state->exiting, 1)) { + return; + } + trace_multifd_send_terminate_threads(err !=3D NULL); =20 if (err) { @@ -497,26 +509,13 @@ static void multifd_send_terminate_threads(Error *err) } } =20 - /* - * We don't want to exit each threads twice. Depending on where - * we get the error, or if there are two independent errors in two - * threads at the same time, we can end calling this function - * twice. - */ - if (qatomic_xchg(&multifd_send_state->exiting, 1)) { - return; - } - for (i =3D 0; i < migrate_multifd_channels(); i++) { MultiFDSendParams *p =3D &multifd_send_state->params[i]; =20 - qemu_mutex_lock(&p->mutex); - p->quit =3D true; qemu_sem_post(&p->sem); if (p->c) { qio_channel_shutdown(p->c, QIO_CHANNEL_SHUTDOWN_BOTH, NULL); } - qemu_mutex_unlock(&p->mutex); } } =20 @@ -615,16 +614,13 @@ int multifd_send_sync_main(void) for (i =3D 0; i < migrate_multifd_channels(); i++) { MultiFDSendParams *p =3D &multifd_send_state->params[i]; =20 - trace_multifd_send_sync_main_signal(p->id); - - qemu_mutex_lock(&p->mutex); - - if (p->quit) { - error_report("%s: channel %d has already quit", __func__, i); - qemu_mutex_unlock(&p->mutex); + if (multifd_send_should_exit()) { return -1; } =20 + trace_multifd_send_sync_main_signal(p->id); + + qemu_mutex_lock(&p->mutex); p->packet_num =3D multifd_send_state->packet_num++; p->flags |=3D MULTIFD_FLAG_SYNC; p->pending_job++; @@ -634,6 +630,10 @@ int multifd_send_sync_main(void) for (i =3D 0; i < migrate_multifd_channels(); i++) { MultiFDSendParams *p =3D &multifd_send_state->params[i]; =20 + if (multifd_send_should_exit()) { + return -1; + } + qemu_sem_wait(&multifd_send_state->channels_ready); trace_multifd_send_sync_main_wait(p->id); qemu_sem_wait(&p->sem_sync); @@ -671,7 +671,7 @@ static void *multifd_send_thread(void *opaque) qemu_sem_post(&multifd_send_state->channels_ready); qemu_sem_wait(&p->sem); =20 - if (qatomic_read(&multifd_send_state->exiting)) { + if (multifd_send_should_exit()) { break; } qemu_mutex_lock(&p->mutex); @@ -786,12 +786,7 @@ static void multifd_tls_outgoing_handshake(QIOTask *ta= sk, =20 trace_multifd_tls_outgoing_handshake_error(ioc, error_get_pretty(err)); =20 - migrate_set_error(migrate_get_current(), err); - /* - * Error happen, mark multifd_send_thread status as 'quit' although it - * is not created, and then tell who pay attention to me. - */ - p->quit =3D true; + multifd_send_terminate_threads(err); multifd_send_kick_main(p); error_free(err); } @@ -857,22 +852,6 @@ static bool multifd_channel_connect(MultiFDSendParams = *p, return true; } =20 -static void multifd_new_send_channel_cleanup(MultiFDSendParams *p, - QIOChannel *ioc, Error *err) -{ - migrate_set_error(migrate_get_current(), err); - /* Error happen, we need to tell who pay attention to me */ - multifd_send_kick_main(p); - /* - * Although multifd_send_thread is not created, but main migration - * thread need to judge whether it is running, so we need to mark - * its status. - */ - p->quit =3D true; - object_unref(OBJECT(ioc)); - error_free(err); -} - static void multifd_new_send_channel_async(QIOTask *task, gpointer opaque) { MultiFDSendParams *p =3D opaque; @@ -889,7 +868,10 @@ static void multifd_new_send_channel_async(QIOTask *ta= sk, gpointer opaque) } =20 trace_multifd_new_send_channel_async_error(p->id, local_err); - multifd_new_send_channel_cleanup(p, ioc, local_err); + multifd_send_terminate_threads(local_err); + multifd_send_kick_main(p); + object_unref(OBJECT(ioc)); + error_free(local_err); } =20 static void multifd_new_send_channel_create(gpointer opaque) @@ -921,7 +903,6 @@ int multifd_save_setup(Error **errp) qemu_mutex_init(&p->mutex); qemu_sem_init(&p->sem, 0); qemu_sem_init(&p->sem_sync, 0); - p->quit =3D false; p->pending_job =3D 0; p->id =3D i; p->pages =3D multifd_pages_init(page_count); --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869858; cv=none; d=zohomail.com; s=zohoarc; b=ZoCqGBDBoDEI7Ca5oguXQuYCTxzR1x30ylINNm01wBVFs7aP5ULDNQbeUlPDVp7ldAzgutmf7TLNS4gfD/BkYe+G5szGbOiE0RYUVKuhoEX7kTH3omQ1lCMxvA0ixtiBWyVdmwOOe3pzsOPGfZ/wsnCNWwAljYZ5N1QInfE7Keg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869858; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=MWl+CT7rBfQ5EHIwdoEgl5y+RskL9DOHcL6phY6qV2M=; b=bCY00Oseba/93PzCNjwdLxBk4lFleWH5c0uffThw/tUJ6SPeHsOl4wfzdsoxsHuf0GTRMQTRTyXeBKS+S/qfUFV0szuakl+lEIghvJyJFYtmQDfPKOjdIEMi4Te5KqidwtjN/FdEFYJY+q5a4PBs42fOPlwkYb1xXzyiWA5E4uQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869858086749.0621209411147; Fri, 2 Feb 2024 02:30:58 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqnv-00089A-KK; Fri, 02 Feb 2024 05:29:51 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqnt-00088y-IQ for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:29:49 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqns-0001an-3O for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:29:49 -0500 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-203-gR7yn4DmPgK3mDhIOKJ0sg-1; Fri, 02 Feb 2024 05:29:41 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 27AC08353E0; Fri, 2 Feb 2024 10:29:41 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 524EB40C9444; Fri, 2 Feb 2024 10:29:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869787; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MWl+CT7rBfQ5EHIwdoEgl5y+RskL9DOHcL6phY6qV2M=; b=NiqnNh6Z1wcuq9+OlvjcuQaTWePIgCjPWu9ex4EykBi7hKW81dXJjTmhOKvT7ps8aRxSgy yq5lRn+/wLDhaUkCL8Q0aOEQwBtP+BVfgMZZvbUWJsv+DUBBdLe4n6LhmIalLvPVVRGEES ZDsK2WMZPDvwodTbljQXEGE1rFRncBs= X-MC-Unique: gR7yn4DmPgK3mDhIOKJ0sg-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 04/23] migration/multifd: Postpone reset of MultiFDPages_t Date: Fri, 2 Feb 2024 18:28:38 +0800 Message-ID: <20240202102857.110210-5-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869859207100003 Content-Type: text/plain; charset="utf-8" From: Peter Xu Now we reset MultiFDPages_t object in the multifd sender thread in the middle of the sending job. That's not necessary, because the "*pages" struct will not be reused anyway until pending_job is cleared. Move that to the end after the job is completed, provide a helper to reset a "*pages" object. Use that same helper when free the object too. This prepares us to keep using p->pages in the follow up patches, where we may drop p->normal[]. Reviewed-by: Fabiano Rosas Signed-off-by: Peter Xu --- migration/multifd.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 2c98023d67..5633ac245a 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -172,6 +172,17 @@ void multifd_register_ops(int method, MultiFDMethods *= ops) multifd_ops[method] =3D ops; } =20 +/* Reset a MultiFDPages_t* object for the next use */ +static void multifd_pages_reset(MultiFDPages_t *pages) +{ + /* + * We don't need to touch offset[] array, because it will be + * overwritten later when reused. + */ + pages->num =3D 0; + pages->block =3D NULL; +} + static int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) { MultiFDInit_t msg =3D {}; @@ -248,9 +259,8 @@ static MultiFDPages_t *multifd_pages_init(uint32_t n) =20 static void multifd_pages_clear(MultiFDPages_t *pages) { - pages->num =3D 0; + multifd_pages_reset(pages); pages->allocated =3D 0; - pages->block =3D NULL; g_free(pages->offset); pages->offset =3D NULL; g_free(pages); @@ -704,8 +714,6 @@ static void *multifd_send_thread(void *opaque) p->flags =3D 0; p->num_packets++; p->total_normal_pages +=3D p->normal_num; - p->pages->num =3D 0; - p->pages->block =3D NULL; qemu_mutex_unlock(&p->mutex); =20 trace_multifd_send(p->id, packet_num, p->normal_num, flags, @@ -732,6 +740,8 @@ static void *multifd_send_thread(void *opaque) =20 stat64_add(&mig_stats.multifd_bytes, p->next_packet_size + p->packet_len); + + multifd_pages_reset(p->pages); p->next_packet_size =3D 0; qemu_mutex_lock(&p->mutex); p->pending_job--; --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869875; cv=none; d=zohomail.com; s=zohoarc; b=BkHeJA40gMGqFvEllV3oQ2nHF5w/I6wkO64hcbTk7Jhdt0l3hjN4Iy9FWL9l8XxdyjySOXl2gOcCn4W4FQonImdAtQISjT2ssyG28YQOp4G384VzZqbX17ooUHOcRA+xT93JdojEkYPjLtpTfYiKLhxh+WYej6TCShjVy2ISSR8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869875; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=YLm1WRjy8Won4jKjMSOId0CsIIQpvrTfJdp6D2eLFP4=; b=dxHi983nBwYEIuxAoXxW0DB3M4UwR/0/a9VHzyWeHoVstTmBdUOOb6xYzzXRh3GhkAKCTCkO5Yqbx8xBgKeJ0iZrj99Hxk2vE7RxZxItmCAhniVkywk5A1P6euRy94C74YNOFvJ0qdwdWw4EC8dCgYiduSauPTdY9DM0F2p2d8k= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869875612872.851905403744; Fri, 2 Feb 2024 02:31:15 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqnz-0008As-6L; Fri, 02 Feb 2024 05:29:55 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqnx-0008A3-TW for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:29:53 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqnw-0001bD-8C for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:29:53 -0500 Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-62-rL8O6IWwMFWj1u29Xsrc1w-1; Fri, 02 Feb 2024 05:29:46 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1393B3C0BE4D; Fri, 2 Feb 2024 10:29:46 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id EC0E340C9444; Fri, 2 Feb 2024 10:29:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869791; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YLm1WRjy8Won4jKjMSOId0CsIIQpvrTfJdp6D2eLFP4=; b=TGOY+GuBlTGmNDkSyqu/zhKDD/dTT/tuqedSr17K9KbR4CDnL/nTlrvg6Cv++cK/exV80J ktmmoneKRrFWh0sUrelgNG1fMdkUfWlu/vEjwmhUJ9OgCsWq050RxeQYWdItvwKcSdyZpS M8Q3FbD1Ec20MpIrhkXRp8qTz99Rj7Q= X-MC-Unique: rL8O6IWwMFWj1u29Xsrc1w-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 05/23] migration/multifd: Drop MultiFDSendParams.normal[] array Date: Fri, 2 Feb 2024 18:28:39 +0800 Message-ID: <20240202102857.110210-6-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869877285100003 Content-Type: text/plain; charset="utf-8" From: Peter Xu This array is redundant when p->pages exists. Now we extended the life of p->pages to the whole period where pending_job is set, it should be safe to always use p->pages->offset[] rather than p->normal[]. Drop the array. Alongside, the normal_num is also redundant, which is the same to p->pages->num. This doesn't apply to recv side, because there's no extra buffering on recv side, so p->normal[] array is still needed. Reviewed-by: Fabiano Rosas Signed-off-by: Peter Xu --- migration/multifd.h | 4 ---- migration/multifd-zlib.c | 7 ++++--- migration/multifd-zstd.c | 7 ++++--- migration/multifd.c | 33 +++++++++++++-------------------- 4 files changed, 21 insertions(+), 30 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index 7c040cb85a..3920bdbcf1 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -122,10 +122,6 @@ typedef struct { struct iovec *iov; /* number of iovs used */ uint32_t iovs_num; - /* Pages that are not zero */ - ram_addr_t *normal; - /* num of non zero pages */ - uint32_t normal_num; /* used for compression methods */ void *data; } MultiFDSendParams; diff --git a/migration/multifd-zlib.c b/migration/multifd-zlib.c index 37ce48621e..100809abc1 100644 --- a/migration/multifd-zlib.c +++ b/migration/multifd-zlib.c @@ -116,17 +116,18 @@ static void zlib_send_cleanup(MultiFDSendParams *p, E= rror **errp) */ static int zlib_send_prepare(MultiFDSendParams *p, Error **errp) { + MultiFDPages_t *pages =3D p->pages; struct zlib_data *z =3D p->data; z_stream *zs =3D &z->zs; uint32_t out_size =3D 0; int ret; uint32_t i; =20 - for (i =3D 0; i < p->normal_num; i++) { + for (i =3D 0; i < pages->num; i++) { uint32_t available =3D z->zbuff_len - out_size; int flush =3D Z_NO_FLUSH; =20 - if (i =3D=3D p->normal_num - 1) { + if (i =3D=3D pages->num - 1) { flush =3D Z_SYNC_FLUSH; } =20 @@ -135,7 +136,7 @@ static int zlib_send_prepare(MultiFDSendParams *p, Erro= r **errp) * with compression. zlib does not guarantee that this is safe, * therefore copy the page before calling deflate(). */ - memcpy(z->buf, p->pages->block->host + p->normal[i], p->page_size); + memcpy(z->buf, p->pages->block->host + pages->offset[i], p->page_s= ize); zs->avail_in =3D p->page_size; zs->next_in =3D z->buf; =20 diff --git a/migration/multifd-zstd.c b/migration/multifd-zstd.c index b471daadcd..2023edd8cc 100644 --- a/migration/multifd-zstd.c +++ b/migration/multifd-zstd.c @@ -113,6 +113,7 @@ static void zstd_send_cleanup(MultiFDSendParams *p, Err= or **errp) */ static int zstd_send_prepare(MultiFDSendParams *p, Error **errp) { + MultiFDPages_t *pages =3D p->pages; struct zstd_data *z =3D p->data; int ret; uint32_t i; @@ -121,13 +122,13 @@ static int zstd_send_prepare(MultiFDSendParams *p, Er= ror **errp) z->out.size =3D z->zbuff_len; z->out.pos =3D 0; =20 - for (i =3D 0; i < p->normal_num; i++) { + for (i =3D 0; i < pages->num; i++) { ZSTD_EndDirective flush =3D ZSTD_e_continue; =20 - if (i =3D=3D p->normal_num - 1) { + if (i =3D=3D pages->num - 1) { flush =3D ZSTD_e_flush; } - z->in.src =3D p->pages->block->host + p->normal[i]; + z->in.src =3D p->pages->block->host + pages->offset[i]; z->in.size =3D p->page_size; z->in.pos =3D 0; =20 diff --git a/migration/multifd.c b/migration/multifd.c index 5633ac245a..8bb1fd95cf 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -90,13 +90,13 @@ static int nocomp_send_prepare(MultiFDSendParams *p, Er= ror **errp) { MultiFDPages_t *pages =3D p->pages; =20 - for (int i =3D 0; i < p->normal_num; i++) { - p->iov[p->iovs_num].iov_base =3D pages->block->host + p->normal[i]; + for (int i =3D 0; i < pages->num; i++) { + p->iov[p->iovs_num].iov_base =3D pages->block->host + pages->offse= t[i]; p->iov[p->iovs_num].iov_len =3D p->page_size; p->iovs_num++; } =20 - p->next_packet_size =3D p->normal_num * p->page_size; + p->next_packet_size =3D pages->num * p->page_size; p->flags |=3D MULTIFD_FLAG_NOCOMP; return 0; } @@ -269,21 +269,22 @@ static void multifd_pages_clear(MultiFDPages_t *pages) static void multifd_send_fill_packet(MultiFDSendParams *p) { MultiFDPacket_t *packet =3D p->packet; + MultiFDPages_t *pages =3D p->pages; int i; =20 packet->flags =3D cpu_to_be32(p->flags); packet->pages_alloc =3D cpu_to_be32(p->pages->allocated); - packet->normal_pages =3D cpu_to_be32(p->normal_num); + packet->normal_pages =3D cpu_to_be32(pages->num); packet->next_packet_size =3D cpu_to_be32(p->next_packet_size); packet->packet_num =3D cpu_to_be64(p->packet_num); =20 - if (p->pages->block) { - strncpy(packet->ramblock, p->pages->block->idstr, 256); + if (pages->block) { + strncpy(packet->ramblock, pages->block->idstr, 256); } =20 - for (i =3D 0; i < p->normal_num; i++) { + for (i =3D 0; i < pages->num; i++) { /* there are architectures where ram_addr_t is 32 bit */ - uint64_t temp =3D p->normal[i]; + uint64_t temp =3D pages->offset[i]; =20 packet->offset[i] =3D cpu_to_be64(temp); } @@ -570,8 +571,6 @@ void multifd_save_cleanup(void) p->packet =3D NULL; g_free(p->iov); p->iov =3D NULL; - g_free(p->normal); - p->normal =3D NULL; multifd_send_state->ops->send_cleanup(p, &local_err); if (local_err) { migrate_set_error(migrate_get_current(), local_err); @@ -688,8 +687,8 @@ static void *multifd_send_thread(void *opaque) =20 if (p->pending_job) { uint64_t packet_num =3D p->packet_num; + MultiFDPages_t *pages =3D p->pages; uint32_t flags; - p->normal_num =3D 0; =20 if (use_zero_copy_send) { p->iovs_num =3D 0; @@ -697,12 +696,7 @@ static void *multifd_send_thread(void *opaque) p->iovs_num =3D 1; } =20 - for (int i =3D 0; i < p->pages->num; i++) { - p->normal[p->normal_num] =3D p->pages->offset[i]; - p->normal_num++; - } - - if (p->normal_num) { + if (pages->num) { ret =3D multifd_send_state->ops->send_prepare(p, &local_er= r); if (ret !=3D 0) { qemu_mutex_unlock(&p->mutex); @@ -713,10 +707,10 @@ static void *multifd_send_thread(void *opaque) flags =3D p->flags; p->flags =3D 0; p->num_packets++; - p->total_normal_pages +=3D p->normal_num; + p->total_normal_pages +=3D pages->num; qemu_mutex_unlock(&p->mutex); =20 - trace_multifd_send(p->id, packet_num, p->normal_num, flags, + trace_multifd_send(p->id, packet_num, pages->num, flags, p->next_packet_size); =20 if (use_zero_copy_send) { @@ -924,7 +918,6 @@ int multifd_save_setup(Error **errp) p->name =3D g_strdup_printf("multifdsend_%d", i); /* We need one extra place for the packet header */ p->iov =3D g_new0(struct iovec, page_count + 1); - p->normal =3D g_new0(ram_addr_t, page_count); p->page_size =3D qemu_target_page_size(); p->page_count =3D page_count; =20 --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869854; cv=none; d=zohomail.com; s=zohoarc; b=S/16IWZAIhm+yugAO76N4vsx5see/STU8b2Kf7kJv2P4dq0iRF1BSvvboQIiDz1dorVoQ8gdrZfVzO09DaND/w3kPq7HIIHQsY8xkQTjtxtIxMQKsyqSdvlHYdcD1yyRvAGxJBtXENhbU6TU0vPqmkgxolwJZDlLXcbdiSXJUzE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869854; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=TXT70xPEWyLM/ia39YTlc0qBsOBdM9SVlb+O5t0RHEk=; b=jWU3qSHMGGVTYupKvgmWOHTN7qn1AVcE9k6a3/TviRXo0NTFMqLXzc8R05H/yiJgYkA5i+SyQGmAHLTV3eMO+bSGkY5ZrRnmCPtiYnX/n57mlQs9XA5gyvRhWxTXuywW+3eBAjbRBwx90TwarDqVJTzMWcJp1xousCdbLiB04hk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869854050873.3131523890645; Fri, 2 Feb 2024 02:30:54 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqo2-0008Bh-EA; Fri, 02 Feb 2024 05:29:58 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqo1-0008BW-Kz for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:29:57 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqnz-0001be-WE for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:29:57 -0500 Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-571-BgaFOXgvNS2I6-yCA-dWYA-1; Fri, 02 Feb 2024 05:29:51 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F36551C04B69; Fri, 2 Feb 2024 10:29:50 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id D734340C95AD; Fri, 2 Feb 2024 10:29:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869794; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TXT70xPEWyLM/ia39YTlc0qBsOBdM9SVlb+O5t0RHEk=; b=TbgbSo3EAo+Xd2BUSFf2yZihDWTaagPBfcKENXE1rhRtNttelqfyiqqPQ2RAmowxvBJYMa Hy7SuhzCuDMTR0AE5ruM+uEan+Of8e0eUj9XX7zjxUO2ZLSuLULl9/dbfdbE+u4kfl6OBY fubRoov/BrevNln2bDSxpxHmgRJp2V4= X-MC-Unique: BgaFOXgvNS2I6-yCA-dWYA-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 06/23] migration/multifd: Separate SYNC request with normal jobs Date: Fri, 2 Feb 2024 18:28:40 +0800 Message-ID: <20240202102857.110210-7-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869855142100003 Content-Type: text/plain; charset="utf-8" From: Peter Xu Multifd provide a threaded model for processing jobs. On sender side, there can be two kinds of job: (1) a list of pages to send, or (2) a sync request. The sync request is a very special kind of job. It never contains a page array, but only a multifd packet telling the dest side to synchronize with sent pages. Before this patch, both requests use the pending_job field, no matter what the request is, it will boost pending_job, while multifd sender thread will decrement it after it finishes one job. However this should be racy, because SYNC is special in that it needs to set p->flags with MULTIFD_FLAG_SYNC, showing that this is a sync request. Consider a sequence of operations where: - migration thread enqueue a job to send some pages, pending_job++ (0->1) - [...before the selected multifd sender thread wakes up...] - migration thread enqueue another job to sync, pending_job++ (1->2), setup p->flags=3DMULTIFD_FLAG_SYNC - multifd sender thread wakes up, found pending_job=3D=3D2 - send the 1st packet with MULTIFD_FLAG_SYNC and list of pages - send the 2nd packet with flags=3D=3D0 and no pages This is not expected, because MULTIFD_FLAG_SYNC should hopefully be done after all the pages are received. Meanwhile, the 2nd packet will be completely useless, which contains zero information. I didn't verify above, but I think this issue is still benign in that at least on the recv side we always receive pages before handling MULTIFD_FLAG_SYNC. However that's not always guaranteed and just tricky. One other reason I want to separate it is using p->flags to communicate between the two threads is also not clearly defined, it's very hard to read and understand why accessing p->flags is always safe; see the current impl of multifd_send_thread() where we tried to cache only p->flags. It doesn't need to be that complicated. This patch introduces pending_sync, a separate flag just to show that the requester needs a sync. Alongside, we remove the tricky caching of p->flags now because after this patch p->flags should only be used by multifd sender thread now, which will be crystal clear. So it is always thread safe to access p->flags. With that, we can also safely convert the pending_job into a boolean, because we don't support >1 pending jobs anyway. Always use atomic ops to access both flags to make sure no cache effect. When at it, drop the initial setting of "pending_job =3D 0" because it's always allocated using g_new0(). Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/multifd.h | 13 +++++++++++-- migration/multifd.c | 39 +++++++++++++++++++++++++-------------- 2 files changed, 36 insertions(+), 16 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index 3920bdbcf1..08f26ef3fe 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -99,8 +99,17 @@ typedef struct { uint32_t flags; /* global number of generated multifd packets */ uint64_t packet_num; - /* thread has work to do */ - int pending_job; + /* + * The sender thread has work to do if either of below boolean is set. + * + * @pending_job: a job is pending + * @pending_sync: a sync request is pending + * + * For both of these fields, they're only set by the requesters, and + * cleared by the multifd sender threads. + */ + bool pending_job; + bool pending_sync; /* array of pages to sent. * The owner of 'pages' depends of 'pending_job' value: * pending_job =3D=3D 0 -> migration_thread can use it. diff --git a/migration/multifd.c b/migration/multifd.c index 8bb1fd95cf..ea25bbe6bd 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -442,8 +442,8 @@ static int multifd_send_pages(void) } p =3D &multifd_send_state->params[i]; qemu_mutex_lock(&p->mutex); - if (!p->pending_job) { - p->pending_job++; + if (qatomic_read(&p->pending_job) =3D=3D false) { + qatomic_set(&p->pending_job, true); next_channel =3D (i + 1) % migrate_multifd_channels(); break; } @@ -631,8 +631,12 @@ int multifd_send_sync_main(void) =20 qemu_mutex_lock(&p->mutex); p->packet_num =3D multifd_send_state->packet_num++; - p->flags |=3D MULTIFD_FLAG_SYNC; - p->pending_job++; + /* + * We should be the only user so far, so not possible to be set by + * others concurrently. + */ + assert(qatomic_read(&p->pending_sync) =3D=3D false); + qatomic_set(&p->pending_sync, true); qemu_mutex_unlock(&p->mutex); qemu_sem_post(&p->sem); } @@ -685,10 +689,9 @@ static void *multifd_send_thread(void *opaque) } qemu_mutex_lock(&p->mutex); =20 - if (p->pending_job) { + if (qatomic_read(&p->pending_job)) { uint64_t packet_num =3D p->packet_num; MultiFDPages_t *pages =3D p->pages; - uint32_t flags; =20 if (use_zero_copy_send) { p->iovs_num =3D 0; @@ -704,13 +707,11 @@ static void *multifd_send_thread(void *opaque) } } multifd_send_fill_packet(p); - flags =3D p->flags; - p->flags =3D 0; p->num_packets++; p->total_normal_pages +=3D pages->num; qemu_mutex_unlock(&p->mutex); =20 - trace_multifd_send(p->id, packet_num, pages->num, flags, + trace_multifd_send(p->id, packet_num, pages->num, p->flags, p->next_packet_size); =20 if (use_zero_copy_send) { @@ -738,12 +739,23 @@ static void *multifd_send_thread(void *opaque) multifd_pages_reset(p->pages); p->next_packet_size =3D 0; qemu_mutex_lock(&p->mutex); - p->pending_job--; + qatomic_set(&p->pending_job, false); qemu_mutex_unlock(&p->mutex); - - if (flags & MULTIFD_FLAG_SYNC) { - qemu_sem_post(&p->sem_sync); + } else if (qatomic_read(&p->pending_sync)) { + p->flags =3D MULTIFD_FLAG_SYNC; + multifd_send_fill_packet(p); + ret =3D qio_channel_write_all(p->c, (void *)p->packet, + p->packet_len, &local_err); + if (ret !=3D 0) { + qemu_mutex_unlock(&p->mutex); + break; } + /* p->next_packet_size will always be zero for a SYNC packet */ + stat64_add(&mig_stats.multifd_bytes, p->packet_len); + p->flags =3D 0; + qatomic_set(&p->pending_sync, false); + qemu_mutex_unlock(&p->mutex); + qemu_sem_post(&p->sem_sync); } else { qemu_mutex_unlock(&p->mutex); /* sometimes there are spurious wakeups */ @@ -907,7 +919,6 @@ int multifd_save_setup(Error **errp) qemu_mutex_init(&p->mutex); qemu_sem_init(&p->sem, 0); qemu_sem_init(&p->sem_sync, 0); - p->pending_job =3D 0; p->id =3D i; p->pages =3D multifd_pages_init(page_count); p->packet_len =3D sizeof(MultiFDPacket_t) --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869820; cv=none; d=zohomail.com; s=zohoarc; b=ajEngwheZIwCAKL8QRiQ6WkJO3O6AFU2ikXcKm29zoUZTVlBqMpASs/YVH4CVFTT8dy0/HqCdkLTOWyFKoZH80cJNRW/TESA57flcz5VSQ39j10XQSPQUcVo07xnDtenmtdQKdcHWHgt061uNPM+MSeD4R+++WsmqsPdOWwQe2k= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869820; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=B2hVNr5Dad1adIDm1bSbcW5cF6eL0qWPX8iO45Hx+VI=; b=UtmwkvlpqocP5sKSK0upEKJSlEgrJbir3h9Glp7Q+PznJV1hvHvKS1BuuaAReKsDYi6bNrhtcQSU18P9pb1XuzWb8xP1tGZe6iBdumC6eUDkVDzCBSMWmz8rgMo2jRGEhboHfovEXWeEh26ZU2IOEHgLLrE+vuncRK/O8kbcpSY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869820438914.0141730473281; Fri, 2 Feb 2024 02:30:20 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqo7-0008Co-TG; Fri, 02 Feb 2024 05:30:05 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqo5-0008CV-Rs for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:01 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqo4-0001c5-BJ for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:01 -0500 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-47-lLAjFLQtPFK4mHBCAWpvvA-1; Fri, 02 Feb 2024 05:29:56 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C58768353E1; Fri, 2 Feb 2024 10:29:55 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id C339440CD14B; Fri, 2 Feb 2024 10:29:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869799; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=B2hVNr5Dad1adIDm1bSbcW5cF6eL0qWPX8iO45Hx+VI=; b=Z22dsqa+WjLjfpJO/32EohYlIEC8ejU8JJ5gArmrilKHrhHlPQDEDEa8CdmvGqqu+P91Su 4AM7/Z7vxH4M5gKAA6q0aFyswt9l0Oankg4NYKOItRWVK6ZNoyRSbEvrSxXiFOzbU9a9eW HwRynx91Sq6T6NzhVbJRhAOgCsGpPCk= X-MC-Unique: lLAjFLQtPFK4mHBCAWpvvA-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 07/23] migration/multifd: Simplify locking in sender thread Date: Fri, 2 Feb 2024 18:28:41 +0800 Message-ID: <20240202102857.110210-8-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869820997100009 Content-Type: text/plain; charset="utf-8" From: Peter Xu The sender thread will yield the p->mutex before IO starts, trying to not block the requester thread. This may be unnecessary lock optimizations, because the requester can already read pending_job safely even without the lock, because the requester is currently the only one who can assign a task. Drop that lock complication on both sides: (1) in the sender thread, always take the mutex until job done (2) in the requester thread, check pending_job clear lockless Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/multifd.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index ea25bbe6bd..4d5a01ed93 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -429,7 +429,9 @@ static int multifd_send_pages(void) return -1; } =20 + /* We wait here, until at least one channel is ready */ qemu_sem_wait(&multifd_send_state->channels_ready); + /* * next_channel can remain from a previous migration that was * using more channels, so ensure it doesn't overflow if the @@ -441,17 +443,26 @@ static int multifd_send_pages(void) return -1; } p =3D &multifd_send_state->params[i]; - qemu_mutex_lock(&p->mutex); + /* + * Lockless read to p->pending_job is safe, because only multifd + * sender thread can clear it. + */ if (qatomic_read(&p->pending_job) =3D=3D false) { - qatomic_set(&p->pending_job, true); next_channel =3D (i + 1) % migrate_multifd_channels(); break; } - qemu_mutex_unlock(&p->mutex); } + + qemu_mutex_lock(&p->mutex); assert(!p->pages->num); assert(!p->pages->block); - + /* + * Double check on pending_job=3D=3Dfalse with the lock. In the futur= e if + * we can have >1 requester thread, we can replace this with a "goto + * retry", but that is for later. + */ + assert(qatomic_read(&p->pending_job) =3D=3D false); + qatomic_set(&p->pending_job, true); p->packet_num =3D multifd_send_state->packet_num++; multifd_send_state->pages =3D p->pages; p->pages =3D pages; @@ -709,8 +720,6 @@ static void *multifd_send_thread(void *opaque) multifd_send_fill_packet(p); p->num_packets++; p->total_normal_pages +=3D pages->num; - qemu_mutex_unlock(&p->mutex); - trace_multifd_send(p->id, packet_num, pages->num, p->flags, p->next_packet_size); =20 @@ -730,6 +739,7 @@ static void *multifd_send_thread(void *opaque) ret =3D qio_channel_writev_full_all(p->c, p->iov, p->iovs_num,= NULL, 0, p->write_flags, &local_er= r); if (ret !=3D 0) { + qemu_mutex_unlock(&p->mutex); break; } =20 @@ -738,7 +748,6 @@ static void *multifd_send_thread(void *opaque) =20 multifd_pages_reset(p->pages); p->next_packet_size =3D 0; - qemu_mutex_lock(&p->mutex); qatomic_set(&p->pending_job, false); qemu_mutex_unlock(&p->mutex); } else if (qatomic_read(&p->pending_sync)) { --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869926; cv=none; d=zohomail.com; s=zohoarc; b=gFOXa9EVc0ih8VEjUH9l6tWoHrq6F0AZX1ZcACgHAmWNT3OW4Wj8knA3wxlGckuzno/wmAsgc6qua1096Uh/eOPplekYzykvnm1qduEhSuZt2m9IpOHupDgml6lkM6jkk0xSQcGy6EVGasIwvduObrusBlYxcVz6HcI6jgM5aBY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869926; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=ZezFfAc9khgs7hj27BwXSI87lihBOShkrXxYtsGi1i8=; b=VUuwhj6n4IhwEq/rlgqZMsB+MH5c6rCYZ2gI4WS+DI6CwXG9kR92nbgGL/83xKsEah3H0c8ZKt6MOpBq7CYKvX3cOXGSvRCJQjoLxFK/qjX9YE4Ku70lCaLMgzVI8F9wMwjyTg4aIZ4MfHYijOThZ919vGA4H1ZYBrNrtMc/7gQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869926097433.95072545978746; Fri, 2 Feb 2024 02:32:06 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqoH-0008Du-EO; Fri, 02 Feb 2024 05:30:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqoF-0008DM-K0 for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:11 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqoE-0001oo-4N for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:11 -0500 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-175-eTppH7PPPISzg2vN_vHX6g-1; Fri, 02 Feb 2024 05:30:01 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B161085A58F; Fri, 2 Feb 2024 10:30:00 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 94F1540C95AD; Fri, 2 Feb 2024 10:29:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869809; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZezFfAc9khgs7hj27BwXSI87lihBOShkrXxYtsGi1i8=; b=QX14E3GX8CiDo5VQwKJgufEBBzjCa63B7kwp5O5suh6/5NnWPW/TQhA6bQVcgeIb/3FVUz 5N5CIc+E207+E+/Eo/ivJQftH4qpDV9fv+cO54+A6ctqU1+wtB6dnRJR/yNXIKg1WqqD0v MOUcbKS2W5JA76V3M+Ei6jW5iFCpYaY= X-MC-Unique: eTppH7PPPISzg2vN_vHX6g-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 08/23] migration/multifd: Drop pages->num check in sender thread Date: Fri, 2 Feb 2024 18:28:42 +0800 Message-ID: <20240202102857.110210-9-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869927390100009 Content-Type: text/plain; charset="utf-8" From: Peter Xu Now with a split SYNC handler, we always have pages->num set for pending_job=3D=3Dtrue. Assert it instead. Reviewed-by: Fabiano Rosas Signed-off-by: Peter Xu --- migration/multifd.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 4d5a01ed93..518f9de723 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -710,13 +710,14 @@ static void *multifd_send_thread(void *opaque) p->iovs_num =3D 1; } =20 - if (pages->num) { - ret =3D multifd_send_state->ops->send_prepare(p, &local_er= r); - if (ret !=3D 0) { - qemu_mutex_unlock(&p->mutex); - break; - } + assert(pages->num); + + ret =3D multifd_send_state->ops->send_prepare(p, &local_err); + if (ret !=3D 0) { + qemu_mutex_unlock(&p->mutex); + break; } + multifd_send_fill_packet(p); p->num_packets++; p->total_normal_pages +=3D pages->num; --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869963; cv=none; d=zohomail.com; s=zohoarc; b=Y3frA0fh4w+mL/QqtCqcujbOYnrW4hoi6MGxhqWo3O7WoOz/w54jF8c9gyGay3wJbHeKrhCl4GQaDGNuKiaJvL2GbiNuoqykgNfFueMoMyFtFiNeq5BW9pdi3DZUB7r2nFeIkBAtnKO/bSYzEXDRg89oVREhA1kofSExnQNBBQU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869963; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=X4xocvaBvjPa4sx3hQRwDEJ7tST+sUp5csWbv+7+6Mg=; b=ghYENlIkE60UwAzxJq7rz7olNqvxiscfS6ibRZUOXy+MXBlx7YTHjoj327fSwguWaFEDBWt5QyHkImvYu9bnTxblBOIx7h/52gtrbJVtLiSpKcQEg6Bpwx5xz94fWjsZ4tcLCx1Epwi2Vd4SPXaFMoS+zDpnbf7A+x9mBmHCDMU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869963811754.1883368392838; Fri, 2 Feb 2024 02:32:43 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqoL-0008EO-BN; Fri, 02 Feb 2024 05:30:17 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqoJ-0008Dz-DJ for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:15 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqoG-0001pJ-J1 for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:15 -0500 Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-677-DtkEB-LGPFe9rX6iWvSlBA-1; Fri, 02 Feb 2024 05:30:06 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9BABB280BC80; Fri, 2 Feb 2024 10:30:05 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8124C400D783; Fri, 2 Feb 2024 10:30:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869811; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X4xocvaBvjPa4sx3hQRwDEJ7tST+sUp5csWbv+7+6Mg=; b=VEox/K5svl7yzhl4gEn4EQT6Sz83Xr/uBOZtUM/oNDzmVcgggX+A0hNbwMfeA3IdI0uD2h laLublHtUub0dSoFh4jj5Mdnempgwr0/RkznLUPDZv1txmDzEf5DljBWZaHpr9zymKXswz wIUkoSvaCpaE/F7KTe7nDe4wBMUx/gQ= X-MC-Unique: DtkEB-LGPFe9rX6iWvSlBA-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 09/23] migration/multifd: Rename p->num_packets and clean it up Date: Fri, 2 Feb 2024 18:28:43 +0800 Message-ID: <20240202102857.110210-10-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869965574100006 Content-Type: text/plain; charset="utf-8" From: Peter Xu This field, no matter whether on src or dest, is only used for debugging purpose. They can even be removed already, unless it still more or less provide some accounting on "how many packets are sent/recved for this thread". The other more important one is called packet_num, which is embeded in the multifd packet headers (MultiFDPacket_t). So let's keep them for now, but make them much easier to understand, by doing below: - Rename both of them to packets_sent / packets_recved, the old name (num_packets) are waaay too confusing when we already have MultiFDPacket_t.packets_num. - Avoid worrying on the "initial packet": we know we will send it, that's good enough. The accounting won't matter a great deal to start with 0 or with 1. - Move them to where we send/recv the packets. They're: - multifd_send_fill_packet() for senders. - multifd_recv_unfill_packet() for receivers. Reviewed-by: Fabiano Rosas Signed-off-by: Peter Xu --- migration/multifd.h | 6 +++--- migration/multifd.c | 13 +++++-------- 2 files changed, 8 insertions(+), 11 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index 08f26ef3fe..2e4ad0dc56 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -124,7 +124,7 @@ typedef struct { /* size of the next packet that contains pages */ uint32_t next_packet_size; /* packets sent through this channel */ - uint64_t num_packets; + uint64_t packets_sent; /* non zero pages sent through this channel */ uint64_t total_normal_pages; /* buffers to send */ @@ -174,8 +174,8 @@ typedef struct { MultiFDPacket_t *packet; /* size of the next packet that contains pages */ uint32_t next_packet_size; - /* packets sent through this channel */ - uint64_t num_packets; + /* packets received through this channel */ + uint64_t packets_recved; /* ramblock */ RAMBlock *block; /* ramblock host address */ diff --git a/migration/multifd.c b/migration/multifd.c index 518f9de723..eca76e2c18 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -288,6 +288,8 @@ static void multifd_send_fill_packet(MultiFDSendParams = *p) =20 packet->offset[i] =3D cpu_to_be64(temp); } + + p->packets_sent++; } =20 static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp) @@ -335,6 +337,7 @@ static int multifd_recv_unfill_packet(MultiFDRecvParams= *p, Error **errp) =20 p->next_packet_size =3D be32_to_cpu(packet->next_packet_size); p->packet_num =3D be64_to_cpu(packet->packet_num); + p->packets_recved++; =20 if (p->normal_num =3D=3D 0) { return 0; @@ -688,8 +691,6 @@ static void *multifd_send_thread(void *opaque) ret =3D -1; goto out; } - /* initial packet */ - p->num_packets =3D 1; =20 while (true) { qemu_sem_post(&multifd_send_state->channels_ready); @@ -719,7 +720,6 @@ static void *multifd_send_thread(void *opaque) } =20 multifd_send_fill_packet(p); - p->num_packets++; p->total_normal_pages +=3D pages->num; trace_multifd_send(p->id, packet_num, pages->num, p->flags, p->next_packet_size); @@ -787,7 +787,7 @@ out: =20 rcu_unregister_thread(); migration_threads_remove(thread); - trace_multifd_send_thread_end(p->id, p->num_packets, p->total_normal_p= ages); + trace_multifd_send_thread_end(p->id, p->packets_sent, p->total_normal_= pages); =20 return NULL; } @@ -1124,7 +1124,6 @@ static void *multifd_recv_thread(void *opaque) p->flags &=3D ~MULTIFD_FLAG_SYNC; trace_multifd_recv(p->id, p->packet_num, p->normal_num, flags, p->next_packet_size); - p->num_packets++; p->total_normal_pages +=3D p->normal_num; qemu_mutex_unlock(&p->mutex); =20 @@ -1150,7 +1149,7 @@ static void *multifd_recv_thread(void *opaque) qemu_mutex_unlock(&p->mutex); =20 rcu_unregister_thread(); - trace_multifd_recv_thread_end(p->id, p->num_packets, p->total_normal_p= ages); + trace_multifd_recv_thread_end(p->id, p->packets_recved, p->total_norma= l_pages); =20 return NULL; } @@ -1252,8 +1251,6 @@ void multifd_recv_new_channel(QIOChannel *ioc, Error = **errp) } p->c =3D ioc; object_ref(OBJECT(ioc)); - /* initial packet */ - p->num_packets =3D 1; =20 p->running =3D true; qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p, --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869932; cv=none; d=zohomail.com; s=zohoarc; b=AwDBNBfalcrEtzebSxurMuJIysyBClsdoce+fHWA6q6euTeF2uG4/DVRgtslegkGPx4lHUQA0DuoNCN2X7qK7SmsSn5WigMBVSBWqhUhVXXsUzHeHAGem8cj6FP2AhlLDPlUpC+45Wvn5TtpMfYkXvULvu4Icw/KHCCALaaGkyA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869932; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=AJ8o52+EAQziqSs2esGXqEn6lIlcKIgYuVs6weMZU7Y=; b=QO5kIHFVx5GESmn8EFI634Wm/u99BDXw7tckboCN0Gj5GPY+aUlemS5OzY4Xrv2iCLSD2wTqOEDI7wnE9siYlhp/wnN7PGz4iINsqhjkR3khdVtrfIlf4j1GGc9mG+gKcPKU8IZncnCexOhlLQn6mfgCtMht7ADmy0ZZquRURg8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869932283412.21155347522927; Fri, 2 Feb 2024 02:32:12 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqoO-0008J7-34; Fri, 02 Feb 2024 05:30:22 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqoK-0008EA-Na for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:16 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqoI-0001pb-OM for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:16 -0500 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-173-eBe0-3IwM3-TzKzFBbujyg-1; Fri, 02 Feb 2024 05:30:10 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8764F8353E0; Fri, 2 Feb 2024 10:30:10 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6BF60400D783; Fri, 2 Feb 2024 10:30:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869814; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AJ8o52+EAQziqSs2esGXqEn6lIlcKIgYuVs6weMZU7Y=; b=SBHnfqS5W341fr/1xe4DDcbbXtmnIq+QXjxVs06hOAJBzjlHbhgwQl14IfLqVF9LcobQef y845l9FHNCSJmJOMEDwuIKl5Hq4d23eKUiWVpU9qpqzny6xn9qkUnn3kpleu6MLBslwag6 x3iJqSs+m4wU67OUAtsT8q4SboaMLqY= X-MC-Unique: eBe0-3IwM3-TzKzFBbujyg-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 10/23] migration/multifd: Move total_normal_pages accounting Date: Fri, 2 Feb 2024 18:28:44 +0800 Message-ID: <20240202102857.110210-11-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869933425100007 Content-Type: text/plain; charset="utf-8" From: Peter Xu Just like the previous patch, move the accounting for total_normal_pages on both src/dst sides into the packet fill/unfill procedures. Reviewed-by: Fabiano Rosas Signed-off-by: Peter Xu --- migration/multifd.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index eca76e2c18..94a0124934 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -290,6 +290,7 @@ static void multifd_send_fill_packet(MultiFDSendParams = *p) } =20 p->packets_sent++; + p->total_normal_pages +=3D pages->num; } =20 static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp) @@ -338,6 +339,7 @@ static int multifd_recv_unfill_packet(MultiFDRecvParams= *p, Error **errp) p->next_packet_size =3D be32_to_cpu(packet->next_packet_size); p->packet_num =3D be64_to_cpu(packet->packet_num); p->packets_recved++; + p->total_normal_pages +=3D p->normal_num; =20 if (p->normal_num =3D=3D 0) { return 0; @@ -720,7 +722,6 @@ static void *multifd_send_thread(void *opaque) } =20 multifd_send_fill_packet(p); - p->total_normal_pages +=3D pages->num; trace_multifd_send(p->id, packet_num, pages->num, p->flags, p->next_packet_size); =20 @@ -1124,7 +1125,6 @@ static void *multifd_recv_thread(void *opaque) p->flags &=3D ~MULTIFD_FLAG_SYNC; trace_multifd_recv(p->id, p->packet_num, p->normal_num, flags, p->next_packet_size); - p->total_normal_pages +=3D p->normal_num; qemu_mutex_unlock(&p->mutex); =20 if (p->normal_num) { --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869929; cv=none; d=zohomail.com; s=zohoarc; b=F73Y8btKu6+Suczh3c41+QouQpVaBxKEzhxLPGk3+/zC3/vjTIhQILNuJS1mlKbjY+OLZPZwtByIslvhX2VKHOzglf8/2MHFAKTig9zzPUpWpPOAqxvvZeVcfRJWJY50+hA/H9Q2oYCSVhFJbyVf9v2FDwEmvCrT74thYo750/s= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869929; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Y72v5vMz+FXPoHE0uZ1Hj7JBuygNXJb/VoP5SwIg+/8=; b=RCobBJVLr1j1gaXEB75m8NAvA1UGAXh9rpuWnQau6liPvV1Apvh1L+fdmAQr/emGiXheY3HmOyK1ZsQwVyDqcVqWg3f2tl54Wj7qBhhLWi1yr5RwcEu0MEzXusKvhoMCSH2L23FdZI2NE8i8fGOstZQgYaDTcUlwp2bQvGyThKU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869929299388.4798312251195; Fri, 2 Feb 2024 02:32:09 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqoZ-00008D-8l; Fri, 02 Feb 2024 05:30:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqoS-0008Qw-3C for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:24 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqoQ-0001sZ-M5 for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:23 -0500 Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-306-6TCVGFyFPhuBBKeoiPYd1Q-1; Fri, 02 Feb 2024 05:30:15 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 71F9D280BC80; Fri, 2 Feb 2024 10:30:15 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5744440C9444; Fri, 2 Feb 2024 10:30:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869821; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y72v5vMz+FXPoHE0uZ1Hj7JBuygNXJb/VoP5SwIg+/8=; b=IIqKtY63Tnm7F9q95PwjXsRGogQHd0cfywNnJPDPrho6zhrD9a9i94EZetkD77hLNg+1q5 5s64hm36NRDcbjOz5DUwRQXSb8sbHNvKTxqJKxbqon85ndxoJXeQyhUhEibjnQYuFoJWAU F3Hm65RaxWxFuDaCqTPmN6X/EJkZzG0= X-MC-Unique: 6TCVGFyFPhuBBKeoiPYd1Q-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 11/23] migration/multifd: Move trace_multifd_send|recv() Date: Fri, 2 Feb 2024 18:28:45 +0800 Message-ID: <20240202102857.110210-12-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869931443100003 Content-Type: text/plain; charset="utf-8" From: Peter Xu Move them into fill/unfill of packets. With that, we can further cleanup the send/recv thread procedure, and remove one more temp var. Reviewed-by: Fabiano Rosas Signed-off-by: Peter Xu --- migration/multifd.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 94a0124934..44163e4e28 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -291,6 +291,9 @@ static void multifd_send_fill_packet(MultiFDSendParams = *p) =20 p->packets_sent++; p->total_normal_pages +=3D pages->num; + + trace_multifd_send(p->id, p->packet_num, pages->num, p->flags, + p->next_packet_size); } =20 static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp) @@ -341,6 +344,9 @@ static int multifd_recv_unfill_packet(MultiFDRecvParams= *p, Error **errp) p->packets_recved++; p->total_normal_pages +=3D p->normal_num; =20 + trace_multifd_recv(p->id, p->packet_num, p->normal_num, p->flags, + p->next_packet_size); + if (p->normal_num =3D=3D 0) { return 0; } @@ -704,7 +710,6 @@ static void *multifd_send_thread(void *opaque) qemu_mutex_lock(&p->mutex); =20 if (qatomic_read(&p->pending_job)) { - uint64_t packet_num =3D p->packet_num; MultiFDPages_t *pages =3D p->pages; =20 if (use_zero_copy_send) { @@ -722,8 +727,6 @@ static void *multifd_send_thread(void *opaque) } =20 multifd_send_fill_packet(p); - trace_multifd_send(p->id, packet_num, pages->num, p->flags, - p->next_packet_size); =20 if (use_zero_copy_send) { /* Send header first, without zerocopy */ @@ -1123,8 +1126,6 @@ static void *multifd_recv_thread(void *opaque) flags =3D p->flags; /* recv methods don't know how to handle the SYNC flag */ p->flags &=3D ~MULTIFD_FLAG_SYNC; - trace_multifd_recv(p->id, p->packet_num, p->normal_num, flags, - p->next_packet_size); qemu_mutex_unlock(&p->mutex); =20 if (p->normal_num) { --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869964; cv=none; d=zohomail.com; s=zohoarc; b=c+ZXU9CFxl9y3YSdtsugBaZNQva4YvcBGvII5Cl9OBgp5HDW54vyadt5DXABvtrrt5zbqIrktwZKGhycmEkJkNKvYK30ry1Fy/NBHkrFMiqDkgX/zmtfl+yL5on6tE2JUUIoVwYNt+UJeXSNz0pCUOafZAYWkCUTNlCIxYoc6BM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869964; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=gQpm8kWXL8qqmYDuhCIkxNF1j+AUaRZ6kwdr9T8tKKM=; b=L75HKQ8ni53JorcXrYH8QCzQMfflbCNHuxDc9O0dNo1Y1s01XrfKnMR2z7zgZoeDHuDlvaR2TjqqnhsX65+s11QSM92q73QS6ljVGBOHkhNjhNiKtccMMFBFHwIqNa9PwaMcsFs7nFvviJNd7kxetr2M/jzhCdo92gQM69EiG1s= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869964735610.5585888441564; Fri, 2 Feb 2024 02:32:44 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqoa-0000Cb-4q; Fri, 02 Feb 2024 05:30:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqoV-0008Rc-BR for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:28 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqoT-0001vG-R9 for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:27 -0500 Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-655-vvlK9RwjOOCMi0KHr1PBxQ-1; Fri, 02 Feb 2024 05:30:20 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2B3BF3C0009A; Fri, 2 Feb 2024 10:30:20 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4241D40C9444; Fri, 2 Feb 2024 10:30:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869824; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gQpm8kWXL8qqmYDuhCIkxNF1j+AUaRZ6kwdr9T8tKKM=; b=H0EyJ/1LWCfVUQJq0JF12PeI1xiU44UT2iL4J0tXHTVDlFpufjIiuzfy5A9bc1Ry/NS50B xDw+lkak4WQ5PYppYl94uIHS8/CBGGDT2O6I1C8B01piXLZWN8o0+f87gbZv1rJt0uL6Lg GWR+66vmG0FTOgkdp5C9Et4y9DKQQVc= X-MC-Unique: vvlK9RwjOOCMi0KHr1PBxQ-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 12/23] migration/multifd: multifd_send_prepare_header() Date: Fri, 2 Feb 2024 18:28:46 +0800 Message-ID: <20240202102857.110210-13-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869965551100005 Content-Type: text/plain; charset="utf-8" From: Peter Xu Introduce a helper multifd_send_prepare_header() to setup the header packet for multifd sender. It's fine to setup the IOV[0] _before_ send_prepare() because the packet buffer is already ready, even if the content is to be filled in. With this helper, we can already slightly clean up the zero copy path. Note that I explicitly put it into multifd.h, because I want it inlined directly into multifd*.c where necessary later. Reviewed-by: Fabiano Rosas Signed-off-by: Peter Xu --- migration/multifd.h | 8 ++++++++ migration/multifd.c | 16 ++++++++-------- 2 files changed, 16 insertions(+), 8 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index 2e4ad0dc56..4ec005f53f 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -209,5 +209,13 @@ typedef struct { =20 void multifd_register_ops(int method, MultiFDMethods *ops); =20 +static inline void multifd_send_prepare_header(MultiFDSendParams *p) +{ + p->iov[0].iov_len =3D p->packet_len; + p->iov[0].iov_base =3D p->packet; + p->iovs_num++; +} + + #endif =20 diff --git a/migration/multifd.c b/migration/multifd.c index 44163e4e28..cd4467aff4 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -712,10 +712,14 @@ static void *multifd_send_thread(void *opaque) if (qatomic_read(&p->pending_job)) { MultiFDPages_t *pages =3D p->pages; =20 - if (use_zero_copy_send) { - p->iovs_num =3D 0; - } else { - p->iovs_num =3D 1; + p->iovs_num =3D 0; + + if (!use_zero_copy_send) { + /* + * Only !zerocopy needs the header in IOV; zerocopy will + * send it separately. + */ + multifd_send_prepare_header(p); } =20 assert(pages->num); @@ -735,10 +739,6 @@ static void *multifd_send_thread(void *opaque) if (ret !=3D 0) { break; } - } else { - /* Send header using the same writev call */ - p->iov[0].iov_len =3D p->packet_len; - p->iov[0].iov_base =3D p->packet; } =20 ret =3D qio_channel_writev_full_all(p->c, p->iov, p->iovs_num,= NULL, --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869890; cv=none; d=zohomail.com; s=zohoarc; b=maXDTY+x9p2Hv5JdmvOMzxVXCalLyNk6G56cwJJk+iMP2svEJTa/5nX/4uyS8W+hg50HavLUGhP1RptLxKeIu//URKyXQ07pCp4v1FtyEF3CYmCwtQWL+jBWYptQdvM3ZDYMnp/QZ0fwC3YZ3dGp5z8OXCV/2LR/R6qhdqhf4AI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869890; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=AzPCPfqn31xVp/9rnVzMd6IeZdsqsES5fCSVKyWK5rc=; b=XBoTow0y920Kl28jLvlgO4823kLlOX/OKdin9alkpjOOwDVNKlJoRp7T4d5roYkcoIoeL22bECJHbHyrWFzoyGKcbOPlSCfsImQ8X7B0Pm3VoyEu2hN0uBXTc4th5FR5Gmno1ip8HtdEQDEE2u9UK3U85mMZCCJ4AvAW1jh9Nhk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869890410936.4284903333875; Fri, 2 Feb 2024 02:31:30 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqob-0000G8-78; Fri, 02 Feb 2024 05:30:33 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqoX-00008K-VZ for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:31 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqoV-0001wS-Dt for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:29 -0500 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-38-zkwcFuleNlmKJrlAGcoYlg-1; Fri, 02 Feb 2024 05:30:25 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C12BA106D100; Fri, 2 Feb 2024 10:30:24 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id EF88340C9444; Fri, 2 Feb 2024 10:30:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869826; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AzPCPfqn31xVp/9rnVzMd6IeZdsqsES5fCSVKyWK5rc=; b=A8paeTIn/aUdFGnVC4PO8A1sQj7wspKwSblVTNxnW/E8axvy0tyhyEcJiJpKeDKpE9GM4/ ixFa7l7ILmw9mGAdzRSWdZYdWXRfCU0i/nxMQGnklSmK3IDSJBoFrTYK6OYmWFYlxt+yhI AkMx6bJs2b0qoskVjcnLa7KogqKFtQc= X-MC-Unique: zkwcFuleNlmKJrlAGcoYlg-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 13/23] migration/multifd: Move header prepare/fill into send_prepare() Date: Fri, 2 Feb 2024 18:28:47 +0800 Message-ID: <20240202102857.110210-14-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869891295100005 Content-Type: text/plain; charset="utf-8" From: Peter Xu This patch redefines the interfacing of ->send_prepare(). It further simplifies multifd_send_thread() especially on zero copy. Now with the new interface, we require the hook to do all the work for preparing the IOVs to send. After it's completed, the IOVs should be ready to be dumped into the specific multifd QIOChannel later. So now the API looks like: p->pages -----------> send_prepare() -------------> IOVs This also prepares for the case where the input can be extended to even not any p->pages. But that's for later. This patch will achieve similar goal of what Fabiano used to propose here: https://lore.kernel.org/r/20240126221943.26628-1-farosas@suse.de However the send() interface may not be necessary. I'm boldly attaching a "Co-developed-by" for Fabiano. Co-developed-by: Fabiano Rosas Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/multifd.h | 1 + migration/multifd-zlib.c | 4 +++ migration/multifd-zstd.c | 4 +++ migration/multifd.c | 61 ++++++++++++++++++---------------------- 4 files changed, 37 insertions(+), 33 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index 4ec005f53f..34a2ecb9f4 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -208,6 +208,7 @@ typedef struct { } MultiFDMethods; =20 void multifd_register_ops(int method, MultiFDMethods *ops); +void multifd_send_fill_packet(MultiFDSendParams *p); =20 static inline void multifd_send_prepare_header(MultiFDSendParams *p) { diff --git a/migration/multifd-zlib.c b/migration/multifd-zlib.c index 100809abc1..012e3bdea1 100644 --- a/migration/multifd-zlib.c +++ b/migration/multifd-zlib.c @@ -123,6 +123,8 @@ static int zlib_send_prepare(MultiFDSendParams *p, Erro= r **errp) int ret; uint32_t i; =20 + multifd_send_prepare_header(p); + for (i =3D 0; i < pages->num; i++) { uint32_t available =3D z->zbuff_len - out_size; int flush =3D Z_NO_FLUSH; @@ -172,6 +174,8 @@ static int zlib_send_prepare(MultiFDSendParams *p, Erro= r **errp) p->next_packet_size =3D out_size; p->flags |=3D MULTIFD_FLAG_ZLIB; =20 + multifd_send_fill_packet(p); + return 0; } =20 diff --git a/migration/multifd-zstd.c b/migration/multifd-zstd.c index 2023edd8cc..dc8fe43e94 100644 --- a/migration/multifd-zstd.c +++ b/migration/multifd-zstd.c @@ -118,6 +118,8 @@ static int zstd_send_prepare(MultiFDSendParams *p, Erro= r **errp) int ret; uint32_t i; =20 + multifd_send_prepare_header(p); + z->out.dst =3D z->zbuff; z->out.size =3D z->zbuff_len; z->out.pos =3D 0; @@ -161,6 +163,8 @@ static int zstd_send_prepare(MultiFDSendParams *p, Erro= r **errp) p->next_packet_size =3D z->out.pos; p->flags |=3D MULTIFD_FLAG_ZSTD; =20 + multifd_send_fill_packet(p); + return 0; } =20 diff --git a/migration/multifd.c b/migration/multifd.c index cd4467aff4..6aa44340de 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -50,15 +50,15 @@ typedef struct { /** * nocomp_send_setup: setup send side * - * For no compression this function does nothing. - * - * Returns 0 for success or -1 for error - * * @p: Params for the channel that we are using * @errp: pointer to an error */ static int nocomp_send_setup(MultiFDSendParams *p, Error **errp) { + if (migrate_zero_copy_send()) { + p->write_flags |=3D QIO_CHANNEL_WRITE_FLAG_ZERO_COPY; + } + return 0; } =20 @@ -88,7 +88,17 @@ static void nocomp_send_cleanup(MultiFDSendParams *p, Er= ror **errp) */ static int nocomp_send_prepare(MultiFDSendParams *p, Error **errp) { + bool use_zero_copy_send =3D migrate_zero_copy_send(); MultiFDPages_t *pages =3D p->pages; + int ret; + + if (!use_zero_copy_send) { + /* + * Only !zerocopy needs the header in IOV; zerocopy will + * send it separately. + */ + multifd_send_prepare_header(p); + } =20 for (int i =3D 0; i < pages->num; i++) { p->iov[p->iovs_num].iov_base =3D pages->block->host + pages->offse= t[i]; @@ -98,6 +108,18 @@ static int nocomp_send_prepare(MultiFDSendParams *p, Er= ror **errp) =20 p->next_packet_size =3D pages->num * p->page_size; p->flags |=3D MULTIFD_FLAG_NOCOMP; + + multifd_send_fill_packet(p); + + if (use_zero_copy_send) { + /* Send header first, without zerocopy */ + ret =3D qio_channel_write_all(p->c, (void *)p->packet, + p->packet_len, errp); + if (ret !=3D 0) { + return -1; + } + } + return 0; } =20 @@ -266,7 +288,7 @@ static void multifd_pages_clear(MultiFDPages_t *pages) g_free(pages); } =20 -static void multifd_send_fill_packet(MultiFDSendParams *p) +void multifd_send_fill_packet(MultiFDSendParams *p) { MultiFDPacket_t *packet =3D p->packet; MultiFDPages_t *pages =3D p->pages; @@ -688,7 +710,6 @@ static void *multifd_send_thread(void *opaque) MigrationThread *thread =3D NULL; Error *local_err =3D NULL; int ret =3D 0; - bool use_zero_copy_send =3D migrate_zero_copy_send(); =20 thread =3D migration_threads_add(p->name, qemu_get_thread_id()); =20 @@ -713,15 +734,6 @@ static void *multifd_send_thread(void *opaque) MultiFDPages_t *pages =3D p->pages; =20 p->iovs_num =3D 0; - - if (!use_zero_copy_send) { - /* - * Only !zerocopy needs the header in IOV; zerocopy will - * send it separately. - */ - multifd_send_prepare_header(p); - } - assert(pages->num); =20 ret =3D multifd_send_state->ops->send_prepare(p, &local_err); @@ -730,17 +742,6 @@ static void *multifd_send_thread(void *opaque) break; } =20 - multifd_send_fill_packet(p); - - if (use_zero_copy_send) { - /* Send header first, without zerocopy */ - ret =3D qio_channel_write_all(p->c, (void *)p->packet, - p->packet_len, &local_err); - if (ret !=3D 0) { - break; - } - } - ret =3D qio_channel_writev_full_all(p->c, p->iov, p->iovs_num,= NULL, 0, p->write_flags, &local_er= r); if (ret !=3D 0) { @@ -945,13 +946,7 @@ int multifd_save_setup(Error **errp) p->iov =3D g_new0(struct iovec, page_count + 1); p->page_size =3D qemu_target_page_size(); p->page_count =3D page_count; - - if (migrate_zero_copy_send()) { - p->write_flags =3D QIO_CHANNEL_WRITE_FLAG_ZERO_COPY; - } else { - p->write_flags =3D 0; - } - + p->write_flags =3D 0; multifd_new_send_channel_create(p); } =20 --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869979; cv=none; d=zohomail.com; s=zohoarc; b=iaEBDnhnUCt9xc1vX+taBabpDBqhAMJvcACVFS+Az6mGTRS30iZDHfAkn1AaqFAnWBu5Tn6wXWFucCCshVH6YBl/oiLF2H90K2kmxDLyMGKMFo3zSSH6qiFfsq2feMR4JUtZY4D/5jLeSKuOy/Uw6TjSTRslmKFyDKbFidokqyo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869979; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=FFlOKHkHp+NTREGXzhlqyouK8g51RQcm5L4VHQC+QI0=; b=Ws2JW4XKkD7xBhj6LxghfB9wst4mkkE1/TRta8F7a4i/PbK4esWFjG5GI6bexjrTdGRApiznwguJFkn72aU4mXlx0Ojq5hoeVSynnSMKRD90f5VD6z541Ee08HYD7LnOH9yHw+1tvLmtHnNrXM3aRgnfOsC0dEnKTypKvrxbDr8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869979577490.9353744060728; Fri, 2 Feb 2024 02:32:59 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqon-0000fS-MZ; Fri, 02 Feb 2024 05:30:46 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqoj-0000Wb-Sw for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:42 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqoc-0001yc-2H for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:35 -0500 Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-90-1VoMIqu7Mw-w5OiBJ9XMmg-1; Fri, 02 Feb 2024 05:30:30 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AB8701C04B6F; Fri, 2 Feb 2024 10:30:29 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 788C0400DF3D; Fri, 2 Feb 2024 10:30:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869833; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FFlOKHkHp+NTREGXzhlqyouK8g51RQcm5L4VHQC+QI0=; b=PM/kVmWsgiUuTGRpmc6BBXUPp1oAFum7FKJGv26cXpEuI2Fp9ykTkoRQnwGhzKGCrQ5yy8 xrvhzDKfCXfsZ/IFH/XjlJPkVJGhBUlMXVoGK4OBZjWzG89is9CbAGYoOgI9Z7TU5yerXm aETpJrSVcrx2ODLSk4yzYqPInkU6k+4= X-MC-Unique: 1VoMIqu7Mw-w5OiBJ9XMmg-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 14/23] migration/multifd: Forbid spurious wakeups Date: Fri, 2 Feb 2024 18:28:48 +0800 Message-ID: <20240202102857.110210-15-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869981568100003 Content-Type: text/plain; charset="utf-8" From: Peter Xu Now multifd's logic is designed to have no spurious wakeup. I still remember a talk to Juan and he seems to agree we should drop it now, and if my memory was right it was there because multifd used to hit that when still debugging. Let's drop it and see what can explode; as long as it's not reaching soft-freeze. Reviewed-by: Fabiano Rosas Signed-off-by: Peter Xu --- migration/multifd.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 6aa44340de..28b54100cd 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -756,7 +756,9 @@ static void *multifd_send_thread(void *opaque) p->next_packet_size =3D 0; qatomic_set(&p->pending_job, false); qemu_mutex_unlock(&p->mutex); - } else if (qatomic_read(&p->pending_sync)) { + } else { + /* If not a normal job, must be a sync request */ + assert(qatomic_read(&p->pending_sync)); p->flags =3D MULTIFD_FLAG_SYNC; multifd_send_fill_packet(p); ret =3D qio_channel_write_all(p->c, (void *)p->packet, @@ -771,9 +773,6 @@ static void *multifd_send_thread(void *opaque) qatomic_set(&p->pending_sync, false); qemu_mutex_unlock(&p->mutex); qemu_sem_post(&p->sem_sync); - } else { - qemu_mutex_unlock(&p->mutex); - /* sometimes there are spurious wakeups */ } } =20 --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869983; cv=none; d=zohomail.com; s=zohoarc; b=QDyhGbTp1G9drOmrALM+Ym760mur0kOAVV/9JwkBM1SXF1irbyR9sZq3CU5A31FUy6RL1UMx8kums01prc/xTq08SbYY79BCgd/nCX9w5SI9Go6bSka5XQ5P4G5CluzYs/JALRMhuvgUzrnu6x4xts6ibO+l2/XnkB5jORPHaT4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869983; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=vhyC/Vnj65CB+A9+jO5hln5Pb24CwPl0CcZBXzEpdsM=; b=HPwVESNhsKVlWPaHKLhuQPsfHoy3F9WBGkgk/FncJVUXL0pIJfUCdfxOPpAV6yhQTOa7UPCCt2heIT8iSoia7fWtwn5+XM+7hsxpK427Ye6m73ejvuiKoaJEWnUx8Bb06mUzDQ04EuD6F9xb4Pn62oCdwTpYwyahOM5FMFMeN4I= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869983319320.16191366314763; Fri, 2 Feb 2024 02:33:03 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqp3-0001GV-9m; Fri, 02 Feb 2024 05:31:01 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqon-0000jT-Tr for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:47 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqoj-0001z6-TF for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:44 -0500 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-204-oSlDp8A-Oaie4-Xx_vHohw-1; Fri, 02 Feb 2024 05:30:35 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 97194868900; Fri, 2 Feb 2024 10:30:34 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7B73D40C9444; Fri, 2 Feb 2024 10:30:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869840; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vhyC/Vnj65CB+A9+jO5hln5Pb24CwPl0CcZBXzEpdsM=; b=hoAYwpcO7/7+HCDV/fJaih0HOc6UNTJoY8+0dlF15LJpkQ6GdDfOxebqoPpA3IHnj0Bow8 MkIZQLHoVVZ1CTVFVMv5f3mgvOm9wBd5RPD0XNY1McOhFlsVGzHhaDhoEg4N3+7FJ1NiaY 1DULHUZ9ZNq4UvJ7wyF9T+s9QX1fYuA= X-MC-Unique: oSlDp8A-Oaie4-Xx_vHohw-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 15/23] migration/multifd: Split multifd_send_terminate_threads() Date: Fri, 2 Feb 2024 18:28:49 +0800 Message-ID: <20240202102857.110210-16-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869983600100005 Content-Type: text/plain; charset="utf-8" From: Peter Xu Split multifd_send_terminate_threads() into two functions: - multifd_send_set_error(): used when an error happened on the sender side, set error and quit state only - multifd_send_terminate_threads(): used only by the main thread to kick all multifd send threads out of sleep, for the last recycling. Use multifd_send_set_error() in the three old call sites where only the error will be set. Use multifd_send_terminate_threads() in the last one where the main thread will kick the multifd threads at last in multifd_save_cleanup(). Both helpers will need to set quitting=3D1. Suggested-by: Fabiano Rosas Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/multifd.c | 27 ++++++++++++++++++--------- migration/trace-events | 2 +- 2 files changed, 19 insertions(+), 10 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 28b54100cd..ba86f9dda5 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -536,10 +536,9 @@ int multifd_queue_page(RAMBlock *block, ram_addr_t off= set) return 1; } =20 -static void multifd_send_terminate_threads(Error *err) +/* Multifd send side hit an error; remember it and prepare to quit */ +static void multifd_send_set_error(Error *err) { - int i; - /* * We don't want to exit each threads twice. Depending on where * we get the error, or if there are two independent errors in two @@ -550,8 +549,6 @@ static void multifd_send_terminate_threads(Error *err) return; } =20 - trace_multifd_send_terminate_threads(err !=3D NULL); - if (err) { MigrationState *s =3D migrate_get_current(); migrate_set_error(s, err); @@ -563,7 +560,19 @@ static void multifd_send_terminate_threads(Error *err) MIGRATION_STATUS_FAILED); } } +} + +static void multifd_send_terminate_threads(void) +{ + int i; + + trace_multifd_send_terminate_threads(); =20 + /* + * Tell everyone we're quitting. No xchg() needed here; we simply + * always set it. + */ + qatomic_set(&multifd_send_state->exiting, 1); for (i =3D 0; i < migrate_multifd_channels(); i++) { MultiFDSendParams *p =3D &multifd_send_state->params[i]; =20 @@ -586,7 +595,7 @@ void multifd_save_cleanup(void) if (!migrate_multifd()) { return; } - multifd_send_terminate_threads(NULL); + multifd_send_terminate_threads(); for (i =3D 0; i < migrate_multifd_channels(); i++) { MultiFDSendParams *p =3D &multifd_send_state->params[i]; =20 @@ -780,7 +789,7 @@ out: if (ret) { assert(local_err); trace_multifd_send_error(p->id); - multifd_send_terminate_threads(local_err); + multifd_send_set_error(local_err); multifd_send_kick_main(p); error_free(local_err); } @@ -816,7 +825,7 @@ static void multifd_tls_outgoing_handshake(QIOTask *tas= k, =20 trace_multifd_tls_outgoing_handshake_error(ioc, error_get_pretty(err)); =20 - multifd_send_terminate_threads(err); + multifd_send_set_error(err); multifd_send_kick_main(p); error_free(err); } @@ -898,7 +907,7 @@ static void multifd_new_send_channel_async(QIOTask *tas= k, gpointer opaque) } =20 trace_multifd_new_send_channel_async_error(p->id, local_err); - multifd_send_terminate_threads(local_err); + multifd_send_set_error(local_err); multifd_send_kick_main(p); object_unref(OBJECT(ioc)); error_free(local_err); diff --git a/migration/trace-events b/migration/trace-events index de4a743c8a..298ad2b0dd 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -141,7 +141,7 @@ multifd_send_error(uint8_t id) "channel %u" multifd_send_sync_main(long packet_num) "packet num %ld" multifd_send_sync_main_signal(uint8_t id) "channel %u" multifd_send_sync_main_wait(uint8_t id) "channel %u" -multifd_send_terminate_threads(bool error) "error %d" +multifd_send_terminate_threads(void) "" multifd_send_thread_end(uint8_t id, uint64_t packets, uint64_t normal_page= s) "channel %u packets %" PRIu64 " normal pages %" PRIu64 multifd_send_thread_start(uint8_t id) "%u" multifd_tls_outgoing_handshake_start(void *ioc, void *tioc, const char *ho= stname) "ioc=3D%p tioc=3D%p hostname=3D%s" --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869919; cv=none; d=zohomail.com; s=zohoarc; b=enGCBq6HPXsn+nen+L8EFQMHVZabgF50Coc/4g705hBjGS8aukKKXVCNSc+hU5V6p0y8dYJZ5ZcGe0flGaC3D+TPMCDIXkDcwfV2X5r6y5ZZRCg/hi0gW4UE8sYra3R3eD1apeLBHHysG246Zn9k38nNoIf8gltsQke35f8ZFes= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869919; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=UrynophngknbN8sfzbL7BkEEPncDlcsjMk31G2ITrG4=; b=FaZxogMxYgSR/mvGU1PZPQlZqIOaZwdiWraXqas1YffrVKgVKzj9P5wjxYuL0t+/iawfEluoCBGQDDOlnwMpe+xIPaXApVqxonqZuCIbWSgC6pPcUERhBxvBzuUUsOkYliPcepYo5mzAQy87WCifcM+EwgQLpsboIw4VYkoG5pI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869919787647.6666086771232; Fri, 2 Feb 2024 02:31:59 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqp1-00018H-4W; Fri, 02 Feb 2024 05:30:59 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqon-0000jS-Sx for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:47 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqok-0001zO-SL for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:44 -0500 Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-695-YQgdbIH8PSGTlmAolE7BAg-1; Fri, 02 Feb 2024 05:30:40 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 85421280BC8B; Fri, 2 Feb 2024 10:30:39 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6719240C9444; Fri, 2 Feb 2024 10:30:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869842; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UrynophngknbN8sfzbL7BkEEPncDlcsjMk31G2ITrG4=; b=XIA9YSO3QX+qeClul8m3wCnG3K4L8Ub+z/4vfq15TKVyCP3jlTKUK3gw7TxcZhm0XqUm7o rAOdREyo9lvTewBdSE/2K4L11tJvyFCLRt69lAFlT0hHmy3YCxMwjxWSehfbBQH1itCxQt EiO0DvqocbVwaNb+x8Wn64afyp6TTHg= X-MC-Unique: YQgdbIH8PSGTlmAolE7BAg-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 16/23] migration/multifd: Change retval of multifd_queue_page() Date: Fri, 2 Feb 2024 18:28:50 +0800 Message-ID: <20240202102857.110210-17-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869921368100003 Content-Type: text/plain; charset="utf-8" From: Peter Xu Using int is an overkill when there're only two options. Change it to a boolean. Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/multifd.h | 2 +- migration/multifd.c | 9 +++++---- migration/ram.c | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index 34a2ecb9f4..a320c53a6f 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -22,7 +22,7 @@ bool multifd_recv_all_channels_created(void); void multifd_recv_new_channel(QIOChannel *ioc, Error **errp); void multifd_recv_sync_main(void); int multifd_send_sync_main(void); -int multifd_queue_page(RAMBlock *block, ram_addr_t offset); +bool multifd_queue_page(RAMBlock *block, ram_addr_t offset); =20 /* Multifd Compression flags */ #define MULTIFD_FLAG_SYNC (1 << 0) diff --git a/migration/multifd.c b/migration/multifd.c index ba86f9dda5..12e587fda8 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -505,7 +505,8 @@ static int multifd_send_pages(void) return 1; } =20 -int multifd_queue_page(RAMBlock *block, ram_addr_t offset) +/* Returns true if enqueue successful, false otherwise */ +bool multifd_queue_page(RAMBlock *block, ram_addr_t offset) { MultiFDPages_t *pages =3D multifd_send_state->pages; bool changed =3D false; @@ -519,21 +520,21 @@ int multifd_queue_page(RAMBlock *block, ram_addr_t of= fset) pages->num++; =20 if (pages->num < pages->allocated) { - return 1; + return true; } } else { changed =3D true; } =20 if (multifd_send_pages() < 0) { - return -1; + return false; } =20 if (changed) { return multifd_queue_page(block, offset); } =20 - return 1; + return true; } =20 /* Multifd send side hit an error; remember it and prepare to quit */ diff --git a/migration/ram.c b/migration/ram.c index d5b7cd5ac2..4649a81204 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -1252,7 +1252,7 @@ static int ram_save_page(RAMState *rs, PageSearchStat= us *pss) =20 static int ram_save_multifd_page(RAMBlock *block, ram_addr_t offset) { - if (multifd_queue_page(block, offset) < 0) { + if (!multifd_queue_page(block, offset)) { return -1; } stat64_add(&mig_stats.normal_pages, 1); --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869939; cv=none; d=zohomail.com; s=zohoarc; b=bIwRsKvkpsx45de82dzLa/ZATyGIgMaO3tV31rW/gShekjfybkKl8kJARfZ2I98ey8sBRBZt8H9cVo4w4WNBCxaSwdMYnfknb4Fq16snnnq4YKN+V7ziwAlEQ1/TApAgytnFt4hrYzE7LoDcfozHyrA+fWeK6fgEnx/c+puYvh8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869939; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=o85LQo0ULDTqdfBYOJEh8DjyVCHlsVN4yMo2YOyBmOU=; b=QPFqZn/xiVYCfoesKNhUtBit4lUKxgp/r96yuTLkEvFi6uexV9vAOVarXDRcmjev9QvD6RPFWgKmv/YifL4ERvlqPvIXH6DKKMIxqla/yJaFOTVGBKd1TEap73YWaOASvt2j3Y97oWyTI7cQ5n/ljIcrHPOriKWhCOkjsa7WpHs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869939372761.5377831205412; Fri, 2 Feb 2024 02:32:19 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqp4-0001Oc-6j; Fri, 02 Feb 2024 05:31:02 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqot-0000yo-Ad for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:51 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqor-00020G-PW for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:51 -0500 Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-460-Rb7XUWABNG6HB7ssHuXsmw-1; Fri, 02 Feb 2024 05:30:44 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6DD7D1C04B7B; Fri, 2 Feb 2024 10:30:44 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 52D6240CD14B; Fri, 2 Feb 2024 10:30:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869849; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=o85LQo0ULDTqdfBYOJEh8DjyVCHlsVN4yMo2YOyBmOU=; b=ZidFxorZKsnxCFkGBNIHwyLFipMY0QldLyhouPB13hvALG3q5BDPLgtpWwRlR9ATvrDMav ZQLDJTdQgLtGJklf6lxRW7EKsU7kqcq5YvGTOCWL4Hl/LWz8uVqyS6Q4ExxOhcUEUiSl4B N9Hmsc5RAArPXuZu+kE4l3OkFXUhq38= X-MC-Unique: Rb7XUWABNG6HB7ssHuXsmw-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 17/23] migration/multifd: Change retval of multifd_send_pages() Date: Fri, 2 Feb 2024 18:28:51 +0800 Message-ID: <20240202102857.110210-18-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869941469100003 Content-Type: text/plain; charset="utf-8" From: Peter Xu Using int is an overkill when there're only two options. Change it to a boolean. Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/multifd.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 12e587fda8..35d4e8ad1f 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -449,9 +449,10 @@ static void multifd_send_kick_main(MultiFDSendParams *= p) * thread is using the channel mutex when changing it, and the channel * have to had finish with its own, otherwise pending_job can't be * false. + * + * Returns true if succeed, false otherwise. */ - -static int multifd_send_pages(void) +static bool multifd_send_pages(void) { int i; static int next_channel; @@ -459,7 +460,7 @@ static int multifd_send_pages(void) MultiFDPages_t *pages =3D multifd_send_state->pages; =20 if (multifd_send_should_exit()) { - return -1; + return false; } =20 /* We wait here, until at least one channel is ready */ @@ -473,7 +474,7 @@ static int multifd_send_pages(void) next_channel %=3D migrate_multifd_channels(); for (i =3D next_channel;; i =3D (i + 1) % migrate_multifd_channels()) { if (multifd_send_should_exit()) { - return -1; + return false; } p =3D &multifd_send_state->params[i]; /* @@ -502,7 +503,7 @@ static int multifd_send_pages(void) qemu_mutex_unlock(&p->mutex); qemu_sem_post(&p->sem); =20 - return 1; + return true; } =20 /* Returns true if enqueue successful, false otherwise */ @@ -526,7 +527,7 @@ bool multifd_queue_page(RAMBlock *block, ram_addr_t off= set) changed =3D true; } =20 - if (multifd_send_pages() < 0) { + if (!multifd_send_pages()) { return false; } =20 @@ -666,7 +667,7 @@ int multifd_send_sync_main(void) return 0; } if (multifd_send_state->pages->num) { - if (multifd_send_pages() < 0) { + if (!multifd_send_pages()) { error_report("%s: multifd_send_pages fail", __func__); return -1; } --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869924; cv=none; d=zohomail.com; s=zohoarc; b=n+Hwfhswt+XohG20BRwR3DIP29pK5EDzZ575IFJabak5YyYzQ0tFfwcAmjpFmPi8R0VDtlaWWiysd/3ugPJZfkGg+KnKFCHpOBU0Drz43Hqfv1t4AdfqCA4P2wDujG9K3Sv5oiSoOfQEIxJWivOWs3ntJlQFiT4qtmc7ApISrjw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869924; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=QCMu/h0Uagcfb4ObsHyaNRe66JC3h0/VZ/U7Orz2xOU=; b=OM+MH3pbDpaG7iPdEXmgdxV9NJs1Ulp/VHhXQ8HLGx12e13UexWj2BkLOt99Ri9JDZMoWvTeIDXeDKQqkm2yLczGQLYoyXBPFU/KSrPpJWx3eF8Pto+8FRT8leSJip0K9nk/+qLC7PWrywaIge8+JLkwjbqYYvwVLRMes2M15aY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869924624163.42133516948775; Fri, 2 Feb 2024 02:32:04 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqp5-0001bF-60; Fri, 02 Feb 2024 05:31:03 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqoy-00018j-7r for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:58 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqov-00020m-Ni for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:55 -0500 Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-260-hZP7jx5fNCCqUNBLQ28FHw-1; Fri, 02 Feb 2024 05:30:49 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 58B973C0BE50; Fri, 2 Feb 2024 10:30:49 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3D5F740C9444; Fri, 2 Feb 2024 10:30:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869853; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QCMu/h0Uagcfb4ObsHyaNRe66JC3h0/VZ/U7Orz2xOU=; b=i+7kKx3ckhTOhNzCHix+P/Q8v2shZJKPNqhrif8/bGDFbM6iTz1iInWfZWGKYK21Uca0MU ySc29LaTFtim5jL65oGaRgGPT4aiyOL0AjX8xm4G2ieZWG8JM1QzSNsGFUVYiaYKw7Cm/m smIwLFYkyXSMwQZgNCyqui3okDt9IQk= X-MC-Unique: hZP7jx5fNCCqUNBLQ28FHw-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 18/23] migration/multifd: Rewrite multifd_queue_page() Date: Fri, 2 Feb 2024 18:28:52 +0800 Message-ID: <20240202102857.110210-19-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869925397100003 Content-Type: text/plain; charset="utf-8" From: Peter Xu The current multifd_queue_page() is not easy to read and follow. It is not good with a few reasons: - No helper at all to show what exactly does a condition mean; in short, readability is low. - Rely on pages->ramblock being cleared to detect an empty queue. It's slightly an overload of the ramblock pointer, per Fabiano [1], which I also agree. - Contains a self recursion, even if not necessary.. Rewrite this function. We add some comments to make it even clearer on what it does. [1] https://lore.kernel.org/r/87wmrpjzew.fsf@suse.de Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/multifd.c | 56 ++++++++++++++++++++++++++++++--------------- 1 file changed, 37 insertions(+), 19 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 35d4e8ad1f..4ab8e6eff2 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -506,35 +506,53 @@ static bool multifd_send_pages(void) return true; } =20 +static inline bool multifd_queue_empty(MultiFDPages_t *pages) +{ + return pages->num =3D=3D 0; +} + +static inline bool multifd_queue_full(MultiFDPages_t *pages) +{ + return pages->num =3D=3D pages->allocated; +} + +static inline void multifd_enqueue(MultiFDPages_t *pages, ram_addr_t offse= t) +{ + pages->offset[pages->num++] =3D offset; +} + /* Returns true if enqueue successful, false otherwise */ bool multifd_queue_page(RAMBlock *block, ram_addr_t offset) { - MultiFDPages_t *pages =3D multifd_send_state->pages; - bool changed =3D false; + MultiFDPages_t *pages; + +retry: + pages =3D multifd_send_state->pages; =20 - if (!pages->block) { + /* If the queue is empty, we can already enqueue now */ + if (multifd_queue_empty(pages)) { pages->block =3D block; + multifd_enqueue(pages, offset); + return true; } =20 - if (pages->block =3D=3D block) { - pages->offset[pages->num] =3D offset; - pages->num++; - - if (pages->num < pages->allocated) { - return true; + /* + * Not empty, meanwhile we need a flush. It can because of either: + * + * (1) The page is not on the same ramblock of previous ones, or, + * (2) The queue is full. + * + * After flush, always retry. + */ + if (pages->block !=3D block || multifd_queue_full(pages)) { + if (!multifd_send_pages()) { + return false; } - } else { - changed =3D true; - } - - if (!multifd_send_pages()) { - return false; - } - - if (changed) { - return multifd_queue_page(block, offset); + goto retry; } =20 + /* Not empty, and we still have space, do it! */ + multifd_enqueue(pages, offset); return true; } =20 --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869971; cv=none; d=zohomail.com; s=zohoarc; b=MQKcwIeiZ9RETWSbKXbwz3wSUTZno182lLq2W1UW9iwDW+jAVJ/RyGoK6QEvLQWGRZmD8PGpVkMzaC4wTPVHOCu5hv2mhEa2AOfNW/3+lUF57knxpBaekGfY3blw3CLiedTfyFXw9Q4B5I1AXW004FoM/igQTeOhmpsbDwaUTOo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869971; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=DVnRs6jVq1WYjkMyJMr1yedgY17NJRMJkVJ6DINU1rc=; b=OyGOF6b8deF0I6WUFayy7IvD3i87dOM7P4Ccb16cc66+J30+CJULGjjcQtIcgOYzj5084KSRnMv/ipaPnrZUijXmNVbCN9zq91yfEQBQGF5eREb8Xc8nnoDEQq70VHBnevbtrRQ0E5E3RZHFvZBnVWeA62pAxhp272tlx8aj8RU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869971239637.464550079958; Fri, 2 Feb 2024 02:32:51 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqp4-0001VD-Q2; Fri, 02 Feb 2024 05:31:02 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqp1-0001Cz-3V for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:59 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqoy-00021A-M6 for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:30:58 -0500 Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-528-E-G0JIS-MyW-baiX_lViog-1; Fri, 02 Feb 2024 05:30:54 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4480E3C0BE50; Fri, 2 Feb 2024 10:30:54 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 291D640C9444; Fri, 2 Feb 2024 10:30:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869856; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DVnRs6jVq1WYjkMyJMr1yedgY17NJRMJkVJ6DINU1rc=; b=bV5AhrejiM5ie6sA/AkxsXv5SvG1z7soLlzIGYTxg9wkZlXvJcKGT1+lG23Yj1bIHWnFuC UsMH8KqiMBblBqzOeLl0lWdS5aXvx0dvt7HjPv57byRRX77C2nDNMMrLUDWFCVaPQY16CN bI/BvIDxxPHFCO7jr5Hhro4yk37bNTw= X-MC-Unique: E-G0JIS-MyW-baiX_lViog-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 19/23] migration/multifd: Cleanup multifd_save_cleanup() Date: Fri, 2 Feb 2024 18:28:53 +0800 Message-ID: <20240202102857.110210-20-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869971588100001 Content-Type: text/plain; charset="utf-8" From: Peter Xu Shrink the function by moving relevant works into helpers: move the thread join()s into multifd_send_terminate_threads(), then create two more helpers to cover channel/state cleanups. Add a TODO entry for the thread terminate process because p->running is still buggy. We need to fix it at some point but not yet covered. Suggested-by: Fabiano Rosas Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/multifd.c | 91 +++++++++++++++++++++++++++++---------------- 1 file changed, 59 insertions(+), 32 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 4ab8e6eff2..4cb0d2cc17 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -593,6 +593,11 @@ static void multifd_send_terminate_threads(void) * always set it. */ qatomic_set(&multifd_send_state->exiting, 1); + + /* + * Firstly, kick all threads out; no matter whether they are just idle, + * or blocked in an IO system call. + */ for (i =3D 0; i < migrate_multifd_channels(); i++) { MultiFDSendParams *p =3D &multifd_send_state->params[i]; =20 @@ -601,6 +606,21 @@ static void multifd_send_terminate_threads(void) qio_channel_shutdown(p->c, QIO_CHANNEL_SHUTDOWN_BOTH, NULL); } } + + /* + * Finally recycle all the threads. + * + * TODO: p->running is still buggy, e.g. we can reach here without the + * corresponding multifd_new_send_channel_async() get invoked yet, + * then a new thread can even be created after this function returns. + */ + for (i =3D 0; i < migrate_multifd_channels(); i++) { + MultiFDSendParams *p =3D &multifd_send_state->params[i]; + + if (p->running) { + qemu_thread_join(&p->thread); + } + } } =20 static int multifd_send_channel_destroy(QIOChannel *send) @@ -608,6 +628,41 @@ static int multifd_send_channel_destroy(QIOChannel *se= nd) return socket_send_channel_destroy(send); } =20 +static bool multifd_send_cleanup_channel(MultiFDSendParams *p, Error **err= p) +{ + if (p->registered_yank) { + migration_ioc_unregister_yank(p->c); + } + multifd_send_channel_destroy(p->c); + p->c =3D NULL; + qemu_mutex_destroy(&p->mutex); + qemu_sem_destroy(&p->sem); + qemu_sem_destroy(&p->sem_sync); + g_free(p->name); + p->name =3D NULL; + multifd_pages_clear(p->pages); + p->pages =3D NULL; + p->packet_len =3D 0; + g_free(p->packet); + p->packet =3D NULL; + g_free(p->iov); + p->iov =3D NULL; + multifd_send_state->ops->send_cleanup(p, errp); + + return *errp =3D=3D NULL; +} + +static void multifd_send_cleanup_state(void) +{ + qemu_sem_destroy(&multifd_send_state->channels_ready); + g_free(multifd_send_state->params); + multifd_send_state->params =3D NULL; + multifd_pages_clear(multifd_send_state->pages); + multifd_send_state->pages =3D NULL; + g_free(multifd_send_state); + multifd_send_state =3D NULL; +} + void multifd_save_cleanup(void) { int i; @@ -615,48 +670,20 @@ void multifd_save_cleanup(void) if (!migrate_multifd()) { return; } + multifd_send_terminate_threads(); - for (i =3D 0; i < migrate_multifd_channels(); i++) { - MultiFDSendParams *p =3D &multifd_send_state->params[i]; =20 - if (p->running) { - qemu_thread_join(&p->thread); - } - } for (i =3D 0; i < migrate_multifd_channels(); i++) { MultiFDSendParams *p =3D &multifd_send_state->params[i]; Error *local_err =3D NULL; =20 - if (p->registered_yank) { - migration_ioc_unregister_yank(p->c); - } - multifd_send_channel_destroy(p->c); - p->c =3D NULL; - qemu_mutex_destroy(&p->mutex); - qemu_sem_destroy(&p->sem); - qemu_sem_destroy(&p->sem_sync); - g_free(p->name); - p->name =3D NULL; - multifd_pages_clear(p->pages); - p->pages =3D NULL; - p->packet_len =3D 0; - g_free(p->packet); - p->packet =3D NULL; - g_free(p->iov); - p->iov =3D NULL; - multifd_send_state->ops->send_cleanup(p, &local_err); - if (local_err) { + if (!multifd_send_cleanup_channel(p, &local_err)) { migrate_set_error(migrate_get_current(), local_err); error_free(local_err); } } - qemu_sem_destroy(&multifd_send_state->channels_ready); - g_free(multifd_send_state->params); - multifd_send_state->params =3D NULL; - multifd_pages_clear(multifd_send_state->pages); - multifd_send_state->pages =3D NULL; - g_free(multifd_send_state); - multifd_send_state =3D NULL; + + multifd_send_cleanup_state(); } =20 static int multifd_zero_copy_flush(QIOChannel *c) --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869903; cv=none; d=zohomail.com; s=zohoarc; b=V/MeDJrBuVpV8KRKN6GRw/TAnoGRwxpLG5J2ss2G4O0Ai/RisWk3i63r8x6RHIzje+Yr7wNjrdqo6zk/mqXcrr61i3wtFa4bpiNx7APir6PyNGuZY7vgOVz+IQk5GLoffkF3RJ73cvuL2oagvs7yoGED5kCuqKw3Ppvt2kCuA4k= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869903; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=w/PVv7xIjUW8mdAijekNJw3s/h7WnKlv6rg47X2gdcY=; b=ePyE0ZtmQlO+1jQiUlVC1od20lZ5I90cbQ2AQYkZEDkMn3rsyemI7F/aX17MMqPyn9IFEWWsiJ4rcRqXXLcXYtcUBiKkyObbwKOeUEoRDpLsjIZ0z6N0Og7cbKecgL/+nkoKz0G3CuKQqimEoZIp4jWvmmYsJFB4IS8y0iaJRv0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869903606640.3219462226668; Fri, 2 Feb 2024 02:31:43 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqp9-000290-C8; Fri, 02 Feb 2024 05:31:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqp7-000225-VC for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:31:05 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqp5-00021n-SN for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:31:05 -0500 Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-295-67lN0alUO9mMoM2mVyE7CQ-1; Fri, 02 Feb 2024 05:30:59 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 167271C05AAD; Fri, 2 Feb 2024 10:30:59 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id EF5CA40C95AD; Fri, 2 Feb 2024 10:30:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869862; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w/PVv7xIjUW8mdAijekNJw3s/h7WnKlv6rg47X2gdcY=; b=FYQXFJ+A/GZHmQ5Z65lj6IjepQYeV0EuAKFYETsphg9COWkS2aJ24c2TsTpjxSVueUf6WY TydCIsBIzh9FRETpni3dtFSOXOwYBc69BBwbLkCN2Nl6UfPCyovwO3igzEQCk9/o67kLyE axJszQhLIWNZOOoIh9HlvA0+L+AuPgI= X-MC-Unique: 67lN0alUO9mMoM2mVyE7CQ-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 20/23] migration/multifd: Cleanup multifd_load_cleanup() Date: Fri, 2 Feb 2024 18:28:54 +0800 Message-ID: <20240202102857.110210-21-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869905303100003 Content-Type: text/plain; charset="utf-8" From: Peter Xu Use similar logic to cleanup the recv side. Note that multifd_recv_terminate_threads() may need some similar rework like the sender side, but let's leave that for later. Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/multifd.c | 52 ++++++++++++++++++++++++++------------------- 1 file changed, 30 insertions(+), 22 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 4cb0d2cc17..e2dd2f6e04 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -1070,6 +1070,34 @@ void multifd_load_shutdown(void) } } =20 +static void multifd_recv_cleanup_channel(MultiFDRecvParams *p) +{ + migration_ioc_unregister_yank(p->c); + object_unref(OBJECT(p->c)); + p->c =3D NULL; + qemu_mutex_destroy(&p->mutex); + qemu_sem_destroy(&p->sem_sync); + g_free(p->name); + p->name =3D NULL; + p->packet_len =3D 0; + g_free(p->packet); + p->packet =3D NULL; + g_free(p->iov); + p->iov =3D NULL; + g_free(p->normal); + p->normal =3D NULL; + multifd_recv_state->ops->recv_cleanup(p); +} + +static void multifd_recv_cleanup_state(void) +{ + qemu_sem_destroy(&multifd_recv_state->sem_sync); + g_free(multifd_recv_state->params); + multifd_recv_state->params =3D NULL; + g_free(multifd_recv_state); + multifd_recv_state =3D NULL; +} + void multifd_load_cleanup(void) { int i; @@ -1092,29 +1120,9 @@ void multifd_load_cleanup(void) qemu_thread_join(&p->thread); } for (i =3D 0; i < migrate_multifd_channels(); i++) { - MultiFDRecvParams *p =3D &multifd_recv_state->params[i]; - - migration_ioc_unregister_yank(p->c); - object_unref(OBJECT(p->c)); - p->c =3D NULL; - qemu_mutex_destroy(&p->mutex); - qemu_sem_destroy(&p->sem_sync); - g_free(p->name); - p->name =3D NULL; - p->packet_len =3D 0; - g_free(p->packet); - p->packet =3D NULL; - g_free(p->iov); - p->iov =3D NULL; - g_free(p->normal); - p->normal =3D NULL; - multifd_recv_state->ops->recv_cleanup(p); + multifd_recv_cleanup_channel(&multifd_recv_state->params[i]); } - qemu_sem_destroy(&multifd_recv_state->sem_sync); - g_free(multifd_recv_state->params); - multifd_recv_state->params =3D NULL; - g_free(multifd_recv_state); - multifd_recv_state =3D NULL; + multifd_recv_cleanup_state(); } =20 void multifd_recv_sync_main(void) --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869890; cv=none; d=zohomail.com; s=zohoarc; b=R0JyTLyBIdT20q0TXLpMrkmbPXNKR2Bj5TD0VS7zC23x7u30BgW2ygx/zRMcpAzdhVYNItpAmaaQWaSzzZYzDoIW1slJ6AFtB0cRpHZwLrSgRlO2ldr3F0eF9ozjuFRL6bu04MFyzsQ12/VC/0yUJn9DVV4B5icYBPSubWEW6f8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869890; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=cO8Dgg9VZJ4FZBN//vNZtsTLsSl5UEe0JKXx5B5TnZ0=; b=hZ/l/2wkSg5X5pJrIjaGd5E8D8KeskI9yDZQ4a5w9otVjwA2gZdwCihB+tJu2/ZwMe9Fcw3emfHnm2xGj/la8TYMOWibqkLYa1nBWhbvBGOWtmdN/moJrnFZziCffiG0gh9rryg2tqC8K1ZziW4zkWxc60wTEbXWL7YmleBIHdg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 170686988992814.932342646174561; Fri, 2 Feb 2024 02:31:29 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqpC-0002FX-5A; Fri, 02 Feb 2024 05:31:10 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqpA-0002Cm-BX for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:31:08 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqp8-000226-Mf for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:31:08 -0500 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-472-BfmbtsJCN9GlXEcxyz0LzQ-1; Fri, 02 Feb 2024 05:31:04 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 01ED7185A784; Fri, 2 Feb 2024 10:31:04 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id DACD340C1231; Fri, 2 Feb 2024 10:30:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869866; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cO8Dgg9VZJ4FZBN//vNZtsTLsSl5UEe0JKXx5B5TnZ0=; b=g/TgD5bjj53y9LCmDIlAUuz4soufeOp1NJFJTTNv74uZVxPsEg+tX8xHLyL9tzj5tArAfX LvTvlfIzeGP+zO8PjzPYVkIATMX4/ElK7jb6HSJLrH4wiV+RAov1We2508NG9MeuIwlnMo EPzo/Uz5O+WMqOOuIxXwt4dd3GvkcAY= X-MC-Unique: BfmbtsJCN9GlXEcxyz0LzQ-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 21/23] migration/multifd: Stick with send/recv on function names Date: Fri, 2 Feb 2024 18:28:55 +0800 Message-ID: <20240202102857.110210-22-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869891312100006 Content-Type: text/plain; charset="utf-8" From: Peter Xu Most of the multifd code uses send/recv to represent the two sides, but some rare cases use save/load. Since send/recv is the majority, replacing the save/load use cases to use send/recv globally. Now we reach a consensus on the naming. Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/multifd.h | 10 +++++----- migration/migration.c | 12 ++++++------ migration/multifd.c | 10 +++++----- 3 files changed, 16 insertions(+), 16 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index a320c53a6f..9b40a53cb6 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -13,11 +13,11 @@ #ifndef QEMU_MIGRATION_MULTIFD_H #define QEMU_MIGRATION_MULTIFD_H =20 -int multifd_save_setup(Error **errp); -void multifd_save_cleanup(void); -int multifd_load_setup(Error **errp); -void multifd_load_cleanup(void); -void multifd_load_shutdown(void); +int multifd_send_setup(Error **errp); +void multifd_send_shutdown(void); +int multifd_recv_setup(Error **errp); +void multifd_recv_cleanup(void); +void multifd_recv_shutdown(void); bool multifd_recv_all_channels_created(void); void multifd_recv_new_channel(QIOChannel *ioc, Error **errp); void multifd_recv_sync_main(void); diff --git a/migration/migration.c b/migration/migration.c index d5f705ceef..ba99772e76 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -311,7 +311,7 @@ void migration_incoming_state_destroy(void) { struct MigrationIncomingState *mis =3D migration_incoming_get_current(= ); =20 - multifd_load_cleanup(); + multifd_recv_cleanup(); compress_threads_load_cleanup(); =20 if (mis->to_src_file) { @@ -662,7 +662,7 @@ static void process_incoming_migration_bh(void *opaque) =20 trace_vmstate_downtime_checkpoint("dst-precopy-bh-announced"); =20 - multifd_load_shutdown(); + multifd_recv_shutdown(); =20 dirty_bitmap_mig_before_vm_start(); =20 @@ -759,7 +759,7 @@ fail: MIGRATION_STATUS_FAILED); qemu_fclose(mis->from_src_file); =20 - multifd_load_cleanup(); + multifd_recv_cleanup(); compress_threads_load_cleanup(); =20 exit(EXIT_FAILURE); @@ -885,7 +885,7 @@ void migration_ioc_process_incoming(QIOChannel *ioc, Er= ror **errp) default_channel =3D !mis->from_src_file; } =20 - if (multifd_load_setup(errp) !=3D 0) { + if (multifd_recv_setup(errp) !=3D 0) { return; } =20 @@ -1331,7 +1331,7 @@ static void migrate_fd_cleanup(MigrationState *s) } bql_lock(); =20 - multifd_save_cleanup(); + multifd_send_shutdown(); qemu_mutex_lock(&s->qemu_file_lock); tmp =3D s->to_dst_file; s->to_dst_file =3D NULL; @@ -3623,7 +3623,7 @@ void migrate_fd_connect(MigrationState *s, Error *err= or_in) return; } =20 - if (multifd_save_setup(&local_err) !=3D 0) { + if (multifd_send_setup(&local_err) !=3D 0) { migrate_set_error(s, local_err); error_report_err(local_err); migrate_set_state(&s->state, MIGRATION_STATUS_SETUP, diff --git a/migration/multifd.c b/migration/multifd.c index e2dd2f6e04..130f86a1fb 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -663,7 +663,7 @@ static void multifd_send_cleanup_state(void) multifd_send_state =3D NULL; } =20 -void multifd_save_cleanup(void) +void multifd_send_shutdown(void) { int i; =20 @@ -965,7 +965,7 @@ static void multifd_new_send_channel_create(gpointer op= aque) socket_send_channel_create(multifd_new_send_channel_async, opaque); } =20 -int multifd_save_setup(Error **errp) +int multifd_send_setup(Error **errp) { int thread_count; uint32_t page_count =3D MULTIFD_PACKET_SIZE / qemu_target_page_size(); @@ -1063,7 +1063,7 @@ static void multifd_recv_terminate_threads(Error *err) } } =20 -void multifd_load_shutdown(void) +void multifd_recv_shutdown(void) { if (migrate_multifd()) { multifd_recv_terminate_threads(NULL); @@ -1098,7 +1098,7 @@ static void multifd_recv_cleanup_state(void) multifd_recv_state =3D NULL; } =20 -void multifd_load_cleanup(void) +void multifd_recv_cleanup(void) { int i; =20 @@ -1213,7 +1213,7 @@ static void *multifd_recv_thread(void *opaque) return NULL; } =20 -int multifd_load_setup(Error **errp) +int multifd_recv_setup(Error **errp) { int thread_count; uint32_t page_count =3D MULTIFD_PACKET_SIZE / qemu_target_page_size(); --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869925; cv=none; d=zohomail.com; s=zohoarc; b=SxVVl9mMTvQVx/VIlDdgeHwcCRV0TQCxGaMM+7PdD0nyxVZ0UHFn0rBtNPSqISJpcXXwKxwo9bA6f/dn2jfYrW8qIDcy0qVlxsmD0UNKIHc9uj0kmjMx8aQLEIIVfZ8KasfaIJYnpZDeRiN0SadxlFd4xaELFPBCjUKKa/E7rHY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869925; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=az32UV8VrqPke7SS/YPfcmNKt+JQAFwDg1s4v7TUxsQ=; b=MjVrcL+8FzCwNQPtPVAxb14HWKEBqtugXsYsRlniAd0HLCiY921tD/KWl1nd39Lko3Igi+7PXSi8aixfBWNwL2M8BR00TNPINFPcgu/f/oj9rjSQOmzcyuUpC2K3gCDII0vPveY6BmYdu+nDNuuMcjU17wAd+nOSngxIiLhFFq0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869925493693.169633527158; Fri, 2 Feb 2024 02:32:05 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqpR-0002ok-Gx; Fri, 02 Feb 2024 05:31:25 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqpH-0002Nz-HI for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:31:16 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqpF-00022f-1O for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:31:14 -0500 Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-662-M2If6aGKPPaOBzIoR_GTpQ-1; Fri, 02 Feb 2024 05:31:09 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C84173869154; Fri, 2 Feb 2024 10:31:08 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id C65EB40C95AD; Fri, 2 Feb 2024 10:31:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869872; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=az32UV8VrqPke7SS/YPfcmNKt+JQAFwDg1s4v7TUxsQ=; b=aewlZBTyofevG2p46v9b+L+b8KDNDS1DREiME5DlvdI3sx/anu2iHx8CmrNAAnuWlOpOoc yVO2LHMif2yCJDbrJllF7gO43dH4tLRZBYhxVm59XVDaq7zzp77WZ7LQHfDK4y6AYxDB4U e4tqALnTSB29T0cSjuWeA5iNzRwJNpU= X-MC-Unique: M2If6aGKPPaOBzIoR_GTpQ-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 22/23] migration/multifd: Fix MultiFDSendParams.packet_num race Date: Fri, 2 Feb 2024 18:28:56 +0800 Message-ID: <20240202102857.110210-23-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869927416100010 Content-Type: text/plain; charset="utf-8" From: Peter Xu As reported correctly by Fabiano [1], MultiFDSendParams.packet_num is buggy to be assigned and stored. Consider two consequent operations of: (1) queue a job into multifd send thread X, then (2) queue another sync request to the same send thread X. Then the MultiFDSendParams.packet_num will be assigned twice, and the first assignment can get lost already. To avoid that, we move the packet_num assignment from p->packet_num into where the thread will fill in the packet. Use atomic operations to protect the field, making sure there's no race. Note that atomic fetch_add() may not be good for scaling purposes, however multifd should be fine as number of threads should normally not go beyond 16 threads. Let's leave that concern for later but fix the issue first. There's also a trick on how to make it always work even on 32 bit hosts for uint64_t packet number. Switching to uintptr_t as of now to simply the case. It will cause packet number to overflow easier on 32 bit, but that shouldn't be a major concern for now as 32 bit systems is not the major audience for any performance concerns like what multifd wants to address. We also need to move multifd_send_state definition upper, so that multifd_send_fill_packet() can reference it. [1] https://lore.kernel.org/r/87o7d1jlu5.fsf@suse.de Reported-by: Fabiano Rosas Signed-off-by: Peter Xu Reported-by: Elena Ufimtseva Reviewed-by: Fabiano Rosas --- migration/multifd.h | 2 -- migration/multifd.c | 56 +++++++++++++++++++++++++++------------------ 2 files changed, 34 insertions(+), 24 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index 9b40a53cb6..98876ff94a 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -97,8 +97,6 @@ typedef struct { bool running; /* multifd flags for each packet */ uint32_t flags; - /* global number of generated multifd packets */ - uint64_t packet_num; /* * The sender thread has work to do if either of below boolean is set. * diff --git a/migration/multifd.c b/migration/multifd.c index 130f86a1fb..b317d57d61 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -45,6 +45,35 @@ typedef struct { uint64_t unused2[4]; /* Reserved for future use */ } __attribute__((packed)) MultiFDInit_t; =20 +struct { + MultiFDSendParams *params; + /* array of pages to sent */ + MultiFDPages_t *pages; + /* + * Global number of generated multifd packets. + * + * Note that we used 'uintptr_t' because it'll naturally support atomic + * operations on both 32bit / 64 bits hosts. It means on 32bit systems + * multifd will overflow the packet_num easier, but that should be + * fine. + * + * Another option is to use QEMU's Stat64 then it'll be 64 bits on all + * hosts, however so far it does not support atomic fetch_add() yet. + * Make it easy for now. + */ + uintptr_t packet_num; + /* send channels ready */ + QemuSemaphore channels_ready; + /* + * Have we already run terminate threads. There is a race when it + * happens that we got one error while we are exiting. + * We will use atomic operations. Only valid values are 0 and 1. + */ + int exiting; + /* multifd ops */ + MultiFDMethods *ops; +} *multifd_send_state; + /* Multifd without compression */ =20 /** @@ -292,13 +321,16 @@ void multifd_send_fill_packet(MultiFDSendParams *p) { MultiFDPacket_t *packet =3D p->packet; MultiFDPages_t *pages =3D p->pages; + uint64_t packet_num; int i; =20 packet->flags =3D cpu_to_be32(p->flags); packet->pages_alloc =3D cpu_to_be32(p->pages->allocated); packet->normal_pages =3D cpu_to_be32(pages->num); packet->next_packet_size =3D cpu_to_be32(p->next_packet_size); - packet->packet_num =3D cpu_to_be64(p->packet_num); + + packet_num =3D qatomic_fetch_inc(&multifd_send_state->packet_num); + packet->packet_num =3D cpu_to_be64(packet_num); =20 if (pages->block) { strncpy(packet->ramblock, pages->block->idstr, 256); @@ -314,7 +346,7 @@ void multifd_send_fill_packet(MultiFDSendParams *p) p->packets_sent++; p->total_normal_pages +=3D pages->num; =20 - trace_multifd_send(p->id, p->packet_num, pages->num, p->flags, + trace_multifd_send(p->id, packet_num, pages->num, p->flags, p->next_packet_size); } =20 @@ -398,24 +430,6 @@ static int multifd_recv_unfill_packet(MultiFDRecvParam= s *p, Error **errp) return 0; } =20 -struct { - MultiFDSendParams *params; - /* array of pages to sent */ - MultiFDPages_t *pages; - /* global number of generated multifd packets */ - uint64_t packet_num; - /* send channels ready */ - QemuSemaphore channels_ready; - /* - * Have we already run terminate threads. There is a race when it - * happens that we got one error while we are exiting. - * We will use atomic operations. Only valid values are 0 and 1. - */ - int exiting; - /* multifd ops */ - MultiFDMethods *ops; -} *multifd_send_state; - static bool multifd_send_should_exit(void) { return qatomic_read(&multifd_send_state->exiting); @@ -497,7 +511,6 @@ static bool multifd_send_pages(void) */ assert(qatomic_read(&p->pending_job) =3D=3D false); qatomic_set(&p->pending_job, true); - p->packet_num =3D multifd_send_state->packet_num++; multifd_send_state->pages =3D p->pages; p->pages =3D pages; qemu_mutex_unlock(&p->mutex); @@ -730,7 +743,6 @@ int multifd_send_sync_main(void) trace_multifd_send_sync_main_signal(p->id); =20 qemu_mutex_lock(&p->mutex); - p->packet_num =3D multifd_send_state->packet_num++; /* * We should be the only user so far, so not possible to be set by * others concurrently. --=20 2.43.0 From nobody Tue Nov 26 13:30:37 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1706869965; cv=none; d=zohomail.com; s=zohoarc; b=X8KE6vJr8u/vhH1CQIxlEl5fNsjWghne/lxrp/MW6rOKPR+y4kSsEXHesST1t3lzvHmaObe5gxSinFYrkUmsKDdx48ovqNVBl+aOOXGRTwXdlGv5AQaT3ttA9BRWpghzjnacAyBLDoVYMvgyRAbN/1EBwYSPxBGKg/0ZFCYfayU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1706869965; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=CbWhraHYG7bp+m3rSnby9URLY7zcB1qtBbP2QwE9kHI=; b=hSdrGhMcmTK/DZ+p7CnGGLHVsu1oBoFIOiv6D1qu/7+Ybz4vSHSJztL3H66jQ9+KKJkF0b7aek4S9A9tUMTiW/Xzf/hD0BTpGDdmaSGXvHV12wqDJJGthOtoL+hcfjwbMJ7wUGwlp0DqlP1kmABqu8+/HC1lEAMQ6acSePnRT1A= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1706869965752239.99631058829982; Fri, 2 Feb 2024 02:32:45 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rVqpq-0003nv-SR; Fri, 02 Feb 2024 05:31:52 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqpU-0003GI-VY for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:31:32 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rVqpJ-00022p-1C for qemu-devel@nongnu.org; Fri, 02 Feb 2024 05:31:28 -0500 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-323-kOOOh0iGMpGsbeVmteh_wQ-1; Fri, 02 Feb 2024 05:31:14 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9D151185A780; Fri, 2 Feb 2024 10:31:13 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 985A7400D783; Fri, 2 Feb 2024 10:31:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706869876; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CbWhraHYG7bp+m3rSnby9URLY7zcB1qtBbP2QwE9kHI=; b=f4BF8SghhyKtx7dPdBKxGxcAI8ojue6PRSTi7WfFZayGMjz67yhZINyqTcG1AflMciepO0 Q4a3fTSQ2Mz/b5tvRS3SUW1kXZ4GibBg9okFEjnd41fC/7sXrHBx+FWN4h252pFfIqOIaf DZcgpwOXZu4ZTq1qffzTl/I3FBLfhqk= X-MC-Unique: kOOOh0iGMpGsbeVmteh_wQ-1 From: peterx@redhat.com To: qemu-devel@nongnu.org Cc: Hao Xiang , Bryan Zhang , Fabiano Rosas , peterx@redhat.com, Avihai Horon , Yuan Liu , Prasad Pandit Subject: [PATCH v2 23/23] migration/multifd: Optimize sender side to be lockless Date: Fri, 2 Feb 2024 18:28:57 +0800 Message-ID: <20240202102857.110210-24-peterx@redhat.com> In-Reply-To: <20240202102857.110210-1-peterx@redhat.com> References: <20240202102857.110210-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.292, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1706869967607100011 Content-Type: text/plain; charset="utf-8" From: Peter Xu When reviewing my attempt to refactor send_prepare(), Fabiano suggested we try out with dropping the mutex in multifd code [1]. I thought about that before but I never tried to change the code. Now maybe it's time to give it a stab. This only optimizes the sender side. The trick here is multifd has a clear provider/consumer model, that the migration main thread publishes requests (either pending_job/pending_sync), while the multifd sender threads are consumers. Here we don't have a lot of comlicated data sharing, and the jobs can logically be submitted lockles= s. Arm the code with atomic weapons. Two things worth mentioning: - For multifd_send_pages(): we can use qatomic_load_acquire() when trying to find a free channel, but that's expensive if we attach one ACQUIRE per channel. Instead, make it atomic_read() on the pending_job flag, but merge the ACQUIRE into one single smp_mb_acquire() later. - For pending_sync: it doesn't have any extra data required since now p->flags are never touched, it should be safe to not use memory barrier. That's different from pending_sync. Provide rich comments for all the lockless operations to state how they are paired. With that, we can remove the mutex. [1] https://lore.kernel.org/r/87o7d1jlu5.fsf@suse.de Suggested-by: Fabiano Rosas Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/multifd.h | 2 -- migration/multifd.c | 51 +++++++++++++++++++++++---------------------- 2 files changed, 26 insertions(+), 27 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index 98876ff94a..78a2317263 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -91,8 +91,6 @@ typedef struct { /* syncs main thread and channels */ QemuSemaphore sem_sync; =20 - /* this mutex protects the following parameters */ - QemuMutex mutex; /* is this channel thread running */ bool running; /* multifd flags for each packet */ diff --git a/migration/multifd.c b/migration/multifd.c index b317d57d61..ef13e2e781 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -501,19 +501,19 @@ static bool multifd_send_pages(void) } } =20 - qemu_mutex_lock(&p->mutex); - assert(!p->pages->num); - assert(!p->pages->block); /* - * Double check on pending_job=3D=3Dfalse with the lock. In the futur= e if - * we can have >1 requester thread, we can replace this with a "goto - * retry", but that is for later. + * Make sure we read p->pending_job before all the rest. Pairs with + * qatomic_store_release() in multifd_send_thread(). */ - assert(qatomic_read(&p->pending_job) =3D=3D false); - qatomic_set(&p->pending_job, true); + smp_mb_acquire(); + assert(!p->pages->num); multifd_send_state->pages =3D p->pages; p->pages =3D pages; - qemu_mutex_unlock(&p->mutex); + /* + * Making sure p->pages is setup before marking pending_job=3Dtrue. Pa= irs + * with the qatomic_load_acquire() in multifd_send_thread(). + */ + qatomic_store_release(&p->pending_job, true); qemu_sem_post(&p->sem); =20 return true; @@ -648,7 +648,6 @@ static bool multifd_send_cleanup_channel(MultiFDSendPar= ams *p, Error **errp) } multifd_send_channel_destroy(p->c); p->c =3D NULL; - qemu_mutex_destroy(&p->mutex); qemu_sem_destroy(&p->sem); qemu_sem_destroy(&p->sem_sync); g_free(p->name); @@ -742,14 +741,12 @@ int multifd_send_sync_main(void) =20 trace_multifd_send_sync_main_signal(p->id); =20 - qemu_mutex_lock(&p->mutex); /* * We should be the only user so far, so not possible to be set by * others concurrently. */ assert(qatomic_read(&p->pending_sync) =3D=3D false); qatomic_set(&p->pending_sync, true); - qemu_mutex_unlock(&p->mutex); qemu_sem_post(&p->sem); } for (i =3D 0; i < migrate_multifd_channels(); i++) { @@ -796,9 +793,12 @@ static void *multifd_send_thread(void *opaque) if (multifd_send_should_exit()) { break; } - qemu_mutex_lock(&p->mutex); =20 - if (qatomic_read(&p->pending_job)) { + /* + * Read pending_job flag before p->pages. Pairs with the + * qatomic_store_release() in multifd_send_pages(). + */ + if (qatomic_load_acquire(&p->pending_job)) { MultiFDPages_t *pages =3D p->pages; =20 p->iovs_num =3D 0; @@ -806,14 +806,12 @@ static void *multifd_send_thread(void *opaque) =20 ret =3D multifd_send_state->ops->send_prepare(p, &local_err); if (ret !=3D 0) { - qemu_mutex_unlock(&p->mutex); break; } =20 ret =3D qio_channel_writev_full_all(p->c, p->iov, p->iovs_num,= NULL, 0, p->write_flags, &local_er= r); if (ret !=3D 0) { - qemu_mutex_unlock(&p->mutex); break; } =20 @@ -822,24 +820,31 @@ static void *multifd_send_thread(void *opaque) =20 multifd_pages_reset(p->pages); p->next_packet_size =3D 0; - qatomic_set(&p->pending_job, false); - qemu_mutex_unlock(&p->mutex); + + /* + * Making sure p->pages is published before saying "we're + * free". Pairs with the qatomic_load_acquire() in + * multifd_send_pages(). + */ + qatomic_store_release(&p->pending_job, false); } else { - /* If not a normal job, must be a sync request */ + /* + * If not a normal job, must be a sync request. Note that + * pending_sync is a standalone flag (unlike pending_job), so + * it doesn't require explicit memory barriers. + */ assert(qatomic_read(&p->pending_sync)); p->flags =3D MULTIFD_FLAG_SYNC; multifd_send_fill_packet(p); ret =3D qio_channel_write_all(p->c, (void *)p->packet, p->packet_len, &local_err); if (ret !=3D 0) { - qemu_mutex_unlock(&p->mutex); break; } /* p->next_packet_size will always be zero for a SYNC packet */ stat64_add(&mig_stats.multifd_bytes, p->packet_len); p->flags =3D 0; qatomic_set(&p->pending_sync, false); - qemu_mutex_unlock(&p->mutex); qemu_sem_post(&p->sem_sync); } } @@ -853,10 +858,7 @@ out: error_free(local_err); } =20 - qemu_mutex_lock(&p->mutex); p->running =3D false; - qemu_mutex_unlock(&p->mutex); - rcu_unregister_thread(); migration_threads_remove(thread); trace_multifd_send_thread_end(p->id, p->packets_sent, p->total_normal_= pages); @@ -998,7 +1000,6 @@ int multifd_send_setup(Error **errp) for (i =3D 0; i < thread_count; i++) { MultiFDSendParams *p =3D &multifd_send_state->params[i]; =20 - qemu_mutex_init(&p->mutex); qemu_sem_init(&p->sem, 0); qemu_sem_init(&p->sem_sync, 0); p->id =3D i; --=20 2.43.0