From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612338384; cv=none; d=zohomail.com; s=zohoarc; b=FzM220iHOpoOXOuGyWjTk/AC8xbZBQhJTGlDgW5Tq/0gcnsh/kfmf/kVEFOyjP0VWlQsareWK3QMfoCqivUnE9Nec7UWVGSy3hS98yq6FklpMOwindrrl3JIctuGFSUHQE9opBmJ/7i+Rl42lmOh0bjOD0XKxqYyr6w7/1dFD8w= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612338384; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=l0L8SE32fPmS/5V0LFgZdOBamzMf1CozD4kKjcI5oOI=; b=lq46a8dvVRrAc7Ofv48xb8tWzXkK/1MdVYCigp2cYfff5fDDbjb/cGgcqDeHj3C9tO0ZcWd4Pdbv13BPzLIeZoxmHqJsxj4L1Ss4IJqQmTIKMpKhW6bkI/EWESlajADwim4YGm44ftc0L6mPOX1RWQcjmOObzZVxN2768egk9ns= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1612338384768695.0724506687978; Tue, 2 Feb 2021 23:46:24 -0800 (PST) Received: from localhost ([::1]:58316 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Crn-0004gD-Ns for importer@patchew.org; Wed, 03 Feb 2021 02:46:23 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34218) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpe-0002xo-8I for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:10 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:3358) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpP-0008EN-V8 for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:08 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DVttr3XjKz163kw; Wed, 3 Feb 2021 15:42:28 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:36 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 01/18] migration/rdma: add the 'migrate_rdma_pin_all' function Date: Wed, 3 Feb 2021 16:01:34 +0800 Message-ID: <1612339311-114805-2-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.190; envelope-from=zhengchuan@huawei.com; helo=szxga04-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng Reviewed-by: Dr. David Alan Gilbert --- migration/migration.c | 9 +++++++++ migration/migration.h | 1 + 2 files changed, 10 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index 1986cb8..447dfb9 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2382,6 +2382,15 @@ bool migrate_use_events(void) return s->enabled_capabilities[MIGRATION_CAPABILITY_EVENTS]; } =20 +bool migrate_rdma_pin_all(void) +{ + MigrationState *s; + + s =3D migrate_get_current(); + + return s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL]; +} + bool migrate_use_multifd(void) { MigrationState *s; diff --git a/migration/migration.h b/migration/migration.h index d096b77..22b36f3 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -316,6 +316,7 @@ bool migrate_ignore_shared(void); bool migrate_validate_uuid(void); =20 bool migrate_auto_converge(void); +bool migrate_rdma_pin_all(void); bool migrate_use_multifd(void); bool migrate_pause_before_switchover(void); int migrate_multifd_channels(void); --=20 1.8.3.1 From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612338814; cv=none; d=zohomail.com; s=zohoarc; b=aAhdQBkLfTutwdeN4FRWSj4JFj3KCm+qTx9U2Rd8x4B1JQ2J6V99wO3GFFxM6ApQSnSSIhBWqB3MAg6FecZIrzFTG9PXq0cTmjRTPMaGbCpi7BRmkCRk55kHfabXe6Z7kD3N0edhzVs2AIewp19+4I/pq9Qe1znvcDpYpB7m44Q= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612338814; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=wpVOo30vlxGANbyKypfGG3toroWsemeoj/p8Nx4PcMg=; b=maD2M2fbx/E2qfMCjD4KE2XqPzvqk6UfTX4gDIzoSHG71BqSIfMwIIscF/KBy1YCnInzyPOePL5E4xo+tnPzPywctFdjgcYEUWCYfK4ve4KliAafA1nViaw+IIQSVAC6rj0x3dY6c59+mSa63wjHuq03bBB0qkuf9KD7oHO7MH0= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1612338814461982.2451247007555; Tue, 2 Feb 2021 23:53:34 -0800 (PST) Received: from localhost ([::1]:55672 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Cyj-0006wF-4p for importer@patchew.org; Wed, 03 Feb 2021 02:53:33 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34296) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpj-00033r-0m for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:15 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:3362) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpc-0008Lu-SW for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:14 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DVttr4jk3z163x2; Wed, 3 Feb 2021 15:42:28 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:37 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 02/18] migration/rdma: judge whether or not the RDMA is used for migration Date: Wed, 3 Feb 2021 16:01:35 +0800 Message-ID: <1612339311-114805-3-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.190; envelope-from=zhengchuan@huawei.com; helo=szxga04-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add enabled_rdma_migration into MigrationState to judge whether or not the RDMA is used for migration. Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/migration.c | 13 +++++++++++++ migration/migration.h | 6 ++++++ 2 files changed, 19 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index 447dfb9..129c81a 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -418,11 +418,13 @@ void migrate_add_address(SocketAddress *address) static void qemu_start_incoming_migration(const char *uri, Error **errp) { const char *p =3D NULL; + MigrationState *s =3D migrate_get_current(); =20 if (!yank_register_instance(MIGRATION_YANK_INSTANCE, errp)) { return; } =20 + s->enabled_rdma_migration =3D false; qapi_event_send_migration(MIGRATION_STATUS_SETUP); if (strstart(uri, "tcp:", &p) || strstart(uri, "unix:", NULL) || @@ -430,6 +432,7 @@ static void qemu_start_incoming_migration(const char *u= ri, Error **errp) socket_start_incoming_migration(p ? p : uri, errp); #ifdef CONFIG_RDMA } else if (strstart(uri, "rdma:", &p)) { + s->enabled_rdma_migration =3D true; rdma_start_incoming_migration(p, errp); #endif } else if (strstart(uri, "exec:", &p)) { @@ -1921,6 +1924,7 @@ void migrate_init(MigrationState *s) s->start_postcopy =3D false; s->postcopy_after_devices =3D false; s->migration_thread_running =3D false; + s->enabled_rdma_migration =3D false; error_free(s->error); s->error =3D NULL; s->hostname =3D NULL; @@ -2162,6 +2166,7 @@ void qmp_migrate(const char *uri, bool has_blk, bool = blk, socket_start_outgoing_migration(s, p ? p : uri, &local_err); #ifdef CONFIG_RDMA } else if (strstart(uri, "rdma:", &p)) { + s->enabled_rdma_migration =3D true; rdma_start_outgoing_migration(s, p, &local_err); #endif } else if (strstart(uri, "exec:", &p)) { @@ -2391,6 +2396,14 @@ bool migrate_rdma_pin_all(void) return s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL]; } =20 +bool migrate_use_rdma(void) +{ + MigrationState *s; + s =3D migrate_get_current(); + + return s->enabled_rdma_migration; +} + bool migrate_use_multifd(void) { MigrationState *s; diff --git a/migration/migration.h b/migration/migration.h index 22b36f3..da5681b 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -280,6 +280,11 @@ struct MigrationState { * This save hostname when out-going migration starts */ char *hostname; + + /* + * Enable RDMA migration + */ + bool enabled_rdma_migration; }; =20 void migrate_set_state(int *state, int old_state, int new_state); @@ -317,6 +322,7 @@ bool migrate_validate_uuid(void); =20 bool migrate_auto_converge(void); bool migrate_rdma_pin_all(void); +bool migrate_use_rdma(void); bool migrate_use_multifd(void); bool migrate_pause_before_switchover(void); int migrate_multifd_channels(void); --=20 1.8.3.1 From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612338521; cv=none; d=zohomail.com; s=zohoarc; b=GBiMt+yS/U0FSeyjXIjWaihSThdWhjB48pVtTRNiHtNets9G2VmWnWTAvmsHJGIKwCkBHLs+ctQL5Z09Ls9lutEpgmeELaLy5hvuK8Tz+MtKhstT9tqitEBVE/KEaOUth7nHLe1ppEF5SLYzkt7ejzTE+hzT4UjXryDv8tfulsU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612338521; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=p6sMw8g1KYuwE257fb0L9ETlFJbYT3CuX5+VftVbrr0=; b=KokqDWl6BlLA/WwmXcH1OXVD1PXVI7/X/r6Rb7MN3diw6d0kZbqt2NtdLbQd1inU8Hdnk+l4Ki/mBdMNXygCmHviWgdnQfkMLXAUFR38syqhqwlvRjYRxwFGe3G6mgNnQWx3pppQ+cqYVEsWlxjOZue/W4Iebsj2PVsfZAcIl0Y= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 161233852157068.18368148562718; Tue, 2 Feb 2021 23:48:41 -0800 (PST) Received: from localhost ([::1]:38564 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Cu0-0008FI-6f for importer@patchew.org; Wed, 03 Feb 2021 02:48:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34302) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpj-00034J-8K for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:15 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:3363) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpd-0008Lt-Hx for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:14 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DVttr4FD4z163wm; Wed, 3 Feb 2021 15:42:28 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:38 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 03/18] migration/rdma: create multifd_setup_ops for Tx/Rx thread Date: Wed, 3 Feb 2021 16:01:36 +0800 Message-ID: <1612339311-114805-4-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.190; envelope-from=zhengchuan@huawei.com; helo=szxga04-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Create multifd_setup_ops for TxRx thread, no logic change. Signed-off-by: Chuan Zheng --- migration/multifd.c | 44 +++++++++++++++++++++++++++++++++++++++----- migration/multifd.h | 7 +++++++ 2 files changed, 46 insertions(+), 5 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 1a1e589..cb1fc01 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -386,6 +386,8 @@ struct { int exiting; /* multifd ops */ MultiFDMethods *ops; + /* multifd setup ops */ + MultiFDSetup *setup_ops; } *multifd_send_state; =20 /* @@ -805,8 +807,9 @@ static bool multifd_channel_connect(MultiFDSendParams *= p, } else { /* update for tls qio channel */ p->c =3D ioc; - qemu_thread_create(&p->thread, p->name, multifd_send_thread, p, - QEMU_THREAD_JOINABLE); + qemu_thread_create(&p->thread, p->name, + multifd_send_state->setup_ops->send_thread, + p, QEMU_THREAD_JOINABLE); } return false; } @@ -854,6 +857,11 @@ cleanup: multifd_new_send_channel_cleanup(p, sioc, local_err); } =20 +static void multifd_send_channel_setup(MultiFDSendParams *p) +{ + socket_send_channel_create(multifd_new_send_channel_async, p); +} + int multifd_save_setup(Error **errp) { int thread_count; @@ -871,6 +879,7 @@ int multifd_save_setup(Error **errp) multifd_send_state->pages =3D multifd_pages_init(page_count); qemu_sem_init(&multifd_send_state->channels_ready, 0); qatomic_set(&multifd_send_state->exiting, 0); + multifd_send_state->setup_ops =3D multifd_setup_ops_init(); multifd_send_state->ops =3D multifd_ops[migrate_multifd_compression()]; =20 for (i =3D 0; i < thread_count; i++) { @@ -890,7 +899,7 @@ int multifd_save_setup(Error **errp) p->packet->version =3D cpu_to_be32(MULTIFD_VERSION); p->name =3D g_strdup_printf("multifdsend_%d", i); p->tls_hostname =3D g_strdup(s->hostname); - socket_send_channel_create(multifd_new_send_channel_async, p); + multifd_send_state->setup_ops->send_channel_setup(p); } =20 for (i =3D 0; i < thread_count; i++) { @@ -917,6 +926,8 @@ struct { uint64_t packet_num; /* multifd ops */ MultiFDMethods *ops; + /* multifd setup ops */ + MultiFDSetup *setup_ops; } *multifd_recv_state; =20 static void multifd_recv_terminate_threads(Error *err) @@ -1117,6 +1128,7 @@ int multifd_load_setup(Error **errp) multifd_recv_state->params =3D g_new0(MultiFDRecvParams, thread_count); qatomic_set(&multifd_recv_state->count, 0); qemu_sem_init(&multifd_recv_state->sem_sync, 0); + multifd_recv_state->setup_ops =3D multifd_setup_ops_init(); multifd_recv_state->ops =3D multifd_ops[migrate_multifd_compression()]; =20 for (i =3D 0; i < thread_count; i++) { @@ -1195,9 +1207,31 @@ bool multifd_recv_new_channel(QIOChannel *ioc, Error= **errp) p->num_packets =3D 1; =20 p->running =3D true; - qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p, - QEMU_THREAD_JOINABLE); + multifd_recv_state->setup_ops->recv_channel_setup(ioc, p); + qemu_thread_create(&p->thread, p->name, + multifd_recv_state->setup_ops->recv_thread, + p, QEMU_THREAD_JOINABLE); qatomic_inc(&multifd_recv_state->count); return qatomic_read(&multifd_recv_state->count) =3D=3D migrate_multifd_channels(); } + +static void multifd_recv_channel_setup(QIOChannel *ioc, MultiFDRecvParams = *p) +{ + return; +} + +static MultiFDSetup multifd_socket_ops =3D { + .send_thread =3D multifd_send_thread, + .recv_thread =3D multifd_recv_thread, + .send_channel_setup =3D multifd_send_channel_setup, + .recv_channel_setup =3D multifd_recv_channel_setup +}; + +MultiFDSetup *multifd_setup_ops_init(void) +{ + MultiFDSetup *ops; + + ops =3D &multifd_socket_ops; + return ops; +} diff --git a/migration/multifd.h b/migration/multifd.h index 8d6751f..1d2dc90 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -166,6 +166,13 @@ typedef struct { int (*recv_pages)(MultiFDRecvParams *p, uint32_t used, Error **errp); } MultiFDMethods; =20 +typedef struct { + void *(*send_thread)(void *opaque); + void *(*recv_thread)(void *opaque); + void (*send_channel_setup)(MultiFDSendParams *p); + void (*recv_channel_setup)(QIOChannel *ioc, MultiFDRecvParams *p); +} MultiFDSetup; + void multifd_register_ops(int method, MultiFDMethods *ops); =20 #endif --=20 1.8.3.1 From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612338384; cv=none; d=zohomail.com; s=zohoarc; b=K5E/flyRJluQaMEoheExxZ+B1X5v0p2XJkiPSMhlfqCv2n1yCoKQ9kj6NTJJ6MwN4nyEZ9eqyX7Kv1l24f4C26Tagem1/MSH5EnVabYZUka/U5pm9zD+TYo14w2jij5O4ASpAiqoPWP5FPXPZQ1cg71bkhdj+H7eVI01vwx6ZDE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612338384; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=AIKfiYbjRN+8u6SYGqjp7ULrULhH/cY7LLRD9Zh4iIw=; b=huZIdBklHrO7rlLtrdqXtUc67mhfg0pmBylXbG/6RgmFfXDnMeGHCpNgRsE7YZofxCs1viGWtWaY+GTbcE3bTHDdtLxgTaP/JtoL8G8AysOfRFI0ezsiiD5umAqpsWaVovAURhfTknK3o5mOiwmcca3YKSeEQNTaJWjF3q4W3ZU= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1612338384789747.6972639514084; Tue, 2 Feb 2021 23:46:24 -0800 (PST) Received: from localhost ([::1]:58264 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Crn-0004f0-EH for importer@patchew.org; Wed, 03 Feb 2021 02:46:23 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34174) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpa-0002xJ-BQ for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:07 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:3360) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpO-0008ER-Ef for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:02 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DVttr3n6xz163l7; Wed, 3 Feb 2021 15:42:28 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:38 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 04/18] migration/rdma: add multifd_setup_ops for rdma Date: Wed, 3 Feb 2021 16:01:37 +0800 Message-ID: <1612339311-114805-5-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.190; envelope-from=zhengchuan@huawei.com; helo=szxga04-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Chuan Zheng Reviewed-by: Dr. David Alan Gilbert --- migration/multifd.c | 6 +++++ migration/multifd.h | 5 ++++ migration/rdma.c | 71 +++++++++++++++++++++++++++++++++++++++++++++++++= ++++ 3 files changed, 82 insertions(+) diff --git a/migration/multifd.c b/migration/multifd.c index cb1fc01..4820702 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -1232,6 +1232,12 @@ MultiFDSetup *multifd_setup_ops_init(void) { MultiFDSetup *ops; =20 +#ifdef CONFIG_RDMA + if (migrate_use_rdma()) { + ops =3D &multifd_rdma_ops; + return ops; + } +#endif ops =3D &multifd_socket_ops; return ops; } diff --git a/migration/multifd.h b/migration/multifd.h index 1d2dc90..e3ab4b0 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -173,6 +173,11 @@ typedef struct { void (*recv_channel_setup)(QIOChannel *ioc, MultiFDRecvParams *p); } MultiFDSetup; =20 +#ifdef CONFIG_RDMA +extern MultiFDSetup multifd_rdma_ops; +#endif +MultiFDSetup *multifd_setup_ops_init(void); + void multifd_register_ops(int method, MultiFDMethods *ops); =20 #endif diff --git a/migration/rdma.c b/migration/rdma.c index 00eac34..e0ea86d 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -19,6 +19,7 @@ #include "qemu/cutils.h" #include "rdma.h" #include "migration.h" +#include "multifd.h" #include "qemu-file.h" #include "ram.h" #include "qemu-file-channel.h" @@ -4139,3 +4140,73 @@ err: g_free(rdma); g_free(rdma_return_path); } + +static void *multifd_rdma_send_thread(void *opaque) +{ + MultiFDSendParams *p =3D opaque; + + while (true) { + WITH_QEMU_LOCK_GUARD(&p->mutex) { + if (p->quit) { + break; + } + } + qemu_sem_wait(&p->sem); + } + + WITH_QEMU_LOCK_GUARD(&p->mutex) { + p->running =3D false; + } + + return NULL; +} + +static void multifd_rdma_send_channel_setup(MultiFDSendParams *p) +{ + Error *local_err =3D NULL; + + if (p->quit) { + error_setg(&local_err, "multifd: send id %d already quit", p->id); + return ; + } + p->running =3D true; + + qemu_thread_create(&p->thread, p->name, multifd_rdma_send_thread, p, + QEMU_THREAD_JOINABLE); +} + +static void *multifd_rdma_recv_thread(void *opaque) +{ + MultiFDRecvParams *p =3D opaque; + + while (true) { + WITH_QEMU_LOCK_GUARD(&p->mutex) { + if (p->quit) { + break; + } + } + qemu_sem_wait(&p->sem_sync); + } + + WITH_QEMU_LOCK_GUARD(&p->mutex) { + p->running =3D false; + } + + return NULL; +} + +static void multifd_rdma_recv_channel_setup(QIOChannel *ioc, + MultiFDRecvParams *p) +{ + QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); + + p->file =3D rioc->file; + return; +} + +MultiFDSetup multifd_rdma_ops =3D { + .send_thread =3D multifd_rdma_send_thread, + .recv_thread =3D multifd_rdma_recv_thread, + .send_channel_setup =3D multifd_rdma_send_channel_setup, + .recv_channel_setup =3D multifd_rdma_recv_channel_setup +}; --=20 1.8.3.1 From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612338672; cv=none; d=zohomail.com; s=zohoarc; b=nQmVGrcei2ehuc2PjqiSNyNVHF8q5aTjjHpLGkWgq8EX2pV5G156aDMVEYt2NG4wULUMBAv1XWY61esTHnYklAFi4ded9J9W1ocahjadAS+7QVnQ+TDu05ut1NHVE3yr8SU57OjEdoT20O4InUVgWKKh6TfNa1g6L7a53+fgcsw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612338672; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=TQ94N/J1jfU8yD1zYJcgOnKZdff+0rUwUP8xAx/aKQ0=; b=jhyHhCE4xlryC0B+ES0numOAEO04WVoL7mqy254pBF/npwVrOzCAW2VzqzCP4ggMyGj0e2dfWzNbYsRbcbpmzXyQa0jbhybbiAdWMRRfe/Vs/yk7VP22yup7N+Rx0F7dV0B7HCZCIAd+t83juSN6QWc55EStMmlHDZeH2Ssx6fI= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1612338672172413.37057820424013; Tue, 2 Feb 2021 23:51:12 -0800 (PST) Received: from localhost ([::1]:47086 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7CwR-0003MX-5Y for importer@patchew.org; Wed, 03 Feb 2021 02:51:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34220) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpe-0002xq-Ag for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:11 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:3361) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpW-0008FK-DK for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:10 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DVttr40nsz163wM; Wed, 3 Feb 2021 15:42:28 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:39 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 05/18] migration/rdma: do not need sync main for rdma Date: Wed, 3 Feb 2021 16:01:38 +0800 Message-ID: <1612339311-114805-6-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.190; envelope-from=zhengchuan@huawei.com; helo=szxga04-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/multifd.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/multifd.c b/migration/multifd.c index 4820702..5d34950 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -583,6 +583,10 @@ void multifd_send_sync_main(QEMUFile *f) if (!migrate_use_multifd()) { return; } + /* Do not need sync for rdma */ + if (migrate_use_rdma()) { + return; + } if (multifd_send_state->pages->used) { if (multifd_send_pages(f) < 0) { error_report("%s: multifd_send_pages fail", __func__); @@ -1024,6 +1028,10 @@ void multifd_recv_sync_main(void) if (!migrate_use_multifd()) { return; } + /* Do not need sync for rdma */ + if (migrate_use_rdma()) { + return; + } for (i =3D 0; i < migrate_multifd_channels(); i++) { MultiFDRecvParams *p =3D &multifd_recv_state->params[i]; =20 --=20 1.8.3.1 From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612338521; cv=none; d=zohomail.com; s=zohoarc; b=HJslWZ9pIKjhBucbMU5EK44+Fd+bcgYZ+cYYHc93dylWDIwgVeyCnaQ4Vf/rmxpFUE+sEPLxdGRetjw5QKiQ5lEYZvWsIjnBYNEr5e4/YyztKnPb/eJtNs+UKNdsQrdgb3zZpT1EPW+XIsN+azHnvpEQ20GTYeKd6DBD8/GorW4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612338521; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=dUHw2NGCg8AjKNRzvN9AgLB46LIC9VRyN+fgBrCCZ2w=; b=PQXpW9q3tpw37VEH1yrMt39W0/SLDAeERRA62NxjdU9GLCevfwTvzHOQc/niv704gUDMYBgW2O3GQ3LemNiM2InQJpC87cyE6QVOqV7lQmv+PNxGksag6yw6uyFYMEFSRSuhdFAbrigak3wSDvwYeYEpROq4BAJWumpHEz0sBhE= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1612338521923970.7141809742033; Tue, 2 Feb 2021 23:48:41 -0800 (PST) Received: from localhost ([::1]:38616 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Cu0-0008Gi-Im for importer@patchew.org; Wed, 03 Feb 2021 02:48:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34200) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpd-0002xZ-2S for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:09 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:3357) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpO-0008EL-E0 for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:06 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DVttr5Cfgz163x7; Wed, 3 Feb 2021 15:42:28 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:39 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 06/18] migration/rdma: export MultiFDSendParams/MultiFDRecvParams Date: Wed, 3 Feb 2021 16:01:39 +0800 Message-ID: <1612339311-114805-7-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.190; envelope-from=zhengchuan@huawei.com; helo=szxga04-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" MultiFDSendParams and MultiFDRecvParams is need for rdma, export it Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/multifd.c | 26 ++++++++++++++++++++++++++ migration/multifd.h | 2 ++ 2 files changed, 28 insertions(+) diff --git a/migration/multifd.c b/migration/multifd.c index 5d34950..ae0b7f0 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -390,6 +390,19 @@ struct { MultiFDSetup *setup_ops; } *multifd_send_state; =20 +int get_multifd_send_param(int id, MultiFDSendParams **param) +{ + int ret =3D 0; + + if (id < 0 || id >=3D migrate_multifd_channels()) { + ret =3D -1; + } else { + *param =3D &(multifd_send_state->params[id]); + } + + return ret; +} + /* * How we use multifd_send_state->pages and channel->pages? * @@ -934,6 +947,19 @@ struct { MultiFDSetup *setup_ops; } *multifd_recv_state; =20 +int get_multifd_recv_param(int id, MultiFDRecvParams **param) +{ + int ret =3D 0; + + if (id < 0 || id >=3D migrate_multifd_channels()) { + ret =3D -1; + } else { + *param =3D &(multifd_recv_state->params[id]); + } + + return ret; +} + static void multifd_recv_terminate_threads(Error *err) { int i; diff --git a/migration/multifd.h b/migration/multifd.h index e3ab4b0..d57756c 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -176,6 +176,8 @@ typedef struct { #ifdef CONFIG_RDMA extern MultiFDSetup multifd_rdma_ops; #endif +int get_multifd_send_param(int id, MultiFDSendParams **param); +int get_multifd_recv_param(int id, MultiFDRecvParams **param); MultiFDSetup *multifd_setup_ops_init(void); =20 void multifd_register_ops(int method, MultiFDMethods *ops); --=20 1.8.3.1 From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612338674; cv=none; d=zohomail.com; s=zohoarc; b=AT1r3Q9PwQYXQnXhTaWZPeoGxpPRN2/7Kcdu3S6GrGzM7zwWi6qdE3eIDwOqvDQYopSjn0M5uhibKCr7I0YRHuo73C/S52SHayyZwLMqu5DOxp+hZnMsTA7vNCbWX8Fmdt8K6dow8aPvNNdtZpV1US7nzT+cM8P2SsO0awRmkNk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612338674; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=BB/gwqKF5twH3kQykgD3SpOhoaWSfdn0aPS2KfHm6Mo=; b=Plq/STKubqiWUzShzoFqFtW2QnMjM1oAj6kic5IDA/jssxZM3z00M7F/zvv0fkqtAe4Ndmr5ax99ziyBdKBtTXpW6QiqE/FrMOutWdPtMvMrnBclQNsrHuqw5O3dZA6bfvi6xThUFpLxSMOLN64u0UEgkFXUsEiwBAwaHVmh0hY= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 161233867446885.68585175632666; Tue, 2 Feb 2021 23:51:14 -0800 (PST) Received: from localhost ([::1]:47352 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7CwT-0003T1-98 for importer@patchew.org; Wed, 03 Feb 2021 02:51:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34282) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpi-00032n-Ha for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:14 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:3359) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpW-0008EO-DN for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:14 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DVttr4yj7z163x5; Wed, 3 Feb 2021 15:42:28 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:40 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 07/18] migration/rdma: add rdma field into multifd send/recv param Date: Wed, 3 Feb 2021 16:01:40 +0800 Message-ID: <1612339311-114805-8-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.190; envelope-from=zhengchuan@huawei.com; helo=szxga04-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Note we do want to export any rdma struct, take void * instead. Signed-off-by: Chuan Zheng --- migration/multifd.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/multifd.h b/migration/multifd.h index d57756c..b17a2c1 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -108,6 +108,10 @@ typedef struct { QemuSemaphore sem_sync; /* used for compression methods */ void *data; + /* used for multifd rdma */ + void *rdma; + /* communication channel */ + QEMUFile *file; } MultiFDSendParams; =20 typedef struct { @@ -147,6 +151,10 @@ typedef struct { QemuSemaphore sem_sync; /* used for de-compression methods */ void *data; + /* used for multifd rdma */ + void *rdma; + /* communication channel */ + QEMUFile *file; } MultiFDRecvParams; =20 typedef struct { --=20 1.8.3.1 From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612338992; cv=none; d=zohomail.com; s=zohoarc; b=ivbM/sOrmjl/j/NJvfxAfMcONZ00msFXQ4N85Sj7SdfoV/9IGX58gu87GKmTIL1b7M8gnN/3Fg3AGnA67FVTbTm4v1JaTqJK9vXg7V0/48laQlfsvtWpNlnZUBMX/KZq0vuFxufnhBLrEfd/8uTh3SRwa4F+SYxj1gyO1HgLIWQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612338992; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=sje0swATjuDYt7JsE9pSsxYhs0CXus8E1ZQqMxvnMyY=; b=UJ7lOhn2gPeDmqYd6jYoeeVHXQBa9bkOcLSXqQmcHFJnGxrIvyaJjyecNquULu2vNeU2fV7YBqc8BUhg6igxPsbp8UtKLYIM5AfgKDFSVLl7JYtnkVIj5dDDMf485UT7TBQ9jLxbsqLNYJ9gsAxV/vtWO4+pKd88Uosf/KX0kHI= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1612338992646674.0102350451704; Tue, 2 Feb 2021 23:56:32 -0800 (PST) Received: from localhost ([::1]:34884 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7D1b-0001kn-F4 for importer@patchew.org; Wed, 03 Feb 2021 02:56:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34316) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpj-000361-RC for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:15 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:3364) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cph-0008MD-BI for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:15 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DVttr5S49z163x8; Wed, 3 Feb 2021 15:42:28 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:41 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 08/18] migration/rdma: export getQIOChannel to get QIOchannel in rdma Date: Wed, 3 Feb 2021 16:01:41 +0800 Message-ID: <1612339311-114805-9-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.190; envelope-from=zhengchuan@huawei.com; helo=szxga04-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/qemu-file.c | 5 +++++ migration/qemu-file.h | 1 + 2 files changed, 6 insertions(+) diff --git a/migration/qemu-file.c b/migration/qemu-file.c index be21518..37f6201 100644 --- a/migration/qemu-file.c +++ b/migration/qemu-file.c @@ -260,6 +260,11 @@ void ram_control_before_iterate(QEMUFile *f, uint64_t = flags) } } =20 +void *getQIOChannel(QEMUFile *f) +{ + return f->opaque; +} + void ram_control_after_iterate(QEMUFile *f, uint64_t flags) { int ret =3D 0; diff --git a/migration/qemu-file.h b/migration/qemu-file.h index a9b6d6c..4cef043 100644 --- a/migration/qemu-file.h +++ b/migration/qemu-file.h @@ -165,6 +165,7 @@ void qemu_file_set_blocking(QEMUFile *f, bool block); void ram_control_before_iterate(QEMUFile *f, uint64_t flags); void ram_control_after_iterate(QEMUFile *f, uint64_t flags); void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data); +void *getQIOChannel(QEMUFile *f); =20 /* Whenever this is found in the data stream, the flags * will be passed to ram_control_load_hook in the incoming-migration --=20 1.8.3.1 From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612339103; cv=none; d=zohomail.com; s=zohoarc; b=hCIds3k1N0sYs9a4041lsMdKgXgfHKQZzNuF9/H1DMNL2ca6bjTCX3ZTTMtVWu55VOr0klfCRqb+5q5kHu/WIPn0xlpvGuuQdi2qPtjUZcuzRtQyMCvj+Rnq5o3DFeW05dNfoDaIFyhEKLXSsRoblpJRJZ3oMnUInqkpOqWsPtw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612339103; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=l8BnxltHOANoBmslSuLazOXDzjLMmPC44KgWAW8FjMI=; b=Cgj7nnTi1rLF3p4OUV3dqoSRtjRG6C/JshFkjosShlv9TWFKveshN0GfR1ObgJf7+3QuPewReQs8pZ9mUfcEQ8gEzGIp0Cj/eTITxcVsgOHavrn2cjUp6H65F1ZRKskWH98trNyNmaKnFfgphNlMmOu13EHRR+gtyo08QtDzyUg= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1612339103913149.5596399149547; Tue, 2 Feb 2021 23:58:23 -0800 (PST) Received: from localhost ([::1]:40400 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7D3O-00045U-Ub for importer@patchew.org; Wed, 03 Feb 2021 02:58:22 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34330) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpl-0003BD-JN for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:17 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3476) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpj-0008MY-Px for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:17 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv06V3xzjJgc; Wed, 3 Feb 2021 15:42:36 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:41 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 09/18] migration/rdma: add multifd_rdma_load_setup() to setup multifd rdma Date: Wed, 3 Feb 2021 16:01:42 +0800 Message-ID: <1612339311-114805-10-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Chuan Zheng Reviewed-by: Dr. David Alan Gilbert --- migration/rdma.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/migration/rdma.c b/migration/rdma.c index e0ea86d..996afb0 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -4011,6 +4011,48 @@ static void rdma_accept_incoming_migration(void *opa= que) } } =20 +static bool multifd_rdma_load_setup(const char *host_port, + RDMAContext *rdma, Error **errp) +{ + int thread_count; + int i; + int idx; + MultiFDRecvParams *multifd_recv_param; + RDMAContext *multifd_rdma; + + if (!migrate_use_multifd()) { + return true; + } + + if (multifd_load_setup(errp) !=3D 0) { + /* + * We haven't been able to create multifd threads + * nothing better to do + */ + return false; + } + + thread_count =3D migrate_multifd_channels(); + for (i =3D 0; i < thread_count; i++) { + if (get_multifd_recv_param(i, &multifd_recv_param) < 0) { + ERROR(errp, "rdma: error getting multifd_recv_param(%d)", i); + return false; + } + + multifd_rdma =3D qemu_rdma_data_init(host_port, errp); + for (idx =3D 0; idx < RDMA_WRID_MAX; idx++) { + multifd_rdma->wr_data[idx].control_len =3D 0; + multifd_rdma->wr_data[idx].control_curr =3D NULL; + } + /* the CM channel and CM id is shared */ + multifd_rdma->channel =3D rdma->channel; + multifd_rdma->listen_id =3D rdma->listen_id; + multifd_recv_param->rdma =3D (void *)multifd_rdma; + } + + return true; +} + void rdma_start_incoming_migration(const char *host_port, Error **errp) { int ret; @@ -4058,6 +4100,16 @@ void rdma_start_incoming_migration(const char *host_= port, Error **errp) qemu_rdma_return_path_dest_init(rdma_return_path, rdma); } =20 + /* multifd rdma setup */ + if (!multifd_rdma_load_setup(host_port, rdma, &local_err)) { + /* + * We haven't been able to create multifd threads + * nothing better to do + */ + error_report_err(local_err); + goto err; + } + qemu_set_fd_handler(rdma->channel->fd, rdma_accept_incoming_migration, NULL, (void *)(intptr_t)rdma); return; --=20 1.8.3.1 From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612338814; cv=none; d=zohomail.com; s=zohoarc; b=n225RQXD/EcNYRUJrMY5vRYCLaDYNmN9cvie7mpjsdmpvZp2VRbkSjeHHelIPlY55wTaJb1syVnDJRCNy7H44xJ5K6eI/1ouCRX56nm8NVmq7ESUFQ9iBa00cweTBmlrueG/B62gr7u3Sd25WWgvJE0VRO4vey6yTq6AE5cY5Vc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612338814; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=FzhmbC5NV7+Q1YwxsoEpWlkIWVu/W6XpAVmpOj1XjNg=; b=JGthL4QsAw2wEa4GTP+py4Mb29E2o+8BHYS94qafagy0wQEDOABfl/oBwk+goBumHDbyA2d+MvxpP7go1oNWBEp+Krx5KxLDltNYwz3MOeSWQzChVaMNkJLxYTqw72zClC9uWyvmACGFhQmCS8iylg4I8Qti/ROe/8T0YN9xaA0= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1612338814317441.6566736872019; Tue, 2 Feb 2021 23:53:34 -0800 (PST) Received: from localhost ([::1]:55584 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Cyj-0006tz-5l for importer@patchew.org; Wed, 03 Feb 2021 02:53:33 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34320) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpk-00037v-EK for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:16 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3477) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpi-0008Ma-0h for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:16 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv06kvFzjJgj; Wed, 3 Feb 2021 15:42:36 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:42 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 10/18] migration/rdma: Create the multifd recv channels for RDMA Date: Wed, 3 Feb 2021 16:01:43 +0800 Message-ID: <1612339311-114805-11-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We still don't transmit anything through them, and we only build the RDMA connections. Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/rdma.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++= ++-- 1 file changed, 67 insertions(+), 2 deletions(-) diff --git a/migration/rdma.c b/migration/rdma.c index 996afb0..ed8a015 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3267,6 +3267,40 @@ static void rdma_cm_poll_handler(void *opaque) } } =20 +static bool qemu_rdma_accept_setup(RDMAContext *rdma) +{ + RDMAContext *multifd_rdma =3D NULL; + int thread_count; + int i; + MultiFDRecvParams *multifd_recv_param; + thread_count =3D migrate_multifd_channels(); + /* create the multifd channels for RDMA */ + for (i =3D 0; i < thread_count; i++) { + if (get_multifd_recv_param(i, &multifd_recv_param) < 0) { + error_report("rdma: error getting multifd_recv_param(%d)", i); + return false; + } + + multifd_rdma =3D (RDMAContext *) multifd_recv_param->rdma; + if (multifd_rdma->cm_id =3D=3D NULL) { + break; + } else { + multifd_rdma =3D NULL; + } + } + + if (multifd_rdma) { + qemu_set_fd_handler(rdma->channel->fd, + rdma_accept_incoming_migration, + NULL, (void *)(intptr_t)multifd_rdma); + } else { + qemu_set_fd_handler(rdma->channel->fd, rdma_cm_poll_handler, + NULL, rdma); + } + + return true; +} + static int qemu_rdma_accept(RDMAContext *rdma) { RDMACapabilities cap; @@ -3366,6 +3400,10 @@ static int qemu_rdma_accept(RDMAContext *rdma) qemu_set_fd_handler(rdma->channel->fd, rdma_accept_incoming_migrat= ion, NULL, (void *)(intptr_t)rdma->return_path); + } else if (migrate_use_multifd()) { + if (!qemu_rdma_accept_setup(rdma)) { + goto err_rdma_dest_wait; + } } else { qemu_set_fd_handler(rdma->channel->fd, rdma_cm_poll_handler, NULL, rdma); @@ -3976,6 +4014,34 @@ static QEMUFile *qemu_fopen_rdma(RDMAContext *rdma, = const char *mode) return rioc->file; } =20 +static void migration_rdma_process_incoming(QEMUFile *f, + RDMAContext *rdma, Error **err= p) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + QIOChannel *ioc =3D NULL; + bool start_migration =3D false; + + if (!migrate_use_multifd()) { + rdma->migration_started_on_destination =3D 1; + migration_fd_process_incoming(f, errp); + return; + } + + if (!mis->from_src_file) { + mis->from_src_file =3D f; + qemu_file_set_blocking(f, false); + } else { + ioc =3D QIO_CHANNEL(getQIOChannel(f)); + /* Multiple connections */ + assert(migrate_use_multifd()); + start_migration =3D multifd_recv_new_channel(ioc, errp); + } + + if (start_migration) { + migration_incoming_process(); + } +} + static void rdma_accept_incoming_migration(void *opaque) { RDMAContext *rdma =3D opaque; @@ -4004,8 +4070,7 @@ static void rdma_accept_incoming_migration(void *opaq= ue) return; } =20 - rdma->migration_started_on_destination =3D 1; - migration_fd_process_incoming(f, &local_err); + migration_rdma_process_incoming(f, rdma, &local_err); if (local_err) { error_reportf_err(local_err, "RDMA ERROR:"); } --=20 1.8.3.1 From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612338672; cv=none; d=zohomail.com; s=zohoarc; b=BMHuf2o/RMV9TrtOZF3ACkaPZlV8Hl2sZask0p+3SvU+yDnm+lnWK1caslkTw89eLp02jOt1Yraz+VUTCiMFO6h8rLJa1Xq+svkGETorNvpzcLMaw88o8RtCLlb2Xjvb7NozgTvAlmC4U00y/DJH31SxovsW44iUEsog8Oo90Mc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612338672; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=JktsDEi8VkaHNvpBUXArayuthGMgRj1csOIgX5KcYLk=; b=aDqMhZKqh7HbP/oaYG/tyUrMbH/5gH9xT40nsj3K3HCBzNgY1HKtGzCJq5pIWZ1LHWDw8xjvkuTxgJvPflbGGDRD+ayQEsUn4LRRKm/1I3Sv8UHpwFYvtsoUPSRQRUFkINwvNFOXR7F660ogc2XkFUNmkbi0yoY5nkUTnsr0vnw= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1612338672164524.1707683079693; Tue, 2 Feb 2021 23:51:12 -0800 (PST) Received: from localhost ([::1]:47126 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7CwR-0003NW-2e for importer@patchew.org; Wed, 03 Feb 2021 02:51:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34264) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cph-000317-Vu for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:14 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3038) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpb-0008J7-HU for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:13 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv05VLBzjJgZ; Wed, 3 Feb 2021 15:42:36 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:43 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 11/18] migration/rdma: record host_port for multifd RDMA Date: Wed, 3 Feb 2021 16:01:44 +0800 Message-ID: <1612339311-114805-12-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Chuan Zheng --- migration/migration.c | 1 + migration/migration.h | 3 +++ migration/rdma.c | 3 +++ 3 files changed, 7 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index 129c81a..b8f4844 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1925,6 +1925,7 @@ void migrate_init(MigrationState *s) s->postcopy_after_devices =3D false; s->migration_thread_running =3D false; s->enabled_rdma_migration =3D false; + s->host_port =3D NULL; error_free(s->error); s->error =3D NULL; s->hostname =3D NULL; diff --git a/migration/migration.h b/migration/migration.h index da5681b..537ee09 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -285,6 +285,9 @@ struct MigrationState { * Enable RDMA migration */ bool enabled_rdma_migration; + + /* Need by Multi-RDMA */ + char *host_port; }; =20 void migrate_set_state(int *state, int old_state, int new_state); diff --git a/migration/rdma.c b/migration/rdma.c index ed8a015..9654b87 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -4206,6 +4206,8 @@ void rdma_start_outgoing_migration(void *opaque, goto err; } =20 + s->host_port =3D g_strdup(host_port); + ret =3D qemu_rdma_source_init(rdma, s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL], errp); =20 @@ -4250,6 +4252,7 @@ void rdma_start_outgoing_migration(void *opaque, =20 s->to_dst_file =3D qemu_fopen_rdma(rdma, "wb"); migrate_fd_connect(s, NULL); + g_free(s->host_port); return; return_path_err: qemu_rdma_cleanup(rdma); --=20 1.8.3.1 From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612338386; cv=none; d=zohomail.com; s=zohoarc; b=HTAbA7R3OTOpcCfMUX3y32OGWsFQtxCMwUeYMYoJz6RWZq2gwK5fvR1c2sWm9k5wQPMVGv49P53gv7NdHc5SMWGptyZAWJ71F4DZ9VhFTgotQWHaqj0cHZcfDtPWMTfMdqd9r6HnbfyUfhLnHSjUcZfMDa3/U92fWhJJ579zh1w= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612338386; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=pozQ0zzTAsh93NwTFl3JJgR8UKcVX7QOy9WebcPal7M=; b=lMmxx9K9AOlr5Zj62OeGTLQkiffZWyTq2Z7ZsN1tkHW98ZeLMIYvjXcUj1/UXxnR9NGwU5kdauqUrIFTMZf30leTorEAI1W3J/n9k0vT5FW27SvDOW78WXyWzGeYdLniMfAmicTIj1fqorhDZbIA//JBKYZUQp5pLTBxtM9M4dk= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1612338386436931.8165947338163; Tue, 2 Feb 2021 23:46:26 -0800 (PST) Received: from localhost ([::1]:58494 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Crp-0004kS-DR for importer@patchew.org; Wed, 03 Feb 2021 02:46:25 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34256) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cph-00030Y-M2 for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:13 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3039) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpY-0008J8-4g for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:13 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv05kgfzjJgf; Wed, 3 Feb 2021 15:42:36 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:43 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 12/18] migration/rdma: Create the multifd send channels for RDMA Date: Wed, 3 Feb 2021 16:01:45 +0800 Message-ID: <1612339311-114805-13-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Chuan Zheng --- migration/multifd.c | 4 ++-- migration/multifd.h | 2 ++ migration/rdma.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++= ++++ 3 files changed, 61 insertions(+), 2 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index ae0b7f0..919a414 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -176,7 +176,7 @@ void multifd_register_ops(int method, MultiFDMethods *o= ps) multifd_ops[method] =3D ops; } =20 -static int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) +int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) { MultiFDInit_t msg =3D {}; int ret; @@ -503,7 +503,7 @@ int multifd_queue_page(QEMUFile *f, RAMBlock *block, ra= m_addr_t offset) return 1; } =20 -static void multifd_send_terminate_threads(Error *err) +void multifd_send_terminate_threads(Error *err) { int i; =20 diff --git a/migration/multifd.h b/migration/multifd.h index b17a2c1..26d4489 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -184,6 +184,8 @@ typedef struct { #ifdef CONFIG_RDMA extern MultiFDSetup multifd_rdma_ops; #endif +void multifd_send_terminate_threads(Error *err); +int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp); int get_multifd_send_param(int id, MultiFDSendParams **param); int get_multifd_recv_param(int id, MultiFDRecvParams **param); MultiFDSetup *multifd_setup_ops_init(void); diff --git a/migration/rdma.c b/migration/rdma.c index 9654b87..cff9446 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -4261,9 +4261,54 @@ err: g_free(rdma_return_path); } =20 +static int multifd_channel_rdma_connect(void *opaque) +{ + MultiFDSendParams *p =3D opaque; + Error *local_err =3D NULL; + int ret =3D 0; + MigrationState *s =3D migrate_get_current(); + + p->rdma =3D qemu_rdma_data_init(s->host_port, &local_err); + if (p->rdma =3D=3D NULL) { + goto out; + } + + ret =3D qemu_rdma_source_init(p->rdma, + migrate_rdma_pin_all(), + &local_err); + if (ret) { + goto out; + } + + ret =3D qemu_rdma_connect(p->rdma, &local_err); + if (ret) { + goto out; + } + + p->file =3D qemu_fopen_rdma(p->rdma, "wb"); + if (p->file =3D=3D NULL) { + goto out; + } + + p->c =3D QIO_CHANNEL(getQIOChannel(p->file)); + +out: + if (local_err) { + trace_multifd_send_error(p->id); + } + + return ret; +} + static void *multifd_rdma_send_thread(void *opaque) { MultiFDSendParams *p =3D opaque; + Error *local_err =3D NULL; + + trace_multifd_send_thread_start(p->id); + if (multifd_send_initial_packet(p, &local_err) < 0) { + goto out; + } =20 while (true) { WITH_QEMU_LOCK_GUARD(&p->mutex) { @@ -4274,6 +4319,12 @@ static void *multifd_rdma_send_thread(void *opaque) qemu_sem_wait(&p->sem); } =20 +out: + if (local_err) { + trace_multifd_send_error(p->id); + multifd_send_terminate_threads(local_err); + } + WITH_QEMU_LOCK_GUARD(&p->mutex) { p->running =3D false; } @@ -4285,6 +4336,12 @@ static void multifd_rdma_send_channel_setup(MultiFDS= endParams *p) { Error *local_err =3D NULL; =20 + if (multifd_channel_rdma_connect(p)) { + error_setg(&local_err, "multifd: rdma channel %d not established", + p->id); + return ; + } + if (p->quit) { error_setg(&local_err, "multifd: send id %d already quit", p->id); return ; --=20 1.8.3.1 From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612338524; cv=none; d=zohomail.com; s=zohoarc; b=O4vcQtTeBuTTRQibeDu8p94j6qEcL3Az973sWFvE79CEOwkpJ9sNcpJ2XsRxyOVG++r66uxZfb1MqUSmOcK6PkMsm5SO2A8zfDVNeSCt7Ghr50YtlV8dgrWfhRjn6W/S2RsRTzTaO916bMqDryjVSPbPaP4o5eOkJo3Jmahrrkw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612338524; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=jJGl2UTZIIOig+iJrT+rWFT7ZlPwibHqo85N3rICvGI=; b=kpFCh9H0cawP3f3H3bfsbYbnZDrLOM9xfUG8c9EhA0zjK957/+6V8ME4plvgzuziFTjFX8Yi8BFQYAOTxIDqMK6WwrBfg5whRaRiUBCxRP8Q/pKfeOyvrfBDIcPJHdojZjc18ClFBjL+5Dy1j0TNZPAu+1ortGJ+q04y00GJ9HI= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 161233852434587.4661161292355; Tue, 2 Feb 2021 23:48:44 -0800 (PST) Received: from localhost ([::1]:38854 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Cu3-0008MK-4F for importer@patchew.org; Wed, 03 Feb 2021 02:48:43 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34266) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpi-00031L-1t for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:14 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3474) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpW-0008J3-Nr for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:13 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv05zzBzjJgg; Wed, 3 Feb 2021 15:42:36 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:44 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 13/18] migration/rdma: Add the function for dynamic page registration Date: Wed, 3 Feb 2021 16:01:46 +0800 Message-ID: <1612339311-114805-14-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add the 'qemu_rdma_registration' function, multifd send threads call it to register memory. Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/rdma.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/migration/rdma.c b/migration/rdma.c index cff9446..1095a8f 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3739,6 +3739,57 @@ out: return ret; } =20 +/* + * Dynamic page registrations for multifd RDMA threads. + */ +static int qemu_rdma_registration(void *opaque) +{ + RDMAContext *rdma =3D opaque; + RDMAControlHeader resp =3D {.type =3D RDMA_CONTROL_RAM_BLOCKS_RESULT }; + RDMALocalBlocks *local =3D &rdma->local_ram_blocks; + int reg_result_idx, i, nb_dest_blocks; + RDMAControlHeader head =3D { .len =3D 0, .repeat =3D 1 }; + int ret =3D 0; + + head.type =3D RDMA_CONTROL_RAM_BLOCKS_REQUEST; + + ret =3D qemu_rdma_exchange_send(rdma, &head, NULL, &resp, + ®_result_idx, rdma->pin_all ? + qemu_rdma_reg_whole_ram_blocks : NULL); + if (ret < 0) { + goto out; + } + + nb_dest_blocks =3D resp.len / sizeof(RDMADestBlock); + + if (local->nb_blocks !=3D nb_dest_blocks) { + rdma->error_state =3D -EINVAL; + ret =3D -1; + goto out; + } + + qemu_rdma_move_header(rdma, reg_result_idx, &resp); + memcpy(rdma->dest_blocks, + rdma->wr_data[reg_result_idx].control_curr, resp.len); + + for (i =3D 0; i < nb_dest_blocks; i++) { + network_to_dest_block(&rdma->dest_blocks[i]); + + /* We require that the blocks are in the same order */ + if (rdma->dest_blocks[i].length !=3D local->block[i].length) { + rdma->error_state =3D -EINVAL; + ret =3D -1; + goto out; + } + local->block[i].remote_host_addr =3D + rdma->dest_blocks[i].remote_host_addr; + local->block[i].remote_rkey =3D rdma->dest_blocks[i].remote_rkey; + } + +out: + return ret; +} + /* Destination: * Called via a ram_control_load_hook during the initial RAM load section = which * lists the RAMBlocks by name. This lets us know the order of the RAMBlo= cks --=20 1.8.3.1 From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612338814; cv=none; d=zohomail.com; s=zohoarc; b=i4DAw9bY3eUee+OAM3o0AezPRYEl4a1lwH1c2QUJkSV55rs6iCUah10yVJLL8Vfx9AIcl2MTStR3b/Od9/up9s4e/ThXgvitRVYkNhnsVgH8cg3aAN77EcbnPE/5mJH8Cp1diuwrTudGSFiHen3v80fu1n+91TdA3op4K7nV/UA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612338814; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=0y/lfSYdVGd57ws8ite+RAW81Fmo3YN5nlOByoZbINQ=; b=VwzoSlGpRSjFAxnkm1+z1WdVw2/jCfBxb+5A+8xfIx1BJMWvi3x8S+Djr8xW82VypPw3HnI5hntfXzfqL9/lk6u24/hQT00G5NbNvgWifJSA+2OYJxqgSTJgmM2nyanrHsP192srcn09Qpd7Fdqx98BCzktjEXgji1BPHMjiSws= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1612338814193194.09559904653133; Tue, 2 Feb 2021 23:53:34 -0800 (PST) Received: from localhost ([::1]:55612 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Cyi-0006ue-US for importer@patchew.org; Wed, 03 Feb 2021 02:53:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34234) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpf-0002yJ-D1 for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:11 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3475) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpW-0008J4-DN for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:11 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv06DphzjJgh; Wed, 3 Feb 2021 15:42:36 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:44 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 14/18] migration/rdma: register memory for multifd RDMA channels Date: Wed, 3 Feb 2021 16:01:47 +0800 Message-ID: <1612339311-114805-15-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/multifd.c | 3 ++ migration/rdma.c | 92 +++++++++++++++++++++++++++++++++++++++++++++++++= ++-- 2 files changed, 93 insertions(+), 2 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 919a414..1186246 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -537,6 +537,9 @@ void multifd_send_terminate_threads(Error *err) qemu_mutex_lock(&p->mutex); p->quit =3D true; qemu_sem_post(&p->sem); + if (migrate_use_rdma()) { + qemu_sem_post(&p->sem_sync); + } qemu_mutex_unlock(&p->mutex); } } diff --git a/migration/rdma.c b/migration/rdma.c index 1095a8f..c906cc7 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3838,6 +3838,19 @@ static int rdma_load_hook(QEMUFile *f, void *opaque,= uint64_t flags, void *data) return rdma_block_notification_handle(opaque, data); =20 case RAM_CONTROL_HOOK: + if (migrate_use_multifd()) { + int i; + MultiFDRecvParams *multifd_recv_param =3D NULL; + int thread_count =3D migrate_multifd_channels(); + /* Inform dest recv_thread to poll */ + for (i =3D 0; i < thread_count; i++) { + if (get_multifd_recv_param(i, &multifd_recv_param)) { + return -1; + } + qemu_sem_post(&multifd_recv_param->sem_sync); + } + } + return qemu_rdma_registration_handle(f, opaque); =20 default: @@ -3910,6 +3923,24 @@ static int qemu_rdma_registration_stop(QEMUFile *f, = void *opaque, head.type =3D RDMA_CONTROL_RAM_BLOCKS_REQUEST; trace_qemu_rdma_registration_stop_ram(); =20 + if (migrate_use_multifd()) { + /* + * Inform the multifd channels to register memory + */ + int i; + int thread_count =3D migrate_multifd_channels(); + MultiFDSendParams *multifd_send_param =3D NULL; + for (i =3D 0; i < thread_count; i++) { + ret =3D get_multifd_send_param(i, &multifd_send_param); + if (ret) { + error_report("rdma: error getting multifd(%d)", i); + return ret; + } + + qemu_sem_post(&multifd_send_param->sem_sync); + } + } + /* * Make sure that we parallelize the pinning on both sides. * For very large guests, doing this serially takes a really @@ -3968,6 +3999,21 @@ static int qemu_rdma_registration_stop(QEMUFile *f, = void *opaque, rdma->dest_blocks[i].remote_host_addr; local->block[i].remote_rkey =3D rdma->dest_blocks[i].remote_rk= ey; } + /* Wait for all multifd channels to complete registration */ + if (migrate_use_multifd()) { + int i; + int thread_count =3D migrate_multifd_channels(); + MultiFDSendParams *multifd_send_param =3D NULL; + for (i =3D 0; i < thread_count; i++) { + ret =3D get_multifd_send_param(i, &multifd_send_param); + if (ret) { + error_report("rdma: error getting multifd(%d)", i); + return ret; + } + + qemu_sem_wait(&multifd_send_param->sem); + } + } } =20 trace_qemu_rdma_registration_stop(flags); @@ -3979,6 +4025,24 @@ static int qemu_rdma_registration_stop(QEMUFile *f, = void *opaque, goto err; } =20 + if (migrate_use_multifd()) { + /* + * Inform src send_thread to send FINISHED signal. + * Wait for multifd RDMA send threads to poll the CQE. + */ + int i; + int thread_count =3D migrate_multifd_channels(); + MultiFDSendParams *multifd_send_param =3D NULL; + for (i =3D 0; i < thread_count; i++) { + ret =3D get_multifd_send_param(i, &multifd_send_param); + if (ret < 0) { + goto err; + } + + qemu_sem_post(&multifd_send_param->sem_sync); + } + } + return 0; err: rdma->error_state =3D ret; @@ -4355,19 +4419,37 @@ static void *multifd_rdma_send_thread(void *opaque) { MultiFDSendParams *p =3D opaque; Error *local_err =3D NULL; + int ret =3D 0; + RDMAControlHeader head =3D { .len =3D 0, .repeat =3D 1 }; =20 trace_multifd_send_thread_start(p->id); if (multifd_send_initial_packet(p, &local_err) < 0) { goto out; } =20 + /* wait for semaphore notification to register memory */ + qemu_sem_wait(&p->sem_sync); + if (qemu_rdma_registration(p->rdma) < 0) { + goto out; + } + /* + * Inform the main RDMA thread to run when multifd + * RDMA thread have completed registration. + */ + qemu_sem_post(&p->sem); while (true) { + qemu_sem_wait(&p->sem_sync); WITH_QEMU_LOCK_GUARD(&p->mutex) { if (p->quit) { break; } } - qemu_sem_wait(&p->sem); + /* Send FINISHED to the destination */ + head.type =3D RDMA_CONTROL_REGISTER_FINISHED; + ret =3D qemu_rdma_exchange_send(p->rdma, &head, NULL, NULL, NULL, = NULL); + if (ret < 0) { + return NULL; + } } =20 out: @@ -4406,14 +4488,20 @@ static void multifd_rdma_send_channel_setup(MultiFD= SendParams *p) static void *multifd_rdma_recv_thread(void *opaque) { MultiFDRecvParams *p =3D opaque; + int ret =3D 0; =20 while (true) { + qemu_sem_wait(&p->sem_sync); WITH_QEMU_LOCK_GUARD(&p->mutex) { if (p->quit) { break; } } - qemu_sem_wait(&p->sem_sync); + ret =3D qemu_rdma_registration_handle(p->file, p->c); + if (ret < 0) { + qemu_file_set_error(p->file, ret); + break; + } } =20 WITH_QEMU_LOCK_GUARD(&p->mutex) { --=20 1.8.3.1 From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612338672; cv=none; d=zohomail.com; s=zohoarc; b=V20YtCt0kwd/k9OZNUvMzxoOBsvgO4noljPWutYUnh5162S+ABDBxjqTetMkCgTEeQ90diTY87Me40vESeKr6icB56/ow3WiXj7Zh392kOH50mReTo0kUBuBBfRnKdpjGKrc/P7wifcFd3953aXJTI27Y8MT0tzqApyJVN7qeGU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612338672; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=+15aJBlN91W8KaUegEecXEuva+ZyCnNuUc70d114CV4=; b=bTn0oijSB1cpot5OyqcZBXY3th4gjMpdJ2V3vIUYjmUu4EtW08uYUlo+EHjZYzbuJQ3z1HGryP8Q6Ck2YeoxzM/Ozjsue0qNpADg/10VXHBJoVo17f2nBWEu2BJwgBnXJ409QQ4BwKwwyU0Vs3tIC8Qi0vyr2DTTtwKzVz+pJsU= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1612338672090327.0460018394871; Tue, 2 Feb 2021 23:51:12 -0800 (PST) Received: from localhost ([::1]:47094 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7CwQ-0003Mh-V3 for importer@patchew.org; Wed, 03 Feb 2021 02:51:10 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34308) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpj-00034j-AX for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:15 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3037) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpW-0008J2-Dk for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:15 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv05FPBzjJgS; Wed, 3 Feb 2021 15:42:36 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:45 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 15/18] migration/rdma: only register the memory for multifd channels Date: Wed, 3 Feb 2021 16:01:48 +0800 Message-ID: <1612339311-114805-16-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" All data is sent by multifd Channels, so we only register its for multifd channels and main channel don't register its. Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/rdma.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/rdma.c b/migration/rdma.c index c906cc7..f5eb563 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3939,6 +3939,12 @@ static int qemu_rdma_registration_stop(QEMUFile *f, = void *opaque, =20 qemu_sem_post(&multifd_send_param->sem_sync); } + + /* + * Use multifd to migrate, we only register memory for + * multifd RDMA channel and main channel don't register it. + */ + goto wait_reg_complete; } =20 /* @@ -3999,6 +4005,8 @@ static int qemu_rdma_registration_stop(QEMUFile *f, v= oid *opaque, rdma->dest_blocks[i].remote_host_addr; local->block[i].remote_rkey =3D rdma->dest_blocks[i].remote_rk= ey; } + +wait_reg_complete: /* Wait for all multifd channels to complete registration */ if (migrate_use_multifd()) { int i; --=20 1.8.3.1 From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612338992; cv=none; d=zohomail.com; s=zohoarc; b=caIuQYcmvGxhWV+dyEGTRV+jO6SKfoQxLzFWK2xNKqGJ/jTYObP0pUWJxWEKf6c4lDZ9dTytFR9/OV4iANUZbMtMB0VBoouLNbD70M+sk2BnOV1IO8GDotl9Gu3G32cjvbgPjw9cU9gios3wq1iSMuTz1S87LjGMUs5W5KoKdbc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612338992; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=AUJk7ZFpadXm0sssPFQ33Ozdefft+AEkpWILsca77Gs=; b=PJ2OZyIQI6bvicoe4UZqif0gWlacyesbPhbyd34IgC4WlSGWJoWCN6JL86TkmfehjxPRVhdUVUO+Km/zOvcyAj0CxERT6NbtbwIYhoEUeto4VX4YmPoMfjdkIur6rUGpvq5MZXtZIFOEZ47Jxlh4UUXou2EP1IrXkaMyHjOe1MI= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1612338992707445.6091268664421; Tue, 2 Feb 2021 23:56:32 -0800 (PST) Received: from localhost ([::1]:34874 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7D1b-0001kZ-Ev for importer@patchew.org; Wed, 03 Feb 2021 02:56:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34358) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpn-0003FL-6P for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:19 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3478) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpk-0008N7-7a for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:18 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv108PJzjJgW; Wed, 3 Feb 2021 15:42:37 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:45 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 16/18] migration/rdma: add rdma_channel into Migrationstate field Date: Wed, 3 Feb 2021 16:01:49 +0800 Message-ID: <1612339311-114805-17-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Multifd RDMA is need to poll when we send data, record it. Signed-off-by: Chuan Zheng --- migration/migration.c | 1 + migration/migration.h | 1 + migration/rdma.c | 14 ++++++++++++++ 3 files changed, 16 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index b8f4844..47bd11d 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1926,6 +1926,7 @@ void migrate_init(MigrationState *s) s->migration_thread_running =3D false; s->enabled_rdma_migration =3D false; s->host_port =3D NULL; + s->rdma_channel =3D 0; error_free(s->error); s->error =3D NULL; s->hostname =3D NULL; diff --git a/migration/migration.h b/migration/migration.h index 537ee09..5ff46e6 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -288,6 +288,7 @@ struct MigrationState { =20 /* Need by Multi-RDMA */ char *host_port; + int rdma_channel; }; =20 void migrate_set_state(int *state, int old_state, int new_state); diff --git a/migration/rdma.c b/migration/rdma.c index f5eb563..2097839 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -183,6 +183,20 @@ typedef struct { } RDMAWorkRequestData; =20 /* + * Get the multifd RDMA channel used to send data. + */ +static int get_multifd_RDMA_channel(void) +{ + int thread_count =3D migrate_multifd_channels(); + MigrationState *s =3D migrate_get_current(); + + s->rdma_channel++; + s->rdma_channel %=3D thread_count; + + return s->rdma_channel; +} + +/* * Negotiate RDMA capabilities during connection-setup time. */ typedef struct { --=20 1.8.3.1 From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612338816; cv=none; d=zohomail.com; s=zohoarc; b=DTTRLyPJ7+WS/RDmxjRA8LVYuYIP/qqoRYHPYPrxb/a9TJWdqMLkbSXF25hCCdrWwxGd+nCknDDPScksQWMqOc1PimYvH4IjeZxT7Q3bta/EWmHj/z+CM7Qca7P2iCwLqb1D/RoIHBSAA7QS4OwmFHDSU8UnCxQeBnCTVHl4XrI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612338816; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=S3KR5ChWOMeZAqzcXWcZBt5GrXXFa+Sg3/rhv1fQ7f4=; b=jwG3j/92ShY2WJRPQPHjHXJY/ho0LvhWMEm39xXkNQM/z6oEVrlLo9eWrUtlOr8VALj+McjszamtFJT1+rCfcXo98t6At7vom9bSv3UvMXDYOxzTzD/DILqZs86rSuBlwLmHeGvov6Xq1ioYX4LlnwZ02ENqXubNbHp6I9KniQ4= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 161233881611597.02428996741412; Tue, 2 Feb 2021 23:53:36 -0800 (PST) Received: from localhost ([::1]:55870 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Cyl-00070s-0y for importer@patchew.org; Wed, 03 Feb 2021 02:53:35 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34362) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpn-0003Fr-DY for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:19 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3479) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpl-0008N6-25 for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:19 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv070QGzjJgd; Wed, 3 Feb 2021 15:42:36 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:46 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 17/18] migration/rdma: send data for both rdma-pin-all and NOT rdma-pin-all mode Date: Wed, 3 Feb 2021 16:01:50 +0800 Message-ID: <1612339311-114805-18-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/rdma.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++= ---- 1 file changed, 61 insertions(+), 4 deletions(-) diff --git a/migration/rdma.c b/migration/rdma.c index 2097839..c19a91f 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -2002,6 +2002,20 @@ static int qemu_rdma_write_one(QEMUFile *f, RDMACont= ext *rdma, .repeat =3D 1, }; =20 + /* use multifd to send data */ + if (migrate_use_multifd()) { + int channel =3D get_multifd_RDMA_channel(); + int ret =3D 0; + MultiFDSendParams *multifd_send_param =3D NULL; + ret =3D get_multifd_send_param(channel, &multifd_send_param); + if (ret) { + error_report("rdma: error getting multifd_send_param(%d)", cha= nnel); + return -EINVAL; + } + rdma =3D (RDMAContext *)multifd_send_param->rdma; + block =3D &(rdma->local_ram_blocks.block[current_index]); + } + retry: sge.addr =3D (uintptr_t)(block->local_host_addr + (current_addr - block->offset)); @@ -2197,6 +2211,27 @@ retry: return 0; } =20 +static int multifd_rdma_write_flush(void) +{ + /* The multifd RDMA threads send data */ + MultiFDSendParams *multifd_send_param =3D NULL; + RDMAContext *rdma =3D NULL; + MigrationState *s =3D migrate_get_current(); + int ret =3D 0; + + ret =3D get_multifd_send_param(s->rdma_channel, + &multifd_send_param); + if (ret) { + error_report("rdma: error getting multifd_send_param(%d)", + s->rdma_channel); + return ret; + } + rdma =3D (RDMAContext *)(multifd_send_param->rdma); + rdma->nb_sent++; + + return ret; +} + /* * Push out any unwritten RDMA operations. * @@ -2219,8 +2254,15 @@ static int qemu_rdma_write_flush(QEMUFile *f, RDMACo= ntext *rdma) } =20 if (ret =3D=3D 0) { - rdma->nb_sent++; - trace_qemu_rdma_write_flush(rdma->nb_sent); + if (migrate_use_multifd()) { + ret =3D multifd_rdma_write_flush(); + if (ret) { + return ret; + } + } else { + rdma->nb_sent++; + trace_qemu_rdma_write_flush(rdma->nb_sent); + } } =20 rdma->current_length =3D 0; @@ -4062,6 +4104,7 @@ wait_reg_complete: } =20 qemu_sem_post(&multifd_send_param->sem_sync); + qemu_sem_wait(&multifd_send_param->sem); } } =20 @@ -4443,6 +4486,7 @@ static void *multifd_rdma_send_thread(void *opaque) Error *local_err =3D NULL; int ret =3D 0; RDMAControlHeader head =3D { .len =3D 0, .repeat =3D 1 }; + RDMAContext *rdma =3D p->rdma; =20 trace_multifd_send_thread_start(p->id); if (multifd_send_initial_packet(p, &local_err) < 0) { @@ -4451,7 +4495,7 @@ static void *multifd_rdma_send_thread(void *opaque) =20 /* wait for semaphore notification to register memory */ qemu_sem_wait(&p->sem_sync); - if (qemu_rdma_registration(p->rdma) < 0) { + if (qemu_rdma_registration(rdma) < 0) { goto out; } /* @@ -4466,12 +4510,25 @@ static void *multifd_rdma_send_thread(void *opaque) break; } } + /* To complete polling(CQE) */ + while (rdma->nb_sent) { + ret =3D qemu_rdma_block_for_wrid(rdma, RDMA_WRID_RDMA_WRITE, N= ULL); + if (ret < 0) { + error_report("multifd RDMA migration: " + "complete polling error!"); + return NULL; + } + } /* Send FINISHED to the destination */ head.type =3D RDMA_CONTROL_REGISTER_FINISHED; - ret =3D qemu_rdma_exchange_send(p->rdma, &head, NULL, NULL, NULL, = NULL); + ret =3D qemu_rdma_exchange_send(rdma, &head, NULL, NULL, NULL, NUL= L); if (ret < 0) { + error_report("multifd RDMA migration: " + "sending remote error!"); return NULL; } + /* sync main thread */ + qemu_sem_post(&p->sem); } =20 out: --=20 1.8.3.1 From nobody Tue May 21 22:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1612338521; cv=none; d=zohomail.com; s=zohoarc; b=GhVdyu4jFqaDNlhkNQeoUtwdowijl5SccHd+N5HvhNLdd4Nhah/pTvB0jrixw8JVWbep8QBMRetKZL3FaLkkZGIBVVBULhT6A64YIrXEI8WxPrNJqwl105+H2UE3Qm1xO5tMILIf4N7nYmxpjaKhPFb50J3kgpE0uwBcwyi0eLk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612338521; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=FWzUQwYkFsXdJRvJwnA/zwFICVsy/jDliHpf9H+Hb84=; b=CePVtt1Bq5FcbR94uqO3DYmuDMAYXjaNicVA7TfoAryNoNag3ZOb0lzfJ5GzmbWYFrHVEIObFKddUJHwD3/TsJdL77wf+KnukqJz6ctGTFVWtzcZDY2EDRnzKpdNK9G9yXLCsIJR/OhZ6xTYIzJ4Adr2t2NefI9War7HWEX0MiU= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1612338521984387.255419252401; Tue, 2 Feb 2021 23:48:41 -0800 (PST) Received: from localhost ([::1]:38614 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Cu0-0008Gf-Lq for importer@patchew.org; Wed, 03 Feb 2021 02:48:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34254) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cph-00030Q-IC for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:13 -0500 Received: from szxga06-in.huawei.com ([45.249.212.32]:2605) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpZ-0008JH-VU for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:12 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga06-in.huawei.com (SkyGuard) with ESMTP id 4DVtvK4qkCzjHYD; Wed, 3 Feb 2021 15:42:53 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:46 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 18/18] migration/rdma: RDMA cleanup for multifd migration Date: Wed, 3 Feb 2021 16:01:51 +0800 Message-ID: <1612339311-114805-19-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.32; envelope-from=zhengchuan@huawei.com; helo=szxga06-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Chuan Zheng --- migration/multifd.c | 6 ++++++ migration/multifd.h | 1 + migration/rdma.c | 16 +++++++++++++++- 3 files changed, 22 insertions(+), 1 deletion(-) diff --git a/migration/multifd.c b/migration/multifd.c index 1186246..4031648 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -577,6 +577,9 @@ void multifd_save_cleanup(void) p->packet_len =3D 0; g_free(p->packet); p->packet =3D NULL; +#ifdef CONFIG_RDMA + multifd_rdma_cleanup(p->rdma); +#endif multifd_send_state->ops->send_cleanup(p, &local_err); if (local_err) { migrate_set_error(migrate_get_current(), local_err); @@ -1039,6 +1042,9 @@ int multifd_load_cleanup(Error **errp) p->packet_len =3D 0; g_free(p->packet); p->packet =3D NULL; +#ifdef CONFIG_RDMA + multifd_rdma_cleanup(p->rdma); +#endif multifd_recv_state->ops->recv_cleanup(p); } qemu_sem_destroy(&multifd_recv_state->sem_sync); diff --git a/migration/multifd.h b/migration/multifd.h index 26d4489..0ecec5e 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -183,6 +183,7 @@ typedef struct { =20 #ifdef CONFIG_RDMA extern MultiFDSetup multifd_rdma_ops; +void multifd_rdma_cleanup(void *opaque); #endif void multifd_send_terminate_threads(Error *err); int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp); diff --git a/migration/rdma.c b/migration/rdma.c index c19a91f..f14357f 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -2369,7 +2369,7 @@ static void qemu_rdma_cleanup(RDMAContext *rdma) { int idx; =20 - if (rdma->cm_id && rdma->connected) { + if (rdma->channel && rdma->cm_id && rdma->connected) { if ((rdma->error_state || migrate_get_current()->state =3D=3D MIGRATION_STATUS_CANCELLI= NG) && !rdma->received_error) { @@ -4599,6 +4599,20 @@ static void multifd_rdma_recv_channel_setup(QIOChann= el *ioc, return; } =20 +void multifd_rdma_cleanup(void *opaque) +{ + RDMAContext *rdma =3D (RDMAContext *)opaque; + + if (!migrate_use_rdma()) { + return; + } + + rdma->listen_id =3D NULL; + rdma->channel =3D NULL; + qemu_rdma_cleanup(rdma); + g_free(rdma); +} + MultiFDSetup multifd_rdma_ops =3D { .send_thread =3D multifd_rdma_send_thread, .recv_thread =3D multifd_rdma_recv_thread, --=20 1.8.3.1