From nobody Thu Apr 25 04:07:12 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602907934; cv=none; d=zohomail.com; s=zohoarc; b=mcv0mJtK02kxN+TExW2ji6FzCUGvh/cQihEYBEO8lWh9lUGDwjEpDCSTLwK+Ao9XFWpJH40006aOPIAN4cAd6ctV2iZs6n3r0RdgVCm5za5oqYGjZRmYsFKuxARMlk7GNPHkJ8zxwyDvDVy10RKYLcxqZa3Th7VcJt4ckoR71F0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602907934; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=5PtBZmv2+/qrc2pwujZOoTOsUgBd1bjAad48l+Hc2f4=; b=Qun9tz90VobuulMZ58AwKngWusZwDttY8UsRipHQqbvWTlKoFuQ/z8IlvCDZ882uz3pE5bYbVgaAuzdmSHV9b+FqhY4L4yz5xJiK2CVTv1z1JRm6wXgmQXJ4LXvNk8hT8JvEaeGFirIQ26DMOji416oZOZ7fnpR6lvCFDKadnZM= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1602907934622179.1289186863387; Fri, 16 Oct 2020 21:12:14 -0700 (PDT) Received: from localhost ([::1]:58666 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdZl-0007aB-Am for importer@patchew.org; Sat, 17 Oct 2020 00:12:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39868) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXd-0005rN-I6 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:01 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5234 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXa-0007BP-ES for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:01 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 89CCADA78D38A7C10B9E; Sat, 17 Oct 2020 12:09:46 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:40 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 01/18] migration/rdma: add the 'migrate_use_rdma_pin_all' function Date: Sat, 17 Oct 2020 12:25:31 +0800 Message-ID: <1602908748-43335-2-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng Reviewed-by: Dr. David Alan Gilbert --- migration/migration.c | 9 +++++++++ migration/migration.h | 1 + 2 files changed, 10 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index 0575ecb..64ae417 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2329,6 +2329,15 @@ bool migrate_use_events(void) return s->enabled_capabilities[MIGRATION_CAPABILITY_EVENTS]; } =20 +bool migrate_use_rdma_pin_all(void) +{ + MigrationState *s; + + s =3D migrate_get_current(); + + return s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL]; +} + bool migrate_use_multifd(void) { MigrationState *s; diff --git a/migration/migration.h b/migration/migration.h index deb411a..74fd790 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -300,6 +300,7 @@ bool migrate_ignore_shared(void); bool migrate_validate_uuid(void); =20 bool migrate_auto_converge(void); +bool migrate_use_rdma_pin_all(void); bool migrate_use_multifd(void); bool migrate_pause_before_switchover(void); int migrate_multifd_channels(void); --=20 1.8.3.1 From nobody Thu Apr 25 04:07:12 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602908341; cv=none; d=zohomail.com; s=zohoarc; b=bTUi3S6RDUnhfVrdloCYpmSJIj0uT7oe5a3LPYvOFECWAg10MvHe/KdylzEjRey6U59u/JM0YlRbzSH9L5V1ITGFs6rt/6zCZlcjEL3wC/IqB/A9LzXl+OYKYV3b0HDirKQR+SSEbbo9CYWyMBmp2mskAYKSgsOc5z/6tURfZCk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602908341; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=xAtShhL5Oz3MiNQ9kokAayag8XbJkWrgzPRIQR+aaZk=; b=iXVVtB/BoJvnX0bSL3v9RcvFBXARTEsInC1y3lo+HGlqFht2esUCIJCuLEg9Aju6ADYjcf5rGZdNZrW3XPFV6R6XUMrO4hAh5iYzZLjqlsFNR8TUt/ZBgQNpourqWoSWoKSnWYr/N0KONKGMwODPsLrctWfzsr8UU9K1Srp7TIo= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1602908341892670.2870183649566; Fri, 16 Oct 2020 21:19:01 -0700 (PDT) Received: from localhost ([::1]:53096 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdgK-0008Tk-Ln for importer@patchew.org; Sat, 17 Oct 2020 00:19:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39936) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXg-0005td-Ho for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:04 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5239 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXe-0007Gb-DE for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:04 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 7B63CD1119B5D6BEA59B; Sat, 17 Oct 2020 12:09:51 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:41 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 02/18] migration/rdma: judge whether or not the RDMA is used for migration Date: Sat, 17 Oct 2020 12:25:32 +0800 Message-ID: <1602908748-43335-3-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add enabled_rdma_migration into MigrationState to judge whether or not the RDMA is used for migration. Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/migration.c | 13 +++++++++++++ migration/migration.h | 6 ++++++ 2 files changed, 19 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index 64ae417..be6166a 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -389,7 +389,9 @@ void migrate_add_address(SocketAddress *address) void qemu_start_incoming_migration(const char *uri, Error **errp) { const char *p =3D NULL; + MigrationState *s =3D migrate_get_current(); =20 + s->enabled_rdma_migration =3D false; qapi_event_send_migration(MIGRATION_STATUS_SETUP); if (!strcmp(uri, "defer")) { deferred_incoming_migration(errp); @@ -399,6 +401,7 @@ void qemu_start_incoming_migration(const char *uri, Err= or **errp) socket_start_incoming_migration(p ? p : uri, errp); #ifdef CONFIG_RDMA } else if (strstart(uri, "rdma:", &p)) { + s->enabled_rdma_migration =3D true; rdma_start_incoming_migration(p, errp); #endif } else if (strstart(uri, "exec:", &p)) { @@ -1887,6 +1890,7 @@ void migrate_init(MigrationState *s) s->start_postcopy =3D false; s->postcopy_after_devices =3D false; s->migration_thread_running =3D false; + s->enabled_rdma_migration =3D false; error_free(s->error); s->error =3D NULL; s->hostname =3D NULL; @@ -2115,6 +2119,7 @@ void qmp_migrate(const char *uri, bool has_blk, bool = blk, socket_start_outgoing_migration(s, p ? p : uri, &local_err); #ifdef CONFIG_RDMA } else if (strstart(uri, "rdma:", &p)) { + s->enabled_rdma_migration =3D true; rdma_start_outgoing_migration(s, p, &local_err); #endif } else if (strstart(uri, "exec:", &p)) { @@ -2329,6 +2334,14 @@ bool migrate_use_events(void) return s->enabled_capabilities[MIGRATION_CAPABILITY_EVENTS]; } =20 +bool migrate_use_rdma(void) +{ + MigrationState *s; + s =3D migrate_get_current(); + + return s->enabled_rdma_migration; +} + bool migrate_use_rdma_pin_all(void) { MigrationState *s; diff --git a/migration/migration.h b/migration/migration.h index 74fd790..e92eb29 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -264,6 +264,11 @@ struct MigrationState * This save hostname when out-going migration starts */ char *hostname; + + /* + * Enable RDMA migration + */ + bool enabled_rdma_migration; }; =20 void migrate_set_state(int *state, int old_state, int new_state); @@ -300,6 +305,7 @@ bool migrate_ignore_shared(void); bool migrate_validate_uuid(void); =20 bool migrate_auto_converge(void); +bool migrate_use_rdma(void); bool migrate_use_rdma_pin_all(void); bool migrate_use_multifd(void); bool migrate_pause_before_switchover(void); --=20 1.8.3.1 From nobody Thu Apr 25 04:07:12 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602908591; cv=none; d=zohomail.com; s=zohoarc; b=KXztjlxtqP9ghV3w09gqGEbDIXJwZ0Di3f46Ov49jBTAJLy6MhFnYg5PE6l9vbwa7nb209GCqoJEoO3qL0SwNoqs5X8Bp9u4hxYS1tc92Nj74NtfyE097VUcmMG6+y+gNYGyqh9SISIJjihhkvmmDE2d03HmXU3GOZTruziCsA8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602908591; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=d23ySpqypJWZYQP8LDZcvDhU+1DcdrK12LF1VHFr7LU=; b=DGbXlWB+GmTMaoKPMQlWNiyz8srSNQyQEXvsjVEe+E250NYeS2QLt0E46rWqhfp+H10mtSNrKunleyX2GpSk2fNTHs6f6GFbUsk4Q80/HKWLimAVUoR0qrCEww5amHSIzvm/PK3BJUbQOBqIsBaP0L9lItBPKWbYc1F/T1026hI= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1602908591498737.25447755302; Fri, 16 Oct 2020 21:23:11 -0700 (PDT) Received: from localhost ([::1]:38000 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdkM-0005b3-AN for importer@patchew.org; Sat, 17 Oct 2020 00:23:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39968) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXh-0005w6-NP for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:05 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5241 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXe-0007Ge-MH for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:05 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 8168DF0A678DFFE6DC66; Sat, 17 Oct 2020 12:09:51 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:41 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 03/18] migration/rdma: create multifd_setup_ops for Tx/Rx thread Date: Sat, 17 Oct 2020 12:25:33 +0800 Message-ID: <1602908748-43335-4-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Create multifd_setup_ops for TxRx thread, no logic change. Signed-off-by: Chuan Zheng --- migration/multifd.c | 44 +++++++++++++++++++++++++++++++++++++++----- migration/multifd.h | 7 +++++++ 2 files changed, 46 insertions(+), 5 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 68b171f..1f82307 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -383,6 +383,8 @@ struct { int exiting; /* multifd ops */ MultiFDMethods *ops; + /* multifd setup ops */ + MultiFDSetup *setup_ops; } *multifd_send_state; =20 /* @@ -790,8 +792,9 @@ static bool multifd_channel_connect(MultiFDSendParams *= p, } else { /* update for tls qio channel */ p->c =3D ioc; - qemu_thread_create(&p->thread, p->name, multifd_send_thread, p, - QEMU_THREAD_JOINABLE); + qemu_thread_create(&p->thread, p->name, + multifd_send_state->setup_ops->send_thread_= setup, + p, QEMU_THREAD_JOINABLE); } return false; } @@ -839,6 +842,11 @@ cleanup: multifd_new_send_channel_cleanup(p, sioc, local_err); } =20 +static void multifd_send_channel_setup(MultiFDSendParams *p) +{ + socket_send_channel_create(multifd_new_send_channel_async, p); +} + int multifd_save_setup(Error **errp) { int thread_count; @@ -856,6 +864,7 @@ int multifd_save_setup(Error **errp) multifd_send_state->pages =3D multifd_pages_init(page_count); qemu_sem_init(&multifd_send_state->channels_ready, 0); qatomic_set(&multifd_send_state->exiting, 0); + multifd_send_state->setup_ops =3D multifd_setup_ops_init(); multifd_send_state->ops =3D multifd_ops[migrate_multifd_compression()]; =20 for (i =3D 0; i < thread_count; i++) { @@ -875,7 +884,7 @@ int multifd_save_setup(Error **errp) p->packet->version =3D cpu_to_be32(MULTIFD_VERSION); p->name =3D g_strdup_printf("multifdsend_%d", i); p->tls_hostname =3D g_strdup(s->hostname); - socket_send_channel_create(multifd_new_send_channel_async, p); + multifd_send_state->setup_ops->send_channel_setup(p); } =20 for (i =3D 0; i < thread_count; i++) { @@ -902,6 +911,8 @@ struct { uint64_t packet_num; /* multifd ops */ MultiFDMethods *ops; + /* multifd setup ops */ + MultiFDSetup *setup_ops; } *multifd_recv_state; =20 static void multifd_recv_terminate_threads(Error *err) @@ -1095,6 +1106,7 @@ int multifd_load_setup(Error **errp) multifd_recv_state->params =3D g_new0(MultiFDRecvParams, thread_count); qatomic_set(&multifd_recv_state->count, 0); qemu_sem_init(&multifd_recv_state->sem_sync, 0); + multifd_recv_state->setup_ops =3D multifd_setup_ops_init(); multifd_recv_state->ops =3D multifd_ops[migrate_multifd_compression()]; =20 for (i =3D 0; i < thread_count; i++) { @@ -1173,9 +1185,31 @@ bool multifd_recv_new_channel(QIOChannel *ioc, Error= **errp) p->num_packets =3D 1; =20 p->running =3D true; - qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p, - QEMU_THREAD_JOINABLE); + multifd_recv_state->setup_ops->recv_channel_setup(ioc, p); + qemu_thread_create(&p->thread, p->name, + multifd_recv_state->setup_ops->recv_thread_setup, + p, QEMU_THREAD_JOINABLE); qatomic_inc(&multifd_recv_state->count); return qatomic_read(&multifd_recv_state->count) =3D=3D migrate_multifd_channels(); } + +static void multifd_recv_channel_setup(QIOChannel *ioc, MultiFDRecvParams = *p) +{ + return; +} + +static MultiFDSetup multifd_socket_ops =3D { + .send_thread_setup =3D multifd_send_thread, + .recv_thread_setup =3D multifd_recv_thread, + .send_channel_setup =3D multifd_send_channel_setup, + .recv_channel_setup =3D multifd_recv_channel_setup +}; + +MultiFDSetup *multifd_setup_ops_init(void) +{ + MultiFDSetup *ops; + + ops =3D &multifd_socket_ops; + return ops; +} diff --git a/migration/multifd.h b/migration/multifd.h index 8d6751f..446315b 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -166,6 +166,13 @@ typedef struct { int (*recv_pages)(MultiFDRecvParams *p, uint32_t used, Error **errp); } MultiFDMethods; =20 +typedef struct { + void *(*send_thread_setup)(void *opaque); + void *(*recv_thread_setup)(void *opaque); + void (*send_channel_setup)(MultiFDSendParams *p); + void (*recv_channel_setup)(QIOChannel *ioc, MultiFDRecvParams *p); +} MultiFDSetup; + void multifd_register_ops(int method, MultiFDMethods *ops); =20 #endif --=20 1.8.3.1 From nobody Thu Apr 25 04:07:12 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602907934; cv=none; d=zohomail.com; s=zohoarc; b=jQq8sDJ+LlCRpJERqaDDWJi4EsAhT2gJG2U4UVR6UrjCHR/jq0jvPVq37HV8N0mwlci2TK9ghzvvPHYEFzGxsNOr+MEsup2uhQGCB5ov2eRmV5FyTSiMh+HhO25XKL3ua3F5JE0PiCcxJE4uDpoQZGXXczabYndQqUgOTY7tnNc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602907934; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=X49YdQL1zi7q7jV9fgFXVP/w9C6lBkXtviFhtSA1PLE=; b=hCLXbSD+1tvHWQpTVS6owVEa6vCYXmfxOeVNrj97ZDbnnH0maApat7AfSPsEKDpocLzQdSWoFVb1bLJuSdgXCYNt2eXfmlG5c/Epoi6gQSeBKqJl1Ge2X3OPsTc1BDgCOuYGqFQLcK7p3ZeRVF8uIU3AKMjnBP9BQUShiA92npY= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1602907934718162.00735757594464; Fri, 16 Oct 2020 21:12:14 -0700 (PDT) Received: from localhost ([::1]:58598 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdZk-0007YX-Vr for importer@patchew.org; Sat, 17 Oct 2020 00:12:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39842) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXc-0005r7-KO for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:00 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5235 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXa-0007CO-74 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:00 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 673E8B90808B864C7865; Sat, 17 Oct 2020 12:09:51 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:42 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 04/18] migration/rdma: add multifd_setup_ops for rdma Date: Sat, 17 Oct 2020 12:25:34 +0800 Message-ID: <1602908748-43335-5-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Chuan Zheng --- migration/multifd.c | 6 ++++ migration/multifd.h | 4 +++ migration/rdma.c | 82 +++++++++++++++++++++++++++++++++++++++++++++++++= ++++ 3 files changed, 92 insertions(+) diff --git a/migration/multifd.c b/migration/multifd.c index 1f82307..0d494df 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -1210,6 +1210,12 @@ MultiFDSetup *multifd_setup_ops_init(void) { MultiFDSetup *ops; =20 +#ifdef CONFIG_RDMA + if (migrate_use_rdma()) { + ops =3D multifd_rdma_setup(); + return ops; + } +#endif ops =3D &multifd_socket_ops; return ops; } diff --git a/migration/multifd.h b/migration/multifd.h index 446315b..62a0b2a 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -173,6 +173,10 @@ typedef struct { void (*recv_channel_setup)(QIOChannel *ioc, MultiFDRecvParams *p); } MultiFDSetup; =20 +#ifdef CONFIG_RDMA +MultiFDSetup *multifd_rdma_setup(void); +#endif +MultiFDSetup *multifd_setup_ops_init(void); void multifd_register_ops(int method, MultiFDMethods *ops); =20 #endif diff --git a/migration/rdma.c b/migration/rdma.c index 0340841..ad4e4ba 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -19,6 +19,7 @@ #include "qemu/cutils.h" #include "rdma.h" #include "migration.h" +#include "multifd.h" #include "qemu-file.h" #include "ram.h" #include "qemu-file-channel.h" @@ -4138,3 +4139,84 @@ err: g_free(rdma); g_free(rdma_return_path); } + +static void *multifd_rdma_send_thread(void *opaque) +{ + MultiFDSendParams *p =3D opaque; + + while (true) { + qemu_mutex_lock(&p->mutex); + if (p->quit) { + qemu_mutex_unlock(&p->mutex); + break; + } + qemu_mutex_unlock(&p->mutex); + qemu_sem_wait(&p->sem); + } + + qemu_mutex_lock(&p->mutex); + p->running =3D false; + qemu_mutex_unlock(&p->mutex); + + return NULL; +} + +static void multifd_rdma_send_channel_setup(MultiFDSendParams *p) +{ + Error *local_err =3D NULL; + + if (p->quit) { + error_setg(&local_err, "multifd: send id %d already quit", p->id); + return ; + } + p->running =3D true; + + qemu_thread_create(&p->thread, p->name, multifd_rdma_send_thread, p, + QEMU_THREAD_JOINABLE); +} + +static void *multifd_rdma_recv_thread(void *opaque) +{ + MultiFDRecvParams *p =3D opaque; + + while (true) { + qemu_mutex_lock(&p->mutex); + if (p->quit) { + qemu_mutex_unlock(&p->mutex); + break; + } + qemu_mutex_unlock(&p->mutex); + qemu_sem_wait(&p->sem_sync); + } + + qemu_mutex_lock(&p->mutex); + p->running =3D false; + qemu_mutex_unlock(&p->mutex); + + return NULL; +} + +static void multifd_rdma_recv_channel_setup(QIOChannel *ioc, + MultiFDRecvParams *p) +{ + QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); + + p->file =3D rioc->file; + return; +} + +static MultiFDSetup multifd_rdma_ops =3D { + .send_thread_setup =3D multifd_rdma_send_thread, + .recv_thread_setup =3D multifd_rdma_recv_thread, + .send_channel_setup =3D multifd_rdma_send_channel_setup, + .recv_channel_setup =3D multifd_rdma_recv_channel_setup +}; + +MultiFDSetup *multifd_rdma_setup(void) +{ + MultiFDSetup *rdma_ops; + + rdma_ops =3D &multifd_rdma_ops; + + return rdma_ops; +} --=20 1.8.3.1 From nobody Thu Apr 25 04:07:12 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602908448; cv=none; d=zohomail.com; s=zohoarc; b=CmWsajNucT/d2Rh8Hnx8ez8n0vO5X7wfCEwzqlEXdIlPvr5VMeizOw4WaP6Jfa+hi44GomxEvpzmfNKp8kUapRTBY347xC+SLoQFE4VhK2wGXOU9UUqVYgVWnvSR5fnLdIl1z1d537O2EaZi5CmjNnRyZjmrehelTnpc5za+9Vo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602908448; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=pE/sD35gs7BvHFG/a6celUybEXBUqo1rsF4VJaoBM0Y=; b=YRFt8oCJ8j0FmH8YXow1ajFPhm8gOM5vXdT9ZVJsBInIwU0Cnv4FC+y26bwpS54mLucOotEppIqrPluGVX9uKBISQItohsxv98qzUYOOlrgno0whcGfMkZpJMBM8ElH+DWBuQUSDBukYtmjDjFcoFXcEDkC/g1rqWg7Uki6BtII= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1602908448698461.36100559283807; Fri, 16 Oct 2020 21:20:48 -0700 (PDT) Received: from localhost ([::1]:59606 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdi3-0002jz-Gz for importer@patchew.org; Sat, 17 Oct 2020 00:20:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39952) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXg-0005uH-TT for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:04 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5240 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXe-0007GZ-Cu for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:04 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 77AEC1621E9FF7088B2F; Sat, 17 Oct 2020 12:09:51 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:43 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 05/18] migration/rdma: do not need sync main for rdma Date: Sat, 17 Oct 2020 12:25:35 +0800 Message-ID: <1602908748-43335-6-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/multifd.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/multifd.c b/migration/multifd.c index 0d494df..8ccfd46 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -580,6 +580,10 @@ void multifd_send_sync_main(QEMUFile *f) if (!migrate_use_multifd()) { return; } + /* Do not need sync for rdma */ + if (migrate_use_rdma()) { + return; + } if (multifd_send_state->pages->used) { if (multifd_send_pages(f) < 0) { error_report("%s: multifd_send_pages fail", __func__); @@ -1002,6 +1006,10 @@ void multifd_recv_sync_main(void) if (!migrate_use_multifd()) { return; } + /* Do not need sync for rdma */ + if (migrate_use_rdma()) { + return; + } for (i =3D 0; i < migrate_multifd_channels(); i++) { MultiFDRecvParams *p =3D &multifd_recv_state->params[i]; =20 --=20 1.8.3.1 From nobody Thu Apr 25 04:07:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602907939; cv=none; d=zohomail.com; s=zohoarc; b=F8tHfSCWNTL04bjnz4ATOXBMSl11c++Q8/fWRZnprwG7sN9YIrC5CpjshZpAo6b197jUlufy8JaeyDerZizYY+JlJTgMdACil8EBs/eBai6UBHq2k5m2/lGo9eNVxKFd9vvyy4LNYKZc7J0xQfp/fDeyRKWLYrrk2FH9WVYZnHA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602907939; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=k6AZ4Z8jCaja5Wm7n+Yqv0/x71H4oUV3QQ7tuiPsbhA=; b=TsfHheDWxE80vURf77m2Ma47trw7T0k6QUZWOKrbfCGZtjR8OlR8I+kk0iuKCEE0S9RVn2ibIHryu9i8bhbqurf/Xz+nC4wPSHrCBQO3fKWdLV6tzcu0XuD9sT/gXMI9u7U1F+k1XZHX9i06Im3NN4obEYTOiZJA2QwS+bEAfeU= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1602907939523580.804488763079; Fri, 16 Oct 2020 21:12:19 -0700 (PDT) Received: from localhost ([::1]:59200 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdZq-0007nA-9W for importer@patchew.org; Sat, 17 Oct 2020 00:12:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39940) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXg-0005tn-Lx for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:04 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5242 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXe-0007Gf-M6 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:04 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 8606FA20CCE762B3E224; Sat, 17 Oct 2020 12:09:51 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:43 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 06/18] migration/rdma: export MultiFDSendParams/MultiFDRecvParams Date: Sat, 17 Oct 2020 12:25:36 +0800 Message-ID: <1602908748-43335-7-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" MultiFDSendParams and MultiFDRecvParams is need for rdma, export it Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/multifd.c | 26 ++++++++++++++++++++++++++ migration/multifd.h | 2 ++ 2 files changed, 28 insertions(+) diff --git a/migration/multifd.c b/migration/multifd.c index 8ccfd46..03f3a1e 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -387,6 +387,19 @@ struct { MultiFDSetup *setup_ops; } *multifd_send_state; =20 +int get_multifd_send_param(int id, MultiFDSendParams **param) +{ + int ret =3D 0; + + if (id < 0 || id >=3D migrate_multifd_channels()) { + ret =3D -1; + } else { + *param =3D &(multifd_send_state->params[id]); + } + + return ret; +} + /* * How we use multifd_send_state->pages and channel->pages? * @@ -919,6 +932,19 @@ struct { MultiFDSetup *setup_ops; } *multifd_recv_state; =20 +int get_multifd_recv_param(int id, MultiFDRecvParams **param) +{ + int ret =3D 0; + + if (id < 0 || id >=3D migrate_multifd_channels()) { + ret =3D -1; + } else { + *param =3D &(multifd_recv_state->params[id]); + } + + return ret; +} + static void multifd_recv_terminate_threads(Error *err) { int i; diff --git a/migration/multifd.h b/migration/multifd.h index 62a0b2a..2f4e585 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -176,6 +176,8 @@ typedef struct { #ifdef CONFIG_RDMA MultiFDSetup *multifd_rdma_setup(void); #endif +int get_multifd_send_param(int id, MultiFDSendParams **param); +int get_multifd_recv_param(int id, MultiFDRecvParams **param); MultiFDSetup *multifd_setup_ops_init(void); void multifd_register_ops(int method, MultiFDMethods *ops); =20 --=20 1.8.3.1 From nobody Thu Apr 25 04:07:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602907936; cv=none; d=zohomail.com; s=zohoarc; b=Vxm4fNp4Hh1SbngWbDXwrjzKmMBXpMAFhpNwwBo50SbgBbH9dcu7alikBlt+nukMCh5bCJhUx0wrLmH2fExuATQcIbgvgOgJcY4cHJs4uJOIoyal9JEcN+MO/w/k14A884ISRjfo4pglPRAeL19vCxJQeSkuRsWymCKL+UCt5zs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602907936; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=VFGf4kwKwdcN4XogNLj7EphLJwIRKGVC1WG61CM7D8g=; b=mwqdknWzOG9eQFlIO6D/312vZmDqHrdlxhOMtMVa39NPNpmQgU7ZxnMocHxtJYQzY8ifm1r7rn/L6mO0LUCsxCrSY2DtOzlndOk+fav9YqfBUu7iKOMf57U0Q66V15cFtgpq4MldmQJE74KZl1XUEc6p3GfEjmFQ3kWfsCEo0lg= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1602907936106107.85028927710505; Fri, 16 Oct 2020 21:12:16 -0700 (PDT) Received: from localhost ([::1]:58856 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdZm-0007em-TI for importer@patchew.org; Sat, 17 Oct 2020 00:12:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39838) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXc-0005r6-K3 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:00 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5236 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXa-0007CP-EO for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:00 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 6C1B6CA8D488661B70AD; Sat, 17 Oct 2020 12:09:51 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:44 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 07/18] migration/rdma: add rdma field into multifd send/recv param Date: Sat, 17 Oct 2020 12:25:37 +0800 Message-ID: <1602908748-43335-8-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Note we do want to export any rdma struct, take void * instead. Signed-off-by: Chuan Zheng --- migration/multifd.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/multifd.h b/migration/multifd.h index 2f4e585..ff80bd5 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -108,6 +108,10 @@ typedef struct { QemuSemaphore sem_sync; /* used for compression methods */ void *data; + /* used for multifd rdma */ + void *rdma; + /* communication channel */ + QEMUFile *file; } MultiFDSendParams; =20 typedef struct { @@ -147,6 +151,10 @@ typedef struct { QemuSemaphore sem_sync; /* used for de-compression methods */ void *data; + /* used for multifd rdma */ + void *rdma; + /* communication channel */ + QEMUFile *file; } MultiFDRecvParams; =20 typedef struct { --=20 1.8.3.1 From nobody Thu Apr 25 04:07:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602908044; cv=none; d=zohomail.com; s=zohoarc; b=WK0fAJYaUIx3nvIkHR953sPIxdBPeHPx+RMR3d2fE6oQ/juVx9HemPzU6Xu6RX8fO0UEJNv1zadVgXPaUiORo4Qw0vw4AhfX4o0lQy0Xt4FHMhtW4qvuXQHpML6mneG+rD+aEezP0r1fj0Fef2K3qJumWVDm9uu0G+5atv15CMY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602908044; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=sje0swATjuDYt7JsE9pSsxYhs0CXus8E1ZQqMxvnMyY=; b=am6vjT/sV5dOzftFn2nfAjcTTcj9I5508FmkX3ooHUp24FRcSBcuw3i/f6cNuQMoSTLZAfdfeBpMqIxcfc0f3oG3AQMrHPQKZpWOfT0/nQ4JjvBgWf8YhHfHua1T3FjGqGtAR/Bck1n2vxMQA7uq5gckKA+282F3lZAibrvbF44= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1602908044983157.62975494648424; Fri, 16 Oct 2020 21:14:04 -0700 (PDT) Received: from localhost ([::1]:38814 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdbX-0002Xz-P8 for importer@patchew.org; Sat, 17 Oct 2020 00:14:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39904) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXf-0005rb-5P for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:03 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5237 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXd-0007CT-Dt for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:02 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 6FBC4D0D39749B1F3E5F; Sat, 17 Oct 2020 12:09:51 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:45 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 08/18] migration/rdma: export getQIOChannel to get QIOchannel in rdma Date: Sat, 17 Oct 2020 12:25:38 +0800 Message-ID: <1602908748-43335-9-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/qemu-file.c | 5 +++++ migration/qemu-file.h | 1 + 2 files changed, 6 insertions(+) diff --git a/migration/qemu-file.c b/migration/qemu-file.c index be21518..37f6201 100644 --- a/migration/qemu-file.c +++ b/migration/qemu-file.c @@ -260,6 +260,11 @@ void ram_control_before_iterate(QEMUFile *f, uint64_t = flags) } } =20 +void *getQIOChannel(QEMUFile *f) +{ + return f->opaque; +} + void ram_control_after_iterate(QEMUFile *f, uint64_t flags) { int ret =3D 0; diff --git a/migration/qemu-file.h b/migration/qemu-file.h index a9b6d6c..4cef043 100644 --- a/migration/qemu-file.h +++ b/migration/qemu-file.h @@ -165,6 +165,7 @@ void qemu_file_set_blocking(QEMUFile *f, bool block); void ram_control_before_iterate(QEMUFile *f, uint64_t flags); void ram_control_after_iterate(QEMUFile *f, uint64_t flags); void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data); +void *getQIOChannel(QEMUFile *f); =20 /* Whenever this is found in the data stream, the flags * will be passed to ram_control_load_hook in the incoming-migration --=20 1.8.3.1 From nobody Thu Apr 25 04:07:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602908202; cv=none; d=zohomail.com; s=zohoarc; b=fX4dj3lUAiB5IuIlKvzUanJLy94D+sPVDJwaH1zApSSMsuXOtkbdmB/9pW5mls0kLrgZJI043WiTjPvS76PXT/1ac6mmH8V2p8nzJugunrmVEvXaxIeTw1+GAsx/95yHfqfcBTVxf5jLUXz9DIVpgYKL+gOXfhZl3gO6lxUwQF0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602908202; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=Td/gHpoUiOMP359qTjtTX+JmXl2PI4yuBoo75A3PfBc=; b=Qs7mbzu+ym99a4gfUVVY0Fzzk1Qj9Nxx+Wii0IRpmwslJXUlnstYtatpq5IgVRwa2a5vhhy9ABhWmaTu5u4VOd8zaSaWFEwxNJEgWvM6qqAzM/58ta0W94CBSg0D7Dh2Yl+wMGLugznzxmyKhqf/DcZCQ6KTFUR1rUYwNzAjiaU= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1602908202046166.73948143687699; Fri, 16 Oct 2020 21:16:42 -0700 (PDT) Received: from localhost ([::1]:46768 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTde4-0005tu-Qy for importer@patchew.org; Sat, 17 Oct 2020 00:16:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39932) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXg-0005t0-4T for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:04 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5238 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXe-0007CU-9D for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:03 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 7393A80C7702E1FEBCA2; Sat, 17 Oct 2020 12:09:51 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:45 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 09/18] migration/rdma: add multifd_rdma_load_setup() to setup multifd rdma Date: Sat, 17 Oct 2020 12:25:39 +0800 Message-ID: <1602908748-43335-10-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Chuan Zheng Reviewed-by: Dr. David Alan Gilbert --- migration/rdma.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/migration/rdma.c b/migration/rdma.c index ad4e4ba..2baa933 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -4010,6 +4010,48 @@ static void rdma_accept_incoming_migration(void *opa= que) } } =20 +static bool multifd_rdma_load_setup(const char *host_port, + RDMAContext *rdma, Error **errp) +{ + int thread_count; + int i; + int idx; + MultiFDRecvParams *multifd_recv_param; + RDMAContext *multifd_rdma; + + if (!migrate_use_multifd()) { + return true; + } + + if (multifd_load_setup(errp) !=3D 0) { + /* + * We haven't been able to create multifd threads + * nothing better to do + */ + return false; + } + + thread_count =3D migrate_multifd_channels(); + for (i =3D 0; i < thread_count; i++) { + if (get_multifd_recv_param(i, &multifd_recv_param) < 0) { + ERROR(errp, "rdma: error getting multifd_recv_param(%d)", i); + return false; + } + + multifd_rdma =3D qemu_rdma_data_init(host_port, errp); + for (idx =3D 0; idx < RDMA_WRID_MAX; idx++) { + multifd_rdma->wr_data[idx].control_len =3D 0; + multifd_rdma->wr_data[idx].control_curr =3D NULL; + } + /* the CM channel and CM id is shared */ + multifd_rdma->channel =3D rdma->channel; + multifd_rdma->listen_id =3D rdma->listen_id; + multifd_recv_param->rdma =3D (void *)multifd_rdma; + } + + return true; +} + void rdma_start_incoming_migration(const char *host_port, Error **errp) { int ret; @@ -4057,6 +4099,16 @@ void rdma_start_incoming_migration(const char *host_= port, Error **errp) qemu_rdma_return_path_dest_init(rdma_return_path, rdma); } =20 + /* multifd rdma setup */ + if (!multifd_rdma_load_setup(host_port, rdma, &local_err)) { + /* + * We haven't been able to create multifd threads + * nothing better to do + */ + error_report_err(local_err); + goto err; + } + qemu_set_fd_handler(rdma->channel->fd, rdma_accept_incoming_migration, NULL, (void *)(intptr_t)rdma); return; --=20 1.8.3.1 From nobody Thu Apr 25 04:07:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602908202; cv=none; d=zohomail.com; s=zohoarc; b=PBjVWV80eOIe7Cxc5u8E4OU0OvT6VTy9YDsZYKe72BFzqwRTyxcxdujqsqQ79h1MeJJgh1qEoeCnfasFc9M4Oh836Ze/UyN4AkMqRFuNKHNF0OK06M+9nhrUYi4ONBcn8RHG4X2FOzeFdPSuKbYtA9jI+c9uhFfPW/7xRzisjCs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602908202; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=rM2dYkEMwcKY9pgqIbMWTG3zaLi/p+4qYjEN7m13jAw=; b=eGe+3TZbBPtmhD7NKKDe/GPaH3DEsQzHZ5F7Ym0WiTU9cEtaD3IqStfErAe97Ow7NrUmUr7egFtX3ZzRnluNFRbQcQrBIbw8GICvIDq1GONW19hmkQrRcvoXYuuMczlzBxZTiTRig/UQLBHRSYny1tEJRvGadwHU10FNN02RKQc= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 160290820267075.88477741803626; Fri, 16 Oct 2020 21:16:42 -0700 (PDT) Received: from localhost ([::1]:46838 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTde5-0005vX-G7 for importer@patchew.org; Sat, 17 Oct 2020 00:16:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40074) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXu-0006Gg-8P for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:19 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5248 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXs-0007Iy-4e for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:17 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 87BF5CC55B61983A2968; Sat, 17 Oct 2020 12:09:56 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:46 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 10/18] migration/rdma: Create the multifd recv channels for RDMA Date: Sat, 17 Oct 2020 12:25:40 +0800 Message-ID: <1602908748-43335-11-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We still don't transmit anything through them, and we only build the RDMA connections. Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/rdma.c | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++= ++-- 1 file changed, 68 insertions(+), 2 deletions(-) diff --git a/migration/rdma.c b/migration/rdma.c index 2baa933..63559f1 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3266,6 +3266,40 @@ static void rdma_cm_poll_handler(void *opaque) } } =20 +static bool qemu_rdma_accept_setup(RDMAContext *rdma) +{ + RDMAContext *multifd_rdma =3D NULL; + int thread_count; + int i; + MultiFDRecvParams *multifd_recv_param; + thread_count =3D migrate_multifd_channels(); + /* create the multifd channels for RDMA */ + for (i =3D 0; i < thread_count; i++) { + if (get_multifd_recv_param(i, &multifd_recv_param) < 0) { + error_report("rdma: error getting multifd_recv_param(%d)", i); + return false; + } + + multifd_rdma =3D (RDMAContext *) multifd_recv_param->rdma; + if (multifd_rdma->cm_id =3D=3D NULL) { + break; + } else { + multifd_rdma =3D NULL; + } + } + + if (multifd_rdma) { + qemu_set_fd_handler(rdma->channel->fd, + rdma_accept_incoming_migration, + NULL, (void *)(intptr_t)multifd_rdma); + } else { + qemu_set_fd_handler(rdma->channel->fd, rdma_cm_poll_handler, + NULL, rdma); + } + + return true; +} + static int qemu_rdma_accept(RDMAContext *rdma) { RDMACapabilities cap; @@ -3365,6 +3399,10 @@ static int qemu_rdma_accept(RDMAContext *rdma) qemu_set_fd_handler(rdma->channel->fd, rdma_accept_incoming_migrat= ion, NULL, (void *)(intptr_t)rdma->return_path); + } else if (migrate_use_multifd()) { + if (!qemu_rdma_accept_setup(rdma)) { + goto err_rdma_dest_wait; + } } else { qemu_set_fd_handler(rdma->channel->fd, rdma_cm_poll_handler, NULL, rdma); @@ -3975,6 +4013,35 @@ static QEMUFile *qemu_fopen_rdma(RDMAContext *rdma, = const char *mode) return rioc->file; } =20 +static void migration_rdma_process_incoming(QEMUFile *f, + RDMAContext *rdma, Error **err= p) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + QIOChannel *ioc =3D NULL; + bool start_migration =3D false; + + /* FIXME: Need refactor */ + if (!migrate_use_multifd()) { + rdma->migration_started_on_destination =3D 1; + migration_fd_process_incoming(f, errp); + return; + } + + if (!mis->from_src_file) { + mis->from_src_file =3D f; + qemu_file_set_blocking(f, false); + } else { + ioc =3D QIO_CHANNEL(getQIOChannel(f)); + /* Multiple connections */ + assert(migrate_use_multifd()); + start_migration =3D multifd_recv_new_channel(ioc, errp); + } + + if (start_migration) { + migration_incoming_process(); + } +} + static void rdma_accept_incoming_migration(void *opaque) { RDMAContext *rdma =3D opaque; @@ -4003,8 +4070,7 @@ static void rdma_accept_incoming_migration(void *opaq= ue) return; } =20 - rdma->migration_started_on_destination =3D 1; - migration_fd_process_incoming(f, &local_err); + migration_rdma_process_incoming(f, rdma, &local_err); if (local_err) { error_reportf_err(local_err, "RDMA ERROR:"); } --=20 1.8.3.1 From nobody Thu Apr 25 04:07:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602908447; cv=none; d=zohomail.com; s=zohoarc; b=VkKfqvFcg+D61SRZre756j6yq9ohZC6Zm/4hdc8PXzTKKfeCMDFerWb2kCP/VargrOiOuK4R5S/M0hvyf/zdN3ar1LhPKULe1ELOWrm98jiuN4WhB8XE751v6KMQMWVBopmTaBxSomoME0khhGtYj+M9aUaNQmiM5Vu5+/2HUqo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602908447; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=tZ9GLILsqVgc7PHIMW9k5JedpjjlzqFSQEMRvW2RORg=; b=chmX5sDqxohyiTzEG0Lfpfs9dFAKkAZb0rntwRk+XP8CC41RodKv7mrFj864HS41pc70Lzaro+rEjUft5tw+Xm/mTxF4WahLFJjhO7VQTQ0hyxjYwo4cUYbbTgSnyW6KyrIddd/HuAmHp3PIHXHq/btHI3sZfDzsLH5toncRQhM= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1602908447033835.5986832250276; Fri, 16 Oct 2020 21:20:47 -0700 (PDT) Received: from localhost ([::1]:59436 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdi1-0002fj-Sz for importer@patchew.org; Sat, 17 Oct 2020 00:20:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40104) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXy-0006JF-D5 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:23 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5249 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXt-0007Jg-Ny for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:22 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 900DEFD1854B5AB8137D; Sat, 17 Oct 2020 12:09:56 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:47 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 11/18] migration/rdma: record host_port for multifd RDMA Date: Sat, 17 Oct 2020 12:25:41 +0800 Message-ID: <1602908748-43335-12-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Chuan Zheng --- migration/migration.c | 1 + migration/migration.h | 3 +++ migration/rdma.c | 3 +++ 3 files changed, 7 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index be6166a..7061410 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1891,6 +1891,7 @@ void migrate_init(MigrationState *s) s->postcopy_after_devices =3D false; s->migration_thread_running =3D false; s->enabled_rdma_migration =3D false; + s->host_port =3D NULL; error_free(s->error); s->error =3D NULL; s->hostname =3D NULL; diff --git a/migration/migration.h b/migration/migration.h index e92eb29..fea63de 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -269,6 +269,9 @@ struct MigrationState * Enable RDMA migration */ bool enabled_rdma_migration; + + /* Need by Multi-RDMA */ + char *host_port; }; =20 void migrate_set_state(int *state, int old_state, int new_state); diff --git a/migration/rdma.c b/migration/rdma.c index 63559f1..dd9f705 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -4206,6 +4206,8 @@ void rdma_start_outgoing_migration(void *opaque, goto err; } =20 + s->host_port =3D g_strdup(host_port); + ret =3D qemu_rdma_source_init(rdma, s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL], errp); =20 @@ -4250,6 +4252,7 @@ void rdma_start_outgoing_migration(void *opaque, =20 s->to_dst_file =3D qemu_fopen_rdma(rdma, "wb"); migrate_fd_connect(s, NULL); + g_free(s->host_port); return; return_path_err: qemu_rdma_cleanup(rdma); --=20 1.8.3.1 From nobody Thu Apr 25 04:07:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602908350; cv=none; d=zohomail.com; s=zohoarc; b=iH+XV4OJHKqkNiNg+MlmRTaxHY1qzdArnwR1nRgQGS5Ksz+A4wQ1p5XjBX93hnURT9LLvCNPoBrTtWq3t8hKtY5LVDdiXeI91MaW0ukQ0Ag1TrngarWV+R0KGCJLP+0ssskgz02GAeNmllXPuBkpylPw8HlR0pOSCublDTCEtz4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602908350; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=Epf6F46XASwg79+TyKsiIycwkQc7AHl0KWjv9VPXG/o=; b=Xb7g/mb2cN49ZUf1jscUDwslBBQNoQhHK65sQIo3PJjmGH/30qNpO94FLfzSrvxxRGAORLHtnEqzCYYuICitHiG0pfXBGf1c2YTwpVZlFS1RsoSg3NOpqpm7ij4QlTVFI3YSYrwTKwz1NM6dITEeUEsf+VAMRjfWeQv1ajChL8A= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1602908350929117.42605782071189; Fri, 16 Oct 2020 21:19:10 -0700 (PDT) Received: from localhost ([::1]:53182 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdgT-00005P-E8 for importer@patchew.org; Sat, 17 Oct 2020 00:19:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40100) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXv-0006HD-J5 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:19 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5245 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXr-0007Ij-5e for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:19 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 7F60251D1C4A51D745D8; Sat, 17 Oct 2020 12:09:56 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:47 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 12/18] migration/rdma: Create the multifd send channels for RDMA Date: Sat, 17 Oct 2020 12:25:42 +0800 Message-ID: <1602908748-43335-13-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Chuan Zheng --- migration/multifd.c | 4 ++-- migration/multifd.h | 2 ++ migration/rdma.c | 56 +++++++++++++++++++++++++++++++++++++++++++++++++= ++++ 3 files changed, 60 insertions(+), 2 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 03f3a1e..9439b3c 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -173,7 +173,7 @@ void multifd_register_ops(int method, MultiFDMethods *o= ps) multifd_ops[method] =3D ops; } =20 -static int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) +int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) { MultiFDInit_t msg =3D {}; int ret; @@ -500,7 +500,7 @@ int multifd_queue_page(QEMUFile *f, RAMBlock *block, ra= m_addr_t offset) return 1; } =20 -static void multifd_send_terminate_threads(Error *err) +void multifd_send_terminate_threads(Error *err) { int i; =20 diff --git a/migration/multifd.h b/migration/multifd.h index ff80bd5..ec9e897 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -184,6 +184,8 @@ typedef struct { #ifdef CONFIG_RDMA MultiFDSetup *multifd_rdma_setup(void); #endif +void multifd_send_terminate_threads(Error *err); +int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp); int get_multifd_send_param(int id, MultiFDSendParams **param); int get_multifd_recv_param(int id, MultiFDRecvParams **param); MultiFDSetup *multifd_setup_ops_init(void); diff --git a/migration/rdma.c b/migration/rdma.c index dd9f705..1af81f5 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -4261,9 +4261,54 @@ err: g_free(rdma_return_path); } =20 +static int multifd_channel_rdma_connect(void *opaque) +{ + MultiFDSendParams *p =3D opaque; + Error *local_err =3D NULL; + int ret =3D 0; + MigrationState *s =3D migrate_get_current(); + + p->rdma =3D qemu_rdma_data_init(s->host_port, &local_err); + if (p->rdma =3D=3D NULL) { + goto out; + } + + ret =3D qemu_rdma_source_init(p->rdma, + migrate_use_rdma_pin_all(), + &local_err); + if (ret) { + goto out; + } + + ret =3D qemu_rdma_connect(p->rdma, &local_err); + if (ret) { + goto out; + } + + p->file =3D qemu_fopen_rdma(p->rdma, "wb"); + if (p->file =3D=3D NULL) { + goto out; + } + + p->c =3D QIO_CHANNEL(getQIOChannel(p->file)); + +out: + if (local_err) { + trace_multifd_send_error(p->id); + } + + return ret; +} + static void *multifd_rdma_send_thread(void *opaque) { MultiFDSendParams *p =3D opaque; + Error *local_err =3D NULL; + + trace_multifd_send_thread_start(p->id); + if (multifd_send_initial_packet(p, &local_err) < 0) { + goto out; + } =20 while (true) { qemu_mutex_lock(&p->mutex); @@ -4275,6 +4320,11 @@ static void *multifd_rdma_send_thread(void *opaque) qemu_sem_wait(&p->sem); } =20 +out: + if (local_err) { + trace_multifd_send_error(p->id); + multifd_send_terminate_threads(local_err); + } qemu_mutex_lock(&p->mutex); p->running =3D false; qemu_mutex_unlock(&p->mutex); @@ -4286,6 +4336,12 @@ static void multifd_rdma_send_channel_setup(MultiFDS= endParams *p) { Error *local_err =3D NULL; =20 + if (multifd_channel_rdma_connect(p)) { + error_setg(&local_err, "multifd: rdma channel %d not established", + p->id); + return ; + } + if (p->quit) { error_setg(&local_err, "multifd: send id %d already quit", p->id); return ; --=20 1.8.3.1 From nobody Thu Apr 25 04:07:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602908339; cv=none; d=zohomail.com; s=zohoarc; b=kqkc5S3rNs72VFgyfXod7gwYee90iHUaAdxZ35sZfumJqVE3pIAkXwea8hlapdH5ihA0rwN9DKLqa/DTehtU0gamnI7VWZ/dQNr3a0w4IHNdD+VyrhC+2/yAYFn49B73QvM7LJwkYAHQZ0xxxbTm6vOq31yxV8X1e7iBVz26jkA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602908339; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=KzToa87ofMd2oqKkhow1X+M9twpTH8Mqwj5uVtG/nTc=; b=T+BRAQnChM+S+Z1eEZU/joNWV5CLKLeUt3zQXy30juFX/5iWHLivMHzlt7ZnPXDOgRND2GvmCRFbwvydI+bqUM4MK8Dbr67sXgCl0MTF3+TS4/MK61gblm+xq2I/GnrsocntsI03kV6xVfzB1r2Bq19mYuzga44Boa12r0Us/pY= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1602908339710780.9587385232177; Fri, 16 Oct 2020 21:18:59 -0700 (PDT) Received: from localhost ([::1]:52960 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdgI-0008QR-9b for importer@patchew.org; Sat, 17 Oct 2020 00:18:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40018) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXr-0006Db-LD for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:15 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5243 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXp-0007Hx-JL for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:15 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 7B2445C3648B4EC72B5E; Sat, 17 Oct 2020 12:09:56 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:48 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 13/18] migration/rdma: Add the function for dynamic page registration Date: Sat, 17 Oct 2020 12:25:43 +0800 Message-ID: <1602908748-43335-14-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add the 'qemu_rdma_registration' function, multifd send threads call it to register memory. Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/rdma.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/migration/rdma.c b/migration/rdma.c index 1af81f5..a366849 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3738,6 +3738,57 @@ out: return ret; } =20 +/* + * Dynamic page registrations for multifd RDMA threads. + */ +static int qemu_rdma_registration(void *opaque) +{ + RDMAContext *rdma =3D opaque; + RDMAControlHeader resp =3D {.type =3D RDMA_CONTROL_RAM_BLOCKS_RESULT }; + RDMALocalBlocks *local =3D &rdma->local_ram_blocks; + int reg_result_idx, i, nb_dest_blocks; + RDMAControlHeader head =3D { .len =3D 0, .repeat =3D 1 }; + int ret =3D 0; + + head.type =3D RDMA_CONTROL_RAM_BLOCKS_REQUEST; + + ret =3D qemu_rdma_exchange_send(rdma, &head, NULL, &resp, + ®_result_idx, rdma->pin_all ? + qemu_rdma_reg_whole_ram_blocks : NULL); + if (ret < 0) { + goto out; + } + + nb_dest_blocks =3D resp.len / sizeof(RDMADestBlock); + + if (local->nb_blocks !=3D nb_dest_blocks) { + rdma->error_state =3D -EINVAL; + ret =3D -1; + goto out; + } + + qemu_rdma_move_header(rdma, reg_result_idx, &resp); + memcpy(rdma->dest_blocks, + rdma->wr_data[reg_result_idx].control_curr, resp.len); + + for (i =3D 0; i < nb_dest_blocks; i++) { + network_to_dest_block(&rdma->dest_blocks[i]); + + /* We require that the blocks are in the same order */ + if (rdma->dest_blocks[i].length !=3D local->block[i].length) { + rdma->error_state =3D -EINVAL; + ret =3D -1; + goto out; + } + local->block[i].remote_host_addr =3D + rdma->dest_blocks[i].remote_host_addr; + local->block[i].remote_rkey =3D rdma->dest_blocks[i].remote_rkey; + } + +out: + return ret; +} + /* Destination: * Called via a ram_control_load_hook during the initial RAM load section = which * lists the RAMBlocks by name. This lets us know the order of the RAMBlo= cks --=20 1.8.3.1 From nobody Thu Apr 25 04:07:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602908050; cv=none; d=zohomail.com; s=zohoarc; b=TpPvoYETnhgW6Ftn8DQqPZpB2zXdaoE0BwKCWFRm6+u+qwsO9Z0gROorIW86sICj5XpyiCBGuKVxNcyJV3lJKz+Le5e7e1BB7Wj6IWoqTMHESqvzmypR0hDxLZHqFFjkuho12L9tMJtEgWXwQ8SQMMXz4vAfoRoP9qXYJhIl+u4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602908050; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=HalsY9n3zig1JiPEeh/j3lCrreks0mSSMWl1CybuoEE=; b=MlbbGrMHsM7tOX7mgsX/M3O1WKVWIGvlJDZ9WsllVk/E9IDvSOYJ6PsV6BG00SX6JWLMsLq7cqSyBxuhmQMSQMmRRMe+dEYofUKHs3KZRn26BMSapCyTz4LCiqVCLO+FQ29u6/kicsmkUeOMMYUrWRAdvisfn3egEO6oEeMMTkc= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 160290805068437.51563831582257; Fri, 16 Oct 2020 21:14:10 -0700 (PDT) Received: from localhost ([::1]:39374 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdbd-0002lY-H6 for importer@patchew.org; Sat, 17 Oct 2020 00:14:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39978) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXh-0005wQ-Sr for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:05 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5244 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXf-0007Hy-I0 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:05 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 7619A73645F7C5433B5A; Sat, 17 Oct 2020 12:09:56 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:49 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 14/18] migration/rdma: register memory for multifd RDMA channels Date: Sat, 17 Oct 2020 12:25:44 +0800 Message-ID: <1602908748-43335-15-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/multifd.c | 3 ++ migration/rdma.c | 94 +++++++++++++++++++++++++++++++++++++++++++++++++= ++-- 2 files changed, 95 insertions(+), 2 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 9439b3c..c4d90ef 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -534,6 +534,9 @@ void multifd_send_terminate_threads(Error *err) qemu_mutex_lock(&p->mutex); p->quit =3D true; qemu_sem_post(&p->sem); + if (migrate_use_rdma()) { + qemu_sem_post(&p->sem_sync); + } qemu_mutex_unlock(&p->mutex); } } diff --git a/migration/rdma.c b/migration/rdma.c index a366849..3210e6e 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3837,6 +3837,19 @@ static int rdma_load_hook(QEMUFile *f, void *opaque,= uint64_t flags, void *data) return rdma_block_notification_handle(opaque, data); =20 case RAM_CONTROL_HOOK: + if (migrate_use_multifd()) { + int i; + MultiFDRecvParams *multifd_recv_param =3D NULL; + int thread_count =3D migrate_multifd_channels(); + /* Inform dest recv_thread to poll */ + for (i =3D 0; i < thread_count; i++) { + if (get_multifd_recv_param(i, &multifd_recv_param)) { + return -1; + } + qemu_sem_post(&multifd_recv_param->sem_sync); + } + } + return qemu_rdma_registration_handle(f, opaque); =20 default: @@ -3909,6 +3922,24 @@ static int qemu_rdma_registration_stop(QEMUFile *f, = void *opaque, head.type =3D RDMA_CONTROL_RAM_BLOCKS_REQUEST; trace_qemu_rdma_registration_stop_ram(); =20 + if (migrate_use_multifd()) { + /* + * Inform the multifd channels to register memory + */ + int i; + int thread_count =3D migrate_multifd_channels(); + MultiFDSendParams *multifd_send_param =3D NULL; + for (i =3D 0; i < thread_count; i++) { + ret =3D get_multifd_send_param(i, &multifd_send_param); + if (ret) { + error_report("rdma: error getting multifd(%d)", i); + return ret; + } + + qemu_sem_post(&multifd_send_param->sem_sync); + } + } + /* * Make sure that we parallelize the pinning on both sides. * For very large guests, doing this serially takes a really @@ -3967,6 +3998,21 @@ static int qemu_rdma_registration_stop(QEMUFile *f, = void *opaque, rdma->dest_blocks[i].remote_host_addr; local->block[i].remote_rkey =3D rdma->dest_blocks[i].remote_rk= ey; } + /* Wait for all multifd channels to complete registration */ + if (migrate_use_multifd()) { + int i; + int thread_count =3D migrate_multifd_channels(); + MultiFDSendParams *multifd_send_param =3D NULL; + for (i =3D 0; i < thread_count; i++) { + ret =3D get_multifd_send_param(i, &multifd_send_param); + if (ret) { + error_report("rdma: error getting multifd(%d)", i); + return ret; + } + + qemu_sem_wait(&multifd_send_param->sem); + } + } } =20 trace_qemu_rdma_registration_stop(flags); @@ -3978,6 +4024,24 @@ static int qemu_rdma_registration_stop(QEMUFile *f, = void *opaque, goto err; } =20 + if (migrate_use_multifd()) { + /* + * Inform src send_thread to send FINISHED signal. + * Wait for multifd RDMA send threads to poll the CQE. + */ + int i; + int thread_count =3D migrate_multifd_channels(); + MultiFDSendParams *multifd_send_param =3D NULL; + for (i =3D 0; i < thread_count; i++) { + ret =3D get_multifd_send_param(i, &multifd_send_param); + if (ret < 0) { + goto err; + } + + qemu_sem_post(&multifd_send_param->sem_sync); + } + } + return 0; err: rdma->error_state =3D ret; @@ -4355,20 +4419,39 @@ static void *multifd_rdma_send_thread(void *opaque) { MultiFDSendParams *p =3D opaque; Error *local_err =3D NULL; + int ret =3D 0; + RDMAControlHeader head =3D { .len =3D 0, .repeat =3D 1 }; =20 trace_multifd_send_thread_start(p->id); if (multifd_send_initial_packet(p, &local_err) < 0) { goto out; } =20 + /* wait for semaphore notification to register memory */ + qemu_sem_wait(&p->sem_sync); + if (qemu_rdma_registration(p->rdma) < 0) { + goto out; + } + /* + * Inform the main RDMA thread to run when multifd + * RDMA thread have completed registration. + */ + qemu_sem_post(&p->sem); while (true) { + qemu_sem_wait(&p->sem_sync); qemu_mutex_lock(&p->mutex); if (p->quit) { qemu_mutex_unlock(&p->mutex); break; } qemu_mutex_unlock(&p->mutex); - qemu_sem_wait(&p->sem); + + /* Send FINISHED to the destination */ + head.type =3D RDMA_CONTROL_REGISTER_FINISHED; + ret =3D qemu_rdma_exchange_send(p->rdma, &head, NULL, NULL, NULL, = NULL); + if (ret < 0) { + return NULL; + } } =20 out: @@ -4406,15 +4489,22 @@ static void multifd_rdma_send_channel_setup(MultiFD= SendParams *p) static void *multifd_rdma_recv_thread(void *opaque) { MultiFDRecvParams *p =3D opaque; + int ret =3D 0; =20 while (true) { + qemu_sem_wait(&p->sem_sync); + qemu_mutex_lock(&p->mutex); if (p->quit) { qemu_mutex_unlock(&p->mutex); break; } qemu_mutex_unlock(&p->mutex); - qemu_sem_wait(&p->sem_sync); + ret =3D qemu_rdma_registration_handle(p->file, p->c); + if (ret < 0) { + qemu_file_set_error(p->file, ret); + break; + } } =20 qemu_mutex_lock(&p->mutex); --=20 1.8.3.1 From nobody Thu Apr 25 04:07:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602908047; cv=none; d=zohomail.com; s=zohoarc; b=TtfApT2zQAuvoQqbaXFAJPE3j1S5BmAi/VQRZdjA93bTwdIR624zST9KV2XMzr8x64lLRPcMjw8huOEjliB+58eJesfhIPhwG71woSiIxIwGYpkYEABETZx6HBDeZwXd8WmK0a5B7YrHonn9naPbxzq8jmqjo+E8f1XrLpcGAM0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602908047; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=cjb/6ootlouoLkeNMfCb6MonvvgjcybBsZ+rBzq6Lvg=; b=eee/WiXALRaQuE3eBIhJvRpiPYNgjgn/dUmL0mlK3Wih/LP+IIVqF9Gbi5TS7lbPp2UK0SFzINKdPzJGRt0Qk5LY/SbBTgnzBlQJsMObKKKBnfRymBtlRSvc3cZuAwYyoZ9X8y7pvKhODt0b1QNl9pndNUKq6p7Z/Ig5RpcgR6I= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1602908047480605.1743512383769; Fri, 16 Oct 2020 21:14:07 -0700 (PDT) Received: from localhost ([::1]:39060 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdba-0002ds-7L for importer@patchew.org; Sat, 17 Oct 2020 00:14:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40070) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXu-0006GX-3q for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:19 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5246 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXr-0007Ig-5U for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:17 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 83D996FE1DD8D46EF0EE; Sat, 17 Oct 2020 12:09:56 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:50 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 15/18] migration/rdma: only register the memory for multifd channels Date: Sat, 17 Oct 2020 12:25:45 +0800 Message-ID: <1602908748-43335-16-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" All data is sent by multifd Channels, so we only register its for multifd channels and main channel don't register its. Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/rdma.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/rdma.c b/migration/rdma.c index 3210e6e..d5d6364 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3938,6 +3938,12 @@ static int qemu_rdma_registration_stop(QEMUFile *f, = void *opaque, =20 qemu_sem_post(&multifd_send_param->sem_sync); } + + /* + * Use multifd to migrate, we only register memory for + * multifd RDMA channel and main channel don't register it. + */ + goto wait_reg_complete; } =20 /* @@ -3998,6 +4004,8 @@ static int qemu_rdma_registration_stop(QEMUFile *f, v= oid *opaque, rdma->dest_blocks[i].remote_host_addr; local->block[i].remote_rkey =3D rdma->dest_blocks[i].remote_rk= ey; } + +wait_reg_complete: /* Wait for all multifd channels to complete registration */ if (migrate_use_multifd()) { int i; --=20 1.8.3.1 From nobody Thu Apr 25 04:07:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602908858; cv=none; d=zohomail.com; s=zohoarc; b=mYuw9pPjo6Tihv1xVPDHjEZAdUuGOAVzfNAKkt+lsJ2dLfj8yhCrzRpY8GHWcZYLIc47rBbSL0qsHqlFVTjeK5PtcMuPNuGT2AEBmmvvB3soPfINtIzCYB6WbZZQyBXM097gLq+MJRohbzXTaV2eoNS/FF8QtNCHi658yAEstoA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602908858; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=kWDDpcrrVeYXpDXagzLXCLhcodpmO1E6HO1JaDLZa/U=; b=V97U4RzkN19jvdCfYxbbWzg6jgAG3Bx1ume94RGYLXepT85aTKXYDa+op4mT/6GF5x4ogRvt3BZ9QWMhcw+jMLQMuDMunM5MVSmJ+w1EFgpyvLAifEeQZZuS2cdVrrGdmqG11xblfMKFPfHznh2LDFlpSeTsA6x6sKL836Iwe6Q= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1602908858939414.49450033249457; Fri, 16 Oct 2020 21:27:38 -0700 (PDT) Received: from localhost ([::1]:53138 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdof-0003U2-Qg for importer@patchew.org; Sat, 17 Oct 2020 00:27:37 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40102) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXy-0006JE-1G for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:23 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5247 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXs-0007Iz-66 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:21 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 8C29B7C90A39DFCFFA82; Sat, 17 Oct 2020 12:09:56 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:50 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 16/18] migration/rdma: add rdma_channel into Migrationstate field Date: Sat, 17 Oct 2020 12:25:46 +0800 Message-ID: <1602908748-43335-17-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Multifd RDMA is need to poll when we send data, record it. Signed-off-by: Chuan Zheng --- migration/migration.c | 1 + migration/migration.h | 1 + migration/rdma.c | 14 ++++++++++++++ 3 files changed, 16 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index 7061410..1ec1dc9 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1892,6 +1892,7 @@ void migrate_init(MigrationState *s) s->migration_thread_running =3D false; s->enabled_rdma_migration =3D false; s->host_port =3D NULL; + s->rdma_channel =3D 0; error_free(s->error); s->error =3D NULL; s->hostname =3D NULL; diff --git a/migration/migration.h b/migration/migration.h index fea63de..5676b23 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -272,6 +272,7 @@ struct MigrationState =20 /* Need by Multi-RDMA */ char *host_port; + int rdma_channel; }; =20 void migrate_set_state(int *state, int old_state, int new_state); diff --git a/migration/rdma.c b/migration/rdma.c index d5d6364..327f80f 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -183,6 +183,20 @@ typedef struct { } RDMAWorkRequestData; =20 /* + * Get the multifd RDMA channel used to send data. + */ +static int get_multifd_RDMA_channel(void) +{ + int thread_count =3D migrate_multifd_channels(); + MigrationState *s =3D migrate_get_current(); + + s->rdma_channel++; + s->rdma_channel %=3D thread_count; + + return s->rdma_channel; +} + +/* * Negotiate RDMA capabilities during connection-setup time. */ typedef struct { --=20 1.8.3.1 From nobody Thu Apr 25 04:07:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602908200; cv=none; d=zohomail.com; s=zohoarc; b=EmAZWDLYhIAGWusOG5x6A75dCdaEHqxVGvb1u06dvu1n2GyJZMvLCgOWGw2TggnR3C/Md6vPHwLlMN2BMsC9xaxcJzTEHDgfBaUMMJ5nxydQxoz6BFJVzaWYh/gTt+AZl+e4+FegmLW4JoYrcOzKeUbqANgd69/qGDI7yXDhqbI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602908200; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=WgN8uU35gIT7QaZiiCSwUgvt5L8t86gIlg2NkmglzeU=; b=TizGBLybWRyi77NJjvkckst9dJXvAaO+3anCVpa4b2RoDNx/TKvS5ACbNs725QCr2npzOOhFA39cElvS130L+u1gGQ34YOvr9mQoclnjPgiQlaLD3DPQdt1cJnh7jzx+3Lj7rGao8nKQRcKgAICDlJJus9MwbErTbQKSWOZltS8= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1602908200279594.285406406511; Fri, 16 Oct 2020 21:16:40 -0700 (PDT) Received: from localhost ([::1]:46638 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTde3-0005qZ-1j for importer@patchew.org; Sat, 17 Oct 2020 00:16:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39988) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXk-000618-BT for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:08 -0400 Received: from szxga07-in.huawei.com ([45.249.212.35]:36936 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXg-0007I3-Q8 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:08 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 8252E61335AD4405501E; Sat, 17 Oct 2020 12:10:01 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:51 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 17/18] migration/rdma: send data for both rdma-pin-all and NOT rdma-pin-all mode Date: Sat, 17 Oct 2020 12:25:47 +0800 Message-ID: <1602908748-43335-18-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.35; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:10:01 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/rdma.c | 67 +++++++++++++++++++++++++++++++++++++++++++++++++++-= ---- 1 file changed, 62 insertions(+), 5 deletions(-) diff --git a/migration/rdma.c b/migration/rdma.c index 327f80f..519fa7a 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -2001,6 +2001,20 @@ static int qemu_rdma_write_one(QEMUFile *f, RDMACont= ext *rdma, .repeat =3D 1, }; =20 + /* use multifd to send data */ + if (migrate_use_multifd()) { + int channel =3D get_multifd_RDMA_channel(); + int ret =3D 0; + MultiFDSendParams *multifd_send_param =3D NULL; + ret =3D get_multifd_send_param(channel, &multifd_send_param); + if (ret) { + error_report("rdma: error getting multifd_send_param(%d)", cha= nnel); + return -EINVAL; + } + rdma =3D (RDMAContext *)multifd_send_param->rdma; + block =3D &(rdma->local_ram_blocks.block[current_index]); + } + retry: sge.addr =3D (uintptr_t)(block->local_host_addr + (current_addr - block->offset)); @@ -2196,6 +2210,27 @@ retry: return 0; } =20 +static int multifd_rdma_write_flush(void) +{ + /* The multifd RDMA threads send data */ + MultiFDSendParams *multifd_send_param =3D NULL; + RDMAContext *rdma =3D NULL; + MigrationState *s =3D migrate_get_current(); + int ret =3D 0; + + ret =3D get_multifd_send_param(s->rdma_channel, + &multifd_send_param); + if (ret) { + error_report("rdma: error getting multifd_send_param(%d)", + s->rdma_channel); + return ret; + } + rdma =3D (RDMAContext *)(multifd_send_param->rdma); + rdma->nb_sent++; + + return ret; +} + /* * Push out any unwritten RDMA operations. * @@ -2218,8 +2253,15 @@ static int qemu_rdma_write_flush(QEMUFile *f, RDMACo= ntext *rdma) } =20 if (ret =3D=3D 0) { - rdma->nb_sent++; - trace_qemu_rdma_write_flush(rdma->nb_sent); + if (migrate_use_multifd()) { + ret =3D multifd_rdma_write_flush(); + if (ret) { + return ret; + } + } else { + rdma->nb_sent++; + trace_qemu_rdma_write_flush(rdma->nb_sent); + } } =20 rdma->current_length =3D 0; @@ -4061,6 +4103,7 @@ wait_reg_complete: } =20 qemu_sem_post(&multifd_send_param->sem_sync); + qemu_sem_wait(&multifd_send_param->sem); } } =20 @@ -4443,6 +4486,7 @@ static void *multifd_rdma_send_thread(void *opaque) Error *local_err =3D NULL; int ret =3D 0; RDMAControlHeader head =3D { .len =3D 0, .repeat =3D 1 }; + RDMAContext *rdma =3D p->rdma; =20 trace_multifd_send_thread_start(p->id); if (multifd_send_initial_packet(p, &local_err) < 0) { @@ -4451,7 +4495,7 @@ static void *multifd_rdma_send_thread(void *opaque) =20 /* wait for semaphore notification to register memory */ qemu_sem_wait(&p->sem_sync); - if (qemu_rdma_registration(p->rdma) < 0) { + if (qemu_rdma_registration(rdma) < 0) { goto out; } /* @@ -4467,13 +4511,26 @@ static void *multifd_rdma_send_thread(void *opaque) break; } qemu_mutex_unlock(&p->mutex); - + /* To complete polling(CQE) */ + while (rdma->nb_sent) { + ret =3D qemu_rdma_block_for_wrid(rdma, RDMA_WRID_RDMA_WRITE, N= ULL); + if (ret < 0) { + error_report("multifd RDMA migration: " + "complete polling error!"); + return NULL; + } + } /* Send FINISHED to the destination */ head.type =3D RDMA_CONTROL_REGISTER_FINISHED; - ret =3D qemu_rdma_exchange_send(p->rdma, &head, NULL, NULL, NULL, = NULL); + ret =3D qemu_rdma_exchange_send(rdma, &head, NULL, NULL, NULL, NUL= L); if (ret < 0) { + error_report("multifd RDMA migration: " + "sending remote error!"); return NULL; } + + /* sync main thread */ + qemu_sem_post(&p->sem); } =20 out: --=20 1.8.3.1 From nobody Thu Apr 25 04:07:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1602908744; cv=none; d=zohomail.com; s=zohoarc; b=aCI5cIeeTkzo5tS2CMl0zGeqSlv7c5f+Z6RB0gnMCFl9hN/WrGMLwWwhcfjlndseY6UC8SvEqBTcHl3EYuCmjP5ihngu1ZpZJZkS9nLwVaCKPQl+NUiFGZ0fg5sVEixZgB44rPVcapwc1J5SeJewwAsTZNplaZa3FQSK8jX57II= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602908744; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=0f1boutGo9oCPvbym1g7TCE5Nda1LJQlDqVvGETtGTs=; b=ULQsarytL/YYv4uObYVQuTiu0XbyUqrQFchTqXYxc73xxYGd1mcAaeojDSkFifRjCovuonw8/CqnDCjbUGpDsWvat5rEWaYpioog2VrvPR41Lhvtq813jY6gDZb5ZxBpzXeeaEIx55kU/QuyxihEKBBvNR146sLVluH/GBQl1HA= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1602908744324976.6070474272761; Fri, 16 Oct 2020 21:25:44 -0700 (PDT) Received: from localhost ([::1]:46638 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdmo-0000hq-Uz for importer@patchew.org; Sat, 17 Oct 2020 00:25:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39984) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXi-0005xv-SW for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:06 -0400 Received: from szxga07-in.huawei.com ([45.249.212.35]:36940 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXg-0007I4-NY for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:06 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 87C02B2F8DAD10F3D499; Sat, 17 Oct 2020 12:10:01 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:51 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 18/18] migration/rdma: RDMA cleanup for multifd migration Date: Sat, 17 Oct 2020 12:25:48 +0800 Message-ID: <1602908748-43335-19-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.35; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:10:01 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Chuan Zheng --- migration/multifd.c | 6 ++++++ migration/multifd.h | 1 + migration/rdma.c | 16 +++++++++++++++- 3 files changed, 22 insertions(+), 1 deletion(-) diff --git a/migration/multifd.c b/migration/multifd.c index c4d90ef..f548122 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -574,6 +574,9 @@ void multifd_save_cleanup(void) p->packet_len =3D 0; g_free(p->packet); p->packet =3D NULL; +#ifdef CONFIG_RDMA + multifd_rdma_cleanup(p->rdma); +#endif multifd_send_state->ops->send_cleanup(p, &local_err); if (local_err) { migrate_set_error(migrate_get_current(), local_err); @@ -1017,6 +1020,9 @@ int multifd_load_cleanup(Error **errp) p->packet_len =3D 0; g_free(p->packet); p->packet =3D NULL; +#ifdef CONFIG_RDMA + multifd_rdma_cleanup(p->rdma); +#endif multifd_recv_state->ops->recv_cleanup(p); } qemu_sem_destroy(&multifd_recv_state->sem_sync); diff --git a/migration/multifd.h b/migration/multifd.h index ec9e897..6fddd4e 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -183,6 +183,7 @@ typedef struct { =20 #ifdef CONFIG_RDMA MultiFDSetup *multifd_rdma_setup(void); +void multifd_rdma_cleanup(void *opaque); #endif void multifd_send_terminate_threads(Error *err); int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp); diff --git a/migration/rdma.c b/migration/rdma.c index 519fa7a..89bf54b 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -2368,7 +2368,7 @@ static void qemu_rdma_cleanup(RDMAContext *rdma) { int idx; =20 - if (rdma->cm_id && rdma->connected) { + if (rdma->channel && rdma->cm_id && rdma->connected) { if ((rdma->error_state || migrate_get_current()->state =3D=3D MIGRATION_STATUS_CANCELLI= NG) && !rdma->received_error) { @@ -4609,6 +4609,20 @@ static MultiFDSetup multifd_rdma_ops =3D { .recv_channel_setup =3D multifd_rdma_recv_channel_setup }; =20 +void multifd_rdma_cleanup(void *opaque) +{ + RDMAContext *rdma =3D (RDMAContext *)opaque; + + if (!migrate_use_rdma()) { + return; + } + + rdma->listen_id =3D NULL; + rdma->channel =3D NULL; + qemu_rdma_cleanup(rdma); + g_free(rdma); +} + MultiFDSetup *multifd_rdma_setup(void) { MultiFDSetup *rdma_ops; --=20 1.8.3.1