From nobody Sat May 18 09:01:19 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1623075202; cv=none; d=zohomail.com; s=zohoarc; b=XUam0dNkQtFbv+7bKpqC+caLINC2N5yhUfdUlgacWcyh5zKgzDkbMBA3QQgfTkaKtZzDTiE6EoXlSwfxYRL9vMcJrv7mSZfslfKiOQfm8Avn/3heTfO9VP1tdyZwwwqyxQ7C47ZFTCYMNZH4y88nJ/fRNS5GggseefPckbaTg48= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1623075202; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=UvyLiFdh5FmurtjO4PsTQPFP/pe6OxezOKTU+Dr1A10=; b=lh7qV9MKPNuBg1YITtV4SF14KBVg9XF0ctjxmZwCzYkIemHiVKHUV3vJeQmpmGD6xtdcTvps+zkp17RaS93iv+ggeTs7Dm4llhLINCwrX+qKHtM3bq13yq+D6bN92/X729YHzEOeu4XX3EcGmXIJDBWOkKLx1WRtWdTPRM16YPA= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1623075201726340.1407590015376; Mon, 7 Jun 2021 07:13:21 -0700 (PDT) Received: from localhost ([::1]:42322 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lqG0F-0003wb-2T for importer@patchew.org; Mon, 07 Jun 2021 10:13:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54300) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lqFzS-0003GW-BA for qemu-devel@nongnu.org; Mon, 07 Jun 2021 10:12:30 -0400 Received: from mail.kingsoft.com ([114.255.44.145]:42942) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lqFzK-00043P-Uu for qemu-devel@nongnu.org; Mon, 07 Jun 2021 10:12:29 -0400 Received: from mail.kingsoft.com (localhost [10.88.1.79]) (using TLS with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mail.kingsoft.com (SMG-2-NODE-85) with SMTP id 91.75.00287.EA52EB06; Mon, 7 Jun 2021 21:57:02 +0800 (HKT) Received: from KSbjmail3.kingsoft.cn (10.88.1.78) by KSBJMAIL4.kingsoft.cn (10.88.1.79) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.14; Mon, 7 Jun 2021 21:57:02 +0800 Received: from KSbjmail3.kingsoft.cn ([fe80::a143:8393:3ff1:cd3]) by KSBJMAIL3.kingsoft.cn ([fe80::a143:8393:3ff1:cd3%10]) with mapi id 15.01.2176.014; Mon, 7 Jun 2021 21:57:02 +0800 X-AuditID: 0a580155-3d1ff7000005011f-0d-60be25ae49a8 From: =?gb2312?B?TElaSEFPWElOMSBbwO7V1fbOXQ==?= To: "qemu-devel@nongnu.org" , "quintela@redhat.com" , "dgilbert@redhat.com" Subject: [PATCH] migration/rdma: Use huge page register VM memory Thread-Topic: [PATCH] migration/rdma: Use huge page register VM memory Thread-Index: AddbpN+kEXe4Oe1WQuqS3KFy+fOdUQ== Date: Mon, 7 Jun 2021 13:57:02 +0000 Message-ID: <51819991cecb42f6a619768bc61d0bfd@kingsoft.com> Accept-Language: zh-CN, en-US Content-Language: zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.88.1.106] Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrBLMWRmVeSWpSXmKPExsXCFcHor7tOdV+CwbXLMha92+6xWxzv3cFi cWdLH5MDs8eTa5uZPN7vu8oWwBTFZZOSmpNZllqkb5fAlfGxYzVzwTOJiu/N/9kaGKdIdDFy ckgImEi0t69m6mLk4hASmM4ksX7fcxYI5zmjxMlplxlBqoQEdjFKTFjJBmKzCXhKfFp1hg2k SESgn1HixvoHYO3MAj1MEr/ftLKCVAkLOEosXz+BBcQWEXCT+LW6E8rWk7i38jZQNwcHi4CK xOurSSBhXgFriYW7X4EtYxSQlZj26D4TiM0sIC4xd9osVohTBSSW7DnPDGGLSrx8/I8VZIyE gLzE6o/CEOVaEvMafkO1KkpM6X7IDjFeUOLkzCcsExhFZiGZOgtJyywkLbOQtCxgZFnFyFKc m260iRES+KE7GGc0fdQ7xMjEwXiIUYKDWUmE10tmT4IQb0piZVVqUX58UWlOavEhRmkOFiVx 3rVGOxKEBNITS1KzU1MLUotgskwcnFINTLymH9KFn5r1pz6bktLaz75j068nClFMPTJ/9Xeb 6Rieu3YgPiqLx/aNq9Bk1RnmMbNsJ0+/ndqmcG35j2SevvN3krhMLZ2it35odm9Y2PvSlS8u aUn70xv7zmvPeLJCWnebb+Ur/yvPzTYlC/3otP12/esXlSQLxdlfsjY9Pn6xY53+x8sJZQsm py3Mmf5/01aGo548jsyPr0xNuLBUcpLT7sKUJY/V02ujlP1FF/x7Fvf99fZzjEvqd4lpiBqp KiQKGGbVnp1kGXK9I1RV5eHvCfPe3g599qu4Z41P3p5TybsXcZu2XH6wq+Hq6VDDvbPmbyvc d41Dwehb0sajXlwB5wRveuzjWMOi99Jvx3IlluKMREMt5qLiRABbqqSn6wIAAA== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=114.255.44.145; envelope-from=LIZHAOXIN1@kingsoft.com; helo=mail.kingsoft.com X-Spam_score_int: 13 X-Spam_score: 1.3 X-Spam_bar: + X-Spam_report: (1.3 / 5.0 requ) BAYES_00=-1.9, CHARSET_FARAWAY_HEADER=3.2, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?gb2312?B?TElaSEFPWElOMSBbwO7V1fbOXQ==?= , =?gb2312?B?c3VuaGFvMiBby+/qu10=?= , =?gb2312?B?REVOR0xJTldFTiBbtcvB1s7EXQ==?= , =?gb2312?B?WUFOR0ZFTkcxIFvR7rflXQ==?= Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" When using libvirt for RDMA live migration, if the VM memory is too large, it will take a lot of time to deregister the VM at the source side, resulti= ng in a long downtime (VM 64G, deregister vm time is about 400ms). =20 Although the VM's memory uses 2M huge pages, the MLNX driver still uses 4K pages for pin memory, as well as for unpin. So we use huge pages to skip the process of pin memory and unpin memory to reduce downtime. =20 The test environment: kernel: linux-5.12 MLNX: ConnectX-4 LX libvirt command: virsh migrate --live --p2p --persistent --copy-storage-inc --listen-address= \ 0.0.0.0 --rdma-pin-all --migrateuri rdma://192.168.0.2 [VM] qemu+tcp://192.= 168.0.2/system =20 Signed-off-by: lizhaoxin diff --git a/migration/rdma.c b/migration/rdma.c index 1cdb4561f3..9823449297 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -1123,13 +1123,26 @@ static int qemu_rdma_reg_whole_ram_blocks(RDMAConte= xt *rdma) RDMALocalBlocks *local =3D &rdma->local_ram_blocks; =20 for (i =3D 0; i < local->nb_blocks; i++) { - local->block[i].mr =3D - ibv_reg_mr(rdma->pd, - local->block[i].local_host_addr, - local->block[i].length, - IBV_ACCESS_LOCAL_WRITE | - IBV_ACCESS_REMOTE_WRITE - ); + if (strcmp(local->block[i].block_name,"pc.ram") =3D=3D 0) { + local->block[i].mr =3D + ibv_reg_mr(rdma->pd, + local->block[i].local_host_addr, + local->block[i].length, + IBV_ACCESS_LOCAL_WRITE | + IBV_ACCESS_REMOTE_WRITE | + IBV_ACCESS_ON_DEMAND | + IBV_ACCESS_HUGETLB + ); + } else { + local->block[i].mr =3D + ibv_reg_mr(rdma->pd, + local->block[i].local_host_addr, + local->block[i].length, + IBV_ACCESS_LOCAL_WRITE | + IBV_ACCESS_REMOTE_WRITE + ); + } + if (!local->block[i].mr) { perror("Failed to register local dest ram block!\n"); break;