From nobody Wed Feb 11 02:07:14 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1500533812954248.99821881359526; Wed, 19 Jul 2017 23:56:52 -0700 (PDT) Received: from localhost ([::1]:36345 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dY5OF-0007sA-Ek for importer@patchew.org; Thu, 20 Jul 2017 02:56:51 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41314) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dY5KU-0004KH-Lu for qemu-devel@nongnu.org; Thu, 20 Jul 2017 02:53:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dY5KR-0005LC-7W for qemu-devel@nongnu.org; Thu, 20 Jul 2017 02:52:58 -0400 Received: from mailout3.w1.samsung.com ([210.118.77.13]:62734) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dY5KQ-0005Jn-Uh for qemu-devel@nongnu.org; Thu, 20 Jul 2017 02:52:55 -0400 Received: from eucas1p2.samsung.com (unknown [182.198.249.207]) by mailout3.w1.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTP id <0OTD00BYXN42FK30@mailout3.w1.samsung.com> for qemu-devel@nongnu.org; Thu, 20 Jul 2017 07:52:50 +0100 (BST) Received: from eusmges5.samsung.com (unknown [203.254.199.245]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20170720065250eucas1p143a8bb0a5f8cf6329b8470a542289d45~S91T6oyGf1113611136eucas1p1U; Thu, 20 Jul 2017 06:52:50 +0000 (GMT) Received: from eucas1p1.samsung.com ( [182.198.249.206]) by eusmges5.samsung.com (EUCPMTA) with SMTP id B4.2A.25577.14350795; Thu, 20 Jul 2017 07:52:49 +0100 (BST) Received: from eusmgms2.samsung.com (unknown [182.198.249.180]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20170720065249eucas1p2cc3e779e2cde2125f90ae25ff03ac646~S91TLAH0u2953229532eucas1p2K; Thu, 20 Jul 2017 06:52:49 +0000 (GMT) Received: from eusync4.samsung.com ( [203.254.199.214]) by eusmgms2.samsung.com (EUCPMTA) with SMTP id 10.AD.20206.14350795; Thu, 20 Jul 2017 07:52:49 +0100 (BST) Received: from aperevalov-ubuntu.rnd.samsung.ru ([106.109.129.199]) by eusync4.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTPA id <0OTD005THN3STW10@eusync4.samsung.com>; Thu, 20 Jul 2017 07:52:49 +0100 (BST) X-AuditID: cbfec7f5-f792f6d0000063e9-61-59705341bd1a From: Alexey Perevalov To: qemu-devel@nongnu.org Date: Thu, 20 Jul 2017 09:52:34 +0300 Message-id: <1500533554-5779-4-git-send-email-a.perevalov@samsung.com> X-Mailer: git-send-email 1.9.1 In-reply-to: <1500533554-5779-1-git-send-email-a.perevalov@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrAIsWRmVeSWpSXmKPExsWy7djPc7qOwQWRBvc6rSzm3j3PYtG77R67 xbTPt9ktrrT/ZLfYsv8bu8Xx3h0sFne29DE5sHs8ubaZyeP9vqtsHn1bVjEGMEdx2aSk5mSW pRbp2yVwZRz4/ZSl4IF5xbof55kbGO9rdTFyckgImEhcOb+KEcIWk7hwbz1bFyMXh5DAUkaJ XU2TWUESQgKfGSUOnDeHadi15CUrRNEyRokZ2xZBFXUzSVw869DFyMHBJmAgse+eLUhYREBS 4nfXaWaQemaBhYwSDfN+M4EkhAXsJTYu/coOYrMIqEosuH+SBcTmFXCTaJm7nA1imZzEyWMQ R3AKuEss+3EEbJCEwHM2if9LLzOBLJMQkJXYdIAZot5FYsaJuSwQtrDEq+Nb2CFsGYnLk7tZ IHrbGSW6d3ayQjgTGCXOTP8LVWUvcermVbDrmAX4JCZtm84MsYBXoqNNCKLEQ6Lp2CqockeJ tavOMEFCYhajxLp5K5kmMMosYGRYxSiSWlqcm55abKpXnJhbXJqXrpecn7uJERivp/8d/7qD cekxq0OMAhyMSjy8Fhn5kUKsiWXFlbmHGCU4mJVEeGUDCyKFeFMSK6tSi/Lji0pzUosPMUpz sCiJ83KduhYhJJCeWJKanZpakFoEk2Xi4JRqYBR9dHXrnGc/eP95pGm6/cybrVq0pVy79Mjl CpVEe402Vb1jVmd2xv0V+RFjONNByLftxnxzI/uVl5O5r4isviH7Jmp/uOG+8h7F9qXlv974 5WuahQtI3T3eJPbrxGLGbR85N0ek5OupVi5VPPfAhu+XUe0Uabd4BebSs9d2Lr3eUahuP/lp qhJLcUaioRZzUXEiAPXWmv7TAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrHLMWRmVeSWpSXmKPExsVy+t/xa7qOwQWRBu0vuCzm3j3PYtG77R67 xbTPt9ktrrT/ZLfYsv8bu8Xx3h0sFne29DE5sHs8ubaZyeP9vqtsHn1bVjEGMEe52WSkJqak Fimk5iXnp2TmpdsqhYa46VooKeQl5qbaKkXo+oYEKSmUJeaUAnlGBmjAwTnAPVhJ3y7BLePA 76csBQ/MK9b9OM/cwHhfq4uRk0NCwERi15KXrBC2mMSFe+vZQGwhgSWMEs+nK0HYvUwS+y/Z dzFycLAJGEjsu2cLEhYRkJT43XWauYuRi4NZYCGjxLvlT5hAEsIC9hIbl35lB7FZBFQlFtw/ yQJi8wq4SbTMXc4GsUtO4uSxyWB7OQXcJZb9OMIMsctNYsOh5SwTGHkXMDKsYhRJLS3OTc8t NtIrTswtLs1L10vOz93ECAzbbcd+btnB2PUu+BCjAAejEg+vRUZ+pBBrYllxZe4hRgkOZiUR XtnAgkgh3pTEyqrUovz4otKc1OJDjKZAR01klhJNzgfGVF5JvKGJobmloZGxhYW5kZGSOO/U D1fChQTSE0tSs1NTC1KLYPqYODilGhi9U/dvWb5t2TrGUpfX1y+9m5nZf+1Ua1i0o2T3j4xL EzoOMq789s3tUdXvK5MK/srqsdi/ffnt9fWM6xIHpNbqG03M/6S4pfbKQ+UPnXGxckpux9/n BE14o3frzyT2uprd/3rmbvdXWnKu2DSmT/TkSqvEsgnZOz7HT5gcuO3jEmtLq2v/+pqvK7EU ZyQaajEXFScCAP0qAuRxAgAA X-MTR: 20000000000000000@CPGS X-CMS-MailID: 20170720065249eucas1p2cc3e779e2cde2125f90ae25ff03ac646 X-Msg-Generator: CA X-Sender-IP: 182.198.249.180 X-Local-Sender: =?UTF-8?B?QWxleGV5IFBlcmV2YWxvdhtTUlItVmlydHVhbGl6YXRpb24g?= =?UTF-8?B?TGFiG+yCvOyEseyghOyekBtTZW5pb3IgRW5naW5lZXI=?= X-Global-Sender: =?UTF-8?B?QWxleGV5IFBlcmV2YWxvdhtTUlItVmlydHVhbGl6YXRpb24g?= =?UTF-8?B?TGFiG1NhbXN1bmcgRWxlY3Ryb25pY3MbU2VuaW9yIEVuZ2luZWVy?= X-Sender-Code: =?UTF-8?B?QzEwG0NJU0hRG0MxMEdEMDFHRDAxMDE1NA==?= CMS-TYPE: 201P X-HopCount: 7 X-CMS-RootMailID: 20170720065249eucas1p2cc3e779e2cde2125f90ae25ff03ac646 X-RootMTR: 20170720065249eucas1p2cc3e779e2cde2125f90ae25ff03ac646 References: <1500533554-5779-1-git-send-email-a.perevalov@samsung.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 210.118.77.13 Subject: [Qemu-devel] [PATCH v8 3/3] migration: add bitmap for received page X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: heetae82.ahn@samsung.com, quintela@redhat.com, dgilbert@redhat.com, peterx@redhat.com, Alexey Perevalov , i.maximets@samsung.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch adds ability to track down already received pages, it's necessary for calculation vCPU block time in postcopy migration feature, maybe for restore after postcopy migration failure. Also it's necessary to solve shared memory issue in postcopy livemigration. Information about received pages will be transferred to the software virtual bridge (e.g. OVS-VSWITCHD), to avoid fallocate (unmap) for already received pages. fallocate syscall is required for remmaped shared memory, due to remmaping itself blocks ioctl(UFFDIO_COPY, ioctl in this case will end with EEXIT error (struct page is exists after remmap). Bitmap is placed into RAMBlock as another postcopy/precopy related bitmaps. Reviewed-by: Peter Xu Signed-off-by: Alexey Perevalov --- include/exec/ram_addr.h | 10 ++++++++++ migration/postcopy-ram.c | 16 +++++++++++----- migration/ram.c | 36 ++++++++++++++++++++++++++++++++++++ migration/ram.h | 5 +++++ 4 files changed, 62 insertions(+), 5 deletions(-) diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h index c04f4f6..bb902bb 100644 --- a/include/exec/ram_addr.h +++ b/include/exec/ram_addr.h @@ -47,6 +47,8 @@ struct RAMBlock { * of the postcopy phase */ unsigned long *unsentmap; + /* bitmap of already received pages in postcopy */ + unsigned long *receivedmap; }; =20 static inline bool offset_in_ramblock(RAMBlock *b, ram_addr_t offset) @@ -60,6 +62,14 @@ static inline void *ramblock_ptr(RAMBlock *block, ram_ad= dr_t offset) return (char *)block->host + offset; } =20 +static inline unsigned long int ramblock_recv_bitmap_offset(void *host_add= r, + RAMBlock *rb) +{ + uint64_t host_addr_offset =3D + (uint64_t)(uintptr_t)(host_addr - (void *)rb->host); + return host_addr_offset >> TARGET_PAGE_BITS; +} + long qemu_getrampagesize(void); unsigned long last_ram_page(void); RAMBlock *qemu_ram_alloc_from_file(ram_addr_t size, MemoryRegion *mr, diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index be497bb..276ce12 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -560,22 +560,27 @@ int postcopy_ram_enable_notify(MigrationIncomingState= *mis) } =20 static int qemu_ufd_copy_ioctl(int userfault_fd, void *host_addr, - void *from_addr, uint64_t pagesize) + void *from_addr, uint64_t pagesize, RAMBloc= k *rb) { + int ret; if (from_addr) { struct uffdio_copy copy_struct; copy_struct.dst =3D (uint64_t)(uintptr_t)host_addr; copy_struct.src =3D (uint64_t)(uintptr_t)from_addr; copy_struct.len =3D pagesize; copy_struct.mode =3D 0; - return ioctl(userfault_fd, UFFDIO_COPY, ©_struct); + ret =3D ioctl(userfault_fd, UFFDIO_COPY, ©_struct); } else { struct uffdio_zeropage zero_struct; zero_struct.range.start =3D (uint64_t)(uintptr_t)host_addr; zero_struct.range.len =3D pagesize; zero_struct.mode =3D 0; - return ioctl(userfault_fd, UFFDIO_ZEROPAGE, &zero_struct); + ret =3D ioctl(userfault_fd, UFFDIO_ZEROPAGE, &zero_struct); + } + if (!ret) { + ramblock_recv_bitmap_set(host_addr, rb); } + return ret; } =20 /* @@ -592,7 +597,7 @@ int postcopy_place_page(MigrationIncomingState *mis, vo= id *host, void *from, * which would be slightly cheaper, but we'd have to be careful * of the order of updating our page state. */ - if (qemu_ufd_copy_ioctl(mis->userfault_fd, host, from, pagesize)) { + if (qemu_ufd_copy_ioctl(mis->userfault_fd, host, from, pagesize, rb)) { int e =3D errno; error_report("%s: %s copy host: %p from: %p (size: %zd)", __func__, strerror(e), host, from, pagesize); @@ -614,7 +619,8 @@ int postcopy_place_page_zero(MigrationIncomingState *mi= s, void *host, trace_postcopy_place_page_zero(host); =20 if (qemu_ram_pagesize(rb) =3D=3D getpagesize()) { - if (qemu_ufd_copy_ioctl(mis->userfault_fd, host, NULL, getpagesize= ())) { + if (qemu_ufd_copy_ioctl(mis->userfault_fd, host, NULL, getpagesize= (), + rb)) { int e =3D errno; error_report("%s: %s zero host: %p", __func__, strerror(e), host); diff --git a/migration/ram.c b/migration/ram.c index 9cc1b17..107ee9d 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -147,6 +147,32 @@ out: return ret; } =20 +static void ramblock_recv_map_init(void) +{ + RAMBlock *rb; + + RAMBLOCK_FOREACH(rb) { + assert(!rb->receivedmap); + rb->receivedmap =3D bitmap_new(rb->max_length >> TARGET_PAGE_BITS); + } +} + +int ramblock_recv_bitmap_test(void *host_addr, RAMBlock *rb) +{ + return test_bit(ramblock_recv_bitmap_offset(host_addr, rb), + rb->receivedmap); +} + +void ramblock_recv_bitmap_set(void *host_addr, RAMBlock *rb) +{ + set_bit_atomic(ramblock_recv_bitmap_offset(host_addr, rb), rb->receive= dmap); +} + +void ramblock_recv_bitmap_clear(void *host_addr, RAMBlock *rb) +{ + clear_bit(ramblock_recv_bitmap_offset(host_addr, rb), rb->receivedmap); +} + /* * An outstanding page request, on the source, having been received * and queued @@ -1793,6 +1819,8 @@ int ram_discard_range(const char *rbname, uint64_t st= art, size_t length) goto err; } =20 + bitmap_clear(rb->receivedmap, start >> TARGET_PAGE_BITS, + length >> TARGET_PAGE_BITS); ret =3D ram_block_discard_range(rb, start, length); =20 err: @@ -2324,13 +2352,20 @@ static int ram_load_setup(QEMUFile *f, void *opaque) { xbzrle_load_setup(); compress_threads_load_setup(); + ramblock_recv_map_init(); return 0; } =20 static int ram_load_cleanup(void *opaque) { + RAMBlock *rb; xbzrle_load_cleanup(); compress_threads_load_cleanup(); + + RAMBLOCK_FOREACH(rb) { + g_free(rb->receivedmap); + rb->receivedmap =3D NULL; + } return 0; } =20 @@ -2545,6 +2580,7 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) ret =3D -EINVAL; break; } + ramblock_recv_bitmap_set(host, block); trace_ram_load_loop(block->idstr, (uint64_t)addr, flags, host); } =20 diff --git a/migration/ram.h b/migration/ram.h index c081fde..b711552 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -52,4 +52,9 @@ int ram_discard_range(const char *block_name, uint64_t st= art, size_t length); int ram_postcopy_incoming_init(MigrationIncomingState *mis); =20 void ram_handle_compressed(void *host, uint8_t ch, uint64_t size); + +int ramblock_recv_bitmap_test(void *host_addr, RAMBlock *rb); +void ramblock_recv_bitmap_set(void *host_addr, RAMBlock *rb); +void ramblock_recv_bitmap_clear(void *host_addr, RAMBlock *rb); + #endif --=20 1.9.1