From nobody Tue Feb 10 01:32:47 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1497640323606659.5175023691373; Fri, 16 Jun 2017 12:12:03 -0700 (PDT) Received: from localhost ([::1]:60446 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dLwf4-0006oS-5f for importer@patchew.org; Fri, 16 Jun 2017 15:12:02 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48334) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dLwcB-0004jP-28 for qemu-devel@nongnu.org; Fri, 16 Jun 2017 15:09:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dLwc6-0004Yq-7y for qemu-devel@nongnu.org; Fri, 16 Jun 2017 15:09:03 -0400 Received: from mailout2.w1.samsung.com ([210.118.77.12]:12153) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dLwc6-0004UU-2z for qemu-devel@nongnu.org; Fri, 16 Jun 2017 15:08:58 -0400 Received: from eucas1p2.samsung.com (unknown [182.198.249.207]) by mailout2.w1.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTP id <0ORN00MIDMIVJJ40@mailout2.w1.samsung.com> for qemu-devel@nongnu.org; Fri, 16 Jun 2017 20:08:55 +0100 (BST) Received: from eusmges5.samsung.com (unknown [203.254.199.245]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20170616190853eucas1p1833817039b7290021b48478775990188~Ir8RQH6SA0583605836eucas1p1f; Fri, 16 Jun 2017 19:08:53 +0000 (GMT) Received: from eucas1p1.samsung.com ( [182.198.249.206]) by eusmges5.samsung.com (EUCPMTA) with SMTP id 1D.D8.25577.5CC24495; Fri, 16 Jun 2017 20:08:53 +0100 (BST) Received: from eusmgms2.samsung.com (unknown [182.198.249.180]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20170616190852eucas1p18bc3671b480012ede7cbd810b40a8263~Ir8QfmGqm0583605836eucas1p1c; Fri, 16 Jun 2017 19:08:52 +0000 (GMT) Received: from eusync4.samsung.com ( [203.254.199.214]) by eusmgms2.samsung.com (EUCPMTA) with SMTP id 4D.C5.20206.4CC24495; Fri, 16 Jun 2017 20:08:52 +0100 (BST) Received: from aperevalov-ubuntu.rnd.samsung.ru ([106.109.129.199]) by eusync4.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTPA id <0ORN00D34MIKUY80@eusync4.samsung.com>; Fri, 16 Jun 2017 20:08:52 +0100 (BST) X-AuditID: cbfec7f5-f792f6d0000063e9-77-59442cc5a8b5 From: Alexey Perevalov To: qemu-devel@nongnu.org Date: Fri, 16 Jun 2017 22:08:40 +0300 Message-id: <1497640120-10729-4-git-send-email-a.perevalov@samsung.com> X-Mailer: git-send-email 1.9.1 In-reply-to: <1497640120-10729-1-git-send-email-a.perevalov@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrLIsWRmVeSWpSXmKPExsWy7djPc7pHdVwiDdYtUbSYe/c8i0Xvtnvs Flfaf7JbbNn/jd3ieO8OFos7W/qYHNg8nlzbzOTxft9VNo++LasYA5ijuGxSUnMyy1KL9O0S uDI2P97HUtDoXbH6x0zmBsZ+iy5GTg4JAROJ3UtPskHYYhIX7q0Hsrk4hASWMkqs/bCXCcL5 zCgxbcYDNpiOq/9boaqWMUp8XDCNEcLpZpJ4dKGDuYuRg4NNwEBi3z1bkAYRAUmJ312nmUFq mAVaGSXa+x4xgySEBewlpvz/xwpiswioSnw/upUFxOYVcJdY9+YrM8Q2OYmTxyaD1XAKeEg0 X7vMDjJIQuA+m8TyJe9ZQZZJCMhKbDoAVe8iMevwRChbWOLV8S3sELaMxOXJ3SwQve2MEt07 O1khnAmMEmem/4Wqspc4dfMqE4jNLMAnMWnbdGaIBbwSHW1CECUeEpf+r2GEsB0l7rccZYb4 fjajxJxTp1gnMMosYGRYxSiSWlqcm55abKpXnJhbXJqXrpecn7uJERipp/8d/7qDcekxq0OM AhyMSjy8DLedI4VYE8uKK3MPMUpwMCuJ8C5kc4kU4k1JrKxKLcqPLyrNSS0+xCjNwaIkzst1 6lqEkEB6YklqdmpqQWoRTJaJg1OqgdHd7aDQmTb+TmWup9vYX361XWC1yO6qWf2KCR1/rys7 sU2tbt71f5Yjiwn3ppfWm/fZSda03X6ZF9i5Uc+xRW5m5irrRpG0B0duXFkQ4rvuW0Zdv937 RQevH9yV1OH7z6bn4/V3t9iuSMvkhCYahf/TdMkIP/7+9kTBNW2nlUVWGq9ZfFAkt1SJpTgj 0VCLuag4EQAah6ou0AIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrFLMWRmVeSWpSXmKPExsVy+t/xa7pHdFwiDXo6dC3m3j3PYtG77R67 xZX2n+wWW/Z/Y7c43ruDxeLOlj4mBzaPJ9c2M3m833eVzaNvyyrGAOYoN5uM1MSU1CKF1Lzk /JTMvHRbpdAQN10LJYW8xNxUW6UIXd+QICWFssScUiDPyAANODgHuAcr6dsluGVsfryPpaDR u2L1j5nMDYz9Fl2MnBwSAiYSV/+3skHYYhIX7q0Hs4UEljBK/Lmo38XIBWT3Mkk0PF7K2sXI wcEmYCCx754tSI2IgKTE767TzCA1zAKtjBJPGmexgySEBewlpvz/xwpiswioSnw/upUFxOYV cJdY9+YrM8QyOYmTxyaD1XAKeEg0X7vMDrHYXWJp30PmCYy8CxgZVjGKpJYW56bnFhvpFSfm Fpfmpesl5+duYgSG67ZjP7fsYOx6F3yIUYCDUYmHN/Kuc6QQa2JZcWXuIUYJDmYlEV57YLAL 8aYkVlalFuXHF5XmpBYfYjQFOmois5Rocj4wlvJK4g1NDM0tDY2MLSzMjYyUxHmnfrgSLiSQ nliSmp2aWpBaBNPHxMEp1cBov/LourVtYc+izZWdY+9/uPYo6sb9jGZO0ZiIVRXpkQEHDnZ6 algVd3cl2XMxfspbU8Dgc/OJ9oT//8RuldSdMrisV52e7KveLpv88s75J9o8i47vjfvNl/W5 +diUw/t/a7wQdV755eD2kLe+E1l2PrZJrJwktahw8gW+k1xuaxsOZEV63RNSYinOSDTUYi4q TgQAe31RNW0CAAA= X-MTR: 20000000000000000@CPGS X-CMS-MailID: 20170616190852eucas1p18bc3671b480012ede7cbd810b40a8263 X-Msg-Generator: CA X-Sender-IP: 182.198.249.180 X-Local-Sender: =?UTF-8?B?QWxleGV5IFBlcmV2YWxvdhtTUlItVmlydHVhbGl6YXRpb24g?= =?UTF-8?B?TGFiG+yCvOyEseyghOyekBtTZW5pb3IgRW5naW5lZXI=?= X-Global-Sender: =?UTF-8?B?QWxleGV5IFBlcmV2YWxvdhtTUlItVmlydHVhbGl6YXRpb24g?= =?UTF-8?B?TGFiG1NhbXN1bmcgRWxlY3Ryb25pY3MbU2VuaW9yIEVuZ2luZWVy?= X-Sender-Code: =?UTF-8?B?QzEwG0NJU0hRG0MxMEdEMDFHRDAxMDE1NA==?= CMS-TYPE: 201P X-HopCount: 7 X-CMS-RootMailID: 20170616190852eucas1p18bc3671b480012ede7cbd810b40a8263 X-RootMTR: 20170616190852eucas1p18bc3671b480012ede7cbd810b40a8263 References: <1497640120-10729-1-git-send-email-a.perevalov@samsung.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 210.118.77.12 Subject: [Qemu-devel] [PATCH v3 3/3] migration: add bitmap for received page X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: i.maximets@samsung.com, dgilbert@redhat.com, Alexey Perevalov , peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch adds ability to track down already received pages, it's necessary for calculation vCPU block time in postcopy migration feature, maybe for restore after postcopy migration failure. Also it's necessary to solve shared memory issue in postcopy livemigration. Information about received pages will be transferred to the software virtual bridge (e.g. OVS-VSWITCHD), to avoid fallocate (unmap) for already received pages. fallocate syscall is required for remmaped shared memory, due to remmaping itself blocks ioctl(UFFDIO_COPY, ioctl in this case will end with EEXIT error (struct page is exists after remmap). Bitmap is placed into RAMBlock as another postcopy/precopy related bitmaps. Signed-off-by: Alexey Perevalov --- include/exec/ram_addr.h | 3 +++ migration/migration.c | 1 + migration/postcopy-ram.c | 12 ++++++--- migration/ram.c | 66 ++++++++++++++++++++++++++++++++++++++++++++= +--- migration/ram.h | 6 +++++ 5 files changed, 82 insertions(+), 6 deletions(-) diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h index 140efa8..67fbb39 100644 --- a/include/exec/ram_addr.h +++ b/include/exec/ram_addr.h @@ -47,6 +47,8 @@ struct RAMBlock { * of the postcopy phase */ unsigned long *unsentmap; + /* bitmap of already received pages in postcopy */ + unsigned long *receivedmap; }; =20 static inline bool offset_in_ramblock(RAMBlock *b, ram_addr_t offset) @@ -60,6 +62,7 @@ static inline void *ramblock_ptr(RAMBlock *block, ram_add= r_t offset) return (char *)block->host + offset; } =20 +unsigned long int ramblock_recv_bitmap_offset(void *host_addr, RAMBlock *r= b); long qemu_getrampagesize(void); unsigned long last_ram_page(void); RAMBlock *qemu_ram_alloc_from_file(ram_addr_t size, MemoryRegion *mr, diff --git a/migration/migration.c b/migration/migration.c index 71e38bc..53fbd41 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -143,6 +143,7 @@ MigrationIncomingState *migration_incoming_get_current(= void) qemu_mutex_init(&mis_current.rp_mutex); qemu_event_init(&mis_current.main_thread_load_event, false); once =3D true; + ramblock_recv_map_init(); } return &mis_current; } diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index d1af2c1..5d2b92d 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -562,8 +562,13 @@ int postcopy_ram_enable_notify(MigrationIncomingState = *mis) } =20 static int qemu_ufd_copy_ioctl(int userfault_fd, void *host_addr, - void *from_addr, uint64_t pagesize) + void *from_addr, uint64_t pagesize, RAMBlock *rb) { + /* received page isn't feature of blocktime calculation, + * it's more general entity, so keep it here, + * but gup betwean two following operation could be high, + * and in this case blocktime for such small interval will be lost */ + ramblock_recv_bitmap_set(host_addr, rb); if (from_addr) { struct uffdio_copy copy_struct; copy_struct.dst =3D (uint64_t)(uintptr_t)host_addr; @@ -594,7 +599,7 @@ int postcopy_place_page(MigrationIncomingState *mis, vo= id *host, void *from, * which would be slightly cheaper, but we'd have to be careful * of the order of updating our page state. */ - if (qemu_ufd_copy_ioctl(mis->userfault_fd, host, from, pagesize)) { + if (qemu_ufd_copy_ioctl(mis->userfault_fd, host, from, pagesize, rb)) { int e =3D errno; error_report("%s: %s copy host: %p from: %p (size: %zd)", __func__, strerror(e), host, from, pagesize); @@ -616,7 +621,8 @@ int postcopy_place_page_zero(MigrationIncomingState *mi= s, void *host, trace_postcopy_place_page_zero(host); =20 if (qemu_ram_pagesize(rb) =3D=3D getpagesize()) { - if (qemu_ufd_copy_ioctl(mis->userfault_fd, host, 0, getpagesize())= ) { + if (qemu_ufd_copy_ioctl(mis->userfault_fd, host, 0, getpagesize(), + rb)) { int e =3D errno; error_report("%s: %s zero host: %p", __func__, strerror(e), host); diff --git a/migration/ram.c b/migration/ram.c index f50479d..fad4dbf 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -151,6 +151,41 @@ out: return ret; } =20 +void ramblock_recv_map_init(void) +{ + RAMBlock *rb; + + RAMBLOCK_FOREACH(rb) { + unsigned long pages; + pages =3D rb->max_length >> TARGET_PAGE_BITS; + assert(!rb->receivedmap); + rb->receivedmap =3D bitmap_new(pages); + } +} + +unsigned long int ramblock_recv_bitmap_offset(void *host_addr, RAMBlock *r= b) +{ + uint64_t host_addr_offset =3D (uint64_t)(uintptr_t)(host_addr + - (void *)rb->host); + return host_addr_offset >> TARGET_PAGE_BITS; +} + +int ramblock_recv_bitmap_test(void *host_addr, RAMBlock *rb) +{ + return test_bit(ramblock_recv_bitmap_offset(host_addr, rb), + rb->receivedmap); +} + +void ramblock_recv_bitmap_set(void *host_addr, RAMBlock *rb) +{ + set_bit_atomic(ramblock_recv_bitmap_offset(host_addr, rb), rb->receive= dmap); +} + +void ramblock_recv_bitmap_clear(void *host_addr, RAMBlock *rb) +{ + clear_bit(ramblock_recv_bitmap_offset(host_addr, rb), rb->receivedmap); +} + /* * An outstanding page request, on the source, having been received * and queued @@ -1773,6 +1808,18 @@ int ram_postcopy_send_discard_bitmap(MigrationState = *ms) return ret; } =20 +static void ramblock_recv_bitmap_clear_range(uint64_t start, size_t length, + RAMBlock *rb) +{ + int i, range_count; + range_count =3D length >> TARGET_PAGE_BITS; + for (i =3D 0; i < range_count; i++) { + ramblock_recv_bitmap_clear((void *)((uint64_t)(intptr_t)rb->host + + start), rb); + start +=3D TARGET_PAGE_SIZE; + } +} + /** * ram_discard_range: discard dirtied pages at the beginning of postcopy * @@ -1797,6 +1844,7 @@ int ram_discard_range(const char *rbname, uint64_t st= art, size_t length) goto err; } =20 + ramblock_recv_bitmap_clear_range(start, length, rb); ret =3D ram_block_discard_range(rb, start, length); =20 err: @@ -2324,8 +2372,14 @@ static int ram_load_setup(QEMUFile *f, void *opaque) =20 static int ram_load_cleanup(void *opaque) { + RAMBlock *rb; xbzrle_load_cleanup(); compress_threads_load_cleanup(); + + RAMBLOCK_FOREACH(rb) { + g_free(rb->receivedmap); + rb->receivedmap =3D NULL; + } return 0; } =20 @@ -2513,6 +2567,7 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) ram_addr_t addr, total_ram_bytes; void *host =3D NULL; uint8_t ch; + RAMBlock *rb; =20 addr =3D qemu_get_be64(f); flags =3D addr & ~TARGET_PAGE_MASK; @@ -2520,15 +2575,15 @@ static int ram_load(QEMUFile *f, void *opaque, int = version_id) =20 if (flags & (RAM_SAVE_FLAG_ZERO | RAM_SAVE_FLAG_PAGE | RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { - RAMBlock *block =3D ram_block_from_stream(f, flags); + rb =3D ram_block_from_stream(f, flags); =20 - host =3D host_from_ram_block_offset(block, addr); + host =3D host_from_ram_block_offset(rb, addr); if (!host) { error_report("Illegal RAM offset " RAM_ADDR_FMT, addr); ret =3D -EINVAL; break; } - trace_ram_load_loop(block->idstr, (uint64_t)addr, flags, host); + trace_ram_load_loop(rb->idstr, (uint64_t)addr, flags, host); } =20 switch (flags & ~RAM_SAVE_FLAG_CONTINUE) { @@ -2582,10 +2637,12 @@ static int ram_load(QEMUFile *f, void *opaque, int = version_id) =20 case RAM_SAVE_FLAG_ZERO: ch =3D qemu_get_byte(f); + ramblock_recv_bitmap_set(host, rb); ram_handle_compressed(host, ch, TARGET_PAGE_SIZE); break; =20 case RAM_SAVE_FLAG_PAGE: + ramblock_recv_bitmap_set(host, rb); qemu_get_buffer(f, host, TARGET_PAGE_SIZE); break; =20 @@ -2596,10 +2653,13 @@ static int ram_load(QEMUFile *f, void *opaque, int = version_id) ret =3D -EINVAL; break; } + + ramblock_recv_bitmap_set(host, rb); decompress_data_with_multi_threads(f, host, len); break; =20 case RAM_SAVE_FLAG_XBZRLE: + ramblock_recv_bitmap_set(host, rb); if (load_xbzrle(f, addr, host) < 0) { error_report("Failed to decompress XBZRLE page at " RAM_ADDR_FMT, addr); diff --git a/migration/ram.h b/migration/ram.h index c081fde..98d68df 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -52,4 +52,10 @@ int ram_discard_range(const char *block_name, uint64_t s= tart, size_t length); int ram_postcopy_incoming_init(MigrationIncomingState *mis); =20 void ram_handle_compressed(void *host, uint8_t ch, uint64_t size); + +void ramblock_recv_map_init(void); +int ramblock_recv_bitmap_test(void *host_addr, RAMBlock *rb); +void ramblock_recv_bitmap_set(void *host_addr, RAMBlock *rb); +void ramblock_recv_bitmap_clear(void *host_addr, RAMBlock *rb); + #endif --=20 1.8.3.1