From nobody Mon Feb 9 18:45:21 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=samsung.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1521742831359974.3180034118176; Thu, 22 Mar 2018 11:20:31 -0700 (PDT) Received: from localhost ([::1]:34151 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ez4p9-0000ZY-0p for importer@patchew.org; Thu, 22 Mar 2018 14:20:27 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53999) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ez4mb-00079H-P3 for qemu-devel@nongnu.org; Thu, 22 Mar 2018 14:17:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ez4mY-0000bR-ID for qemu-devel@nongnu.org; Thu, 22 Mar 2018 14:17:49 -0400 Received: from mailout1.w1.samsung.com ([210.118.77.11]:60165) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ez4mY-0000ab-3f for qemu-devel@nongnu.org; Thu, 22 Mar 2018 14:17:46 -0400 Received: from eucas1p2.samsung.com (unknown [182.198.249.207]) by mailout1.w1.samsung.com (KnoxPortal) with ESMTP id 20180322181743euoutp0172692277a5f615fe2bada13dd20a3462~eUOO_tdqs0379303793euoutp01K; Thu, 22 Mar 2018 18:17:43 +0000 (GMT) Received: from eusmges1new.samsung.com (unknown [203.254.199.242]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20180322181742eucas1p15597af98dd1a170bdaf044fa56b07e8e~eUOOTdYFK1715217152eucas1p1D; Thu, 22 Mar 2018 18:17:42 +0000 (GMT) Received: from eucas1p2.samsung.com ( [182.198.249.207]) by eusmges1new.samsung.com (EUCPMTA) with SMTP id 25.B7.05700.643F3BA5; Thu, 22 Mar 2018 18:17:42 +0000 (GMT) Received: from eusmgms2.samsung.com (unknown [182.198.249.180]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20180322181741eucas1p1d80d609a6d0aca10e459563b715b37d6~eUONjIHel0211502115eucas1p1F; Thu, 22 Mar 2018 18:17:41 +0000 (GMT) Received: from eusync4.samsung.com ( [203.254.199.214]) by eusmgms2.samsung.com (EUCPMTA) with SMTP id E5.AC.04183.543F3BA5; Thu, 22 Mar 2018 18:17:41 +0000 (GMT) Received: from aperevalov-ubuntu.rnd.samsung.ru ([106.109.129.199]) by eusync4.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTPA id <0P60001OD857D350@eusync4.samsung.com>; Thu, 22 Mar 2018 18:17:41 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.w1.samsung.com 20180322181743euoutp0172692277a5f615fe2bada13dd20a3462~eUOO_tdqs0379303793euoutp01K DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1521742663; bh=JVGyMUYj5okNvtP6rivYzs7au9yWZE0EGQC9gUfIIfg=; h=From:To:Cc:Subject:Date:In-reply-to:References:From; b=QLCTfcJ+TWubmxpqcGOtvVRG2jwy3ca7hyrfqxCxq/yNYUUaqNrRSt6RbRLe0lekw XoDRvGB5rsbJ3g9B8gkRyQIplyHLfWcmrDB5OCW7P/gLItdi+11WInsT8dvseHQWOj CKIVtz4An+cyEbao42yT/kMIraEP4QezIejirCXs= X-AuditID: cbfec7f2-1c1ff70000011644-53-5ab3f34683d8 From: Alexey Perevalov To: qemu-devel@nongnu.org, dgilbert@redhat.com Date: Thu, 22 Mar 2018 21:17:24 +0300 Message-id: <1521742647-25550-4-git-send-email-a.perevalov@samsung.com> X-Mailer: git-send-email 2.7.4 In-reply-to: <1521742647-25550-1-git-send-email-a.perevalov@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrFIsWRmVeSWpSXmKPExsWy7djP87punzdHGSybw24x9+55FouVk5cw WfRuu8ducWLTQWaLTw1SFlv2f2O3ON67g8XizpY+JovTNzezO3B6HHixkN3jybXNTB7v911l 8+jbsooxgCWKyyYlNSezLLVI3y6BK+PV11nsBVN9Kva/3sbWwLjKsouRk0NCwETi/INuxi5G Lg4hgRWMElv7T7JAOJ8ZJX4dPsEKU7W+pYMZIrGMUeLFlONQTjeTxKzHT4EcDg42AQOJffds QUwRAT2JCVcjQUqYBTYySuy8854FZJCwgLtEy9Gz7CA2i4CqxNYVfxlBbF6g+Pzdv5kglslJ 3DzXyQxicwp4SOy9tQnsPAmBCWwSa29OgbrIReL1rQVsELawxKvjW9ghbBmJy5O7WSAa2hkl 7m68ygrVzSixbvIxZogqe4lTN6+CrWMW4JOYtG062AcSArwSHW1CECUeEovOrIca6iix8lMD 2GIhgdmMEnOuSk1glFrAyLCKUTy1tDg3PbXYMC+1XK84Mbe4NC9dLzk/dxMjMEZP/zv+aQfj 10tJhxgFOBiVeHhnHNocJcSaWFZcmXuIUYKDWUmEN38tUIg3JbGyKrUoP76oNCe1+BCjNAeL kjhvnEZdlJBAemJJanZqakFqEUyWiYNTqoHRc+07I7F1pee3dG488mer2jYGsWt992XW7E5w 8bvhmnq9hje6N2ipTIJi4IRa78X/hArcQn3SNnMtO7xxws9WHmb7M3oKp1q2ivQxBspN1zz5 +CnHu1efv/zkn3/k5/KrFx9bLzzAcvijx8z0j2lV6b2uT9ccVfn/ReLj5YWC/7Rm/HhudOf+ CiWW4oxEQy3mouJEAGT6P6jNAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupmluLIzCtJLcpLzFFi42I5/e/4NV3Xz5ujDDZNUrWYe/c8i8XKyUuY LHq33WO3OLHpILPFpwYpiy37v7FbHO/dwWJxZ0sfk8Xpm5vZHTg9DrxYyO7x5NpmJo/3+66y efRtWcUYwBLFZZOSmpNZllqkb5fAlfHq6yz2gqk+Fftfb2NrYFxl2cXIySEhYCKxvqWDuYuR i0NIYAmjREv3RxYIp5dJYsr9B0xdjBwcbAIGEvvu2YKYIgJ6EhOuRoKUMAtsZJT4NeMuK8gg YQF3iZajZ9lBbBYBVYmtK/4ygti8QPH5u38zQSyTk7h5rpMZxOYU8JDYe2sTWI0QUM2uC9eY JzDyLGBkWMUoklpanJueW2ykV5yYW1yal66XnJ+7iREYRNuO/dyyg7HrXfAhRgEORiUe3hmH NkcJsSaWFVfmHmKU4GBWEuHNXwsU4k1JrKxKLcqPLyrNSS0+xCjNwaIkznveoDJKSCA9sSQ1 OzW1ILUIJsvEwSnVwLh61/X1108t/rHGY86xf9xXFq+QmyPmfCuubl/rxybF2ONW7tcuzr4w 9cJdh806PduErRcFtU4q5/t+0VQx6uiPnc6BCntqHjhaqL4Q89vDcfXr4SrhU0vMDmc3eVSd nlQ7b0V46eYZERnyLi53o3LEQ9hnNKau52u4xx+Rm79qX85H49le+zcosRRnJBpqMRcVJwIA NILwqB4CAAA= X-CMS-MailID: 20180322181741eucas1p1d80d609a6d0aca10e459563b715b37d6 X-Msg-Generator: CA CMS-TYPE: 201P X-CMS-RootMailID: 20180322181741eucas1p1d80d609a6d0aca10e459563b715b37d6 X-RootMTR: 20180322181741eucas1p1d80d609a6d0aca10e459563b715b37d6 References: <1521742647-25550-1-git-send-email-a.perevalov@samsung.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-Received-From: 210.118.77.11 Subject: [Qemu-devel] [PATCH v2 3/6] migration: calculate vCPU blocktime on dst side X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, v.kuramshin@samsung.com, quintela@redhat.com, ash.billore@samsung.com, Alexey Perevalov , peterx@redhat.com, f4bug@amsat.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch provides blocktime calculation per vCPU, as a summary and as a overlapped value for all vCPUs. This approach was suggested by Peter Xu, as an improvements of previous approch where QEMU kept tree with faulted page address and cpus bi= tmask in it. Now QEMU is keeping array with faulted page address as value and vCPU as index. It helps to find proper vCPU at UFFD_COPY time. Also it keeps list for blocktime per vCPU (could be traced with page_fault_addr) Blocktime will not calculated if postcopy_blocktime field of MigrationIncomingState wasn't initialized. Signed-off-by: Alexey Perevalov Reviewed-by: Dr. David Alan Gilbert Reviewed-by: Juan Quintela Signed-off-by: Juan Quintela --- migration/postcopy-ram.c | 151 +++++++++++++++++++++++++++++++++++++++++++= +++- migration/trace-events | 5 +- 2 files changed, 154 insertions(+), 2 deletions(-) diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 66f1df9..6b01884 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -636,6 +636,148 @@ int postcopy_request_shared_page(struct PostCopyFD *p= cfd, RAMBlock *rb, return 0; } =20 +static int get_mem_fault_cpu_index(uint32_t pid) +{ + CPUState *cpu_iter; + + CPU_FOREACH(cpu_iter) { + if (cpu_iter->thread_id =3D=3D pid) { + trace_get_mem_fault_cpu_index(cpu_iter->cpu_index, pid); + return cpu_iter->cpu_index; + } + } + trace_get_mem_fault_cpu_index(-1, pid); + return -1; +} + +static uint32_t get_low_time_offset(PostcopyBlocktimeContext *dc) +{ + int64_t start_time_offset =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIME) - + dc->start_time; + return start_time_offset < 1 ? 1 : start_time_offset & UINT32_MAX; +} + +/* + * This function is being called when pagefault occurs. It + * tracks down vCPU blocking time. + * + * @addr: faulted host virtual address + * @ptid: faulted process thread id + * @rb: ramblock appropriate to addr + */ +static void mark_postcopy_blocktime_begin(uintptr_t addr, uint32_t ptid, + RAMBlock *rb) +{ + int cpu, already_received; + MigrationIncomingState *mis =3D migration_incoming_get_current(); + PostcopyBlocktimeContext *dc =3D mis->blocktime_ctx; + uint32_t low_time_offset; + + if (!dc || ptid =3D=3D 0) { + return; + } + cpu =3D get_mem_fault_cpu_index(ptid); + if (cpu < 0) { + return; + } + + low_time_offset =3D get_low_time_offset(dc); + if (dc->vcpu_addr[cpu] =3D=3D 0) { + atomic_inc(&dc->smp_cpus_down); + } + + atomic_xchg(&dc->last_begin, low_time_offset); + atomic_xchg(&dc->page_fault_vcpu_time[cpu], low_time_offset); + atomic_xchg(&dc->vcpu_addr[cpu], addr); + + /* check it here, not at the begining of the function, + * due to, check could accur early than bitmap_set in + * qemu_ufd_copy_ioctl */ + already_received =3D ramblock_recv_bitmap_test(rb, (void *)addr); + if (already_received) { + atomic_xchg(&dc->vcpu_addr[cpu], 0); + atomic_xchg(&dc->page_fault_vcpu_time[cpu], 0); + atomic_dec(&dc->smp_cpus_down); + } + trace_mark_postcopy_blocktime_begin(addr, dc, dc->page_fault_vcpu_time= [cpu], + cpu, already_received); +} + +/* + * This function just provide calculated blocktime per cpu and trace it. + * Total blocktime is calculated in mark_postcopy_blocktime_end. + * + * + * Assume we have 3 CPU + * + * S1 E1 S1 E1 + * -----***********------------xxx***************------------------------>= CPU1 + * + * S2 E2 + * ------------****************xxx--------------------------------------->= CPU2 + * + * S3 E3 + * ------------------------****xxx********------------------------------->= CPU3 + * + * We have sequence S1,S2,E1,S3,S1,E2,E3,E1 + * S2,E1 - doesn't match condition due to sequence S1,S2,E1 doesn't includ= e CPU3 + * S3,S1,E2 - sequence includes all CPUs, in this case overlap will be S1,= E2 - + * it's a part of total blocktime. + * S1 - here is last_begin + * Legend of the picture is following: + * * - means blocktime per vCPU + * x - means overlapped blocktime (total blocktime) + * + * @addr: host virtual address + */ +static void mark_postcopy_blocktime_end(uintptr_t addr) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + PostcopyBlocktimeContext *dc =3D mis->blocktime_ctx; + int i, affected_cpu =3D 0; + bool vcpu_total_blocktime =3D false; + uint32_t read_vcpu_time, low_time_offset; + + if (!dc) { + return; + } + + low_time_offset =3D get_low_time_offset(dc); + /* lookup cpu, to clear it, + * that algorithm looks straighforward, but it's not + * optimal, more optimal algorithm is keeping tree or hash + * where key is address value is a list of */ + for (i =3D 0; i < smp_cpus; i++) { + uint32_t vcpu_blocktime =3D 0; + + read_vcpu_time =3D atomic_fetch_add(&dc->page_fault_vcpu_time[i], = 0); + if (atomic_fetch_add(&dc->vcpu_addr[i], 0) !=3D addr || + read_vcpu_time =3D=3D 0) { + continue; + } + atomic_xchg(&dc->vcpu_addr[i], 0); + vcpu_blocktime =3D low_time_offset - read_vcpu_time; + affected_cpu +=3D 1; + /* we need to know is that mark_postcopy_end was due to + * faulted page, another possible case it's prefetched + * page and in that case we shouldn't be here */ + if (!vcpu_total_blocktime && + atomic_fetch_add(&dc->smp_cpus_down, 0) =3D=3D smp_cpus) { + vcpu_total_blocktime =3D true; + } + /* continue cycle, due to one page could affect several vCPUs */ + dc->vcpu_blocktime[i] +=3D vcpu_blocktime; + } + + atomic_sub(&dc->smp_cpus_down, affected_cpu); + if (vcpu_total_blocktime) { + dc->total_blocktime +=3D low_time_offset - atomic_fetch_add( + &dc->last_begin, 0); + } + trace_mark_postcopy_blocktime_end(addr, dc, dc->total_blocktime, + affected_cpu); +} + /* * Handle faults detected by the USERFAULT markings */ @@ -742,7 +884,12 @@ static void *postcopy_ram_fault_thread(void *opaque) rb_offset &=3D ~(qemu_ram_pagesize(rb) - 1); trace_postcopy_ram_fault_thread_request(msg.arg.pagefault.addr= ess, qemu_ram_get_idstr(rb), - rb_offset); + rb_offset, + msg.arg.pagefault.feat.pti= d); + mark_postcopy_blocktime_begin( + (uintptr_t)(msg.arg.pagefault.address), + msg.arg.pagefault.feat.ptid, rb); + /* * Send the request to the source - we want to request one * of our host page sizes (which is >=3D TPS) @@ -889,6 +1036,8 @@ static int qemu_ufd_copy_ioctl(int userfault_fd, void = *host_addr, if (!ret) { ramblock_recv_bitmap_set_range(rb, host_addr, pagesize / qemu_target_page_size()); + mark_postcopy_blocktime_end((uintptr_t)host_addr); + } return ret; } diff --git a/migration/trace-events b/migration/trace-events index a180d7b..368bc4b 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -115,6 +115,8 @@ process_incoming_migration_co_end(int ret, int ps) "ret= =3D%d postcopy-state=3D%d" process_incoming_migration_co_postcopy_end_main(void) "" migration_set_incoming_channel(void *ioc, const char *ioctype) "ioc=3D%p i= octype=3D%s" migration_set_outgoing_channel(void *ioc, const char *ioctype, const char = *hostname, void *err) "ioc=3D%p ioctype=3D%s hostname=3D%s err=3D%p" +mark_postcopy_blocktime_begin(uint64_t addr, void *dd, uint32_t time, int = cpu, int received) "addr: 0x%" PRIx64 ", dd: %p, time: %u, cpu: %d, already= _received: %d" +mark_postcopy_blocktime_end(uint64_t addr, void *dd, uint32_t time, int af= fected_cpu) "addr: 0x%" PRIx64 ", dd: %p, time: %u, affected_cpu: %d" =20 # migration/rdma.c qemu_rdma_accept_incoming_migration(void) "" @@ -193,7 +195,7 @@ postcopy_ram_fault_thread_exit(void) "" postcopy_ram_fault_thread_fds_core(int baseufd, int quitfd) "ufd: %d quitf= d: %d" postcopy_ram_fault_thread_fds_extra(size_t index, const char *name, int fd= ) "%zd/%s: %d" postcopy_ram_fault_thread_quit(void) "" -postcopy_ram_fault_thread_request(uint64_t hostaddr, const char *ramblock,= size_t offset) "Request for HVA=3D0x%" PRIx64 " rb=3D%s offset=3D0x%zx" +postcopy_ram_fault_thread_request(uint64_t hostaddr, const char *ramblock,= size_t offset, uint32_t pid) "Request for HVA=3D0x%" PRIx64 " rb=3D%s offs= et=3D0x%zx pid=3D%u" postcopy_ram_incoming_cleanup_closeuf(void) "" postcopy_ram_incoming_cleanup_entry(void) "" postcopy_ram_incoming_cleanup_exit(void) "" @@ -206,6 +208,7 @@ save_xbzrle_page_skipping(void) "" save_xbzrle_page_overflow(void) "" ram_save_iterate_big_wait(uint64_t milliconds, int iterations) "big wait: = %" PRIu64 " milliseconds, %d iterations" ram_load_complete(int ret, uint64_t seq_iter) "exit_code %d seq iteration = %" PRIu64 +get_mem_fault_cpu_index(int cpu, uint32_t pid) "cpu: %d, pid: %u" =20 # migration/exec.c migration_exec_outgoing(const char *cmd) "cmd=3D%s" --=20 2.7.4