From nobody Tue Feb 10 02:00:43 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 149336288318829.103771726804325; Fri, 28 Apr 2017 00:01:23 -0700 (PDT) Received: from localhost ([::1]:35475 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d3zu4-0003eU-KZ for importer@patchew.org; Fri, 28 Apr 2017 03:01:20 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44846) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d3zqr-0001Jm-MG for qemu-devel@nongnu.org; Fri, 28 Apr 2017 02:58:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d3zqp-00036A-0w for qemu-devel@nongnu.org; Fri, 28 Apr 2017 02:58:01 -0400 Received: from mailout1.w1.samsung.com ([210.118.77.11]:63296) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1d3zqo-00033A-Qy for qemu-devel@nongnu.org; Fri, 28 Apr 2017 02:57:58 -0400 Received: from eucas1p2.samsung.com (unknown [182.198.249.207]) by mailout1.w1.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTP id <0OP3008EDY0KZ220@mailout1.w1.samsung.com> for qemu-devel@nongnu.org; Fri, 28 Apr 2017 07:57:56 +0100 (BST) Received: from eusmges5.samsung.com (unknown [203.254.199.245]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20170428065755eucas1p14f64cac3a9a1fc950a178120706937fe~5fXEJBXpr0350003500eucas1p1Q; Fri, 28 Apr 2017 06:57:55 +0000 (GMT) Received: from eucas1p2.samsung.com ( [182.198.249.207]) by eusmges5.samsung.com (EUCPMTA) with SMTP id 71.6B.25577.3F7E2095; Fri, 28 Apr 2017 07:57:55 +0100 (BST) Received: from eusmgms2.samsung.com (unknown [182.198.249.180]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20170428065755eucas1p2ff9aa17eaa294e741d8c65f8d58a71fb~5fXDjqKhI1887918879eucas1p2R; Fri, 28 Apr 2017 06:57:55 +0000 (GMT) Received: from eusync3.samsung.com ( [203.254.199.213]) by eusmgms2.samsung.com (EUCPMTA) with SMTP id AF.CC.20206.E08E2095; Fri, 28 Apr 2017 07:58:22 +0100 (BST) Received: from aperevalov-ubuntu.rnd.samsung.ru ([106.109.129.199]) by eusync3.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTPA id <0OP300L3YY07A050@eusync3.samsung.com>; Fri, 28 Apr 2017 07:57:54 +0100 (BST) X-AuditID: cbfec7f5-f792f6d0000063e9-df-5902e7f364c5 From: Alexey Perevalov To: qemu-devel@nongnu.org Date: Fri, 28 Apr 2017 09:57:37 +0300 Message-id: <1493362658-8179-6-git-send-email-a.perevalov@samsung.com> X-Mailer: git-send-email 1.9.1 In-reply-to: <1493362658-8179-1-git-send-email-a.perevalov@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrPIsWRmVeSWpSXmKPExsWy7djP87qfnzNFGvQ9VrOYe/c8i0Xvtnvs Fic2HWS2uNL+k91iy/5v7BbHe3ewOLB5HHixkN3jybXNTB7v911l8+jbsooxgCWKyyYlNSez LLVI3y6BK2Pl34KCTy4Ve5YdYmlgnGPaxcjJISFgItGzfh87hC0mceHeerYuRi4OIYGljBIr vuyFcj4zSnw/+p4FpuP/krvsEIlljBLr9r1kgnC6mSQ2L7sFlOHgYBMwkNh3zxakQURAUuJ3 12lmEJtZIE/i0eTzYLawgIfEzY3nmEBsFgFViUs3/7CC2LwCbhLTd31hglgmJ3Hy2GSwOKeA u8Tux7fYIOL32YCGhoKskhCQldh0gBki7CIx+043lC0s8er4FqjPZCQuT+5mATlTQqCdUaJ7 ZycrhDOBUeLM9L9QVfYSp25eZYI4lE9i0rbpzBALeCU62oQgSjwkfn/pgVrgKPF+9gNWiN9n MUosaOhjmsAos4CRYRWjSGppcW56arGpXnFibnFpXrpecn7uJkZgtJ7+d/zrDsalx6wOMQpw MCrx8DJ8YIwUYk0sK67MPcQowcGsJMIrmcgUKcSbklhZlVqUH19UmpNafIhRmoNFSZyX69S1 CCGB9MSS1OzU1ILUIpgsEwenVAPj7FXeoe1rf3r+7N+cFRPjpm4+c3HqntAlhQ83/P7wXHW7 1OYH0ufPrPlgxejA927h9AlbC+aH7d47xfjL/X9fzv8+UfbyQPPLxCkvNW+JWxW/5qvY/9x6 us+x7dpzqjqPyRXdiShc9Sp6eWJ2fn3GQeecR+l3HkqYb7BgX73AUHx1k3d3eWZXshJLcUai oRZzUXEiAGUwfz/SAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrDLMWRmVeSWpSXmKPExsVy+t/xq7p8L5giDWYuE7WYe/c8i0Xvtnvs Fic2HWS2uNL+k91iy/5v7BbHe3ewOLB5HHixkN3jybXNTB7v911l8+jbsooxgCXKzSYjNTEl tUghNS85PyUzL91WKTTETddCSSEvMTfVVilC1zckSEmhLDGnFMgzMkADDs4B7sFK+nYJbhkr /xYUfHKp2LPsEEsD4xzTLkZODgkBE4n/S+6yQ9hiEhfurWcDsYUEljBKvOpih7B7mSSOvhbs YuTgYBMwkNh3zxYkLCIgKfG76zQziM0skCexcsZ/RhBbWMBD4ubGc0wgNouAqsSlm39YQWxe ATeJ6bu+MEGskpM4eWwyWJxTwF1i9+NbUGvdJJa8nsY8gZF3ASPDKkaR1NLi3PTcYiO94sTc 4tK8dL3k/NxNjMCQ3Xbs55YdjF3vgg8xCnAwKvHwRnxijBRiTSwrrsw9xCjBwawkwiuZyBQp xJuSWFmVWpQfX1Sak1p8iNEU6KiJzFKiyfnAeMoriTc0MTS3NDQytrAwNzJSEued+uFKuJBA emJJanZqakFqEUwfEwenVAOj1VHTP+IZvBbeMr/fiKcs1/pznm1++PtVd3eKP6461nd2+tvc +XwHl1amr7s/g0UzI9/khe/e+fFvGAUiG05GdEYpdh9iu2Vtu/671deWJvGPZ9Y32XEERF1Q aFgazcVw7qJM003lH0vZd4gaLbmrW3DEYc6eO6Fpr/admvVIe8VzVQGmjYUlSizFGYmGWsxF xYkAdCDYiW8CAAA= X-MTR: 20000000000000000@CPGS X-CMS-MailID: 20170428065755eucas1p2ff9aa17eaa294e741d8c65f8d58a71fb X-Msg-Generator: CA X-Sender-IP: 182.198.249.180 X-Local-Sender: =?UTF-8?B?QWxleGV5IFBlcmV2YWxvdhtTUlItVmlydHVhbGl6YXRpb24g?= =?UTF-8?B?TGFiG+yCvOyEseyghOyekBtTZW5pb3IgRW5naW5lZXI=?= X-Global-Sender: =?UTF-8?B?QWxleGV5IFBlcmV2YWxvdhtTUlItVmlydHVhbGl6YXRpb24g?= =?UTF-8?B?TGFiG1NhbXN1bmcgRWxlY3Ryb25pY3MbU2VuaW9yIEVuZ2luZWVy?= X-Sender-Code: =?UTF-8?B?QzEwG0NJU0hRG0MxMEdEMDFHRDAxMDE1NA==?= CMS-TYPE: 201P X-HopCount: 7 X-CMS-RootMailID: 20170428065755eucas1p2ff9aa17eaa294e741d8c65f8d58a71fb X-RootMTR: 20170428065755eucas1p2ff9aa17eaa294e741d8c65f8d58a71fb References: <1493362658-8179-1-git-send-email-a.perevalov@samsung.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 210.118.77.11 Subject: [Qemu-devel] [PATCH RESEND V3 5/6] migration: calculate downtime on dst side X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: i.maximets@samsung.com, f4bug@amsat.org, dgilbert@redhat.com, peterx@redhat.com, a.perevalov@samsung.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch provides downtime calculation per vCPU, as a summary and as a overlapped value for all vCPUs. This approach was suggested by Peter Xu, as an improvements of previous approch where QEMU kept tree with faulted page address and cpus bi= tmask in it. Now QEMU is keeping array with faulted page address as value and vCPU as index. It helps to find proper vCPU at UFFD_COPY time. Also it keeps list for downtime per vCPU (could be traced with page_fault_addr) For more details see comments for get_postcopy_total_downtime implementation. Downtime will not calculated if postcopy_downtime field of MigrationIncomingState wasn't initialized. Signed-off-by: Alexey Perevalov --- include/migration/migration.h | 3 ++ migration/migration.c | 103 ++++++++++++++++++++++++++++++++++++++= ++++ migration/postcopy-ram.c | 20 +++++++- migration/trace-events | 6 ++- 4 files changed, 130 insertions(+), 2 deletions(-) diff --git a/include/migration/migration.h b/include/migration/migration.h index e8fb68f..a22f9ce 100644 --- a/include/migration/migration.h +++ b/include/migration/migration.h @@ -139,6 +139,9 @@ void migration_incoming_state_destroy(void); * Functions to work with downtime context */ struct DowntimeContext *downtime_context_new(void); +void mark_postcopy_downtime_begin(uint64_t addr, int cpu); +void mark_postcopy_downtime_end(uint64_t addr); +uint64_t get_postcopy_total_downtime(void); =20 struct MigrationState { diff --git a/migration/migration.c b/migration/migration.c index ec76e5c..2c6f150 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2150,3 +2150,106 @@ PostcopyState postcopy_state_set(PostcopyState new_= state) return atomic_xchg(&incoming_postcopy_state, new_state); } =20 +void mark_postcopy_downtime_begin(uint64_t addr, int cpu) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + DowntimeContext *dc; + if (!mis->downtime_ctx || cpu < 0) { + return; + } + dc =3D mis->downtime_ctx; + dc->vcpu_addr[cpu] =3D addr; + dc->last_begin =3D dc->page_fault_vcpu_time[cpu] =3D + qemu_clock_get_ms(QEMU_CLOCK_REALTIME); + + trace_mark_postcopy_downtime_begin(addr, dc, dc->page_fault_vcpu_time[= cpu], + cpu); +} + +void mark_postcopy_downtime_end(uint64_t addr) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + DowntimeContext *dc; + int i; + bool all_vcpu_down =3D true; + int64_t now; + + if (!mis->downtime_ctx) { + return; + } + dc =3D mis->downtime_ctx; + now =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIME); + + /* check all vCPU down, + * QEMU has bitmap.h, but even with bitmap_and + * will be a cycle */ + for (i =3D 0; i < smp_cpus; i++) { + if (dc->vcpu_addr[i]) { + continue; + } + all_vcpu_down =3D false; + break; + } + + if (all_vcpu_down) { + dc->total_downtime +=3D now - dc->last_begin; + } + + /* lookup cpu, to clear it */ + for (i =3D 0; i < smp_cpus; i++) { + uint64_t vcpu_downtime; + + if (dc->vcpu_addr[i] !=3D addr) { + continue; + } + + vcpu_downtime =3D now - dc->page_fault_vcpu_time[i]; + + dc->vcpu_addr[i] =3D 0; + dc->vcpu_downtime[i] +=3D vcpu_downtime; + } + + trace_mark_postcopy_downtime_end(addr, dc, dc->total_downtime); +} + +/* + * This function just provide calculated before downtime per cpu and trace= it. + * Total downtime is calculated in mark_postcopy_downtime_end. + * + * + * Assume we have 3 CPU + * + * S1 E1 S1 E1 + * -----***********------------xxx***************------------------------>= CPU1 + * + * S2 E2 + * ------------****************xxx--------------------------------------->= CPU2 + * + * S3 E3 + * ------------------------****xxx********------------------------------->= CPU3 + * + * We have sequence S1,S2,E1,S3,S1,E2,E3,E1 + * S2,E1 - doesn't match condition due to sequence S1,S2,E1 doesn't includ= e CPU3 + * S3,S1,E2 - sequence includes all CPUs, in this case overlap will be S1,= E2 - + * it's a part of total downtime. + * S1 - here is last_begin + * Legend of the picture is following: + * * - means downtime per vCPU + * x - means overlapped downtime (total downtime) + */ +uint64_t get_postcopy_total_downtime(void) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + + if (!mis->downtime_ctx) { + return 0; + } + + if (trace_event_get_state(TRACE_DOWNTIME_PER_CPU)) { + int i; + for (i =3D 0; i < smp_cpus; i++) { + trace_downtime_per_cpu(i, mis->downtime_ctx->vcpu_downtime[i]); + } + } + return mis->downtime_ctx->total_downtime; +} diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index f3688f5..cf2b935 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -23,6 +23,7 @@ #include "migration/postcopy-ram.h" #include "sysemu/sysemu.h" #include "sysemu/balloon.h" +#include #include "qemu/error-report.h" #include "trace.h" =20 @@ -468,6 +469,19 @@ static int ram_block_enable_notify(const char *block_n= ame, void *host_addr, return 0; } =20 +static int get_mem_fault_cpu_index(uint32_t pid) +{ + CPUState *cpu_iter; + + CPU_FOREACH(cpu_iter) { + if (cpu_iter->thread_id =3D=3D pid) { + return cpu_iter->cpu_index; + } + } + trace_get_mem_fault_cpu_index(pid); + return -1; +} + /* * Handle faults detected by the USERFAULT markings */ @@ -545,8 +559,11 @@ static void *postcopy_ram_fault_thread(void *opaque) rb_offset &=3D ~(qemu_ram_pagesize(rb) - 1); trace_postcopy_ram_fault_thread_request(msg.arg.pagefault.address, qemu_ram_get_idstr(rb), - rb_offset); + rb_offset, + msg.arg.pagefault.feat.pti= d); =20 + mark_postcopy_downtime_begin((uintptr_t)(msg.arg.pagefault.address= ), + get_mem_fault_cpu_index(msg.arg.pagefault.feat.pt= id)); /* * Send the request to the source - we want to request one * of our host page sizes (which is >=3D TPS) @@ -641,6 +658,7 @@ int postcopy_place_page(MigrationIncomingState *mis, vo= id *host, void *from, =20 return -e; } + mark_postcopy_downtime_end((uint64_t)host); =20 trace_postcopy_place_page(host); return 0; diff --git a/migration/trace-events b/migration/trace-events index b8f01a2..d338810 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -110,6 +110,9 @@ process_incoming_migration_co_end(int ret, int ps) "ret= =3D%d postcopy-state=3D%d" process_incoming_migration_co_postcopy_end_main(void) "" migration_set_incoming_channel(void *ioc, const char *ioctype) "ioc=3D%p i= octype=3D%s" migration_set_outgoing_channel(void *ioc, const char *ioctype, const char = *hostname) "ioc=3D%p ioctype=3D%s hostname=3D%s" +mark_postcopy_downtime_begin(uint64_t addr, void *dd, int64_t time, int cp= u) "addr 0x%" PRIx64 " dd %p time %" PRId64 " cpu %d" +mark_postcopy_downtime_end(uint64_t addr, void *dd, int64_t time) "addr 0x= %" PRIx64 " dd %p time %" PRId64 +downtime_per_cpu(int cpu_index, int64_t downtime) "downtime cpu[%d]=3D%" P= RId64 =20 # migration/rdma.c qemu_rdma_accept_incoming_migration(void) "" @@ -186,7 +189,7 @@ postcopy_ram_enable_notify(void) "" postcopy_ram_fault_thread_entry(void) "" postcopy_ram_fault_thread_exit(void) "" postcopy_ram_fault_thread_quit(void) "" -postcopy_ram_fault_thread_request(uint64_t hostaddr, const char *ramblock,= size_t offset) "Request for HVA=3D%" PRIx64 " rb=3D%s offset=3D%zx" +postcopy_ram_fault_thread_request(uint64_t hostaddr, const char *ramblock,= size_t offset, uint32_t pid) "Request for HVA=3D%" PRIx64 " rb=3D%s offset= =3D%zx %u" postcopy_ram_incoming_cleanup_closeuf(void) "" postcopy_ram_incoming_cleanup_entry(void) "" postcopy_ram_incoming_cleanup_exit(void) "" @@ -195,6 +198,7 @@ save_xbzrle_page_skipping(void) "" save_xbzrle_page_overflow(void) "" ram_save_iterate_big_wait(uint64_t milliconds, int iterations) "big wait: = %" PRIu64 " milliseconds, %d iterations" ram_load_complete(int ret, uint64_t seq_iter) "exit_code %d seq iteration = %" PRIu64 +get_mem_fault_cpu_index(uint32_t pid) "pid %u is not vCPU" =20 # migration/exec.c migration_exec_outgoing(const char *cmd) "cmd=3D%s" --=20 1.9.1