From nobody Mon Feb 9 18:45:19 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1494588989979333.6005602157742; Fri, 12 May 2017 04:36:29 -0700 (PDT) Received: from localhost ([::1]:53006 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d98s0-0006HY-H6 for importer@patchew.org; Fri, 12 May 2017 07:36:28 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57374) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d98oM-0003Xe-0a for qemu-devel@nongnu.org; Fri, 12 May 2017 07:32:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d98oI-0000tk-QQ for qemu-devel@nongnu.org; Fri, 12 May 2017 07:32:42 -0400 Received: from mailout3.w1.samsung.com ([210.118.77.13]:19970) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1d98oI-0000ix-JO for qemu-devel@nongnu.org; Fri, 12 May 2017 07:32:38 -0400 Received: from eucas1p2.samsung.com (unknown [182.198.249.207]) by mailout3.w1.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTP id <0OPU00JR881ZQU80@mailout3.w1.samsung.com> for qemu-devel@nongnu.org; Fri, 12 May 2017 12:32:23 +0100 (BST) Received: from eusmges4.samsung.com (unknown [203.254.199.244]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20170512113223eucas1p2f575857c6d9a49c25fedc037eb66d673~92IsI4uUI1055610556eucas1p2A; Fri, 12 May 2017 11:32:23 +0000 (GMT) Received: from eucas1p1.samsung.com ( [182.198.249.206]) by eusmges4.samsung.com (EUCPMTA) with SMTP id C8.13.04729.64D95195; Fri, 12 May 2017 12:32:22 +0100 (BST) Received: from eusmgms2.samsung.com (unknown [182.198.249.180]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20170512113222eucas1p2f0964bb82e343438798c1c341c1cfe8a~92IrV8-lO1054210542eucas1p2d; Fri, 12 May 2017 11:32:22 +0000 (GMT) Received: from eusync2.samsung.com ( [203.254.199.212]) by eusmgms2.samsung.com (EUCPMTA) with SMTP id 89.A1.20206.66D95195; Fri, 12 May 2017 12:32:54 +0100 (BST) Received: from aperevalov-ubuntu.rnd.samsung.ru ([106.109.129.199]) by eusync2.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTPA id <0OPU0053K81CCS30@eusync2.samsung.com>; Fri, 12 May 2017 12:32:21 +0100 (BST) Date: Fri, 12 May 2017 14:31:41 +0300 From: Alexey Perevalov In-reply-to: <1494588703-9076-1-git-send-email-a.perevalov@samsung.com> To: qemu-devel@nongnu.org Message-id: <1494588703-9076-8-git-send-email-a.perevalov@samsung.com> MIME-version: 1.0 X-Mailer: git-send-email 1.9.1 Content-transfer-encoding: quoted-printable X-Sender-IP: 182.198.249.180 CMS-TYPE: 201P X-AuditID: cbfec7f4-f79806d000001279-69-59159d4641ce X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrDIsWRmVeSWpSXmKPExsWy7djPc7puc0UjDX5261vMvXuexaJ32z12 iyvtP9kttuz/xm5xvHcHiwOrx5Nrm5k83u+7yubRt2UVYwBzFJdNSmpOZllqkb5dAlfG5pOd LAV/rCvaDug1MH7Q62Lk5JAQMJG4PeEuM4QtJnHh3nq2LkYuDiGBpYwSp5pWs0A4nxklzl86 zgjT0XbxAFRiGaPEjRX/wRK8AoISPybfA0pwcDALyEscuZQNEmYW0JTYuns9O0R9N5PEvkun wWrYBAwk9t2zBakREZCU+N11mhmiPlbi7fVrbCC2sIC7xO+nE8DiLAKqEn9e7GaDWOUmsXHq XDaIe+QkTh6bzApicwLVf1iznBFkl4RAM7vEt+OvmEF2SQjISmw6APWli8Si3V3sELawxKvj W6BsGYnLk7tZIHrbGSW6d3ayQjgTGCXOTP8LVWUvcermVSaIS/kkJm2bDrWAV6KjTQiixEPi 0a02aGA5SqyYvIwJ4vlZjBK7J/awTGCUn4UUXrMQ4TULKbwWMDKvYhRJLS3OTU8tNtErTswt Ls1L10vOz93ECEwTp/8d/7KDcfExq0OMAhyMSjy8ElUikUKsiWXFlbmHGCU4mJVEeFfEi0YK 8aYkVlalFuXHF5XmpBYfYpTmYFES5+U6dS1CSCA9sSQ1OzW1ILUIJsvEwSnVwGg0dWHeuRhv Q4VFKbt9vVbo3rpbe3HnqZ/tKxQa3z8ue/LxkbzQwyX82xnW6Vterz6jc3JB4xaDX/GZBRxP X7ZemN8d+n2q7Tub1kVOtiyHm8+/NWU7dHzappURWZdSX8VMry8WMf0kt9ZB7llk7L2HEyI4 tOJmTlrVP737stqrz9vvzu3qeKmqxFKckWioxVxUnAgA41VPsQ8DAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprJIsWRmVeSWpSXmKPExsVy+t/xK7ppc0UjDa52MFvMvXuexaJ32z12 iyvtP9kttuz/xm5xvHcHiwOrx5Nrm5k83u+7yubRt2UVYwBzlJtNRmpiSmqRQmpecn5KZl66 rVJoiJuuhZJCXmJuqq1ShK5vSJCSQlliTimQZ2SABhycA9yDlfTtEtwyNp/sZCn4Y13RdkCv gfGDXhcjJ4eEgIlE28UDLBC2mMSFe+vZuhi5OIQEljBKNMx+xAaS4BUQlPgx+R5QEQcHs4C8 xJFL2SBhZgF1iUnzFjFD1PcySdx6vo8RpIZNwEBi3z1bkBoRAUmJ312nmSHqYyUuTXgBNlJY wF3i99MJYHEWAVWJPy92Q61yk9g4dS4bxD1yEiePTWYFsTmB6j+sWQ42XgioZu6/2AmMArOQ HDcL4bhZSI5bwMi8ilEktbQ4Nz232EivODG3uDQvXS85P3cTIzBath37uWUHY9e74EOMAhyM Sjy8ElUikUKsiWXFlbmHGCU4mJVEeFfEi0YK8aYkVlalFuXHF5XmpBYfYjQFOnsis5Rocj4w kvNK4g1NDM0tDY2MLSzMjYyUxHmnfrgSLiSQnliSmp2aWpBaBNPHxMEp1cC4VnTfhIN+RxYk 7GbI0BF+ZZG5InbHVfVtM75wOm/ZcrPNWGS5EVtCu+l3htnqLn8tHn1NO7M4fn0c15947z2V Mgu27zhmNUXZveWzR+jhslnZO85/PJ5uqb3o8DOzjonWO65fm8H8oDxi8fbwEpfb4R1Lqu7P 0J547Hf/Ta/NC16clltx8pDMOyWW4oxEQy3mouJEAB+1mSesAgAA X-MTR: 20000000000000000@CPGS X-CMS-MailID: 20170512113222eucas1p2f0964bb82e343438798c1c341c1cfe8a X-Msg-Generator: CA X-Local-Sender: =?UTF-8?B?QWxleGV5IFBlcmV2YWxvdhtTUlItVmlydHVhbGl6YXRpb24g?= =?UTF-8?B?TGFiG+yCvOyEseyghOyekBtTZW5pb3IgRW5naW5lZXI=?= X-Global-Sender: =?UTF-8?B?QWxleGV5IFBlcmV2YWxvdhtTUlItVmlydHVhbGl6YXRpb24g?= =?UTF-8?B?TGFiG1NhbXN1bmcgRWxlY3Ryb25pY3MbU2VuaW9yIEVuZ2luZWVy?= X-Sender-Code: =?UTF-8?B?QzEwG0NJU0hRG0MxMEdEMDFHRDAxMDE1NA==?= X-HopCount: 7 X-CMS-RootMailID: 20170512113222eucas1p2f0964bb82e343438798c1c341c1cfe8a X-RootMTR: 20170512113222eucas1p2f0964bb82e343438798c1c341c1cfe8a References: <1494588703-9076-1-git-send-email-a.perevalov@samsung.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 210.118.77.13 Subject: [Qemu-devel] [PATCH V4 7/9] migration: calculate vCPU blocktime on dst side X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: i.maximets@samsung.com, dgilbert@redhat.com, peterx@redhat.com, a.perevalov@samsung.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" This patch provides blocktime calculation per vCPU, as a summary and as a overlapped value for all vCPUs. This approach was suggested by Peter Xu, as an improvements of previous approch where QEMU kept tree with faulted page address and cpus bi= tmask in it. Now QEMU is keeping array with faulted page address as value and vCPU as index. It helps to find proper vCPU at UFFD_COPY time. Also it keeps list for blocktime per vCPU (could be traced with page_fault_addr) Blocktime will not calculated if postcopy_blocktime field of MigrationIncomingState wasn't initialized. PostcopyBlocktimeContext is not protected by lock, due to structure itself = is not changing except postcopy end callback, but even in that time will be query-migrate request, query-migrate is not touching page_fault_vcpu_time & vcpu_addr.=20 Also atomic for int64_t is using here, but QEMU's atomic_read/set is not allowing operation on types which bigger than pointer size, although clang/= gcc supports it ("=E2=80=98__atomic=E2=80=99 builtins can be used with any integral scalar = or pointer type that is 1, 2, 4, or 8 bytes in length) also on 32 platforms such as AR= M. Maybe could be a problem with mingw, but postcopy is under linux ifdef. Signed-off-by: Alexey Perevalov --- migration/postcopy-ram.c | 87 ++++++++++++++++++++++++++++++++++++++++++++= +++- migration/trace-events | 5 ++- 2 files changed, 90 insertions(+), 2 deletions(-) diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index fd5aadd..db95747 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -23,6 +23,7 @@ #include "postcopy-ram.h" #include "sysemu/sysemu.h" #include "sysemu/balloon.h" +#include #include "qemu/error-report.h" #include "trace.h" =20 @@ -541,6 +542,86 @@ static int ram_block_enable_notify(const char *block_n= ame, void *host_addr, return 0; } =20 +static int get_mem_fault_cpu_index(uint32_t pid) +{ + CPUState *cpu_iter; + + CPU_FOREACH(cpu_iter) { + if (cpu_iter->thread_id =3D=3D pid) { + return cpu_iter->cpu_index; + } + } + trace_get_mem_fault_cpu_index(pid); + return -1; +} + +static void mark_postcopy_blocktime_begin(uint64_t addr, int cpu) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + PostcopyBlocktimeContext *dc; + int64_t now_ms; + if (!mis->blocktime_ctx || cpu < 0) { + return; + } + now_ms =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIME); + dc =3D mis->blocktime_ctx; + if (dc->vcpu_addr[cpu] =3D=3D 0) { + atomic_inc(&dc->smp_cpus_down); + } + + atomic_xchg__nocheck(&dc->vcpu_addr[cpu], addr); + atomic_xchg__nocheck(&dc->last_begin, now_ms); + atomic_xchg__nocheck(&dc->page_fault_vcpu_time[cpu], now_ms); + + trace_mark_postcopy_blocktime_begin(addr, dc, dc->page_fault_vcpu_time= [cpu], + cpu); +} + +static void mark_postcopy_blocktime_end(uint64_t addr) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + PostcopyBlocktimeContext *dc; + int i, affected_cpu =3D 0; + int64_t now_ms; + bool vcpu_total_blocktime =3D false; + + if (!mis->blocktime_ctx) { + return; + } + dc =3D mis->blocktime_ctx; + now_ms =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIME); + + /* lookup cpu, to clear it, + * that algorithm looks straighforward, but it's not + * optimal, more optimal algorithm is keeping tree or hash + * where key is address value is a list of */ + for (i =3D 0; i < smp_cpus; i++) { + uint64_t vcpu_blocktime =3D 0; + if (atomic_fetch_add(&dc->vcpu_addr[i], 0) !=3D addr) { + continue; + } + atomic_xchg__nocheck(&dc->vcpu_addr[i], 0); + vcpu_blocktime =3D now_ms - + atomic_fetch_add(&dc->page_fault_vcpu_time[i], 0); + affected_cpu +=3D 1; + /* we need to know is that mark_postcopy_end was due to + * faulted page, another possible case it's prefetched + * page and in that case we shouldn't be here */ + if (!vcpu_total_blocktime && + atomic_fetch_add(&dc->smp_cpus_down, 0) =3D=3D smp_cpus) { + vcpu_total_blocktime =3D true; + } + /* continue cycle, due to one page could affect several vCPUs */ + dc->vcpu_blocktime[i] +=3D vcpu_blocktime; + } + + atomic_sub(&dc->smp_cpus_down, affected_cpu); + if (vcpu_total_blocktime) { + dc->total_blocktime +=3D now_ms - atomic_fetch_add(&dc->last_begin= , 0); + } + trace_mark_postcopy_blocktime_end(addr, dc, dc->total_blocktime); +} + /* * Handle faults detected by the USERFAULT markings */ @@ -618,8 +699,11 @@ static void *postcopy_ram_fault_thread(void *opaque) rb_offset &=3D ~(qemu_ram_pagesize(rb) - 1); trace_postcopy_ram_fault_thread_request(msg.arg.pagefault.address, qemu_ram_get_idstr(rb), - rb_offset); + rb_offset, + msg.arg.pagefault.feat.pti= d); =20 + mark_postcopy_blocktime_begin((uintptr_t)(msg.arg.pagefault.addres= s), + get_mem_fault_cpu_index(msg.arg.pagefault.feat.pt= id)); /* * Send the request to the source - we want to request one * of our host page sizes (which is >=3D TPS) @@ -714,6 +798,7 @@ int postcopy_place_page(MigrationIncomingState *mis, vo= id *host, void *from, =20 return -e; } + mark_postcopy_blocktime_end((uint64_t)(uintptr_t)host); =20 trace_postcopy_place_page(host); return 0; diff --git a/migration/trace-events b/migration/trace-events index 5b8ccf3..7bdadbb 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -112,6 +112,8 @@ process_incoming_migration_co_end(int ret, int ps) "ret= =3D%d postcopy-state=3D%d" process_incoming_migration_co_postcopy_end_main(void) "" migration_set_incoming_channel(void *ioc, const char *ioctype) "ioc=3D%p i= octype=3D%s" migration_set_outgoing_channel(void *ioc, const char *ioctype, const char = *hostname) "ioc=3D%p ioctype=3D%s hostname=3D%s" +mark_postcopy_blocktime_begin(uint64_t addr, void *dd, int64_t time, int c= pu) "addr 0x%" PRIx64 " dd %p time %" PRId64 " cpu %d" +mark_postcopy_blocktime_end(uint64_t addr, void *dd, int64_t time) "addr 0= x%" PRIx64 " dd %p time %" PRId64 =20 # migration/rdma.c qemu_rdma_accept_incoming_migration(void) "" @@ -188,7 +190,7 @@ postcopy_ram_enable_notify(void) "" postcopy_ram_fault_thread_entry(void) "" postcopy_ram_fault_thread_exit(void) "" postcopy_ram_fault_thread_quit(void) "" -postcopy_ram_fault_thread_request(uint64_t hostaddr, const char *ramblock,= size_t offset) "Request for HVA=3D%" PRIx64 " rb=3D%s offset=3D%zx" +postcopy_ram_fault_thread_request(uint64_t hostaddr, const char *ramblock,= size_t offset, uint32_t pid) "Request for HVA=3D%" PRIx64 " rb=3D%s offset= =3D%zx %u" postcopy_ram_incoming_cleanup_closeuf(void) "" postcopy_ram_incoming_cleanup_entry(void) "" postcopy_ram_incoming_cleanup_exit(void) "" @@ -197,6 +199,7 @@ save_xbzrle_page_skipping(void) "" save_xbzrle_page_overflow(void) "" ram_save_iterate_big_wait(uint64_t milliconds, int iterations) "big wait: = %" PRIu64 " milliseconds, %d iterations" ram_load_complete(int ret, uint64_t seq_iter) "exit_code %d seq iteration = %" PRIu64 +get_mem_fault_cpu_index(uint32_t pid) "pid %u is not vCPU" =20 # migration/exec.c migration_exec_outgoing(const char *cmd) "cmd=3D%s" --=20 1.8.3.1