From nobody Fri Apr 19 01:52:18 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1547188848043186.6468002289056; Thu, 10 Jan 2019 22:40:48 -0800 (PST) Received: from localhost ([127.0.0.1]:54842 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghqUo-0004W6-Vq for importer@patchew.org; Fri, 11 Jan 2019 01:40:47 -0500 Received: from eggs.gnu.org ([209.51.188.92]:57819) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghqS2-0002Zt-GB for qemu-devel@nongnu.org; Fri, 11 Jan 2019 01:37:55 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ghqS1-0007YM-ED for qemu-devel@nongnu.org; Fri, 11 Jan 2019 01:37:54 -0500 Received: from mail-pf1-x42a.google.com ([2607:f8b0:4864:20::42a]:36699) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ghqS1-0007V4-7S for qemu-devel@nongnu.org; Fri, 11 Jan 2019 01:37:53 -0500 Received: by mail-pf1-x42a.google.com with SMTP id b85so6501256pfc.3 for ; Thu, 10 Jan 2019 22:37:53 -0800 (PST) Received: from localhost.localdomain ([203.205.141.36]) by smtp.gmail.com with ESMTPSA id 78sm141460933pft.184.2019.01.10.22.37.48 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 10 Jan 2019 22:37:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=LmgMEMti+z/l1IMU5qcc3+pH5l7mn1xGFRW4iyILREA=; b=TAAONjqCOLuD0+E5Ao/svOsR2IBEs3yUgnoersZhXQZU8zvfbFDSzFqoYKHfD2AC0n hi2ciNegrOlsq4Mtgta0Yyfs54mYFR9gTUgdYU0yPwIeOZCEkMPdJalISioUJrCxVp3t kOA9mLhiSNsHFaZMKpRnKOXofrnUsy4/9iW924VrNCe15nHxyJ6z3O3vWhiUrDThNJ8f mCYEZtj5xaGiYYC8Fx/pPwHfXpCn7YUMylv++LBQjZpZJVc8zz2hS6mH0ITZ58IRgh9q iBHT4I6JK1gGljfuvUkuh8ZVqdx/L8yGnoGLVGxYC5Z9DT2sm0J9fwTtVHV7nOI+OIDg nggg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=LmgMEMti+z/l1IMU5qcc3+pH5l7mn1xGFRW4iyILREA=; b=Kqq/44lA0tRQEz0IBxu0Hj1i++FJ0FMlZSa7BaQp8zTrZe4TT4XcNNjrSPwP9wtfdh 6DKK4qL5lkgEoYiNycOHtaWHrXeNx846riUg+Vl+fDjFeIyW9aj3lbo7ltVevnRc+hZd cZxefxN2Y3+e0J0xBm/8vs539vD5Tw/xNDLrqem+8O8E7AnPYAeYwcsC09efm9Xo0tAv 8ytEH/XZ1msfPXsns7jOtZrjzEF04TM8d27tFdJQIqHGNEIFeA6xr6iVHtkAXt98UU/B rHY+npTN5u/Lfq0CddRwops0zg7TmjhPuJiPhoWiRAUKQO9jG2ioAhEPLzL0EABgg5i3 EAFg== X-Gm-Message-State: AJcUukdylxdRlq9YDonmFAkG3q1OL2f/9Yrk5KuclxTZ6fz2SNgIKF4V WimROYYXPX6s05t0sqC9hHI= X-Google-Smtp-Source: ALg8bN4J4MYJLbz8KUzD94oX1guKoUicQny9e6bYQWAnF6u7SF4/WqF8Oqsmo73CZRWHqiNXGP2iyQ== X-Received: by 2002:a63:de04:: with SMTP id f4mr9934252pgg.292.1547188672170; Thu, 10 Jan 2019 22:37:52 -0800 (PST) From: guangrong.xiao@gmail.com X-Google-Original-From: xiaoguangrong@tencent.com To: pbonzini@redhat.com, mst@redhat.com, mtosatti@redhat.com Date: Fri, 11 Jan 2019 14:37:30 +0800 Message-Id: <20190111063732.10484-2-xiaoguangrong@tencent.com> X-Mailer: git-send-email 2.14.5 In-Reply-To: <20190111063732.10484-1-xiaoguangrong@tencent.com> References: <20190111063732.10484-1-xiaoguangrong@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::42a Subject: [Qemu-devel] [PATCH v2 1/3] migration: introduce pages-per-second X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, quintela@redhat.com, Xiao Guangrong , qemu-devel@nongnu.org, peterx@redhat.com, dgilbert@redhat.com, wei.w.wang@intel.com, cota@braap.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Xiao Guangrong It introduces a new statistic, pages-per-second, as bandwidth or mbps is not enough to measure the performance of posting pages out as we have compression, xbzrle, which can significantly reduce the amount of the data size, instead, pages-per-second is the one we want Signed-off-by: Xiao Guangrong Reviewed-by: Dr. David Alan Gilbert --- hmp.c | 2 ++ migration/migration.c | 11 ++++++++++- migration/migration.h | 8 ++++++++ migration/ram.c | 6 ++++++ qapi/migration.json | 5 ++++- 5 files changed, 30 insertions(+), 2 deletions(-) diff --git a/hmp.c b/hmp.c index 80aa5ab504..944e3e072d 100644 --- a/hmp.c +++ b/hmp.c @@ -236,6 +236,8 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict) info->ram->page_size >> 10); monitor_printf(mon, "multifd bytes: %" PRIu64 " kbytes\n", info->ram->multifd_bytes >> 10); + monitor_printf(mon, "pages-per-second: %" PRIu64 "\n", + info->ram->pages_per_second); =20 if (info->ram->dirty_pages_rate) { monitor_printf(mon, "dirty pages rate: %" PRIu64 " pages\n", diff --git a/migration/migration.c b/migration/migration.c index ffc4d9e556..a82d594f29 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -777,6 +777,7 @@ static void populate_ram_info(MigrationInfo *info, Migr= ationState *s) info->ram->postcopy_requests =3D ram_counters.postcopy_requests; info->ram->page_size =3D qemu_target_page_size(); info->ram->multifd_bytes =3D ram_counters.multifd_bytes; + info->ram->pages_per_second =3D s->pages_per_second; =20 if (migrate_use_xbzrle()) { info->has_xbzrle_cache =3D true; @@ -1563,6 +1564,7 @@ void migrate_init(MigrationState *s) s->rp_state.from_dst_file =3D NULL; s->rp_state.error =3D false; s->mbps =3D 0.0; + s->pages_per_second =3D 0.0; s->downtime =3D 0; s->expected_downtime =3D 0; s->setup_time =3D 0; @@ -2881,7 +2883,7 @@ static void migration_calculate_complete(MigrationSta= te *s) static void migration_update_counters(MigrationState *s, int64_t current_time) { - uint64_t transferred, time_spent; + uint64_t transferred, transferred_pages, time_spent; uint64_t current_bytes; /* bytes transferred since the beginning */ double bandwidth; =20 @@ -2898,6 +2900,11 @@ static void migration_update_counters(MigrationState= *s, s->mbps =3D (((double) transferred * 8.0) / ((double) time_spent / 1000.0)) / 1000.0 / 1000.0; =20 + transferred_pages =3D ram_get_total_transferred_pages() - + s->iteration_initial_pages; + s->pages_per_second =3D (double) transferred_pages / + (((double) time_spent / 1000.0)); + /* * if we haven't sent anything, we don't want to * recalculate. 10000 is a small enough number for our purposes @@ -2910,6 +2917,7 @@ static void migration_update_counters(MigrationState = *s, =20 s->iteration_start_time =3D current_time; s->iteration_initial_bytes =3D current_bytes; + s->iteration_initial_pages =3D ram_get_total_transferred_pages(); =20 trace_migrate_transferred(transferred, time_spent, bandwidth, s->threshold_size); @@ -3314,6 +3322,7 @@ static void migration_instance_init(Object *obj) =20 ms->state =3D MIGRATION_STATUS_NONE; ms->mbps =3D -1; + ms->pages_per_second =3D -1; qemu_sem_init(&ms->pause_sem, 0); qemu_mutex_init(&ms->error_mutex); =20 diff --git a/migration/migration.h b/migration/migration.h index e413d4d8b6..810effc384 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -126,6 +126,12 @@ struct MigrationState */ QemuSemaphore rate_limit_sem; =20 + /* pages already send at the beggining of current interation */ + uint64_t iteration_initial_pages; + + /* pages transferred per second */ + double pages_per_second; + /* bytes already send at the beggining of current interation */ uint64_t iteration_initial_bytes; /* time at the start of current iteration */ @@ -271,6 +277,8 @@ bool migrate_use_block_incremental(void); int migrate_max_cpu_throttle(void); bool migrate_use_return_path(void); =20 +uint64_t ram_get_total_transferred_pages(void); + bool migrate_use_compression(void); int migrate_compress_level(void); int migrate_compress_threads(void); diff --git a/migration/ram.c b/migration/ram.c index 7e7deec4d8..7e429b0502 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -1593,6 +1593,12 @@ uint64_t ram_pagesize_summary(void) return summary; } =20 +uint64_t ram_get_total_transferred_pages(void) +{ + return ram_counters.normal + ram_counters.duplicate + + compression_counters.pages + xbzrle_counters.pages; +} + static void migration_update_rates(RAMState *rs, int64_t end_time) { uint64_t page_count =3D rs->target_page_count - rs->target_page_count_= prev; diff --git a/qapi/migration.json b/qapi/migration.json index 31b589ec26..c5babd03b0 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -41,6 +41,9 @@ # # @multifd-bytes: The number of bytes sent through multifd (since 3.0) # +# @pages-per-second: the number of memory pages transferred per second +# (Since 3.2) +# # Since: 0.14.0 ## { 'struct': 'MigrationStats', @@ -49,7 +52,7 @@ 'normal-bytes': 'int', 'dirty-pages-rate' : 'int', 'mbps' : 'number', 'dirty-sync-count' : 'int', 'postcopy-requests' : 'int', 'page-size' : 'int', - 'multifd-bytes' : 'uint64' } } + 'multifd-bytes' : 'uint64', 'pages-per-second' : 'uint64' } } =20 ## # @XBZRLECacheStats: --=20 2.14.5 From nobody Fri Apr 19 01:52:18 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1547188797124515.9945858114837; Thu, 10 Jan 2019 22:39:57 -0800 (PST) Received: from localhost ([127.0.0.1]:54626 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghqU0-00040D-2P for importer@patchew.org; Fri, 11 Jan 2019 01:39:56 -0500 Received: from eggs.gnu.org ([209.51.188.92]:57836) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghqSB-0002eu-RB for qemu-devel@nongnu.org; Fri, 11 Jan 2019 01:38:04 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ghqS9-0007gK-NH for qemu-devel@nongnu.org; Fri, 11 Jan 2019 01:38:03 -0500 Received: from mail-pl1-x634.google.com ([2607:f8b0:4864:20::634]:46000) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ghqS9-0007dk-9i for qemu-devel@nongnu.org; Fri, 11 Jan 2019 01:38:01 -0500 Received: by mail-pl1-x634.google.com with SMTP id a14so6316509plm.12 for ; Thu, 10 Jan 2019 22:37:57 -0800 (PST) Received: from localhost.localdomain ([203.205.141.36]) by smtp.gmail.com with ESMTPSA id 78sm141460933pft.184.2019.01.10.22.37.52 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 10 Jan 2019 22:37:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=EaEm7GhKPLbm8j7I19095wyJJWd37TUtwImyNV9Ffpw=; b=tkh3duRO6nuUWaTs25iLN/Fb1O3BUz89l/vxM97eUGMtyS5k2qfgxVO+Xi0E/jkF0K cOSbw2cfy6fLF7i5AG4HoYRqtf2RYMKKxRiQrKm0REvGNZnZQ3qytdL/l8Q3ml3ar3oC 9J8IKxM6VkziJaL3tkxCP7ujn2LdXmTBFWCLUaAvXg8U36lqMZ+jvbtcuG82ghJBra9z kyw4LFRJOKhl9wtLS4yrh6Mfzn6MbBevqsOA0F3z2RlW+a6EqQEANI9c+yNQLeyz1W/k yuRmoWLPgpnOw7m04VXc5+gv8kk1pWYsqDlsrfOHnewbwvrw5NevTjW+4Mx/NvsT/U5x MUcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=EaEm7GhKPLbm8j7I19095wyJJWd37TUtwImyNV9Ffpw=; b=S1mEpH6dv6X9YyykyDxbinzexMh0JeUzQ1wstlt2mREGFdeKqsMNKT3ab7pt+Ohrdh 48kTiTmMNMe+C/dkZ13NiijM6Eg2rdVnA7VmK9Eu2omE+SKNWP6CrPqUXo3NuNnL/oXH oH9n+9rhQcMMMBhXJ5kcTML9Nc031sBMGXFk74qjXaDZR3l1qpBVlARxKl/KKigViaO3 Kluep0kY3T4iJT1T76Q/QBgvAhuAvzhb2rWeZEDy4iA550+cE9CxQ6KRbJ2iyVxLtUZ8 Z1ciLq8L1DxAh4uk3NHXETZQugW7ETmuVduuWUoWoodB89+74q2I5DxHCS3sjANWGI9k 07Dg== X-Gm-Message-State: AJcUukedYvtL8lvtXPjmKqdbjNUgao9l51Il5NfpbdIkJ307HcAmQXQR LmWWMjqTsuCLeHZp1Ow/OX4= X-Google-Smtp-Source: ALg8bN63YWcQNfj+sMHNFX2B3oEp62u/Y/ZVIgH5NNUFqH0ne4ue/kHwigepgVVSasTo7YW2+IyB7A== X-Received: by 2002:a17:902:34a:: with SMTP id 68mr13751093pld.268.1547188676338; Thu, 10 Jan 2019 22:37:56 -0800 (PST) From: guangrong.xiao@gmail.com X-Google-Original-From: xiaoguangrong@tencent.com To: pbonzini@redhat.com, mst@redhat.com, mtosatti@redhat.com Date: Fri, 11 Jan 2019 14:37:31 +0800 Message-Id: <20190111063732.10484-3-xiaoguangrong@tencent.com> X-Mailer: git-send-email 2.14.5 In-Reply-To: <20190111063732.10484-1-xiaoguangrong@tencent.com> References: <20190111063732.10484-1-xiaoguangrong@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::634 Subject: [Qemu-devel] [PATCH v2 2/3] migration: fix memory leak when updating tls-creds and tls-hostname X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, quintela@redhat.com, Xiao Guangrong , qemu-devel@nongnu.org, peterx@redhat.com, dgilbert@redhat.com, wei.w.wang@intel.com, cota@braap.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Xiao Guangrong If we update parameter, tls-creds and tls-hostname, these string values are duplicated to local variables in migrate_params_test_apply() by using g_strdup(), however these new allocated memory are missed to be freed Actually, they are not used to check anything, we can directly skip them Signed-off-by: Xiao Guangrong --- migration/migration.c | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index a82d594f29..fb39d7bec1 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1145,16 +1145,6 @@ static void migrate_params_test_apply(MigrateSetPara= meters *params, dest->cpu_throttle_increment =3D params->cpu_throttle_increment; } =20 - if (params->has_tls_creds) { - assert(params->tls_creds->type =3D=3D QTYPE_QSTRING); - dest->tls_creds =3D g_strdup(params->tls_creds->u.s); - } - - if (params->has_tls_hostname) { - assert(params->tls_hostname->type =3D=3D QTYPE_QSTRING); - dest->tls_hostname =3D g_strdup(params->tls_hostname->u.s); - } - if (params->has_max_bandwidth) { dest->max_bandwidth =3D params->max_bandwidth; } --=20 2.14.5 From nobody Fri Apr 19 01:52:18 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1547188922139612.9780481073439; Thu, 10 Jan 2019 22:42:02 -0800 (PST) Received: from localhost ([127.0.0.1]:55165 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghqW1-0005Pq-1S for importer@patchew.org; Fri, 11 Jan 2019 01:42:01 -0500 Received: from eggs.gnu.org ([209.51.188.92]:57850) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghqSD-0002gm-VV for qemu-devel@nongnu.org; Fri, 11 Jan 2019 01:38:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ghqSB-0007hL-Om for qemu-devel@nongnu.org; Fri, 11 Jan 2019 01:38:05 -0500 Received: from mail-pf1-x441.google.com ([2607:f8b0:4864:20::441]:46937) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ghqS9-0007fr-Ur for qemu-devel@nongnu.org; Fri, 11 Jan 2019 01:38:03 -0500 Received: by mail-pf1-x441.google.com with SMTP id c73so6480726pfe.13 for ; Thu, 10 Jan 2019 22:38:01 -0800 (PST) Received: from localhost.localdomain ([203.205.141.36]) by smtp.gmail.com with ESMTPSA id 78sm141460933pft.184.2019.01.10.22.37.56 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 10 Jan 2019 22:38:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=v8TI0xcCkkfBJaNx2AA3N0vTEd+Ei48B84lFTkfWIQ8=; b=YIjeUeVU+SJIepoTZiCCSWOLk5n/QA351RP7VpcPXfF48V3ngz3xSz3aykP2/Vx/+v 41Z8u4U+TN/M7QbAMR7ub3plzCQZgDiaeyROTWIHWwa0BB5pe+OaAnTkH6gxLJmyd+0A KVp/neOTZtsQ51cTWhqCwKVGXU/wvSNuQ0kvy1McXuQ976Gsr7MhXXYMKpJ0FrzpEwG5 ZfVoh0jlW7Eh1hHLCI43TSGcf8OYY0j6DHXeCRFwHiwED1EM24iWgZ/0UV2OXseTGmzl oy/UkZwWJme8iJyYd4meJSsU/GLI4TpI2IzQ8hkj6fcExVDPI/sHEPB+nSmxq+Zr6atO yfQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=v8TI0xcCkkfBJaNx2AA3N0vTEd+Ei48B84lFTkfWIQ8=; b=VjjLwJARPOikFH3QMObmnPX+r6Ngnpe+Ko66UNgy/hX2IumeSycAAoVwNtMywEIKZ3 9kKUYbdQSDFvS3Hm/NjKlQf8exWS6wxQCE/zBK2KqgNvJzE6omNnP7CuFnNXYqCrAmH5 5s5GkdQnE44QxIzum3wut25IhO7HUa3ZHegnlNYu1EzE2Lkn0B+/NJ5H6MSOubEeQlfB kglDIoEjsCE8v5hcXYqTKaIlI5zMj/4/0Nvq9teBdcoSQGz+aGg31L4XrbG/g4KpHdbi XrPQuZ7RqKGAoAUS0AZqwehd8MJ83brTi1Et+sftDyT0mNB0jluy0A5xaLNg/DdjfKPG 9qQQ== X-Gm-Message-State: AJcUukfpspZEPek3dHHb1BUrTISWPA2VGyZUlv0SGoe7gfx7/Yq9jB5t NteK5YxlS0UHNO0maMYmPfVMYOD4 X-Google-Smtp-Source: ALg8bN6BcktU3G0XNVeWpK1E4sAq8G5sEM4CMagJMBc2zi+HbGQ/Hhll1kFp6IEZ2WTXjY4jeSH3DQ== X-Received: by 2002:a63:6d48:: with SMTP id i69mr11780057pgc.215.1547188680651; Thu, 10 Jan 2019 22:38:00 -0800 (PST) From: guangrong.xiao@gmail.com X-Google-Original-From: xiaoguangrong@tencent.com To: pbonzini@redhat.com, mst@redhat.com, mtosatti@redhat.com Date: Fri, 11 Jan 2019 14:37:32 +0800 Message-Id: <20190111063732.10484-4-xiaoguangrong@tencent.com> X-Mailer: git-send-email 2.14.5 In-Reply-To: <20190111063732.10484-1-xiaoguangrong@tencent.com> References: <20190111063732.10484-1-xiaoguangrong@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::441 Subject: [Qemu-devel] [PATCH v2 3/3] migration: introduce adaptive model for waiting thread X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, quintela@redhat.com, Xiao Guangrong , qemu-devel@nongnu.org, peterx@redhat.com, dgilbert@redhat.com, wei.w.wang@intel.com, cota@braap.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Xiao Guangrong Currently we have two behaviors if all threads are busy to do compression, the main thread mush wait one of them becoming free if @compress-wait-thread set to on or the main thread can directly return without wait and post the page out as normal one Both of them have its profits and short-comes, however, if the bandwidth is not limited extremely so that compression can not use out all of it bandwid= th, at the same time, the migration process is easily throttled if we posted too may pages as normal pages. None of them can work properly under this case In order to use the bandwidth more effectively, we introduce the third behavior, adaptive, which make the main thread wait if there is no bandwidth left or let the page go out as normal page if there has enough bandwidth to make sure the migration process will not be throttled Signed-off-by: Xiao Guangrong --- hmp.c | 6 ++- migration/migration.c | 116 ++++++++++++++++++++++++++++++++++++++++++++--= ---- migration/migration.h | 13 ++++++ migration/ram.c | 116 +++++++++++++++++++++++++++++++++++++++++++++-= ---- qapi/migration.json | 39 +++++++++++------ 5 files changed, 251 insertions(+), 39 deletions(-) diff --git a/hmp.c b/hmp.c index 944e3e072d..0705833e14 100644 --- a/hmp.c +++ b/hmp.c @@ -284,6 +284,8 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict) info->compression->compressed_size); monitor_printf(mon, "compression rate: %0.2f\n", info->compression->compression_rate); + monitor_printf(mon, "compress-no-wait-weight: %"PRIu64"\n", + info->compression->compress_no_wait_weight); } =20 if (info->has_cpu_throttle_percentage) { @@ -345,7 +347,7 @@ void hmp_info_migrate_parameters(Monitor *mon, const QD= ict *qdict) assert(params->has_compress_wait_thread); monitor_printf(mon, "%s: %s\n", MigrationParameter_str(MIGRATION_PARAMETER_COMPRESS_WAIT_THREA= D), - params->compress_wait_thread ? "on" : "off"); + params->compress_wait_thread); assert(params->has_decompress_threads); monitor_printf(mon, "%s: %u\n", MigrationParameter_str(MIGRATION_PARAMETER_DECOMPRESS_THREADS), @@ -1679,7 +1681,7 @@ void hmp_migrate_set_parameter(Monitor *mon, const QD= ict *qdict) break; case MIGRATION_PARAMETER_COMPRESS_WAIT_THREAD: p->has_compress_wait_thread =3D true; - visit_type_bool(v, param, &p->compress_wait_thread, &err); + visit_type_str(v, param, &p->compress_wait_thread, &err); break; case MIGRATION_PARAMETER_DECOMPRESS_THREADS: p->has_decompress_threads =3D true; diff --git a/migration/migration.c b/migration/migration.c index fb39d7bec1..0be0b02c8a 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -31,6 +31,7 @@ #include "migration/vmstate.h" #include "block/block.h" #include "qapi/error.h" +#include "qapi/string-input-visitor.h" #include "qapi/qapi-commands-migration.h" #include "qapi/qapi-events-migration.h" #include "qapi/qmp/qerror.h" @@ -705,7 +706,7 @@ MigrationParameters *qmp_query_migrate_parameters(Error= **errp) params->has_compress_threads =3D true; params->compress_threads =3D s->parameters.compress_threads; params->has_compress_wait_thread =3D true; - params->compress_wait_thread =3D s->parameters.compress_wait_thread; + params->compress_wait_thread =3D g_strdup(s->parameters.compress_wait_= thread); params->has_decompress_threads =3D true; params->decompress_threads =3D s->parameters.decompress_threads; params->has_cpu_throttle_initial =3D true; @@ -800,6 +801,8 @@ static void populate_ram_info(MigrationInfo *info, Migr= ationState *s) compression_counters.compressed_size; info->compression->compression_rate =3D compression_counters.compression_rate; + info->compression->compress_no_wait_weight =3D + compression_counters.compress_no_wait_weig= ht; } =20 if (cpu_throttle_active()) { @@ -1016,6 +1019,68 @@ void qmp_migrate_set_capabilities(MigrationCapabilit= yStatusList *params, } } =20 +static int get_compress_wait_thread(const MigrationParameters *params) +{ + Visitor *v =3D string_input_visitor_new(params->compress_wait_thread); + Error *err =3D NULL; + int wait_thread =3D COMPRESS_WAIT_THREAD_ERR; + char *value; + bool wait; + + visit_type_str(v, "compress-wait-thread", &value, &err); + if (err) { + goto exit; + } + + if (!strcmp(value, "adaptive")) { + wait_thread =3D COMPRESS_WAIT_THREAD_ADAPTIVE; + goto free_value; + } + + visit_type_bool(v, "compress-wait-thread", &wait, &err); + if (!err) { + wait_thread =3D wait; + } + +free_value: + g_free(value); +exit: + visit_free(v); + error_free(err); + return wait_thread; +} + +static bool +check_compress_wait_thread(MigrationParameters *params, Error **errp) +{ + if (!params->has_compress_wait_thread) { + return true; + } + + if (get_compress_wait_thread(params) =3D=3D COMPRESS_WAIT_THREAD_ERR) { + error_setg(errp, + "Parameter 'compress-wait-thread' expects 'adaptive' or a bool va= lue"); + return false; + } + + return true; +} + +static void update_compress_wait_thread(MigrationState *s) +{ + s->compress_wait_thread =3D get_compress_wait_thread(&s->parameters); + assert(s->compress_wait_thread !=3D COMPRESS_WAIT_THREAD_ERR); +} + +int migrate_compress_wait_thread(void) +{ + MigrationState *s; + + s =3D migrate_get_current(); + + return s->compress_wait_thread; +} + /* * Check whether the parameters are valid. Error will be put into errp * (if provided). Return true if valid, otherwise false. @@ -1036,6 +1101,10 @@ static bool migrate_params_check(MigrationParameters= *params, Error **errp) return false; } =20 + if (!check_compress_wait_thread(params, errp)) { + return false; + } + if (params->has_decompress_threads && (params->decompress_threads < 1)= ) { error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "decompress_threads", @@ -1130,7 +1199,7 @@ static void migrate_params_test_apply(MigrateSetParam= eters *params, } =20 if (params->has_compress_wait_thread) { - dest->compress_wait_thread =3D params->compress_wait_thread; + dest->compress_wait_thread =3D g_strdup(params->compress_wait_thre= ad); } =20 if (params->has_decompress_threads) { @@ -1177,6 +1246,14 @@ static void migrate_params_test_apply(MigrateSetPara= meters *params, } } =20 +static void migrate_params_test_destroy(MigrateSetParameters *params, + MigrationParameters *dest) +{ + if (params->has_compress_wait_thread) { + g_free(dest->compress_wait_thread); + } +} + static void migrate_params_apply(MigrateSetParameters *params, Error **err= p) { MigrationState *s =3D migrate_get_current(); @@ -1192,7 +1269,10 @@ static void migrate_params_apply(MigrateSetParameter= s *params, Error **errp) } =20 if (params->has_compress_wait_thread) { - s->parameters.compress_wait_thread =3D params->compress_wait_threa= d; + g_free(s->parameters.compress_wait_thread); + s->parameters.compress_wait_thread =3D + g_strdup(params->compress_wait_thr= ead); + update_compress_wait_thread(s); } =20 if (params->has_decompress_threads) { @@ -1282,10 +1362,12 @@ void qmp_migrate_set_parameters(MigrateSetParameter= s *params, Error **errp) =20 if (!migrate_params_check(&tmp, errp)) { /* Invalid parameter */ - return; + goto exit; } =20 migrate_params_apply(params, errp); +exit: + migrate_params_test_destroy(params, &tmp); } =20 =20 @@ -1571,6 +1653,7 @@ void migrate_init(MigrationState *s) s->vm_was_running =3D false; s->iteration_initial_bytes =3D 0; s->threshold_size =3D 0; + update_compress_wait_thread(s); } =20 static GSList *migration_blockers; @@ -1917,40 +2000,40 @@ bool migrate_postcopy_blocktime(void) return s->enabled_capabilities[MIGRATION_CAPABILITY_POSTCOPY_BLOCKTIME= ]; } =20 -bool migrate_use_compression(void) +int64_t migrate_max_bandwidth(void) { MigrationState *s; =20 s =3D migrate_get_current(); =20 - return s->enabled_capabilities[MIGRATION_CAPABILITY_COMPRESS]; + return s->parameters.max_bandwidth; } =20 -int migrate_compress_level(void) +bool migrate_use_compression(void) { MigrationState *s; =20 s =3D migrate_get_current(); =20 - return s->parameters.compress_level; + return s->enabled_capabilities[MIGRATION_CAPABILITY_COMPRESS]; } =20 -int migrate_compress_threads(void) +int migrate_compress_level(void) { MigrationState *s; =20 s =3D migrate_get_current(); =20 - return s->parameters.compress_threads; + return s->parameters.compress_level; } =20 -int migrate_compress_wait_thread(void) +int migrate_compress_threads(void) { MigrationState *s; =20 s =3D migrate_get_current(); =20 - return s->parameters.compress_wait_thread; + return s->parameters.compress_threads; } =20 int migrate_decompress_threads(void) @@ -2895,6 +2978,8 @@ static void migration_update_counters(MigrationState = *s, s->pages_per_second =3D (double) transferred_pages / (((double) time_spent / 1000.0)); =20 + compress_adaptive_update(s->mbps); + /* * if we haven't sent anything, we don't want to * recalculate. 10000 is a small enough number for our purposes @@ -3228,8 +3313,8 @@ static Property migration_properties[] =3D { DEFINE_PROP_UINT8("x-compress-threads", MigrationState, parameters.compress_threads, DEFAULT_MIGRATE_COMPRESS_THREAD_COUNT), - DEFINE_PROP_BOOL("x-compress-wait-thread", MigrationState, - parameters.compress_wait_thread, true), + DEFINE_PROP_STRING("x-compress-wait-thread", MigrationState, + parameters.compress_wait_thread), DEFINE_PROP_UINT8("x-decompress-threads", MigrationState, parameters.decompress_threads, DEFAULT_MIGRATE_DECOMPRESS_THREAD_COUNT), @@ -3297,6 +3382,7 @@ static void migration_instance_finalize(Object *obj) qemu_mutex_destroy(&ms->qemu_file_lock); g_free(params->tls_hostname); g_free(params->tls_creds); + g_free(params->compress_wait_thread); qemu_sem_destroy(&ms->rate_limit_sem); qemu_sem_destroy(&ms->pause_sem); qemu_sem_destroy(&ms->postcopy_pause_sem); @@ -3318,10 +3404,12 @@ static void migration_instance_init(Object *obj) =20 params->tls_hostname =3D g_strdup(""); params->tls_creds =3D g_strdup(""); + params->compress_wait_thread =3D g_strdup("on"); =20 /* Set has_* up only for parameter checks */ params->has_compress_level =3D true; params->has_compress_threads =3D true; + params->has_compress_wait_thread =3D true; params->has_decompress_threads =3D true; params->has_cpu_throttle_initial =3D true; params->has_cpu_throttle_increment =3D true; diff --git a/migration/migration.h b/migration/migration.h index 810effc384..f4958a9354 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -146,6 +146,8 @@ struct MigrationState /* params from 'migrate-set-parameters' */ MigrationParameters parameters; =20 + int compress_wait_thread; + int state; =20 /* State related to return path */ @@ -276,13 +278,24 @@ bool migrate_use_block(void); bool migrate_use_block_incremental(void); int migrate_max_cpu_throttle(void); bool migrate_use_return_path(void); +int64_t migrate_max_bandwidth(void); =20 uint64_t ram_get_total_transferred_pages(void); =20 bool migrate_use_compression(void); int migrate_compress_level(void); int migrate_compress_threads(void); + +enum { + COMPRESS_WAIT_THREAD_OFF =3D 0, + COMPRESS_WAIT_THREAD_ON =3D 1, + COMPRESS_WAIT_THREAD_ADAPTIVE =3D 2, + COMPRESS_WAIT_THREAD_ERR =3D 3, +}; int migrate_compress_wait_thread(void); + +void compress_adaptive_update(double mbps); + int migrate_decompress_threads(void); bool migrate_use_events(void); bool migrate_postcopy_blocktime(void); diff --git a/migration/ram.c b/migration/ram.c index 7e429b0502..869724a70e 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -276,6 +276,8 @@ struct RAMSrcPageRequest { QSIMPLEQ_ENTRY(RAMSrcPageRequest) next_req; }; =20 +#define COMPRESS_BUSY_COUNT_PERIOD 200 + /* State of RAM for migration */ struct RAMState { /* QEMUFile used for this migration */ @@ -292,6 +294,19 @@ struct RAMState { bool ram_bulk_stage; /* How many times we have dirty too many pages */ int dirty_rate_high_cnt; + + /* used by by compress-wait-thread-adaptive */ + /* + * the count for the case that all compress threads are busy to + * handle a page in a period + */ + uint8_t compress_busy_count; + /* + * the number of pages that can be directly posted as normal page when + * all compress threads are busy in a period + */ + uint8_t compress_no_wait_left; + /* these variables are used for bitmap sync */ /* last time we did a full bitmap_sync */ int64_t time_last_bitmap_sync; @@ -470,6 +485,8 @@ static void compress_threads_save_cleanup(void) comp_param =3D NULL; } =20 +static void compress_adaptive_init(void); + static int compress_threads_save_setup(void) { int i, thread_count; @@ -477,6 +494,9 @@ static int compress_threads_save_setup(void) if (!migrate_use_compression()) { return 0; } + + compress_adaptive_init(); + thread_count =3D migrate_compress_threads(); compress_threads =3D g_new0(QemuThread, thread_count); comp_param =3D g_new0(CompressParam, thread_count); @@ -1599,6 +1619,68 @@ uint64_t ram_get_total_transferred_pages(void) compression_counters.pages + xbzrle_counters.pages; } =20 +static void compress_adaptive_init(void) +{ + /* fully wait on default. */ + compression_counters.compress_no_wait_weight =3D 0; + ram_state->compress_no_wait_left =3D 0; + ram_state->compress_busy_count =3D 0; +} + +void compress_adaptive_update(double mbps) +{ + int64_t rate_limit, remain_bw, max_bw =3D migrate_max_bandwidth(); + int compress_wait_thread =3D migrate_compress_wait_thread(); + + if (!migrate_use_compression() || + !(compress_wait_thread =3D=3D COMPRESS_WAIT_THREAD_ADAPTIVE)) { + return; + } + + /* no bandwith is set to the file then we can not do adaptive adjustme= nt */ + rate_limit =3D qemu_file_get_rate_limit(migrate_get_current()->to_dst_= file); + if (rate_limit =3D=3D 0 || rate_limit =3D=3D INT64_MAX) { + return; + } + + max_bw =3D (max_bw >> 20) * 8; + remain_bw =3D abs(max_bw - (int64_t)(mbps)); + if (remain_bw <=3D (max_bw / 20)) { + /* if we have used all the bandwidth, let's compress more. */ + if (compression_counters.compress_no_wait_weight) { + compression_counters.compress_no_wait_weight--; + } + goto exit; + } + + /* have enough bandwidth left, do not need to compress so aggressively= */ + if (compression_counters.compress_no_wait_weight !=3D + COMPRESS_BUSY_COUNT_PERIOD) { + compression_counters.compress_no_wait_weight++; + } + +exit: + ram_state->compress_busy_count =3D 0; + ram_state->compress_no_wait_left =3D + compression_counters.compress_no_wait_weight; +} + +static bool compress_adaptive_need_wait(void) +{ + if (++ram_state->compress_busy_count =3D=3D COMPRESS_BUSY_COUNT_PERIOD= ) { + ram_state->compress_busy_count =3D 0; + ram_state->compress_no_wait_left =3D + compression_counters.compress_no_wait_weight; + } + + if (ram_state->compress_no_wait_left) { + ram_state->compress_no_wait_left--; + return false; + } + + return true; +} + static void migration_update_rates(RAMState *rs, int64_t end_time) { uint64_t page_count =3D rs->target_page_count - rs->target_page_count_= prev; @@ -1975,7 +2057,11 @@ static int compress_page_with_multi_thread(RAMState = *rs, RAMBlock *block, ram_addr_t offset) { int idx, thread_count, bytes_xmit =3D -1, pages =3D -1; - bool wait =3D migrate_compress_wait_thread(); + int compress_wait_thread =3D migrate_compress_wait_thread(); + bool wait, adaptive; + + wait =3D (adaptive =3D=3D COMPRESS_WAIT_THREAD_ON); + adaptive =3D (adaptive =3D=3D COMPRESS_WAIT_THREAD_ADAPTIVE); =20 thread_count =3D migrate_compress_threads(); qemu_mutex_lock(&comp_done_lock); @@ -1990,20 +2076,29 @@ retry: qemu_mutex_unlock(&comp_param[idx].mutex); pages =3D 1; update_compress_thread_counts(&comp_param[idx], bytes_xmit); - break; + goto exit; } } =20 + if (adaptive && !wait) { + /* it is the first time go to the loop */ + wait =3D compress_adaptive_need_wait(); + } + /* - * wait for the free thread if the user specifies 'compress-wait-threa= d', - * otherwise we will post the page out in the main thread as normal pa= ge. + * wait for the free thread if the user specifies + * 'compress-wait-thread-adaptive' that detected the bandwidth is + * not enough or compress-wait-thread', otherwise we will post the + * page out in the main thread as normal page. */ - if (pages < 0 && wait) { + if (wait) { qemu_cond_wait(&comp_done_cond, &comp_done_lock); goto retry; } - qemu_mutex_unlock(&comp_done_lock); =20 + +exit: + qemu_mutex_unlock(&comp_done_lock); return pages; } =20 @@ -3153,19 +3248,18 @@ static int ram_save_setup(QEMUFile *f, void *opaque) RAMState **rsp =3D opaque; RAMBlock *block; =20 - if (compress_threads_save_setup()) { - return -1; - } - /* migration has already setup the bitmap, reuse it. */ if (!migration_in_colo_state()) { if (ram_init_all(rsp) !=3D 0) { - compress_threads_save_cleanup(); return -1; } } (*rsp)->f =3D f; =20 + if (compress_threads_save_setup()) { + return -1; + } + rcu_read_lock(); =20 qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE); diff --git a/qapi/migration.json b/qapi/migration.json index c5babd03b0..0220a0945b 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -93,11 +93,16 @@ # # @compression-rate: rate of compressed size # +# @compress-no-wait-weight: it controls how many pages are directly posted +# out as normal page when all compression threads are currently busy. +# Only available if compress-wait-thread =3D adaptive. (Since 3.2) +# # Since: 3.1 ## { 'struct': 'CompressionStats', 'data': {'pages': 'int', 'busy': 'int', 'busy-rate': 'number', - 'compressed-size': 'int', 'compression-rate': 'number' } } + 'compressed-size': 'int', 'compression-rate': 'number', + 'compress-no-wait-weight': 'int'} } =20 ## # @MigrationStatus: @@ -489,9 +494,13 @@ # the compression thread count is an integer between 1 and 255. # # @compress-wait-thread: Controls behavior when all compression threads are -# currently busy. If true (default), wait for a free -# compression thread to become available; otherwise, -# send the page uncompressed. (Since 3.1) +# currently busy. If 'true/on' (default), wait for a free +# compression thread to become available; if 'false/off', send the +# page uncompressed. (Since 3.1) +# If it is 'adaptive', the behavior is adaptively controlled bas= ed on +# the rate limit. If it has enough bandwidth, it acts as +# compress-wait-thread is off. (Since 3.2) +# # # @decompress-threads: Set decompression thread count to be used in live # migration, the decompression thread count is an integer between= 1 @@ -577,9 +586,12 @@ # @compress-threads: compression thread count # # @compress-wait-thread: Controls behavior when all compression threads are -# currently busy. If true (default), wait for a free -# compression thread to become available; otherwise, -# send the page uncompressed. (Since 3.1) +# currently busy. If 'true/on' (default), wait for a free +# compression thread to become available; if 'false/off', send the +# page uncompressed. (Since 3.1) +# If it is 'adaptive', the behavior is adaptively controlled bas= ed on +# the rate limit. If it has enough bandwidth, it acts as +# compress-wait-thread is off. (Since 3.2) # # @decompress-threads: decompression thread count # @@ -655,7 +667,7 @@ { 'struct': 'MigrateSetParameters', 'data': { '*compress-level': 'int', '*compress-threads': 'int', - '*compress-wait-thread': 'bool', + '*compress-wait-thread': 'str', '*decompress-threads': 'int', '*cpu-throttle-initial': 'int', '*cpu-throttle-increment': 'int', @@ -697,9 +709,12 @@ # @compress-threads: compression thread count # # @compress-wait-thread: Controls behavior when all compression threads are -# currently busy. If true (default), wait for a free -# compression thread to become available; otherwise, -# send the page uncompressed. (Since 3.1) +# currently busy. If 'true/on' (default), wait for a free +# compression thread to become available; if 'false/off', send the +# page uncompressed. (Since 3.1) +# If it is 'adaptive', the behavior is adaptively controlled bas= ed on +# the rate limit. If it has enough bandwidth, it acts as +# compress-wait-thread is off. (Since 3.2) # # @decompress-threads: decompression thread count # @@ -771,7 +786,7 @@ { 'struct': 'MigrationParameters', 'data': { '*compress-level': 'uint8', '*compress-threads': 'uint8', - '*compress-wait-thread': 'bool', + '*compress-wait-thread': 'str', '*decompress-threads': 'uint8', '*cpu-throttle-initial': 'uint8', '*cpu-throttle-increment': 'uint8', --=20 2.14.5