From nobody Sun May 5 15:25:30 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nutanix.com ARC-Seal: i=1; a=rsa-sha256; t=1560530461; cv=none; d=zoho.com; s=zohoarc; b=CfPldgh2/0HMJope6nP6KZcomsPtOoC/daoCUqHih/46fH42KIVC/QJIyUwmDDU0fZe8huXorljHC9jT41++NLgpU38L5v1naKsYgxNBX8WU/LHLn9JtRcPxqQ8CnLIFEJ9KmFgqjn6dQzCnlRrGUDgnrTwx/JMTcjT2u50UNro= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1560530461; h=Cc:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:Sender:Subject:To:ARC-Authentication-Results; bh=HoeG9ZfctzeS3yDgUR11zpkHcJL8IyWay78iZ5F3mU0=; b=Nv7rZVMpxAFyozV3AaeH3Q+FWznqksLaZLcwxN2noM0ZQ1bk7ZFT3vS5W43qVDPqm0Y0qlcyP256yQkjISQsjLi8bfmRS7VoTgWXozPhrsHt846nWJNt47Fi9XWzs0jEJpJX3W7OqIWhUCspdHQ3miJjOgLE1rtlPhIB+BRZKtY= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1560530461524997.832887797318; Fri, 14 Jun 2019 09:41:01 -0700 (PDT) Received: from localhost ([::1]:53374 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hbpG5-0005l8-J3 for importer@patchew.org; Fri, 14 Jun 2019 12:40:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38655) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hboud-0007n8-JM for qemu-devel@nongnu.org; Fri, 14 Jun 2019 12:18:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hbouc-0000Y6-0e for qemu-devel@nongnu.org; Fri, 14 Jun 2019 12:18:47 -0400 Received: from [192.146.154.1] (port=32898 helo=mcp01.nutanix.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hboub-0000TM-N6 for qemu-devel@nongnu.org; Fri, 14 Jun 2019 12:18:45 -0400 Received: from cosmin-marin.dev.nutanix.com (unknown [10.41.25.11]) by mcp01.nutanix.com (Postfix) with ESMTP id 090661008CAD; Fri, 14 Jun 2019 16:11:13 +0000 (UTC) From: Cosmin Marin To: qemu-devel@nongnu.org Date: Fri, 14 Jun 2019 09:11:06 -0700 Message-Id: <20190614161106.218854-1-cosmin@nutanix.com> X-Mailer: git-send-email 2.16.5 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 192.146.154.1 Subject: [Qemu-devel] [PATCH] migration: Improve accuracy of vCPU throttling with per-vCPU timers X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , Richard Henderson , Cosmin Marin Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" During auto-convergence live migration, the configured throttling rate is not matched in practice. Experimental measurements of throughput for a memory-write intensive workload indicate disparities between expected and measured throttle rate - when set to 99%, the actual throttle rate was 95%. The workload spawns multiple threads (#threads equals #vCPUs) that dirty most of the VM's memory in an infinite loop. The root cause is the usage of a VM-wide timer to schedule and execute asynchronously cpu_throttle_thread() on the vCPUs. Firstly, there are scalability limitations at scheduling time as a VM-wide (global) loop must iterate over all vCPUs while running atomic operations (i.e., may induce delays between vCPUs); moreover, if a vCPU is already running cpu_throttle_thread() (!DONE) it is skipped (i.e., may induce uneven aggregate sleep times across vCPUs). Secondly, there is a race condition between the vCPU threads and the 'scheduling' (migration) thread as a vCPU thread needs to release the iothread lock, sleep, reacquire the lock and mark "itself" as completed (DONE). Configuring correct per-vCPU sleep intervals using this model is non-trivial. To address the above issues, per-vCPU timers replace the per-VM timer. The migration thread globally decides the throttling level while each vCPU thread calculates the equivalent sleep times and sleeps accordingly. The following table summarizes the results obtained by running the workload on a 22vCPUs/45GB VM in both scenarios. +----------------------------------------------------------------+ | | per-VM Timer | per-vCPU Timer | | Target |=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D|=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D| | Throttle | Throughput | Actual | Throughput | Actual | | (%) | (GBps) | Throttle(%) | (GBps) | Throttle(%) | |----------|------------|-------------|------------|-------------| | 0| ~493.50| 0| ~493.50| 0 | | 20| 395.65| 19.81| 390.35| 20.88| | 30| 356.43| 27.76| 342.39| 30.60| | 40| 317.26| 35.69| 293.99| 40.41| | 50| 268.78| 45.52| 244.95| 50.35| | 60| 214.61| 56.50| 195.23| 60.43| | 70| 164.72| 66.61| 147.55| 70.09| | 80| 112.62| 77.17| 98.52| 80.03| | 90| 57.09| 88.43| 47.90| 90.29| | 99| 26.87| 94.55| 3.11| 99.36| +----------------------------------------------------------------+ The results support a per-vCPU timer model as it produces more accurate throttling. Signed-off-by: Cosmin Marin --- cpus.c | 29 +++++++++++++++-------------- include/qom/cpu.h | 4 ++-- 2 files changed, 17 insertions(+), 16 deletions(-) diff --git a/cpus.c b/cpus.c index dde3b7b981..c2bd3babf6 100644 --- a/cpus.c +++ b/cpus.c @@ -80,7 +80,6 @@ int64_t max_delay; int64_t max_advance; =20 /* vcpu throttling controls */ -static QEMUTimer *throttle_timer; static unsigned int throttle_percentage; =20 #define CPU_THROTTLE_PCT_MIN 1 @@ -792,40 +791,42 @@ static void cpu_throttle_thread(CPUState *cpu, run_on= _cpu_data opaque) qemu_mutex_unlock_iothread(); g_usleep(sleeptime_ns / 1000); /* Convert ns to us for usleep call */ qemu_mutex_lock_iothread(); - atomic_set(&cpu->throttle_thread_scheduled, 0); } =20 static void cpu_throttle_timer_tick(void *opaque) { - CPUState *cpu; + CPUState *cpu =3D (CPUState *)opaque; double pct; =20 /* Stop the timer if needed */ if (!cpu_throttle_get_percentage()) { return; } - CPU_FOREACH(cpu) { - if (!atomic_xchg(&cpu->throttle_thread_scheduled, 1)) { - async_run_on_cpu(cpu, cpu_throttle_thread, - RUN_ON_CPU_NULL); - } - } + =20 + async_run_on_cpu(cpu, cpu_throttle_thread, RUN_ON_CPU_NULL); =20 pct =3D (double)cpu_throttle_get_percentage()/100; - timer_mod(throttle_timer, qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL_RT) + + timer_mod(cpu->throttle_timer, qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL_RT= ) + CPU_THROTTLE_TIMESLICE_NS / (1-pct)); } =20 void cpu_throttle_set(int new_throttle_pct) { + CPUState *cpu; + double pct; + /* Ensure throttle percentage is within valid range */ new_throttle_pct =3D MIN(new_throttle_pct, CPU_THROTTLE_PCT_MAX); new_throttle_pct =3D MAX(new_throttle_pct, CPU_THROTTLE_PCT_MIN); =20 atomic_set(&throttle_percentage, new_throttle_pct); =20 - timer_mod(throttle_timer, qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL_RT) + - CPU_THROTTLE_TIMESLICE_NS); + pct =3D (double)new_throttle_pct/100; + CPU_FOREACH(cpu) { + timer_mod_anticipate(cpu->throttle_timer, + qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL_RT) + + CPU_THROTTLE_TIMESLICE_NS / (1-pct)); + } } =20 void cpu_throttle_stop(void) @@ -848,8 +849,6 @@ void cpu_ticks_init(void) seqlock_init(&timers_state.vm_clock_seqlock); qemu_spin_init(&timers_state.vm_clock_lock); vmstate_register(NULL, 0, &vmstate_timers, &timers_state); - throttle_timer =3D timer_new_ns(QEMU_CLOCK_VIRTUAL_RT, - cpu_throttle_timer_tick, NULL); } =20 void configure_icount(QemuOpts *opts, Error **errp) @@ -1267,6 +1266,8 @@ static void *qemu_kvm_cpu_thread_fn(void *arg) qemu_thread_get_self(cpu->thread); cpu->thread_id =3D qemu_get_thread_id(); cpu->can_do_io =3D 1; + cpu->throttle_timer =3D timer_new_ns(QEMU_CLOCK_VIRTUAL_RT, + cpu_throttle_timer_tick, cpu); current_cpu =3D cpu; =20 r =3D kvm_init_vcpu(cpu); diff --git a/include/qom/cpu.h b/include/qom/cpu.h index 5ee0046b62..5a11baec69 100644 --- a/include/qom/cpu.h +++ b/include/qom/cpu.h @@ -439,10 +439,10 @@ struct CPUState { /* shared by kvm, hax and hvf */ bool vcpu_dirty; =20 - /* Used to keep track of an outstanding cpu throttle thread for migrat= ion + /* Used to cyclically trigger vCPU throttling during VM migration * autoconverge */ - bool throttle_thread_scheduled; + QEMUTimer *throttle_timer; =20 bool ignore_memory_transaction_failures; =20 --=20 2.16.5