From nobody Fri May 3 16:49:10 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1600452405; cv=none; d=zohomail.com; s=zohoarc; b=KVZCEZmezvMyGAGM7v3pn8gdUM7sYgK3dBl15+ZX80MSfoICBKE9wGInmGMPMP/d3MstrHRcXYfvOejpSlrE0MoZQ/XBVuJC1NgngBeniILvA+hj/yZcoQd5RR+M7i+Rb8WImmCXPChplG4k/JLUakvnAQDPr9l9mY/su2SGkSU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600452405; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=TGenVNsFHAsSYoxzYlDtrDLuy8WLSGh5++O8wHq504k=; b=GjmMXoSom+EsiIFrBEnfpUjMvSqzvS5XgGix+fNPSqMnoSx2rz+S2BVGusXD7f8R6slsuw88Zda5GWBL5GOhJGATE1diX7M3HPKItgFz2fO3xusXVd2rNh6B0tX/w+Rz4P/d9Cl2yiRTHnwLOJuFJ0VLNiUVoX7UyYH54d3sIWo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1600452405177481.6864806412781; Fri, 18 Sep 2020 11:06:45 -0700 (PDT) Received: from localhost ([::1]:35706 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kJKmR-0001MB-Mw for importer@patchew.org; Fri, 18 Sep 2020 14:06:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40738) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kJKks-0000MF-T9 for qemu-devel@nongnu.org; Fri, 18 Sep 2020 14:05:06 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:42047) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kJKkp-0002nP-MU for qemu-devel@nongnu.org; Fri, 18 Sep 2020 14:05:06 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-503-C-LytfRHM9u0xjxpasFifg-1; Fri, 18 Sep 2020 14:04:56 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D3C7D5703F; Fri, 18 Sep 2020 18:04:54 +0000 (UTC) Received: from localhost (ovpn-119-217.rdu2.redhat.com [10.10.119.217]) by smtp.corp.redhat.com (Postfix) with ESMTP id 38C1B1002D53; Fri, 18 Sep 2020 18:04:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1600452303; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TGenVNsFHAsSYoxzYlDtrDLuy8WLSGh5++O8wHq504k=; b=MzwyTus7QO8fNDO9Z/Pyd1Asw4LzRZKiPfaIahkCEUO1pnSbrIi9Ld+uar94KJlnrsfA+B AImcnEV5LDfe6OITvSc16f79ggF4YYnyfvwpempRulP4v6EqV/tFkHpPEpFTuVqRgQouE0 j/SnPf6bON48LKxNxlfQLZ1ZnjlDArQ= X-MC-Unique: C-LytfRHM9u0xjxpasFifg-1 From: Eduardo Habkost To: Cleber Rosa , Peter Maydell , qemu-devel@nongnu.org Subject: [PULL 1/2] scripts/simplebench: compare write request performance Date: Fri, 18 Sep 2020 14:04:50 -0400 Message-Id: <20200918180451.1121055-2-ehabkost@redhat.com> In-Reply-To: <20200918180451.1121055-1-ehabkost@redhat.com> References: <20200918180451.1121055-1-ehabkost@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=ehabkost@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=ehabkost@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/18 01:32:10 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -50 X-Spam_score: -5.1 X-Spam_bar: ----- X-Spam_report: (-5.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.999, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Andrey Shinkevich , Vladimir Sementsov-Ogievskiy , Eduardo Habkost , "Denis V . Lunev" Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Andrey Shinkevich The script 'bench_write_req.py' allows comparing performances of write request for two qemu-img binary files. An example with (qemu-img binary 1) and without (qemu-img binary 2) the applied patch "qcow2: skip writing zero buffers to empty COW areas" (git commit ID: c8bb23cbdbe32f5) has the following results: SSD: Reviewed-by: Vladimir Sementsov-Ogievskiy Suggested-by: Denis V. Lunev Suggested-by: Vladimir Sementsov-Ogievskiy ---------------- ------------------- ------------------- 0.10 +- 0.00 8.16 +- 0.65 0.10 +- 0.00 7.37 +- 1.10 7.40 +- 1.08 21.97 +- 4.19 2.14 +- 0.94 8.48 +- 1.66 ---------------- ------------------- ------------------- HDD: ---------------- ------------------- ------------------- 2.30 +- 0.01 6.19 +- 0.06 2.20 +- 0.09 6.20 +- 0.06 8.32 +- 0.16 8.26 +- 0.14 8.20 +- 0.05 6.26 +- 0.10 ---------------- ------------------- ------------------- Suggested-by: Denis V. Lunev Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy Message-Id: <1594741846-475697-1-git-send-email-andrey.shinkevich@virtuozzo= .com> Signed-off-by: Eduardo Habkost --- scripts/simplebench/bench_write_req.py | 170 +++++++++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100755 scripts/simplebench/bench_write_req.py diff --git a/scripts/simplebench/bench_write_req.py b/scripts/simplebench/b= ench_write_req.py new file mode 100755 index 0000000000..ca1178fd68 --- /dev/null +++ b/scripts/simplebench/bench_write_req.py @@ -0,0 +1,170 @@ +#!/usr/bin/env python3 +# +# Test to compare performance of write requests for two qemu-img binary fi= les. +# +# The idea of the test comes from intention to check the benefit of c8bb23= cbdbe +# "qcow2: skip writing zero buffers to empty COW areas". +# +# Copyright (c) 2020 Virtuozzo International GmbH. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + + +import sys +import os +import subprocess +import simplebench + + +def bench_func(env, case): + """ Handle one "cell" of benchmarking table. """ + return bench_write_req(env['qemu_img'], env['image_name'], + case['block_size'], case['block_offset'], + case['cluster_size']) + + +def qemu_img_pipe(*args): + '''Run qemu-img and return its output''' + subp =3D subprocess.Popen(list(args), + stdout=3Dsubprocess.PIPE, + stderr=3Dsubprocess.STDOUT, + universal_newlines=3DTrue) + exitcode =3D subp.wait() + if exitcode < 0: + sys.stderr.write('qemu-img received signal %i: %s\n' + % (-exitcode, ' '.join(list(args)))) + return subp.communicate()[0] + + +def bench_write_req(qemu_img, image_name, block_size, block_offset, + cluster_size): + """Benchmark write requests + + The function creates a QCOW2 image with the given path/name. Then it r= uns + the 'qemu-img bench' command and makes series of write requests on the + image clusters. Finally, it returns the total time of the write operat= ions + on the disk. + + qemu_img -- path to qemu_img executable file + image_name -- QCOW2 image name to create + block_size -- size of a block to write to clusters + block_offset -- offset of the block in clusters + cluster_size -- size of the image cluster + + Returns {'seconds': int} on success and {'error': str} on failure. + Return value is compatible with simplebench lib. + """ + + if not os.path.isfile(qemu_img): + print(f'File not found: {qemu_img}') + sys.exit(1) + + image_dir =3D os.path.dirname(os.path.abspath(image_name)) + if not os.path.isdir(image_dir): + print(f'Path not found: {image_name}') + sys.exit(1) + + image_size =3D 1024 * 1024 * 1024 + + args_create =3D [qemu_img, 'create', '-f', 'qcow2', '-o', + f'cluster_size=3D{cluster_size}', + image_name, str(image_size)] + + count =3D int(image_size / cluster_size) - 1 + step =3D str(cluster_size) + + args_bench =3D [qemu_img, 'bench', '-w', '-n', '-t', 'none', '-c', + str(count), '-s', f'{block_size}', '-o', str(block_offse= t), + '-S', step, '-f', 'qcow2', image_name] + + try: + qemu_img_pipe(*args_create) + except OSError as e: + os.remove(image_name) + return {'error': 'qemu_img create failed: ' + str(e)} + + try: + ret =3D qemu_img_pipe(*args_bench) + except OSError as e: + os.remove(image_name) + return {'error': 'qemu_img bench failed: ' + str(e)} + + os.remove(image_name) + + if 'seconds' in ret: + ret_list =3D ret.split() + index =3D ret_list.index('seconds.') + return {'seconds': float(ret_list[index-1])} + else: + return {'error': 'qemu_img bench failed: ' + ret} + + +if __name__ =3D=3D '__main__': + + if len(sys.argv) < 4: + program =3D os.path.basename(sys.argv[0]) + print(f'USAGE: {program} ' + ' ' + '') + exit(1) + + # Test-cases are "rows" in benchmark resulting table, 'id' is a caption + # for the row, other fields are handled by bench_func. + test_cases =3D [ + { + 'id': '', + 'block_size': 4096, + 'block_offset': 0, + 'cluster_size': 1048576 + }, + { + 'id': '', + 'block_size': 4096, + 'block_offset': 524288, + 'cluster_size': 1048576 + }, + { + 'id': '', + 'block_size': 1048576, + 'block_offset': 4096, + 'cluster_size': 1048576 + }, + { + 'id': '', + 'block_size': 4096, + 'block_offset': 0, + 'cluster_size': 65536 + }, + ] + + # Test-envs are "columns" in benchmark resulting table, 'id is a capti= on + # for the column, other fields are handled by bench_func. + # Set the paths below to desired values + test_envs =3D [ + { + 'id': '', + 'qemu_img': f'{sys.argv[1]}', + 'image_name': f'{sys.argv[3]}' + }, + { + 'id': '', + 'qemu_img': f'{sys.argv[2]}', + 'image_name': f'{sys.argv[3]}' + }, + ] + + result =3D simplebench.bench(bench_func, test_envs, test_cases, count= =3D3, + initial_run=3DFalse) + print(simplebench.ascii(result)) --=20 2.26.2 From nobody Fri May 3 16:49:10 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1600452486; cv=none; d=zohomail.com; s=zohoarc; b=ICRnrcwtz1GMW12BR0Vf+p3JZ2LzCt1oNtLbHB05KdDdKmKHCmPKJzA0qLSr1i5H+ns54jpSbqSzdlDdO5qHdQBoHqW4fai3aGFfS/4g35+PHPE8rnMeg2rW6PmJ8WynjsfnO++53+5worVbcgdzQCjkbazIJoT7kuWilT+2EE8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600452486; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=syFAbH20ps3bZJnXHQ5hKzOCwTqYrnEvIr2Q/VfgrK0=; b=RqPqZOuKEfEnmLAk302WCPTjjREEfPuElvq/ygUXRxUT5DdflF2CUytt194zSPUIdAlZOJleGp/FttibVJanojnbgQr56mixbOQU8nnNxV/E0Iqx6h4U6eCS1eO6u+LoxfC26cq2eRcVNaAsKYKpAkq5BrTisb+xTB/YL5M+vAg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1600452486732944.6095232647084; Fri, 18 Sep 2020 11:08:06 -0700 (PDT) Received: from localhost ([::1]:38646 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kJKnl-0002aA-Ep for importer@patchew.org; Fri, 18 Sep 2020 14:08:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40736) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kJKks-0000Lr-7w for qemu-devel@nongnu.org; Fri, 18 Sep 2020 14:05:06 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:45264) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kJKkq-0002nW-I5 for qemu-devel@nongnu.org; Fri, 18 Sep 2020 14:05:05 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-189-TrH9_ArSMo-vyNLMfzqZJQ-1; Fri, 18 Sep 2020 14:04:57 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 75A0B18BE163; Fri, 18 Sep 2020 18:04:56 +0000 (UTC) Received: from localhost (ovpn-119-217.rdu2.redhat.com [10.10.119.217]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1163E1002D58; Fri, 18 Sep 2020 18:04:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1600452303; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=syFAbH20ps3bZJnXHQ5hKzOCwTqYrnEvIr2Q/VfgrK0=; b=XqGq7HnVMyCNS6JD01soHIVLs6oDBwt+occP4ymTNlrycl2wfnZhHLUU1Eog+TSQmW+ToO sihz7lrey2SRPuw4OrgtZ0fFdhoypxfsqC8RonHE+Ex1gghLvjrpsgEwa6svyLo5Tuf5Ca gxbJrePbBKIDOmZWA+xZu4aol3m2Zr8= X-MC-Unique: TrH9_ArSMo-vyNLMfzqZJQ-1 From: Eduardo Habkost To: Cleber Rosa , Peter Maydell , qemu-devel@nongnu.org Subject: [PULL 2/2] analyze-migration.py: fix read_migration_debug_json() return type Date: Fri, 18 Sep 2020 14:04:51 -0400 Message-Id: <20200918180451.1121055-3-ehabkost@redhat.com> In-Reply-To: <20200918180451.1121055-1-ehabkost@redhat.com> References: <20200918180451.1121055-1-ehabkost@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=ehabkost@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=ehabkost@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/18 01:32:10 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -50 X-Spam_score: -5.1 X-Spam_bar: ----- X-Spam_report: (-5.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.999, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alexey Kirillov , Vladimir Sementsov-Ogievskiy , Eduardo Habkost Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Alexey Kirillov Since we use result of read_migration_debug_json() as JSON formatted string, we must provide proper type. Before Python 3.6 json.loads() method support only str typed input. Signed-off-by: Alexey Kirillov Message-Id: <20200715152135.20287-1-lekiravi@yandex-team.ru> [ehabkost: added comment explaining why decode() is needed} Signed-off-by: Eduardo Habkost --- scripts/analyze-migration.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/scripts/analyze-migration.py b/scripts/analyze-migration.py index 95838cbff3..d70ec7d33a 100755 --- a/scripts/analyze-migration.py +++ b/scripts/analyze-migration.py @@ -97,7 +97,8 @@ class MigrationFile(object): # Seek back to where we were at the beginning self.file.seek(entrypos, 0) =20 - return data[jsonpos:jsonpos + jsonlen] + # explicit decode() needed for Python 3.5 compatibility + return data[jsonpos:jsonpos + jsonlen].decode("utf-8") =20 def close(self): self.file.close() --=20 2.26.2