From nobody Sun Nov 16 04:05:48 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1598611381; cv=none; d=zohomail.com; s=zohoarc; b=EvHu86h/gC9Y1qM8smIQxfjttleC86YQRehMkyzlRocgiZmtHxnKtPNg/K1j9NKaMFx2TowTLSccCKPrzBNBl6n6uftOo+/6CnpJdRm0VHYvjhC8vZRBtd3HCU9dm27jm3wmvROS3FGT8rkbqPnu00jBu5EjFZ3a3/OoVTMmuvQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1598611381; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=GGrLy3IbvYhI9UUc6EFtGpu2pNno9f5HSPArLmtNmFM=; b=JdvIueEQ9aOusVGg5ZNeX7G0WRtJJUKF/fTYjhO6nTLHe2GrQI5t+Ip8123BKT4Om3I0HGYg6n/4pZ5X5heCZWf1H3kQScN2ZffdooAerHaRVQmkLF4TX/FInj+jcip+qFT3XyDwJPCBk63gdjoorZAnvfIsMyg6XScX+Hm5lVk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1598611381474949.0592556375858; Fri, 28 Aug 2020 03:43:01 -0700 (PDT) Received: from localhost ([::1]:56658 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kBbqV-0004wk-TE for importer@patchew.org; Fri, 28 Aug 2020 06:42:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34384) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kBbpB-0003MZ-Ex for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:37 -0400 Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]:42827) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kBbp8-0005fA-Ql for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:37 -0400 Received: by mail-wr1-x442.google.com with SMTP id c18so833999wrm.9 for ; Fri, 28 Aug 2020 03:41:34 -0700 (PDT) Received: from localhost.localdomain ([197.58.77.158]) by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:41:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=GGrLy3IbvYhI9UUc6EFtGpu2pNno9f5HSPArLmtNmFM=; b=ULG82S0L3kCib5VT4GaouqmZivM+bay/mzr4qks3za+D7dtBqR++RqUjn5NKpkuVLE /3WjJcnTTdwSFiaGo8qwVLUJahcQ7YdGddB3xqtaldRO0TWnk8RyT7Cau2kciF5uc7V7 Nk/GzfRznwhn6IjcEL5I8C3h1O59R/HBQrVnu7SUJ3+UrmTDzW1TF7XsuaY1bR65VcNx sZ3AFsVXCOcMZP1jtQhLb8qc/1cQPTORJ/AxlxH/uybNPNQO95i10kyWpJ8XPe87hLJA AQLF6MfAcC6rwAgym1SY74AMz2zNBol60n2pAN+xDpkD9mAz2NmUuitE7eVfTahJz+cX 4BPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=GGrLy3IbvYhI9UUc6EFtGpu2pNno9f5HSPArLmtNmFM=; b=edhvkGgaW9FNDljG3BIyWtLg3vd0nXEB+/VdohZOpFWeWNs3yQU+HeWPI6aFOXHqip w9iA0fpMObgbfEg/q9ms4kU1Td1b/+f5x4Ujjxb4tbeSLEfHVuBXTCrmeu+dzRBVRdOd KTaTcdGN+h854DLMezFJJJE+lqYJyY7gep7kYzCj5BzIqTS9cqSIIVPBBcFB1hC+3U4d 7RX1jMrvlxe7E5Qh1aZIcQ2ntgdnxUDhsbZhfIezgHAVJt9rbhqI/58mbqcy9rwhmJLr FXKvpSlFTuI4+ereHs7Wv3rBgAizzUm+s/azfzQAc6n7LkmlymlVZCZkfCZ4KHaN585s GQig== X-Gm-Message-State: AOAM532cer+BEXV1Y4xxukMb6m+oXF80a4vYTa7iq27DCeVULpPw8SUI B7Snuz/ED3TY/vhWtxrk80YWy3tKpaP0UA== X-Google-Smtp-Source: ABdhPJy+UxZw0jQnURExU+I0JXJh37Lpqu5CD/VUPQVnG0uk7ICLiSSbKv/D/w+rB/FxGK0ERvMoDA== X-Received: by 2002:adf:a1d6:: with SMTP id v22mr954931wrv.185.1598611292937; Fri, 28 Aug 2020 03:41:32 -0700 (PDT) From: Ahmed Karaman To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com, alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com, jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com Subject: [PATCH 1/9] scripts/performance: Refactor topN_perf.py Date: Fri, 28 Aug 2020 12:40:54 +0200 Message-Id: <20200828104102.4490-2-ahmedkhaledkaraman@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::442; envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x442.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_PDS_OTHER_BAD_TLD=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ahmed Karaman Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @gmail.com) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" - Apply pylint and flake8 formatting rules to the script. - Use 'tempfile' instead of '/tmp' for creating temporary files. Signed-off-by: Ahmed Karaman Reviewed-by: Aleksandar Markovic --- scripts/performance/topN_perf.py | 174 +++++++++++++++---------------- 1 file changed, 87 insertions(+), 87 deletions(-) diff --git a/scripts/performance/topN_perf.py b/scripts/performance/topN_pe= rf.py index 07be195fc8..56b100da87 100755 --- a/scripts/performance/topN_perf.py +++ b/scripts/performance/topN_perf.py @@ -1,72 +1,77 @@ #!/usr/bin/env python3 =20 -# Print the top N most executed functions in QEMU using perf. -# Syntax: -# topN_perf.py [-h] [-n] -- \ -# [] \ -# [] -# -# [-h] - Print the script arguments help message. -# [-n] - Specify the number of top functions to print. -# - If this flag is not specified, the tool defaults to 25. -# -# Example of usage: -# topN_perf.py -n 20 -- qemu-arm coulomb_double-arm -# -# This file is a part of the project "TCG Continuous Benchmarking". -# -# Copyright (C) 2020 Ahmed Karaman -# Copyright (C) 2020 Aleksandar Markovic -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . +""" +Print the top N most executed functions in QEMU using perf. + +Syntax: +topN_perf.py [-h] [-n ] -- \ + [] \ + [] + +[-h] - Print the script arguments help message. +[-n] - Specify the number of top functions to print. + - If this flag is not specified, the tool defaults to 25. + +Example of usage: +topN_perf.py -n 20 -- qemu-arm coulomb_double-arm + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman +Copyright (C) 2020 Aleksandar Markovic + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" =20 import argparse import os import subprocess import sys +import tempfile =20 =20 # Parse the command line arguments -parser =3D argparse.ArgumentParser( - usage=3D'topN_perf.py [-h] [-n] = -- ' +PARSER =3D argparse.ArgumentParser( + usage=3D'topN_perf.py [-h] [-n ] --= ' ' [] ' ' []') =20 -parser.add_argument('-n', dest=3D'top', type=3Dint, default=3D25, +PARSER.add_argument('-n', dest=3D'top', type=3Dint, default=3D25, help=3D'Specify the number of top functions to print.') =20 -parser.add_argument('command', type=3Dstr, nargs=3D'+', help=3Dargparse.SU= PPRESS) +PARSER.add_argument('command', type=3Dstr, nargs=3D'+', help=3Dargparse.SU= PPRESS) =20 -args =3D parser.parse_args() +ARGS =3D PARSER.parse_args() =20 # Extract the needed variables from the args -command =3D args.command -top =3D args.top +COMMAND =3D ARGS.command +TOP =3D ARGS.top =20 # Insure that perf is installed -check_perf_presence =3D subprocess.run(["which", "perf"], - stdout=3Dsubprocess.DEVNULL) -if check_perf_presence.returncode: +CHECK_PERF_PRESENCE =3D subprocess.run(["which", "perf"], + stdout=3Dsubprocess.DEVNULL, + check=3DFalse) +if CHECK_PERF_PRESENCE.returncode: sys.exit("Please install perf before running the script!") =20 # Insure user has previllage to run perf -check_perf_executability =3D subprocess.run(["perf", "stat", "ls", "/"], +CHECK_PERF_EXECUTABILITY =3D subprocess.run(["perf", "stat", "ls", "/"], stdout=3Dsubprocess.DEVNULL, - stderr=3Dsubprocess.DEVNULL) -if check_perf_executability.returncode: - sys.exit( -""" + stderr=3Dsubprocess.DEVNULL, + check=3DFalse) +if CHECK_PERF_EXECUTABILITY.returncode: + sys.exit(""" Error: You may not have permission to collect stats. =20 @@ -85,43 +90,42 @@ To make this setting permanent, edit /etc/sysctl.conf t= oo, e.g.: kernel.perf_event_paranoid =3D -1 =20 * Alternatively, you can run this script under sudo privileges. -""" -) - -# Run perf record -perf_record =3D subprocess.run((["perf", "record", "--output=3D/tmp/perf.d= ata"] + - command), - stdout=3Dsubprocess.DEVNULL, - stderr=3Dsubprocess.PIPE) -if perf_record.returncode: - os.unlink('/tmp/perf.data') - sys.exit(perf_record.stderr.decode("utf-8")) - -# Save perf report output to /tmp/perf_report.out -with open("/tmp/perf_report.out", "w") as output: - perf_report =3D subprocess.run( - ["perf", "report", "--input=3D/tmp/perf.data", "--stdio"], - stdout=3Doutput, - stderr=3Dsubprocess.PIPE) - if perf_report.returncode: - os.unlink('/tmp/perf.data') - output.close() - os.unlink('/tmp/perf_report.out') - sys.exit(perf_report.stderr.decode("utf-8")) - -# Read the reported data to functions[] -functions =3D [] -with open("/tmp/perf_report.out", "r") as data: - # Only read lines that are not comments (comments start with #) - # Only read lines that are not empty - functions =3D [line for line in data.readlines() if line and line[0] - !=3D '#' and line[0] !=3D "\n"] - -# Limit the number of top functions to "top" -number_of_top_functions =3D top if len(functions) > top else len(functions) - -# Store the data of the top functions in top_functions[] -top_functions =3D functions[:number_of_top_functions] +""") + +# Run perf and save all intermediate files in a temporary directory +with tempfile.TemporaryDirectory() as tmpdir: + RECORD_PATH =3D os.path.join(tmpdir, "record.data") + REPORT_PATH =3D os.path.join(tmpdir, "report.txt") + + PERF_RECORD =3D subprocess.run((["perf", "record", "--output=3D"+RECOR= D_PATH] + + COMMAND), + stdout=3Dsubprocess.DEVNULL, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if PERF_RECORD.returncode: + sys.exit(PERF_RECORD.stderr.decode("utf-8")) + + with open(REPORT_PATH, "w") as output: + PERF_REPORT =3D subprocess.run( + ["perf", "report", "--input=3D"+RECORD_PATH, "--stdio"], + stdout=3Doutput, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if PERF_REPORT.returncode: + sys.exit(PERF_REPORT.stderr.decode("utf-8")) + + # Save the reported data to FUNCTIONS[] + with open(REPORT_PATH, "r") as data: + # Only read lines that are not comments (comments start with #) + # Only read lines that are not empty + FUNCTIONS =3D [line for line in data.readlines() if line and + line[0] !=3D '#' and line[0] !=3D "\n"] + +# Limit the number of top functions to "TOP" +NO_TOP_FUNCTIONS =3D TOP if len(FUNCTIONS) > TOP else len(FUNCTIONS) + +# Store the data of the top functions in TOP_FUNCTIONS[] +TOP_FUNCTIONS =3D FUNCTIONS[:NO_TOP_FUNCTIONS] =20 # Print table header print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.', @@ -134,7 +138,7 @@ print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.forma= t('No.', '-' * 25)) =20 # Print top N functions -for (index, function) in enumerate(top_functions, start=3D1): +for (index, function) in enumerate(TOP_FUNCTIONS, start=3D1): function_data =3D function.split() function_percentage =3D function_data[0] function_name =3D function_data[-1] @@ -143,7 +147,3 @@ for (index, function) in enumerate(top_functions, start= =3D1): function_percentage, function_name, function_invoker)) - -# Remove intermediate files -os.unlink('/tmp/perf.data') -os.unlink('/tmp/perf_report.out') --=20 2.17.1 From nobody Sun Nov 16 04:05:48 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1598611472; cv=none; d=zohomail.com; s=zohoarc; b=TuKgDXGV4t613J2ZcRPBsRaweEwHj58dsjQJS2b3YjDazfz53Bc5bqVGT1IOIXvAtMjMIMJCZ883+QRrpUUzdTIk5Jl57bMQrSHBjXshJkgFNoTfT7qsd8qePwfhgOhgQfGfafnR7F1OuD+OBYzYoiEGp369BVBcW2ruKeJyhqo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1598611472; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=6VnUOaw47q1sBFWEPDkKi8Rl+L6HJXn0J6GhoPyLYWM=; b=Gak5QKLU7uGR+tM0Oyr4i0WdjLGIheRSL9C4V47sPhIMNosEYnWOKXkQl9NjXulVksHoLmVlDqu6kR2GX0A4DWSHZ0q2mE0BuT/7yEkOD7TKOYqeBsmqwEtw85lonDMmjZL+MQDsG/8ue9nWw1rqGGXw91QLHnElMLXtD9hlN1w= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1598611472611251.12486478976257; Fri, 28 Aug 2020 03:44:32 -0700 (PDT) Received: from localhost ([::1]:36646 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kBbrz-0008Gm-AM for importer@patchew.org; Fri, 28 Aug 2020 06:44:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34414) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kBbpD-0003Pe-Ia for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:39 -0400 Received: from mail-wr1-x443.google.com ([2a00:1450:4864:20::443]:45770) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kBbpA-0005gS-RI for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:39 -0400 Received: by mail-wr1-x443.google.com with SMTP id h15so830654wrt.12 for ; Fri, 28 Aug 2020 03:41:36 -0700 (PDT) Received: from localhost.localdomain ([197.58.77.158]) by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:41:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=6VnUOaw47q1sBFWEPDkKi8Rl+L6HJXn0J6GhoPyLYWM=; b=HK/+cklW0orMXNl4Is5lsX4XkmBhx776gGavdwJf8IBs89t4vZJzXPAaCtxc6yKIvc RzQB1Qus6S9Bn6Mzp/bHtBKVuU99Vcl4/E54OUhOshOmMFSRImyfLeEu3yKgjaZNuKZr Q/R91V2p8iDPXFPeLzOrwTJzcHoK+Rwdtb5Izj+iATpzclxbu7oMxWbdaKr5AbmjWolH WQEPE/DkQ7BvxFwG2cz0hL0+6qnx3otbnGYt7t0mxUeUiEXYIxZeDOGK0qs9WxCdaBSo ZNaz31UqFtszJ4YvYS+YHjq3NuI1ipWTa24bHx8MYl6pALyPRYiokQMv3shLjDT4F9bH NkiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=6VnUOaw47q1sBFWEPDkKi8Rl+L6HJXn0J6GhoPyLYWM=; b=NriiN+nz4yZkfD74HZulD9tIomLmngDWDA8gazf8+8/WJHHvFU5zcT5gGhzY5GBOQQ E+WlbiNH+y2SDEIjCdO7wq/KE26t8X/PZaUepCvMWKRVxyxifYmXvJrfQMn89RuDPkIJ sYzFCFV+adVzh8kg4R26iZWSpisLfzwYbeoiBsUvP3DdYwcYKzywaia/4Tu+yLTK7a+/ 11BbMJiuu9+c8iIjVEX4LUBEhEg8bL/fSEfxWtOJtIVqfld2ausPS8moJRsmx93PLtNC n7liJJuATdjssXl9WRFwM+zup86OT3AXSAmeKCOe59p1sI5+ACWWMErExjyzkKAKXZRY iTdA== X-Gm-Message-State: AOAM533QXLo3R0Uz/AxpcfakWt4+/QsIVEYutCI00pdz4sFd8rW4P2Rl rZ3xit9u9SaskK8XOYJn6gdLKIUiLPu4/A== X-Google-Smtp-Source: ABdhPJzJsAjybpcBhmfxWDs+uG3yuDlTJWs5Nnd98OPlyvZdKNUSixe42FeYCDweCVGzHmXJt3aKPA== X-Received: by 2002:adf:9d44:: with SMTP id o4mr864841wre.99.1598611294899; Fri, 28 Aug 2020 03:41:34 -0700 (PDT) From: Ahmed Karaman To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com, alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com, jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com Subject: [PATCH 2/9] scripts/performance: Refactor topN_callgrind.py Date: Fri, 28 Aug 2020 12:40:55 +0200 Message-Id: <20200828104102.4490-3-ahmedkhaledkaraman@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::443; envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x443.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_PDS_OTHER_BAD_TLD=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ahmed Karaman Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @gmail.com) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" - Apply pylint and flake8 formatting rules to the script. - Use 'tempfile' instead of '/tmp' for creating temporary files. Signed-off-by: Ahmed Karaman Reviewed-by: Aleksandar Markovic --- scripts/performance/topN_callgrind.py | 169 +++++++++++++------------- 1 file changed, 87 insertions(+), 82 deletions(-) diff --git a/scripts/performance/topN_callgrind.py b/scripts/performance/to= pN_callgrind.py index 67c59197af..f8a554f393 100755 --- a/scripts/performance/topN_callgrind.py +++ b/scripts/performance/topN_callgrind.py @@ -1,113 +1,122 @@ #!/usr/bin/env python3 =20 -# Print the top N most executed functions in QEMU using callgrind. -# Syntax: -# topN_callgrind.py [-h] [-n] -- \ -# [] \ -# [] -# -# [-h] - Print the script arguments help message. -# [-n] - Specify the number of top functions to print. -# - If this flag is not specified, the tool defaults to 25. -# -# Example of usage: -# topN_callgrind.py -n 20 -- qemu-arm coulomb_double-arm -# -# This file is a part of the project "TCG Continuous Benchmarking". -# -# Copyright (C) 2020 Ahmed Karaman -# Copyright (C) 2020 Aleksandar Markovic -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . +""" +Print the top N most executed functions in QEMU using callgrind. + +Syntax: +topN_callgrind.py [-h] [-n ] -- \ + [] \ + [] + +[-h] - Print the script arguments help message. +[-n] - Specify the number of top functions to print. + - If this flag is not specified, the tool defaults to 25. + +Example of usage: +topN_callgrind.py -n 20 -- qemu-arm coulomb_double-arm + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman +Copyright (C) 2020 Aleksandar Markovic + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" =20 import argparse import os import subprocess import sys +import tempfile =20 =20 # Parse the command line arguments -parser =3D argparse.ArgumentParser( - usage=3D'topN_callgrind.py [-h] [-n] -- ' +PARSER =3D argparse.ArgumentParser( + usage=3D'topN_callgrind.py [-h] [-n] -- ' ' [] ' ' []') =20 -parser.add_argument('-n', dest=3D'top', type=3Dint, default=3D25, +PARSER.add_argument('-n', dest=3D'top', type=3Dint, default=3D25, help=3D'Specify the number of top functions to print.') =20 -parser.add_argument('command', type=3Dstr, nargs=3D'+', help=3Dargparse.SU= PPRESS) +PARSER.add_argument('command', type=3Dstr, nargs=3D'+', help=3Dargparse.SU= PPRESS) =20 -args =3D parser.parse_args() +ARGS =3D PARSER.parse_args() =20 # Extract the needed variables from the args -command =3D args.command -top =3D args.top +COMMAND =3D ARGS.command +TOP =3D ARGS.top =20 # Insure that valgrind is installed -check_valgrind_presence =3D subprocess.run(["which", "valgrind"], - stdout=3Dsubprocess.DEVNULL) -if check_valgrind_presence.returncode: +CHECK_VALGRIND_PRESENCE =3D subprocess.run(["which", "valgrind"], + stdout=3Dsubprocess.DEVNULL, + check=3DFalse) +if CHECK_VALGRIND_PRESENCE.returncode: sys.exit("Please install valgrind before running the script!") =20 -# Run callgrind -callgrind =3D subprocess.run(( - ["valgrind", "--tool=3Dcallgrind", "--callgrind-out-file=3D/tmp/callgr= ind.data"] - + command), - stdout=3Dsubprocess.DEVNULL, - stderr=3Dsubprocess.PIPE) -if callgrind.returncode: - sys.exit(callgrind.stderr.decode("utf-8")) - -# Save callgrind_annotate output to /tmp/callgrind_annotate.out -with open("/tmp/callgrind_annotate.out", "w") as output: - callgrind_annotate =3D subprocess.run(["callgrind_annotate", - "/tmp/callgrind.data"], - stdout=3Doutput, - stderr=3Dsubprocess.PIPE) - if callgrind_annotate.returncode: - os.unlink('/tmp/callgrind.data') - output.close() - os.unlink('/tmp/callgrind_annotate.out') - sys.exit(callgrind_annotate.stderr.decode("utf-8")) - -# Read the callgrind_annotate output to callgrind_data[] -callgrind_data =3D [] -with open('/tmp/callgrind_annotate.out', 'r') as data: - callgrind_data =3D data.readlines() +# Run callgrind and save all intermediate files in a temporary directory +with tempfile.TemporaryDirectory() as tmpdir: + CALLGRIND_DATA_PATH =3D os.path.join(tmpdir, "callgrind.data") + ANNOTATE_OUT_PATH =3D os.path.join(tmpdir, "callgrind_annotate.out") + + # Run callgrind + CALLGRIND =3D subprocess.run((["valgrind", + "--tool=3Dcallgrind", + "--callgrind-out-file=3D"+CALLGRIND_DATA_= PATH] + + COMMAND), + stdout=3Dsubprocess.DEVNULL, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if CALLGRIND.returncode: + sys.exit(CALLGRIND.stderr.decode("utf-8")) + + with open(ANNOTATE_OUT_PATH, "w") as output: + CALLGRIND_ANNOTATE =3D subprocess.run(["callgrind_annotate", + CALLGRIND_DATA_PATH], + stdout=3Doutput, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if CALLGRIND_ANNOTATE.returncode: + sys.exit(CALLGRIND_ANNOTATE.stderr.decode("utf-8")) + + # Read the callgrind_annotate output to CALLGRIND_DATA[] + CALLGRIND_DATA =3D [] + with open(ANNOTATE_OUT_PATH, 'r') as data: + CALLGRIND_DATA =3D data.readlines() =20 # Line number with the total number of instructions -total_instructions_line_number =3D 20 +TOTAL_INSTRUCTIONS_LINE_NO =3D 20 =20 # Get the total number of instructions -total_instructions_line_data =3D callgrind_data[total_instructions_line_nu= mber] -total_number_of_instructions =3D total_instructions_line_data.split(' ')[0] -total_number_of_instructions =3D int( - total_number_of_instructions.replace(',', '')) +TOTAL_INSTRUCTIONS_LINE_DATA =3D CALLGRIND_DATA[TOTAL_INSTRUCTIONS_LINE_NO] +TOTAL_NUMBER_OF_INSTRUCTIONS =3D TOTAL_INSTRUCTIONS_LINE_DATA.split(' ')[0] +TOTAL_NUMBER_OF_INSTRUCTIONS =3D int( + TOTAL_NUMBER_OF_INSTRUCTIONS.replace(',', '')) =20 # Line number with the top function -first_func_line =3D 25 +FIRST_FUNC_LINE =3D 25 =20 # Number of functions recorded by callgrind, last two lines are always emp= ty -number_of_functions =3D len(callgrind_data) - first_func_line - 2 +NUMBER_OF_FUNCTIONS =3D len(CALLGRIND_DATA) - FIRST_FUNC_LINE - 2 =20 # Limit the number of top functions to "top" -number_of_top_functions =3D (top if number_of_functions > - top else number_of_functions) +NUMBER_OF_TOP_FUNCTIONS =3D (TOP if NUMBER_OF_FUNCTIONS > + TOP else NUMBER_OF_FUNCTIONS) =20 # Store the data of the top functions in top_functions[] -top_functions =3D callgrind_data[first_func_line: - first_func_line + number_of_top_functions] +TOP_FUNCTIONS =3D CALLGRIND_DATA[FIRST_FUNC_LINE: + FIRST_FUNC_LINE + NUMBER_OF_TOP_FUNCTIONS] =20 # Print table header print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.', @@ -121,12 +130,12 @@ print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.for= mat('No.', )) =20 # Print top N functions -for (index, function) in enumerate(top_functions, start=3D1): +for (index, function) in enumerate(TOP_FUNCTIONS, start=3D1): function_data =3D function.split() # Calculate function percentage function_instructions =3D float(function_data[0].replace(',', '')) function_percentage =3D (function_instructions / - total_number_of_instructions)*100 + TOTAL_NUMBER_OF_INSTRUCTIONS)*100 # Get function name and source files path function_source_file, function_name =3D function_data[1].split(':') # Print extracted data @@ -134,7 +143,3 @@ for (index, function) in enumerate(top_functions, start= =3D1): round(function_percentage,= 3), function_name, function_source_file)) - -# Remove intermediate files -os.unlink('/tmp/callgrind.data') -os.unlink('/tmp/callgrind_annotate.out') --=20 2.17.1 From nobody Sun Nov 16 04:05:48 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1598611396; cv=none; d=zohomail.com; s=zohoarc; b=HmYKjJsBQTq4CIrLnnfUb9eA4NbtQ0bNmHM1bOrDfDyFMZRg+kmEyXprgmAt84VRzC5HOzxcJJBqeBzOlqd8juTObBHl+I4Y8CkjquhoetW5EC7Z0IaFxKIDJQTvUBjIi3HvP/7eTmXmr8l8rsY7ZvN8Wsry9Vy+ougLcddTzIw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1598611396; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=EJjPWrg0i8aqjuW0NGnomyE8WzQ7HVuoMsi3T2ewGf0=; b=aj8JiIQ7JtLg983+D8clviC1C01QPPN0PYNpXB0PjNM/rS0aFvio5mCxGELAnaMVB935O90F/CLPLbIB7k4UdnwnqR8gY4K7fghUNi8qYeK1S/Giis0YcmZ6MCrsVKSyGrYiJTPcL8NST8Tr+BewdvtaEk8demttUWw2I0YPszg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1598611396368372.1725726624603; Fri, 28 Aug 2020 03:43:16 -0700 (PDT) Received: from localhost ([::1]:57140 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kBbqk-00058c-Tq for importer@patchew.org; Fri, 28 Aug 2020 06:43:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34422) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kBbpE-0003Re-M4 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:40 -0400 Received: from mail-wr1-x42a.google.com ([2a00:1450:4864:20::42a]:46399) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kBbpC-0005gg-Fx for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:40 -0400 Received: by mail-wr1-x42a.google.com with SMTP id r15so828425wrp.13 for ; Fri, 28 Aug 2020 03:41:38 -0700 (PDT) Received: from localhost.localdomain ([197.58.77.158]) by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:41:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=EJjPWrg0i8aqjuW0NGnomyE8WzQ7HVuoMsi3T2ewGf0=; b=O+XmnZCn19gSQ9LhKKOi5ptnw7DICMn0z17F4uo1VVTkzXwb2ScJWwleT9vSnBgGuj wTh3Zc+jwzHhwmOcdb3U7pjhZdoamRmOZ/jSFUzcvbsk8eweunRayngHIL7rjoa+/eVK JPVI2q09eKw7LJI4qwfB3JT32NfIjKFoVj9IlbeoO2PpiA3p0v/DxmEgN49zvbDOjZ4/ U82KeCfn4VYTbhe7HBaxx0E41dnyqI1Z8piJ0fVdLyuDy7IR0BtHs0XyS/LrAaAxyeYL ZIFudBPRpRM5aH7hyqB4Fg+pLfQUbmwhKdiQLNvIUOiPYI2hU/rRrIUvZLYejCck3JeX 3EAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=EJjPWrg0i8aqjuW0NGnomyE8WzQ7HVuoMsi3T2ewGf0=; b=hEyCFR39ji16AXv7G3xOMFY/PE6FDpY4nCkffrNxVPkdR3W1JtI5lgBougtjiWjkU5 9zrFm02UubIOpHxxjZVSDZxSDneZ/rlyfHuiI1Tdv/ESKBEu5kcfTBk/QBd4ez9+D4eR gcjRAO5DTxsNgskFPXcgFRIsO+OPuqAAgm9I3zdSn4QqRxfBQra625WqRQQUbb7i7mB6 5FN1Y9sp78xF19UVdfTzXZj+T0BaE/cP+m8pCM1QET3Vtb7eQ1FwM4ih4tpt29MwCbmT VaKjXajHzsQVTs8Kz6LKwJiB83qj1Akmf7T7E+MLjGBvHFAZeSi0jdQdA25Z0n1Z0Z4V J8hg== X-Gm-Message-State: AOAM533VW2tu8hEUDUoLsrG35SbLapUUjYIKfvp2tXhnkPLyfsUtIh0g h/WfOSOk5W/mmec4o5yD5lGPjvsBqXvPBw== X-Google-Smtp-Source: ABdhPJwYHcBmyWUqISf1+MDWEgjeN49KGbx78XbWEICTBGoADTQXsb4aYsE666vllJTQwD7y/kKXvw== X-Received: by 2002:adf:ea0f:: with SMTP id q15mr904644wrm.113.1598611296846; Fri, 28 Aug 2020 03:41:36 -0700 (PDT) From: Ahmed Karaman To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com, alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com, jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com Subject: [PATCH 3/9] scripts/performance: Refactor dissect.py Date: Fri, 28 Aug 2020 12:40:56 +0200 Message-Id: <20200828104102.4490-4-ahmedkhaledkaraman@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::42a; envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x42a.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ahmed Karaman Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @gmail.com) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" - Apply pylint and flake8 formatting rules to the script. - Move syntax and usage exmaple to main() docstring. - Update get_jit_line() to only detect the main jit call. - Use mypy for specifying parameters and return types in functions. Signed-off-by: Ahmed Karaman Reviewed-by: Aleksandar Markovic --- scripts/performance/dissect.py | 123 ++++++++++++++++++--------------- 1 file changed, 68 insertions(+), 55 deletions(-) diff --git a/scripts/performance/dissect.py b/scripts/performance/dissect.py index bf24f50922..d4df884b75 100755 --- a/scripts/performance/dissect.py +++ b/scripts/performance/dissect.py @@ -1,34 +1,27 @@ #!/usr/bin/env python3 =20 -# Print the percentage of instructions spent in each phase of QEMU -# execution. -# -# Syntax: -# dissect.py [-h] -- [] \ -# [] -# -# [-h] - Print the script arguments help message. -# -# Example of usage: -# dissect.py -- qemu-arm coulomb_double-arm -# -# This file is a part of the project "TCG Continuous Benchmarking". -# -# Copyright (C) 2020 Ahmed Karaman -# Copyright (C) 2020 Aleksandar Markovic -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . +""" +Print the percentage of instructions spent in each phase of QEMU +execution. + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman +Copyright (C) 2020 Aleksandar Markovic + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" =20 import argparse import os @@ -36,23 +29,26 @@ import subprocess import sys import tempfile =20 +from typing import List + =20 -def get_JIT_line(callgrind_data): +def get_jit_line(callgrind_data: List[str]) -> int: """ Search for the first instance of the JIT call in the callgrind_annotate output when ran using --tree=3Dcaller This is equivalent to the self number of instructions of JIT. =20 Parameters: - callgrind_data (list): callgrind_annotate output + callgrind_data (List[str]): callgrind_annotate output =20 Returns: (int): Line number """ line =3D -1 - for i in range(len(callgrind_data)): - if callgrind_data[i].strip('\n') and \ - callgrind_data[i].split()[-1] =3D=3D "[???]": + for (i, callgrind_datum) in enumerate(callgrind_data): + if callgrind_datum.strip('\n') and \ + callgrind_datum.split()[-1] =3D=3D "[???]" and \ + callgrind_datum.split()[1] =3D=3D "*": line =3D i break if line =3D=3D -1: @@ -61,6 +57,18 @@ def get_JIT_line(callgrind_data): =20 =20 def main(): + """ + Parse the command line arguments then start the execution. + Syntax: + dissect.py [-h] -- [] \ + [] + + [-h] - Print the script arguments help message. + + Example of usage: + dissect.py -- qemu-arm coulomb_double-arm + """ + # Parse the command line arguments parser =3D argparse.ArgumentParser( usage=3D'dissect.py [-h] -- ' @@ -76,7 +84,7 @@ def main(): =20 # Insure that valgrind is installed check_valgrind =3D subprocess.run( - ["which", "valgrind"], stdout=3Dsubprocess.DEVNULL) + ["which", "valgrind"], stdout=3Dsubprocess.DEVNULL, check=3DFalse) if check_valgrind.returncode: sys.exit("Please install valgrind before running the script.") =20 @@ -93,7 +101,8 @@ def main(): "--callgrind-out-file=3D" + data_path] + command), stdout=3Dsubprocess.DEVNULL, - stderr=3Dsubprocess.PIPE) + stderr=3Dsubprocess.PIPE, + check=3DFalse) if callgrind.returncode: sys.exit(callgrind.stderr.decode("utf-8")) =20 @@ -102,7 +111,8 @@ def main(): callgrind_annotate =3D subprocess.run( ["callgrind_annotate", data_path, "--tree=3Dcaller"], stdout=3Doutput, - stderr=3Dsubprocess.PIPE) + stderr=3Dsubprocess.PIPE, + check=3DFalse) if callgrind_annotate.returncode: sys.exit(callgrind_annotate.stderr.decode("utf-8")) =20 @@ -120,25 +130,28 @@ def main(): total_instructions =3D int(total_instructions.replace(',', '')) =20 # Line number with the JIT self number of instructions - JIT_self_instructions_line_number =3D get_JIT_line(callgrind_data) + jit_self_instructions_line_number =3D get_jit_line(callgrind_data) # Get the JIT self number of instructions - JIT_self_instructions_line_data =3D \ - callgrind_data[JIT_self_instructions_line_number] - JIT_self_instructions =3D JIT_self_instructions_line_data.split()[= 0] - JIT_self_instructions =3D int(JIT_self_instructions.replace(',', '= ')) + jit_self_instructions_line_data =3D \ + callgrind_data[jit_self_instructions_line_number] + jit_self_instructions =3D jit_self_instructions_line_data.split()[= 0] + jit_self_instructions =3D int(jit_self_instructions.replace(',', '= ')) =20 # Line number with the JIT self + inclusive number of instructions - # It's the line above the first JIT call when running with --tree= =3Dcaller - JIT_total_instructions_line_number =3D JIT_self_instructions_line_= number-1 + # It's the line above the first JIT call when running with + # --tree=3Dcaller + jit_total_instructions_line_number =3D \ + jit_self_instructions_line_number-1 # Get the JIT self + inclusive number of instructions - JIT_total_instructions_line_data =3D \ - callgrind_data[JIT_total_instructions_line_number] - JIT_total_instructions =3D JIT_total_instructions_line_data.split(= )[0] - JIT_total_instructions =3D int(JIT_total_instructions.replace(',',= '')) + jit_total_instructions_line_data =3D \ + callgrind_data[jit_total_instructions_line_number] + jit_total_instructions =3D jit_total_instructions_line_data.split(= )[0] + jit_total_instructions =3D int(jit_total_instructions.replace(',',= '')) =20 # Calculate number of instructions in helpers and code generation - helpers_instructions =3D JIT_total_instructions-JIT_self_instructi= ons - code_generation_instructions =3D total_instructions-JIT_total_inst= ructions + helpers_instructions =3D jit_total_instructions-jit_self_instructi= ons + code_generation_instructions =3D \ + total_instructions-jit_total_instructions =20 # Print results (Insert commas in large numbers) # Print total number of instructions @@ -149,12 +162,12 @@ def main(): print('{:<20}{:>20}\t{:>6.3f}%'. format("Code Generation:", format(code_generation_instructions, ","), - (code_generation_instructions / total_instructions) *= 100)) - # Print JIT instructions and percentage + (code_generation_instructions/total_instructions)*100= )) + # Print jit instructions and percentage print('{:<20}{:>20}\t{:>6.3f}%'. - format("JIT Execution:", - format(JIT_self_instructions, ","), - (JIT_self_instructions / total_instructions) * 100)) + format("jit Execution:", + format(jit_self_instructions, ","), + (jit_self_instructions / total_instructions) * 100)) # Print helpers instructions and percentage print('{:<20}{:>20}\t{:>6.3f}%'. format("Helpers:", --=20 2.17.1 From nobody Sun Nov 16 04:05:48 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1598611396; cv=none; d=zohomail.com; s=zohoarc; b=ZvXo6gcAIc75+lr3MnZJvfMkJKdrOK5VcCO388U78J7T+6BYm5fQCd29x8TxQfMMzrrd0Pi5Ro3RhT8gL5L2pgh6l1khmv8BI+mTQBH7cCGD/pGypTLdN7IeDx3/QWxD1Ile9orhxmzJSGPrDXyieXvRiKTclcGavdGOBJ6M6PI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1598611396; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=BLJGQzaC58i7Jzawy2mcPTfB5Z7zUVTcLSsOZpQQw9o=; b=Yq3Q9oUSpQHHW72doEzXMaM9KI8GhIXkxDHn79VDSIy2RX8fss30LjXhXXoiMj5/lSrpgMW6B1ti1HW7xDQRcRn2lfjMiPlFOManoOUJJOsf51p0L1OYjzipdfP8LFtxZmk+M83Ce3w5iPr0nnh+PE0Xtu7sboYWWXaSRtNA1No= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1598611396522729.9845480282758; Fri, 28 Aug 2020 03:43:16 -0700 (PDT) Received: from localhost ([::1]:57154 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kBbql-00058u-0M for importer@patchew.org; Fri, 28 Aug 2020 06:43:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34440) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kBbpH-0003To-FF for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:43 -0400 Received: from mail-wm1-x329.google.com ([2a00:1450:4864:20::329]:39586) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kBbpE-0005gz-JQ for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:43 -0400 Received: by mail-wm1-x329.google.com with SMTP id b79so461882wmb.4 for ; Fri, 28 Aug 2020 03:41:40 -0700 (PDT) Received: from localhost.localdomain ([197.58.77.158]) by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:41:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BLJGQzaC58i7Jzawy2mcPTfB5Z7zUVTcLSsOZpQQw9o=; b=Ee1yLB37j//pyLubHmHwGE8x2jNKvKKRnrZaLj1/xd5pSgwuowx6nfn0DOwOpSopLR lGrY3IzEbRA9n29sUMWiAFi5Nj2kl0YaJ/mhdAW/uodYb0B7NNvBctZw+DZXFQ3yidzC EP9i1sM/0hQ2O7xlHkMT6RC679fmLATZv4VoQMCxTWt+zhdBTs6I14b7TXYDDG/KkRCZ 86si0/dF9yibzmS08RLtZXmlOAvo88BSy/Ly0dVn0/ALObIPfXIAO08mTXce39zOOXZf kv9JgEYhlYyp+y5XTcjCCx0XcGHyghgbGVBw4WAQTtaQYY+5HAiIs0lfR5QrPs6MPB7N CXvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BLJGQzaC58i7Jzawy2mcPTfB5Z7zUVTcLSsOZpQQw9o=; b=A7mJaKGoZVAedVWUOv2sMnSRbe+WRYr5r+gHJs3hCyDC0+PItsbn+MH7zYaac3qPgU oXzUp0EByUSSp383wDWES+vsiNquyZMwj0qQBErrwFtHis8nKRZYZOg6VplliMw+rey8 1v8zDcljO6rTiM10nM2HomWd2ry3rSz6hVcMbj5B9LoUQDaYIT9yHbzPJknThkaoSjuN CezMwJ3yhn2icmsk54U0xDTLCx2zXMnIff2WNrfSTxSEklbr8D8it5TrXi2O9GdckmS1 muKWR3W4ToEFsnOhLcpQyVO+E8Yk1MooiftFUKQVwo5U+YgZBt4UOSE7eIKtR55iZ7cV yu1Q== X-Gm-Message-State: AOAM5333ZetjEBn2uL10Mll/K7q/nQzpX04uhpuT9XH4KXcjM0STbCF0 gsvUkaXDkT+NMIoPkYA2qeJoP0JO5oQqHg== X-Google-Smtp-Source: ABdhPJxzVzFgK/pQN+NIm9cfQdPnRVmfEJwuo9hH0KmgVb+A2Odn35Uv2iFuBuHTUbF0FlSOjwCBsQ== X-Received: by 2002:a1c:e919:: with SMTP id q25mr953381wmc.123.1598611298903; Fri, 28 Aug 2020 03:41:38 -0700 (PDT) From: Ahmed Karaman To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com, alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com, jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com Subject: [PATCH 4/9] scripts/performance: Add list_fn_callees.py script Date: Fri, 28 Aug 2020 12:40:57 +0200 Message-Id: <20200828104102.4490-5-ahmedkhaledkaraman@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::329; envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wm1-x329.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ahmed Karaman Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @gmail.com) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Python script that prints the callees of a given list of QEMU functions. Syntax: list_fn_callees.py [-h] -f FUNCTION [FUNCTION ...] -- \ [] \ [] [-h] - Print the script arguments help message. -f FUNCTION [FUNCTION ...] - List of function names Example of usage: list_fn_callees.py -f helper_float_sub_d helper_float_mul_d -- \ qemu-mips coulomb_double-mips -n10 Example output: Total number of instructions: 108,952,851 Callees of helper_float_sub_d: No. Instructions Percentage Calls Ins/Call Function Name Source File --- ------------ ---------- ------ -------- ------------- --------------- 1 153,160 0.141% 1,305 117 float64_sub /fpu/softf= loat.c Callees of helper_float_mul_d: No. Instructions Percentage Calls Ins/Call Function Name Source File --- ------------ ---------- ------ -------- ------------- --------------- 1 131,137 0.120% 1,014 129 float64_mul /fpu/softf= loat.c Signed-off-by: Ahmed Karaman --- scripts/performance/list_fn_callees.py | 245 +++++++++++++++++++++++++ 1 file changed, 245 insertions(+) create mode 100755 scripts/performance/list_fn_callees.py diff --git a/scripts/performance/list_fn_callees.py b/scripts/performance/l= ist_fn_callees.py new file mode 100755 index 0000000000..6aa8f6b6ca --- /dev/null +++ b/scripts/performance/list_fn_callees.py @@ -0,0 +1,245 @@ +#!/usr/bin/env python3 + +""" +Print the callees of a given list of QEMU functions. + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman +Copyright (C) 2020 Aleksandar Markovic + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" + +import argparse +import os +import subprocess +import sys +import tempfile + +from typing import List, Union + + +def find_function_lines(function_name: str, + callgrind_data: List[str]) -> List[int]: + """ + Search for the line with the function name in the + callgrind_annotate output when ran using --tre=3Dcalling. + All the function callees should be listed after that line. + + Parameters: + function_name (string): The desired function name to print its callees + callgrind_data (List[str]): callgrind_annotate output + + Returns: + (List[int]): List of function line numbers + """ + lines =3D [] + for (i, callgrind_datum) in enumerate(callgrind_data): + split_line =3D callgrind_datum.split() + if len(split_line) > 2 and \ + split_line[1] =3D=3D "*" and \ + split_line[2].split(":")[-1] =3D=3D function_name: + # Function might be in the callgrind_annotate output more than + # once, so don't break after finding an instance + if callgrind_data[i + 1] !=3D "\n": + # Only append the line number if the found instance has + # callees + lines.append(i) + return lines + + +def get_function_calles( + function_lines: List[int], + callgrind_data: List[str]) -> List[List[Union[str, int]]]: + """ + Get all callees data for a function given its list of line numbers in + callgrind_annotate output. + + Parameters: + function_lines (List[int]): Line numbers of the function to get its ca= llees + callgrind_data (List[str]): callgrind_annotate output + + Returns: + (List[List[Union[str, int]]]):[[number_of_instructions(int), + callee_name(str), + number_of_calls(int), + source_file(str)], + ...] + """ + callees: List[List[Union[str, int]]] =3D [] + for function_line in function_lines: + next_callee =3D function_line + 1 + while callgrind_data[next_callee] !=3D "\n": + split_line =3D callgrind_data[next_callee].split() + number_of_instructions =3D int(split_line[0].replace(",", "")) + source_file =3D split_line[2].split(":")[0] + callee_name =3D split_line[2].split(":")[1] + number_of_calls =3D int(split_line[3][1:-2]) + callees.append([number_of_instructions, callee_name, + number_of_calls, source_file]) + next_callee +=3D 1 + return sorted(callees, reverse=3DTrue) + + +def main(): + """ + Parse the command line arguments then start execution. + + Syntax: + list_fn_callees.py [-h] -f FUNCTION [FUNCTION ...] -- \ + [] \ + [] + + [-h] - Print the script arguments help message. + -f FUNCTION [FUNCTION ...] - List of function names + + Example of usage: + list_fn_callees.py -f helper_float_sub_d helper_float_mul_d -- \ + qemu-mips coulomb_double-mips + """ + + # Parse the command line arguments + parser =3D argparse.ArgumentParser( + usage=3D"list_fn_callees.py [-h] -f FUNCTION [FUNCTION ...] -- " + " [] " + " []") + + parser.add_argument("-f", dest=3D"function", type=3Dstr, + nargs=3D"+", required=3DTrue, + help=3D"list of function names to print their call= ees") + + parser.add_argument("command", type=3Dstr, nargs=3D"+", help=3Dargpars= e.SUPPRESS) + + args =3D parser.parse_args() + + # Extract the needed variables from the args + command =3D args.command + function_names =3D args.function + + # Insure that valgrind is installed + check_valgrind =3D subprocess.run( + ["which", "valgrind"], stdout=3Dsubprocess.DEVNULL, check=3DFalse) + if check_valgrind.returncode: + sys.exit("Please install valgrind before running the script.") + + # Save all intermediate files in a temporary directory + with tempfile.TemporaryDirectory() as tmpdirname: + # callgrind output file path + data_path =3D os.path.join(tmpdirname, "callgrind.data") + # callgrind_annotate output file path + annotate_out_path =3D os.path.join(tmpdirname, "callgrind_annotate= .out") + + # Run callgrind + callgrind =3D subprocess.run((["valgrind", + "--tool=3Dcallgrind", + "--callgrind-out-file=3D" + data_path] + + command), + stdout=3Dsubprocess.DEVNULL, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if callgrind.returncode: + sys.exit(callgrind.stderr.decode("utf-8")) + + # Save callgrind_annotate output + with open(annotate_out_path, "w") as output: + callgrind_annotate =3D subprocess.run(["callgrind_annotate", + data_path, + "--threshold=3D100", + "--tree=3Dcalling"], + stdout=3Doutput, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if callgrind_annotate.returncode: + sys.exit(callgrind_annotate.stderr.decode("utf-8")) + + # Read the callgrind_annotate output to callgrind_data[] + callgrind_data =3D [] + with open(annotate_out_path, "r") as data: + callgrind_data =3D data.readlines() + + # Line number with the total number of instructions + total_instructions_line_number =3D 20 + # Get the total number of instructions + total_instructions_line_data =3D \ + callgrind_data[total_instructions_line_number] + total_instructions =3D total_instructions_line_data.split()[0] + + print("Total number of instructions: {}\n".format(total_instructio= ns)) + + # Remove commas and convert to int + total_instructions =3D int(total_instructions.replace(",", "")) + + for function_name in function_names: + # Line numbers with the desired function + function_lines =3D find_function_lines(function_name, callgrin= d_data) + + if len(function_lines) =3D=3D 0: + print("Couldn't locate function: {}.\n".format( + function_name)) + continue + + # Get function callees + function_callees =3D get_function_calles( + function_lines, callgrind_data) + + print("Callees of {}:\n".format(function_name)) + + # Print table header + print("{:>4} {:>15} {:>10} {:>15} {:>10} {:<25} {}". + format( + "No.", + "Instructions", + "Percentage", + "Calls", + "Ins/Call", + "Function Name", + "Source File") + ) + + print("{:>4} {:>15} {:>10} {:>15} {:>10} {:<25} {}". + format( + "-" * 4, + "-" * 15, + "-" * 10, + "-" * 15, + "-" * 10, + "-" * 25, + "-" * 30) + ) + + for (index, callee) in enumerate(function_callees, start=3D1): + instructions =3D callee[0] + percentage =3D (callee[0] / total_instructions) * 100 + calls =3D callee[2] + instruction_per_call =3D int(callee[0] / callee[2]) + function_name =3D callee[1] + source_file =3D callee[3] + # Print extracted data + print("{:>4} {:>15} {:>9.3f}% {:>15} {:>10} {:<25} {= }". + format( + index, + format(instructions, ","), + round(percentage, 3), + format(calls, ","), + format(instruction_per_call, ","), + function_name, + source_file) + ) + + print("\n") + + +if __name__ =3D=3D "__main__": + main() --=20 2.17.1 From nobody Sun Nov 16 04:05:48 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1598611486; cv=none; d=zohomail.com; s=zohoarc; b=co61ViHzgEsC2syIvlMzklj0rThuJIXgfkhw8nxZtgpMpasat7c032PSawz/3b8gQOMZBn919laK4kZM/9R2/tlwIQMGCt8PNENS+cRqzpnXwbwQ4OGFUPBpNXJP/keFGtxj4yAScFmDeVjzytZre2mNsVA7OeZAwFbsI1zzZaE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1598611486; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=v7lvcz/T9X8UTSmvha1Vg7mQ7t6EJ96o/jbqvgFXbUM=; b=cRepqVCh8Njo5hRpFTVGAqLEltyKfLjaKz1itmBRuyb1eppcZ4MF3fQ4VCh7NyRKiNYoP/ahgtg7/FOt6Nefw71jKWBT1hpj3jXEz/+rSCCOhTHAV5w+1kh9hbz0q6KiqGsP5/oMSaUpnEPrIIjta+tH/SMBYtCQ0OKyXdbSvSE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1598611486714357.1567278250718; Fri, 28 Aug 2020 03:44:46 -0700 (PDT) Received: from localhost ([::1]:37044 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kBbsC-0008Qn-NN for importer@patchew.org; Fri, 28 Aug 2020 06:44:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34456) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kBbpJ-0003W3-Do for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:45 -0400 Received: from mail-wr1-x444.google.com ([2a00:1450:4864:20::444]:42832) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kBbpH-0005hF-45 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:45 -0400 Received: by mail-wr1-x444.google.com with SMTP id c18so834373wrm.9 for ; Fri, 28 Aug 2020 03:41:42 -0700 (PDT) Received: from localhost.localdomain ([197.58.77.158]) by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:41:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=v7lvcz/T9X8UTSmvha1Vg7mQ7t6EJ96o/jbqvgFXbUM=; b=Zm/LCScjFpPa/OlSC90gjfzM/dO1Cbbd5MhzgHc9Av9kwuVbpJIujUhey643OVEB1d TT/D7HlKFQmnS9Q/4boVcHsch3eljfQZQHjoVG01OgX2Vdp2N5J1I5FawkA49Jm962DL MY2UmlBxSy2HvOFv7r8nQrmQZd/lYbDSYj0HpMra0uIMN6rxb5Srtyen7yRJUWmV3CXT SFpiLG2qgNMuX6st734EPDycMo0Rm3pYVC/MRiZjHBz7QeE84BnSfz9H+vUoxlO+OZ3y Gddeph3LBqzyWTF4LCmBxde3BbQjLQRmp/E0tmh44npvEsgrCxE8UL7qqEc+59O4mga8 lgKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=v7lvcz/T9X8UTSmvha1Vg7mQ7t6EJ96o/jbqvgFXbUM=; b=Tv8P2f6ILztUAIY7DoyREs1wKjd4qaFtO+ybMAvqN/B/2obnhNhnjMwvF6uzXh/9Ok MvyR7/K4LLEkyJT9laJJNGByRPGqNyVhaEKNEizKIjIFCGyk6xJ93ollHM1GvxFD38uN hZSZozq9FwHdeABqYhST1vv1FOdTEqUBtlmevdIcxSxsR09ggmyuYTi3le4fbazzW8Om gRu7UUGFFEvdYTjHvxm/PwDziGi1zRk9cphZi6/WsGVjxqm00uJ7m4g1YPs6FkvVihxO K9nnjueyO6KixGE+XOawdCqFDvTYbYp+Er9/1kpMmcWXLmPic4X4TnYUKF7fnC+2dSEq aXLQ== X-Gm-Message-State: AOAM531UJu/+YYBcpjhRLoVahiLfyT/9gTEV1omwSqS+J7MiGclnEIJG wyGiHz+6QWyYwHxPomP+uFk34gN9hKaU+g== X-Google-Smtp-Source: ABdhPJyyk0V9ys0F1gVgrOARWvF6p8zAVaB4JNP7nTqmgazkjjzo1v0ik5fyIsGCLdQhEQDDkO3rdg== X-Received: by 2002:a5d:4c90:: with SMTP id z16mr1016638wrs.170.1598611300883; Fri, 28 Aug 2020 03:41:40 -0700 (PDT) From: Ahmed Karaman To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com, alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com, jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com Subject: [PATCH 5/9] scripts/performance: Add list_helpers.py script Date: Fri, 28 Aug 2020 12:40:58 +0200 Message-Id: <20200828104102.4490-6-ahmedkhaledkaraman@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::444; envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x444.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ahmed Karaman Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @gmail.com) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Python script that prints executed helpers of a QEMU invocation. Syntax: list_helpers.py [-h] -- \ [] \ [] [-h] - Print the script arguments help message. Example of usage: list_helpers.py -- qemu-mips coulomb_double-mips -n10 Example output: Total number of instructions: 108,933,695 Executed QEMU Helpers: No. Ins Percent Calls Ins/Call Helper Name Source File --- ------- ------- ------ -------- -------------------- --------------- 1 183,021 0.168% 1,305 140 helper_float_sub_d /target/= mips/fpu_helper.c 2 177,111 0.163% 770 230 helper_float_madd_d /target/= mips/fpu_helper.c 3 171,537 0.157% 1,014 169 helper_float_mul_d /target/= mips/fpu_helper.c 4 157,298 0.144% 2,443 64 helper_lookup_tb_ptr /accel/t= cg/tcg-runtime.c 5 138,123 0.127% 897 153 helper_float_add_d /target/= mips/fpu_helper.c 6 47,083 0.043% 207 227 helper_float_msub_d /target/= mips/fpu_helper.c 7 24,062 0.022% 487 49 helper_cmp_d_lt /target/= mips/fpu_helper.c 8 22,910 0.021% 150 152 helper_float_div_d /target/= mips/fpu_helper.c 9 15,497 0.014% 321 48 helper_cmp_d_eq /target/= mips/fpu_helper.c 10 9,100 0.008% 52 175 helper_float_trunc_w_d /target/= mips/fpu_helper.c 11 7,059 0.006% 10 705 helper_float_sqrt_d /target/= mips/fpu_helper.c 12 3,000 0.003% 40 75 helper_cmp_d_ule /target/= mips/fpu_helper.c 13 2,720 0.002% 20 136 helper_float_cvtd_w /target/= mips/fpu_helper.c 14 2,477 0.002% 27 91 helper_swl /target/= mips/op_helper.c 15 2,000 0.002% 40 50 helper_cmp_d_le /target/= mips/fpu_helper.c 16 1,800 0.002% 40 45 helper_cmp_d_un /target/= mips/fpu_helper.c 17 1,164 0.001% 12 97 helper_raise_exception_ /target/= mips/op_helper.c 18 720 0.001% 10 72 helper_cmp_d_ult /target/= mips/fpu_helper.c 19 560 0.001% 140 4 helper_cfc1 /target/= mips/fpu_helper.c Signed-off-by: Ahmed Karaman --- scripts/performance/list_helpers.py | 221 ++++++++++++++++++++++++++++ 1 file changed, 221 insertions(+) create mode 100755 scripts/performance/list_helpers.py diff --git a/scripts/performance/list_helpers.py b/scripts/performance/list= _helpers.py new file mode 100755 index 0000000000..823b1cef66 --- /dev/null +++ b/scripts/performance/list_helpers.py @@ -0,0 +1,221 @@ +#!/usr/bin/env python3 + +""" +Print the executed helpers of a QEMU invocation. + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman +Copyright (C) 2020 Aleksandar Markovic + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" + +import argparse +import os +import subprocess +import sys +import tempfile + +from typing import List, Union + + +def find_jit_line(callgrind_data: List[str]) -> int: + """ + Search for the line with the JIT call in the callgrind_annotate + output when ran using --tre=3Dcalling. + All the helpers should be listed after that line. + + Parameters: + callgrind_data (List[str]): callgrind_annotate output + + Returns: + (int): Line number of JIT call + """ + line =3D -1 + for (i, callgrind_datum) in enumerate(callgrind_data): + split_line =3D callgrind_datum.split() + if len(split_line) > 2 and \ + split_line[1] =3D=3D "*" and \ + split_line[-1] =3D=3D "[???]": + line =3D i + break + return line + + +def get_helpers(jit_line: int, + callgrind_data: List[str]) -> List[List[Union[str, int]]]: + """ + Get all helpers data given the line number of the JIT call. + + Parameters: + jit_line (int): Line number of the JIT call + callgrind_data (List[str]): callgrind_annotate output + + Returns: + (List[List[Union[str, int]]]):[[number_of_instructions(int), + helper_name(str), + number_of_calls(int), + source_file(str)], + ...] + """ + helpers: List[List[Union[str, int]]] =3D [] + next_helper =3D jit_line + 1 + while callgrind_data[next_helper] !=3D "\n": + split_line =3D callgrind_data[next_helper].split() + number_of_instructions =3D int(split_line[0].replace(",", "")) + source_file =3D split_line[2].split(":")[0] + callee_name =3D split_line[2].split(":")[1] + number_of_calls =3D int(split_line[3][1:-2]) + helpers.append([number_of_instructions, callee_name, + number_of_calls, source_file]) + next_helper +=3D 1 + return sorted(helpers, reverse=3DTrue) + + +def main(): + """ + Parse the command line arguments then start execution + + Syntax: + list_helpers.py [-h] -- \ + [] \ + [] + + [-h] - Print the script arguments help message. + + Example of usage: + list_helpers.py -- qemu-mips coulomb_double-mips + """ + # Parse the command line arguments + parser =3D argparse.ArgumentParser( + usage=3D"list_helpers.py [-h] -- " + " [] " + " []") + + parser.add_argument("command", type=3Dstr, nargs=3D"+", help=3Dargpars= e.SUPPRESS) + + args =3D parser.parse_args() + + # Extract the needed variables from the args + command =3D args.command + + # Insure that valgrind is installed + check_valgrind =3D subprocess.run( + ["which", "valgrind"], stdout=3Dsubprocess.DEVNULL, check=3DFalse) + if check_valgrind.returncode: + sys.exit("Please install valgrind before running the script.") + + # Save all intermediate files in a temporary directory + with tempfile.TemporaryDirectory() as tmpdirname: + # callgrind output file path + data_path =3D os.path.join(tmpdirname, "callgrind.data") + # callgrind_annotate output file path + annotate_out_path =3D os.path.join(tmpdirname, "callgrind_annotate= .out") + + # Run callgrind + callgrind =3D subprocess.run((["valgrind", + "--tool=3Dcallgrind", + "--callgrind-out-file=3D" + data_path] + + command), + stdout=3Dsubprocess.DEVNULL, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if callgrind.returncode: + sys.exit(callgrind.stderr.decode("utf-8")) + + # Save callgrind_annotate output + with open(annotate_out_path, "w") as output: + callgrind_annotate =3D subprocess.run(["callgrind_annotate", + data_path, + "--threshold=3D100", + "--tree=3Dcalling"], + stdout=3Doutput, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if callgrind_annotate.returncode: + sys.exit(callgrind_annotate.stderr.decode("utf-8")) + + # Read the callgrind_annotate output to callgrind_data[] + callgrind_data =3D [] + with open(annotate_out_path, "r") as data: + callgrind_data =3D data.readlines() + + # Line number with the total number of instructions + total_instructions_line_number =3D 20 + # Get the total number of instructions + total_instructions_line_data =3D \ + callgrind_data[total_instructions_line_number] + total_instructions =3D total_instructions_line_data.split()[0] + + print("Total number of instructions: {}\n".format(total_instructio= ns)) + + # Remove commas and convert to int + total_instructions =3D int(total_instructions.replace(",", "")) + + # Line number with the JIT call + jit_line =3D find_jit_line(callgrind_data) + if jit_line =3D=3D -1: + sys.exit("Couldn't locate the JIT call ... Exiting") + + # Get helpers + helpers =3D get_helpers(jit_line, callgrind_data) + + print("Executed QEMU Helpers:\n") + + # Print table header + print("{:>4} {:>15} {:>10} {:>15} {:>10} {:<25} {}". + format( + "No.", + "Instructions", + "Percentage", + "Calls", + "Ins/Call", + "Helper Name", + "Source File") + ) + + print("{:>4} {:>15} {:>10} {:>15} {:>10} {:<25} {}". + format( + "-" * 4, + "-" * 15, + "-" * 10, + "-" * 15, + "-" * 10, + "-" * 25, + "-" * 30) + ) + + for (index, callee) in enumerate(helpers, start=3D1): + instructions =3D callee[0] + percentage =3D (callee[0] / total_instructions) * 100 + calls =3D callee[2] + instruction_per_call =3D int(callee[0] / callee[2]) + helper_name =3D callee[1] + source_file =3D callee[3] + # Print extracted data + print("{:>4} {:>15} {:>9.3f}% {:>15} {:>10} {:<25} {}". + format( + index, + format(instructions, ","), + round(percentage, 3), + format(calls, ","), + format(instruction_per_call, ","), + helper_name, + source_file) + ) + + +if __name__ =3D=3D "__main__": + main() --=20 2.17.1 From nobody Sun Nov 16 04:05:48 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1598611625; cv=none; d=zohomail.com; s=zohoarc; b=DDkzZ288mX2J+o003Ino4de6tVX58FTDzz80xnsZbPpKU4yxdA+9aMQBKM6+r53e0E1MipjyACLZofz2MQO66n+DlupofaAlCfx2LoOK4o7ETEzmj5sLB2alsmuoGz74otY/G/koqJJnbTyDnuaRtVtqIYtA7+Rsdn59CflthqM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1598611625; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=LIf0cxQtW4hzKm0O7L8caZ594n9XcAZSf6i1rCeNjZk=; b=U6W9zM2aJsjhcd1eIuvgzsIf8rH5f5h8ljPfEGsGyAfibHRCFRFbQqAznCAF19FSpkWOpGSVP+wdjkJ1mcFg6/qpDB+ZZUu9A7YC/ygab94/b0+M0O1D7Q9TM1v74UfgXNnRsfB3WsnlhGodcOwSL3VRIA474hjGwdJCIWWHP1c= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1598611625410972.671533445616; Fri, 28 Aug 2020 03:47:05 -0700 (PDT) Received: from localhost ([::1]:42302 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kBbuS-0002Fi-4K for importer@patchew.org; Fri, 28 Aug 2020 06:47:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34486) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kBbpM-0003Yn-4j for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:48 -0400 Received: from mail-wr1-x441.google.com ([2a00:1450:4864:20::441]:40951) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kBbpJ-0005hg-3a for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:47 -0400 Received: by mail-wr1-x441.google.com with SMTP id b18so843147wrs.7 for ; Fri, 28 Aug 2020 03:41:44 -0700 (PDT) Received: from localhost.localdomain ([197.58.77.158]) by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:41:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=LIf0cxQtW4hzKm0O7L8caZ594n9XcAZSf6i1rCeNjZk=; b=d22AIMjKeRUgBAq6Me+AslRt0Z7/E/+lfAYk3Vhyj8k1DRf8LQV5OXxTbpE/8tCkRC frlF68m70idHOn5t4oyqLtwzxdjs7EiR0UzvYpV4K+XqBu0PKE6PHCzb2jhS7CDPuLFr e6l82FARRMrbNCzVWFcNX8xoYPvXz7IV8nrAr3gN+zTMuytZyPFx4RtHP0uIevCpmbCn d0C2HPig5+nf8mvn+Gq8UER1gcI3Vi7YAbNXUfQkj4v5mP40brC6JnPd/dO51+ZfXlqP KKel+EL8jPYe0ojuwXc/Qyj3AGzSWETumaaUC7xLZ+91uV1sYj4smmidCJfTClMEunx5 /eIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=LIf0cxQtW4hzKm0O7L8caZ594n9XcAZSf6i1rCeNjZk=; b=feEeLct9rkHjZ88mvTPUA4l7AWPH55VmgjDlav8Grcx26XOoq2RlMTk0buX2lCXkeg x03GsHgfixSS59dkFF+/2nle12X8jwBSvsFvpB87A4LoIlMs8EIKEwbnbZ1yNodWAPo2 d2vn4PJnBZOucu+GUl65aOMYUmE4fXi9mAUTMiCsYU+fyEwezbp+PNohHVXvftwqVtLZ DktxSXdJnE90NuV6PrZQW3w2VA0kB0r1uIC5Xlr+/0074aqDGEHOwnlL8VTdiUP28s5F 3cPk8OzFs+1H1dvz6R9FyhxsF8YgFH/tQlbnT1msTIhrxK/WrJA9w39fpNlnjkYnT9kt XlqA== X-Gm-Message-State: AOAM532DQqWgeRH7B5EkZJz1RXs6MuN8PCxUHB3U7wvnmQLDRfFpfYpE VjAsOWrAltggJMxmA+zqq90nsyKBD7J7Rw== X-Google-Smtp-Source: ABdhPJzLgdt+VUg1kK5s5dAGQlueSyKj9ZHTrjhfUnKU3dUnC08oX7yEINk3+0r1h3AlB5U2mS4WfA== X-Received: by 2002:a5d:6087:: with SMTP id w7mr916406wrt.290.1598611303029; Fri, 28 Aug 2020 03:41:43 -0700 (PDT) From: Ahmed Karaman To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com, alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com, jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com Subject: [PATCH 6/9] scripts/performance: Add bisect.py script Date: Fri, 28 Aug 2020 12:40:59 +0200 Message-Id: <20200828104102.4490-7-ahmedkhaledkaraman@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::441; envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x441.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ahmed Karaman Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @gmail.com) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Python script that locates the commit that caused a performance degradation or improvement in QEMU using the git bisect command (binary search). Syntax: bisect.py [-h] -s,--start START [-e,--end END] [-q,--qemu QEMU] \ --target TARGET --tool {perf,callgrind} -- \ [] [-h] - Print the script arguments help message -s,--start START - First commit hash in the search range [-e,--end END] - Last commit hash in the search range (default: Latest commit) [-q,--qemu QEMU] - QEMU path. (default: Path to a GitHub QEMU clone) --target TARGET - QEMU target name --tool {perf,callgrind} - Underlying tool used for measurements Example of usage: bisect.py --start=3Dfdd76fecdd --qemu=3D/path/to/qemu --target=3Dppc \ --tool=3Dperf -- coulomb_double-ppc -n 1000 Example output: Start Commit Instructions: 12,710,790,060 End Commit Instructions: 13,031,083,512 Performance Change: -2.458% Estimated Number of Steps: 10 *****************BISECT STEP 1***************** Instructions: 13,031,097,790 Status: slow commit *****************BISECT STEP 2***************** Instructions: 12,710,805,265 Status: fast commit *****************BISECT STEP 3***************** Instructions: 13,031,028,053 Status: slow commit *****************BISECT STEP 4***************** Instructions: 12,711,763,211 Status: fast commit *****************BISECT STEP 5***************** Instructions: 13,031,027,292 Status: slow commit *****************BISECT STEP 6***************** Instructions: 12,711,748,738 Status: fast commit *****************BISECT STEP 7***************** Instructions: 12,711,748,788 Status: fast commit *****************BISECT STEP 8***************** Instructions: 13,031,100,493 Status: slow commit *****************BISECT STEP 9***************** Instructions: 12,714,472,954 Status: fast commit ****************BISECT STEP 10***************** Instructions: 12,715,409,153 Status: fast commit ****************BISECT STEP 11***************** Instructions: 12,715,394,739 Status: fast commit *****************BISECT RESULT***************** commit 0673ecdf6cb2b1445a85283db8cbacb251c46516 Author: Richard Henderson Date: Tue May 5 10:40:23 2020 -0700 softfloat: Inline float64 compare specializations Replace the float64 compare specializations with inline functions that call the standard float64_compare{,_quiet} functions. Use bool as the return type. *********************************************** Signed-off-by: Ahmed Karaman --- scripts/performance/bisect.py | 425 ++++++++++++++++++++++++++++++++++ 1 file changed, 425 insertions(+) create mode 100755 scripts/performance/bisect.py diff --git a/scripts/performance/bisect.py b/scripts/performance/bisect.py new file mode 100755 index 0000000000..0c60be22ce --- /dev/null +++ b/scripts/performance/bisect.py @@ -0,0 +1,425 @@ +#!/usr/bin/env python3 + +""" +Locate the commit that caused a performance degradation or improvement in +QEMU using the git bisect command (binary search). + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman +Copyright (C) 2020 Aleksandar Markovic + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" + +import argparse +import multiprocessing +import tempfile +import os +import shutil +import subprocess +import sys + +from typing import List + + +# --------------------------- GIT WRAPPERS -------------------------- +def git_bisect(qemu_path: str, qemu_build_path: str, command: str, + args: List[str] =3D None) -> str: + """ + Wrapper function for running git bisect. + + Parameters: + qemu_path (str): QEMU path + qemu_build_path (str): Path to the build directory with configuration = files + command (str): bisect command (start|fast|slow|reset) + args (list): Optional arguments + + Returns: + (str): git bisect stdout. + """ + process =3D ["git", "bisect", command] + if args: + process +=3D args + bisect =3D subprocess.run(process, + cwd=3Dqemu_path, + stdout=3Dsubprocess.PIPE, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if bisect.returncode: + clean_exit(qemu_build_path, bisect.stderr.decode("utf-8")) + + return bisect.stdout.decode("utf-8") + + +def git_checkout(commit: str, qemu_path: str, qemu_build_path: str) -> Non= e: + """ + Wrapper function for checking out a given git commit. + + Parameters: + commit (str): Commit hash of a git commit + qemu_path (str): QEMU path + qemu_build_path (str): Path to the build directory with configuration = files + """ + checkout_commit =3D subprocess.run(["git", + "checkout", + commit], + cwd=3Dqemu_path, + stdout=3Dsubprocess.DEVNULL, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if checkout_commit.returncode: + clean_exit(qemu_build_path, checkout_commit.stderr.decode("utf-8")) + + +def git_clone(qemu_path: str) -> None: + """ + Wrapper function for cloning QEMU git repo from GitHub. + + Parameters: + qemu_path (str): Path to clone the QEMU repo to + """ + clone_qemu =3D subprocess.run(["git", + "clone", + "https://github.com/qemu/qemu.git", + qemu_path], + stderr=3Dsubprocess.STDOUT, + check=3DFalse) + if clone_qemu.returncode: + sys.exit("Failed to clone QEMU!") +# ------------------------------------------------------------------- + + +def check_requirements(tool: str) -> None: + """ + Verify that all script requirements are installed (perf|callgrind & gi= t). + + Parameters: + tool (str): Tool used for the measurement (perf or callgrind) + """ + if tool =3D=3D "perf": + check_perf_installation =3D subprocess.run(["which", "perf"], + stdout=3Dsubprocess.DEVNU= LL, + check=3DFalse) + if check_perf_installation.returncode: + sys.exit("Please install perf before running the script.") + + # Insure user has previllage to run perf + check_perf_executability =3D subprocess.run(["perf", "stat", "ls",= "/"], + stdout=3Dsubprocess.DEVN= ULL, + stderr=3Dsubprocess.DEVN= ULL, + check=3DFalse) + if check_perf_executability.returncode: + sys.exit(""" + Error: + You may not have permission to collect stats. + Consider tweaking /proc/sys/kernel/perf_event_paranoid, + which controls use of the performance events system by + unprivileged users (without CAP_SYS_ADMIN). + -1: Allow use of (almost) all events by all users + Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_L= OCK + 0: Disallow ftrace function tracepoint by users without CAP_SYS_AD= MIN + Disallow raw tracepoint access by users without CAP_SYS_ADMIN + 1: Disallow CPU event access by users without CAP_SYS_ADMIN + 2: Disallow kernel profiling by users without CAP_SYS_ADMIN + To make this setting permanent, edit /etc/sysctl.conf too, e.g.: + kernel.perf_event_paranoid =3D -1 + + *Alternatively, you can run this script under sudo privileges. + """) + elif tool =3D=3D "callgrind": + check_valgrind_installation =3D subprocess.run(["which", "valgrind= "], + stdout=3Dsubprocess.D= EVNULL, + check=3DFalse) + if check_valgrind_installation.returncode: + sys.exit("Please install valgrind before running the script.") + + # Insure that git is installed + check_git_installation =3D subprocess.run(["which", "git"], + stdout=3Dsubprocess.DEVNULL, + check=3DFalse) + if check_git_installation.returncode: + sys.exit("Please install git before running the script.") + + +def clean_exit(qemu_build_path: str, error_message: str) -> None: + """ + Clean up intermediate files and exit. + + Parameters: + qemu_build_path (str): Path to the build directory with configuration = files + error_message (str): Error message to display after exiting + """ + shutil.rmtree(qemu_build_path) + sys.exit(error_message) + + +def make(qemu_build_path: str) -> None: + """ + Build QEMU by running the Makefile. + + Parameters: + qemu_build_path (str): Path to the build directory with configuration = files + """ + run_make =3D subprocess.run(["make", + "-j", + str(multiprocessing.cpu_count())], + cwd=3Dqemu_build_path, + stdout=3Dsubprocess.DEVNULL, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if run_make.returncode: + clean_exit(qemu_build_path, run_make.stderr.decode("utf-8")) + + +def measure_instructions(tool: str, qemu_build_path: str, target: str, + command: List[str]) -> int: + """ + Measure the number of instructions when running an program with QEMU. + + Parameters: + tool (str): Tool used for the measurement (perf|callgrind) + qemu_build_path (str): Path to the build directory with configuration = files + target (str): QEMU target + command (list): Program path and arguments + + Returns: + (int): Number of instructions. + """ + qemu_exe_path =3D os.path.join(qemu_build_path, + "{}-linux-user".format(target), + "qemu-{}".format(target)) + instructions =3D 0 + if tool =3D=3D "perf": + run_perf =3D subprocess.run((["perf", + "stat", + "-x", + " ", + "-e", + "instructions", + qemu_exe_path] + + command), + stdout=3Dsubprocess.DEVNULL, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if run_perf.returncode: + clean_exit(qemu_build_path, run_perf.stderr.decode("utf-8")) + + else: + perf_output =3D run_perf.stderr.decode("utf-8").split(" ") + instructions =3D int(perf_output[0]) + + elif tool =3D=3D "callgrind": + with tempfile.NamedTemporaryFile() as tmpfile: + run_callgrind =3D subprocess.run((["valgrind", + "--tool=3Dcallgrind", + "--callgrind-out-file=3D{}".f= ormat( + tmpfile.name), + qemu_exe_path] + + command), + stdout=3Dsubprocess.DEVNULL, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if run_callgrind.returncode: + clean_exit(qemu_build_path, run_callgrind.stderr.decode("utf-8= ")) + + else: + callgrind_output =3D run_callgrind.stderr.decode("utf-8").spli= t("\n") + instructions =3D int(callgrind_output[8].split(" ")[-1]) + + return instructions + + +def main(): + """ + Parse the command line arguments then start the execution. + + Syntax: + bisect.py [-h] -s,--start START [-e,--end END] [-q,--qemu QEMU] \ + --target TARGET --tool {perf,callgrind} -- \ + [] + + Arguments: + [-h] - Print the script arguments help message + -s,--start START - First commit hash in the search range + [-e,--end END] - Last commit hash in the search range + (default: Latest commit) + [-q,--qemu QEMU] - QEMU path. + (default: Path to a GitHub QEMU clone) + --target TARGET - QEMU target name + --tool {perf,callgrind} - Underlying tool used for measurements + + Example of usage: + bisect.py --start=3Dfdd76fecdd --qemu=3D/path/to/qemu --target=3Dp= pc \ + --tool=3Dperf coulomb_double-ppc -n 1000 + """ + + # Parse the command line arguments + parser =3D argparse.ArgumentParser( + usage=3D"bisect.py [-h] -s,--start START [-e,--end END] [-q,--qemu= QEMU]" + " --target TARGET --tool {perf,callgrind} -- " + " []") + + parser.add_argument("-s", "--start", dest=3D"start", type=3Dstr, requi= red=3DTrue, + help=3D"First commit hash in the search range") + parser.add_argument("-e", "--end", dest=3D"end", type=3Dstr, default= =3D"master", + help=3D"Last commit hash in the search range") + parser.add_argument("-q", "--qemu", dest=3D"qemu", type=3Dstr, default= =3D"", + help=3D"QEMU path") + parser.add_argument("--target", dest=3D"target", type=3Dstr, required= =3DTrue, + help=3D"QEMU target") + parser.add_argument("--tool", dest=3D"tool", choices=3D["perf", "callg= rind"], + required=3DTrue, help=3D"Tool used for measurement= s") + + parser.add_argument("command", type=3Dstr, nargs=3D"+", help=3Dargpars= e.SUPPRESS) + + args =3D parser.parse_args() + + # Extract the needed variables from the args + start_commit =3D args.start + end_commit =3D args.end + qemu =3D args.qemu + target =3D args.target + tool =3D args.tool + command =3D args.command + + # Set QEMU path + if qemu =3D=3D "": + # Create a temp directory for cloning QEMU + tmpdir =3D tempfile.TemporaryDirectory() + qemu_path =3D os.path.join(tmpdir.name, "qemu") + + # Clone QEMU into the temporary directory + print("Fetching QEMU: ", end=3D"", flush=3DTrue) + git_clone(qemu_path) + print() + else: + qemu_path =3D qemu + + # Create the build directory + qemu_build_path =3D os.path.join(qemu_path, "tmp-build-gcc") + + if not os.path.exists(qemu_build_path): + os.mkdir(qemu_build_path) + else: + sys.exit("A build directory with the same name (tmp-build-gcc) use= d in" + " the script is already in the provided QEMU path.") + + # Configure QEMU + configure =3D subprocess.run(["../configure", + "--target-list=3D{}-linux-user".format(tar= get)], + cwd=3Dqemu_build_path, + stdout=3Dsubprocess.DEVNULL, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if configure.returncode: + clean_exit(qemu_build_path, configure.stderr.decode("utf-8")) + + # Do performance measurements for the start commit + git_checkout(start_commit, qemu_path, qemu_build_path) + make(qemu_build_path) + start_commit_instructions =3D measure_instructions(tool, + qemu_build_path, + target, + command) + print("{:<30} {}".format("Start Commit Instructions:", + format(start_commit_instructions, ","))) + + # Do performance measurements for the end commit + git_checkout(end_commit, qemu_path, qemu_build_path) + make(qemu_build_path) + end_commit_instructions =3D measure_instructions(tool, + qemu_build_path, + target, + command) + print("{:<30} {}".format("End Commit Instructions:", + format(end_commit_instructions, ","))) + + # Calculate performance difference between start and end commits + performance_difference =3D \ + (start_commit_instructions - end_commit_instructions) / \ + max(end_commit_instructions, start_commit_instructions) * 100 + performance_change =3D "+" if performance_difference > 0 else "-" + print("{:<30} {}".format("Performance Change:", + performance_change + + str(round(abs(performance_difference), 3))+"%= ")) + + # Set the custom terms used for progressing in "git bisect" + term_old =3D "fast" if performance_difference < 0 else "slow" + term_new =3D "slow" if term_old =3D=3D "fast" else "fast" + + # Start git bisect + git_bisect(qemu_path, qemu_build_path, "start", + ["--term-old", term_old, "--term-new", term_new]) + # Set start commit state + git_bisect(qemu_path, qemu_build_path, term_old, [start_commit]) + # Set end commit state + bisect_output =3D git_bisect( + qemu_path, qemu_build_path, term_new, [end_commit]) + # Print estimated bisect steps + print("\n{:<30} {}\n".format( + "Estimated Number of Steps:", bisect_output.split()[9])) + + # Initialize bisect_count to track the number of performed + bisect_count =3D 1 + + while True: + print("**************BISECT STEP {}**************". + format(bisect_count)) + + make(qemu_build_path) + + instructions =3D measure_instructions(tool, + qemu_build_path, + target, + command) + # Find the difference between the current instructions and start/e= nd + # instructions. + diff_end =3D abs(instructions - end_commit_instructions) + diff_start =3D abs(instructions - start_commit_instructions) + + # If current number of insructions is closer to that of start, + # set current commit as term_old. + # Else, set current commit as term_new. + if diff_end > diff_start: + bisect_command =3D term_old + else: + bisect_command =3D term_new + + print("{:<20} {}".format("Instructions:", format(instructions, ","= ))) + print("{:<20} {}".format("Status:", "{} commit". + format(bisect_command))) + + bisect_output =3D git_bisect(qemu_path, qemu_build_path, bisect_co= mmand) + + # Continue if still bisecting, + # else, print result and break. + if not bisect_output.split(" ")[0] =3D=3D "Bisecting:": + print("\n*****************BISECT RESULT*****************") + commit_message_start =3D bisect_output.find("commit\n") + 7 + commit_message_end =3D bisect_output.find(":040000") - 1 + print(bisect_output[commit_message_start:commit_message_end]) + break + + bisect_count +=3D 1 + + # Reset git bisect + git_bisect(qemu_path, qemu_build_path, "reset") + + # Delete temp build directory + shutil.rmtree(qemu_build_path) + + +if __name__ =3D=3D "__main__": + main() --=20 2.17.1 From nobody Sun Nov 16 04:05:48 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1598611699; cv=none; d=zohomail.com; s=zohoarc; b=mkUyxkS0sGKeLobmaXceCUeMXItQXdas67Q4G6POzZF2xBb5xCvzzB1IFT7qITsBF3xW/3DGG96C4mY6SOZV9DYzM6d/y0AfJrRP4pt55WZDU6nNOiYpVUVMm2xwzq6/PGPeVEAvnGZiQWgid4eZm10wfyRaoV3ESh58nPeKDOQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1598611699; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=ztdp06tHpUTupGG5kZnKkBKNJQ2GBi6jOrEePLCl0Kw=; b=Tnw25jzAGM1pXxCariBllaeVmV/s+gKVyEibRo1HjQzRyltdnLiIZ/eA5RaHBjPwWnK5ae9qiZgkHVPtTXgGIrOKz+t/U9YG0W6NKmtkOTmSYsIieUhLV/RQ44CGC0BJda2sKcv3bCZn9zN2YKs+FlefXHZrk03P1qeEw4+XtgY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1598611699545995.9118947744789; Fri, 28 Aug 2020 03:48:19 -0700 (PDT) Received: from localhost ([::1]:46104 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kBbvd-0003qb-Lz for importer@patchew.org; Fri, 28 Aug 2020 06:48:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34590) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kBbpU-0003rt-Tz for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:56 -0400 Received: from mail-wr1-x433.google.com ([2a00:1450:4864:20::433]:34674) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kBbpP-0005jN-Rs for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:56 -0400 Received: by mail-wr1-x433.google.com with SMTP id f7so857616wrw.1 for ; Fri, 28 Aug 2020 03:41:50 -0700 (PDT) Received: from localhost.localdomain ([197.58.77.158]) by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:41:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ztdp06tHpUTupGG5kZnKkBKNJQ2GBi6jOrEePLCl0Kw=; b=dX41ymz9tc93PMf0ZRirpDHXn9NI60cKLfOFjuDhBM1gUH6SuQItoBy6JDX/CJOvk6 bDz6/aq3ZOE1IxebLaizUmTQkWn7Mz5zHGHn+HrNOhHXjjGoeVdPGn+5PoPp6HRUwEfp FTV+fQ+S9yrf7NDm0zHDjoGsJACo53WQDRRCEIDKIhvrv/aWtZl+w1PJinQyqz1z+gh9 M79aCKCCrSWxwJA6/g2LHsSvIXFrDCu3Tnu0TlZ1ieQV3oyVWhj0HmDcOh7sqeqlGpCk fKYMHuC2qqtaF4h4gvEzINeRW9s6JlztrNEID7Gjo8L/Uw84Mc3tfOZCi0zDSdCUd0rp ABAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ztdp06tHpUTupGG5kZnKkBKNJQ2GBi6jOrEePLCl0Kw=; b=mb1l57QQfTdbklIqsDkSWNR77EcX+iAQSb3xsZd+/RZUBx4PY0mxygyGjRHAUYIZKG wY0fJbM065LqBCBI8VGSAn7g6Pk83rxyhdi75dAbgfvPthAzh4KUovl7B2OC9934c1N2 pCvnP5GlX+jfeljiqgKlmownBCdrJ0CmIJorDl1OOif0JRfSWnmHOUAwkOObP5dSsw2Z BfXn2MYH+gUUVIZmJ1xB5LaCgejcJuMzgUssWN1GQK4pax4EGLhsO3sMF2NSW+x7k532 HUjy3XViWHJfNTsSoAQ4d5EuZN+3o6jDAb1KL5ztdSy60T5r2LEAOEyT7yK4GVlRMfZr 6Ofg== X-Gm-Message-State: AOAM531X5tgLPxIq1FUHahh1HI+Ptk03nI1WDbyHLPkJLviYBMkh1W8j Xc4C27V5KgC0IjvJwaFFHYnnNjQeIFtW8g== X-Google-Smtp-Source: ABdhPJxZOldAOxbmDbmBbcG+/cHy/Z3PNSWRU55JhPE+ZYjn/P9C2pkVeNhEcVnBDOtvT5yTaUrhXA== X-Received: by 2002:a5d:4145:: with SMTP id c5mr986131wrq.18.1598611306982; Fri, 28 Aug 2020 03:41:46 -0700 (PDT) From: Ahmed Karaman To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com, alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com, jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com Subject: [PATCH 7/9] tests/performance: Add nightly tests Date: Fri, 28 Aug 2020 12:41:00 +0200 Message-Id: <20200828104102.4490-8-ahmedkhaledkaraman@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::433; envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x433.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ahmed Karaman Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @gmail.com) A nightly performance testing system to monitor any change in QEMU performance across seventeen different targets. The system includes eight different benchmarks to provide a variety of testing workloads. dijkstra_double: Find the shortest path between the source node and all other nodes using Dijkstra=E2=80=99s algorithm. The graph contains n nodes where all nxn distances are double values. The value of n can be specified using the -n flag. The default value is 2000. dijkstra_int32: Find the shortest path between the source node and all other nodes using Dijkstra=E2=80=99s algorithm. The graph contains n nodes where all nxn distances are int32 values. The value of n can be specified using the -n flag. The default value is 2000. matmult_double: Standard matrix multiplication of an n*n matrix of randomly generated double numbers from 0 to 100. The value of n is passed as an argument with the -n flag. The default value is 200. matmult_int32: Standard matrix multiplication of an n*n matrix of randomly generated integer numbers from 0 to 100. The value of n is passed as an argument with the -n flag. The default value is 200. qsort_double: Quick sort of an array of n randomly generated double numbers from 0 to 1000. The value of n is passed as an argument with the -n flag. The default value is 300000. qsort_int32: Quick sort of an array of n randomly generated integer numbers from 0 to 50000000. The value of n is passed as an argument with the -n flag.The default value is 300000. qsort_string: Quick sort of an array of 10000 randomly generated strings of size 8 (including null terminating character). The sort process is repeated n number of times. The value of n is passed as an argument with the -n flag. The default value is 20. search_string: Search for the occurrence of a small string in a much larger random string (=E2=80=9Cneedle in a hay=E2=80=9D). The search process is repeated = n number of times and each time, a different large random string (=E2=80=9Chay=E2=80= =9D) is generated. The value of n can be specified using the -n flag. The default value is 20. Syntax: nightly_tests_core.py [-h] [-r REF] Optional arguments: -h, --help Show this help message and exit -r REF, --reference REF Reference QEMU version - Default is v5.1.0 Example of usage: nightly_tests_core.py -r v5.1.0 2>log.txt The following report includes detailed setup and execution details of the system: https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Perfor= mance-Tests/ Signed-off-by: Ahmed Karaman Reviewed-by: Aleksandar Markovic --- tests/performance/nightly-tests/README.md | 243 +++++ .../source/dijkstra_double/dijkstra_double.c | 194 ++++ .../source/dijkstra_int32/dijkstra_int32.c | 192 ++++ .../source/matmult_double/matmult_double.c | 123 +++ .../source/matmult_int32/matmult_int32.c | 121 +++ .../source/qsort_double/qsort_double.c | 104 ++ .../source/qsort_int32/qsort_int32.c | 103 ++ .../source/qsort_string/qsort_string.c | 122 +++ .../source/search_string/search_string.c | 110 +++ .../scripts/nightly_tests_core.py | 920 ++++++++++++++++++ .../scripts/run_nightly_tests.py | 135 +++ .../nightly-tests/scripts/send_email.py | 56 ++ 12 files changed, 2423 insertions(+) create mode 100644 tests/performance/nightly-tests/README.md create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijks= tra_double/dijkstra_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijks= tra_int32/dijkstra_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmu= lt_double/matmult_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmu= lt_int32/matmult_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort= _double/qsort_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort= _int32/qsort_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort= _string/qsort_string.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/searc= h_string/search_string.c create mode 100755 tests/performance/nightly-tests/scripts/nightly_tests_c= ore.py create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_tes= ts.py create mode 100644 tests/performance/nightly-tests/scripts/send_email.py diff --git a/tests/performance/nightly-tests/README.md b/tests/performance/= nightly-tests/README.md new file mode 100644 index 0000000000..6db3b351b3 --- /dev/null +++ b/tests/performance/nightly-tests/README.md @@ -0,0 +1,243 @@ +### QEMU Nightly Tests + +**Required settings:** + +Update the `GMAIL_USER` object in `send_email.py` with your credentials. + +For more details on how the system works, please check the [eighth report]= (https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Perfo= rmance-Tests/) of the "TCG Continuos Benchmarking" series. + +**Running the System:** + +The default reference version is v5.1.0. To specify a custom version, plea= se use the `-r, --reference` flag. + +```bash +./run_nightly_tests.py +``` + +**Output:** + +``` +Host CPU : Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz +Host Memory : 15.49 GB + +Start Time (UTC) : 2020-08-25 21:30:01 +End Time (UTC) : 2020-08-25 22:02:37 +Execution Time : 0:32:35.896990 + +Status : SUCCESS + +Note: +Changes denoted by '-----' are less than 0.01%. + +-------------------------------------------------------- + SUMMARY REPORT - COMMIT d1a2b51f +-------------------------------------------------------- + AVERAGE RESULTS +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 158 355 274 ----- +1.693% +alpha 1 914 967 171 ----- +3.524% +arm 8 076 402 940 ----- +2.304% +hppa 4 261 685 987 -0.182% +3.164% +m68k 2 690 273 044 ----- +7.131% +mips 1 862 033 667 ----- +2.494% +mipsel 2 008 211 069 ----- +2.674% +mips64 1 918 635 565 ----- +2.818% +mips64el 2 051 565 677 ----- +3.026% +ppc 2 480 141 217 ----- +3.107% +ppc64 2 576 713 959 ----- +3.143% +ppc64le 2 558 853 539 ----- +3.173% +riscv64 1 406 704 050 ----- +2.65% +s390x 3 158 140 046 ----- +3.118% +sh4 2 364 449 748 ----- +3.33% +sparc64 3 318 544 783 ----- +3.851% +x86_64 1 775 844 158 ----- +2.156% +-------------------------------------------------------- + + DETAILED RESULTS +-------------------------------------------------------- +Test Program: dijkstra_double +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 3 062 583 464 ----- +1.424% +alpha 3 191 864 698 ----- +3.696% +arm 16 357 157 526 ----- +2.347% +hppa 7 228 376 315 -0.139% +3.086% +m68k 4 294 016 587 ----- +9.692% +mips 3 051 419 166 ----- +2.427% +mipsel 3 231 509 618 ----- +2.869% +mips64 3 245 837 754 ----- +2.596% +mips64el 3 414 195 796 ----- +3.021% +ppc 4 914 520 972 -0.041% +4.74% +ppc64 5 098 154 311 ----- +4.565% +ppc64le 5 082 419 054 ----- +4.58% +riscv64 2 192 294 915 ----- +1.955% +s390x 4 584 503 977 ----- +2.896% +sh4 3 949 036 447 ----- +3.464% +sparc64 4 586 203 546 ----- +4.237% +x86_64 2 484 092 105 ----- +1.75% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: dijkstra_int32 +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 210 194 577 ----- +1.493% +alpha 1 494 133 274 ----- +2.15% +arm 8 262 935 967 ----- +2.665% +hppa 5 207 318 306 ----- +3.047% +m68k 1 725 856 962 ----- +2.527% +mips 1 495 227 032 ----- +1.492% +mipsel 1 497 147 869 ----- +1.479% +mips64 1 715 388 570 ----- +1.892% +mips64el 1 695 276 864 ----- +1.913% +ppc 2 014 557 389 ----- +1.819% +ppc64 2 206 267 901 ----- +2.139% +ppc64le 2 197 998 781 ----- +2.146% +riscv64 1 354 912 745 ----- +2.396% +s390x 2 916 247 062 ----- +1.241% +sh4 1 990 532 533 ----- +2.669% +sparc64 2 872 231 051 ----- +3.758% +x86_64 1 553 981 241 ----- +2.12% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: matmult_double +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 1 412 273 223 ----- +0.302% +alpha 3 233 991 649 ----- +7.473% +arm 8 545 173 979 ----- +1.088% +hppa 3 483 597 802 -1.267% +4.468% +m68k 3 919 065 529 ----- +18.431% +mips 2 344 774 894 ----- +4.091% +mipsel 3 329 886 464 ----- +5.177% +mips64 2 359 046 988 ----- +4.076% +mips64el 3 343 664 785 ----- +5.167% +ppc 3 209 457 051 ----- +3.246% +ppc64 3 287 503 981 ----- +3.173% +ppc64le 3 287 189 065 ----- +3.173% +riscv64 1 221 603 682 ----- +0.277% +s390x 2 874 199 923 ----- +5.827% +sh4 3 543 943 634 ----- +6.416% +sparc64 3 426 150 004 ----- +7.139% +x86_64 1 248 917 276 ----- +0.322% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: matmult_int32 +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 598 681 621 ----- +0.585% +alpha 372 437 418 ----- +0.677% +arm 746 583 193 ----- +1.462% +hppa 674 278 359 ----- +1.183% +m68k 410 495 553 ----- +0.9% +mips 499 698 837 ----- +0.531% +mipsel 499 500 429 ----- +0.497% +mips64 481 554 664 ----- +0.599% +mips64el 465 057 054 ----- +0.619% +ppc 341 334 603 ----- +0.944% +ppc64 393 796 203 ----- +0.966% +ppc64le 393 977 298 ----- +0.965% +riscv64 351 709 769 ----- +0.785% +s390x 494 427 384 ----- +0.599% +sh4 402 668 444 ----- +0.899% +sparc64 495 952 959 ----- +1.192% +x86_64 402 928 461 ----- +0.833% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: qsort_double +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 709 683 624 ----- +2.417% +alpha 1 969 460 172 ----- +3.68% +arm 8 322 998 390 ----- +2.587% +hppa 3 188 301 995 -0.047% +2.9% +m68k 4 953 930 065 ----- +15.153% +mips 2 123 919 587 ----- +3.055% +mipsel 2 124 212 187 ----- +3.048% +mips64 1 999 047 826 ----- +3.405% +mips64el 1 996 426 772 ----- +3.409% +ppc 2 819 267 902 -0.021% +5.435% +ppc64 2 768 186 548 ----- +5.513% +ppc64le 2 724 803 772 -0.011% +5.603% +riscv64 1 638 328 937 ----- +4.021% +s390x 2 519 081 708 ----- +3.362% +sh4 2 595 545 154 ----- +2.994% +sparc64 3 988 986 587 ----- +2.747% +x86_64 2 033 468 588 ----- +3.234% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: qsort_int32 +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 193 392 565 ----- +2.916% +alpha 1 521 291 933 ----- +4.193% +arm 3 465 445 043 ----- +2.756% +hppa 2 280 034 340 ----- +3.821% +m68k 1 843 189 041 ----- +3.583% +mips 1 558 024 873 ----- +3.863% +mipsel 1 560 583 980 ----- +3.846% +mips64 1 563 415 749 ----- +4.412% +mips64el 1 542 677 320 ----- +4.474% +ppc 1 728 698 880 ----- +3.665% +ppc64 1 842 444 545 ----- +3.555% +ppc64le 1 791 822 067 ----- +3.661% +riscv64 1 348 866 430 ----- +4.654% +s390x 2 184 073 151 ----- +3.319% +sh4 1 946 492 539 ----- +3.624% +sparc64 3 452 215 585 ----- +2.937% +x86_64 1 813 544 414 ----- +3.537% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: qsort_string +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 592 218 418 ----- +2.468% +alpha 1 855 834 626 ----- +3.487% +arm 7 347 721 165 ----- +2.682% +hppa 4 758 753 926 ----- +3.543% +m68k 2 376 811 462 ----- +3.567% +mips 2 166 608 045 ----- +2.532% +mipsel 2 163 392 541 ----- +2.528% +mips64 2 029 251 969 ----- +3.117% +mips64el 2 011 628 621 ----- +3.145% +ppc 2 492 942 463 ----- +2.673% +ppc64 2 464 702 554 ----- +2.488% +ppc64le 2 445 253 307 ----- +2.505% +riscv64 1 625 053 328 ----- +3.953% +s390x 4 194 608 798 ----- +6.623% +sh4 2 164 142 539 ----- +3.166% +sparc64 4 299 516 539 ----- +4.065% +x86_64 2 940 456 780 ----- +2.649% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: search_string +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 487 814 704 ----- +1.94% +alpha 1 680 723 605 ----- +2.835% +arm 11 563 208 260 ----- +2.848% +hppa 7 272 826 858 ----- +3.263% +m68k 1 998 819 159 ----- +3.198% +mips 1 656 596 909 ----- +1.959% +mipsel 1 659 455 464 ----- +1.947% +mips64 1 955 541 001 ----- +2.447% +mips64el 1 943 598 207 ----- +2.462% +ppc 2 320 350 477 ----- +2.332% +ppc64 2 552 655 634 ----- +2.742% +ppc64le 2 547 364 971 ----- +2.748% +riscv64 1 520 862 601 ----- +3.159% +s390x 5 497 978 370 ----- +1.078% +sh4 2 323 236 696 ----- +3.41% +sparc64 3 427 101 999 ----- +4.73% +x86_64 1 729 364 402 ----- +2.806% +-------------------------------------------------------- +``` diff --git a/tests/performance/nightly-tests/benchmarks/source/dijkstra_dou= ble/dijkstra_double.c b/tests/performance/nightly-tests/benchmarks/source/d= ijkstra_double/dijkstra_double.c new file mode 100644 index 0000000000..9c0bb804ac --- /dev/null +++ b/tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dij= kstra_double.c @@ -0,0 +1,194 @@ +/* + * Source file of a benchmark program involving calculations of the + * shortest distances between a source node and all other nodes in a + * graph of n nodes in which all nxn distances are defined as "double". + * The number n can be given via command line, and the default is 2000. + * The algorithm used is Dijsktra's. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman + * Copyright (C) 2020 Aleksandar Markovic + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + */ + +#include +#include +#include +#include +#include +#include + +/* Number of columns and rows in all matrixes*/ +#define DEFAULT_NODE_COUNT 2000 +#define MIN_NODE_COUNT 3 +#define MAX_NODE_COUNT 10000 + + +int32_t closest_index(int32_t count, double *distances, bool *flags) +{ + int32_t closest; + double minimum =3D DBL_MAX; + + for (size_t i =3D 0; i < count; i++) { + if (flags[i] =3D=3D false && distances[i] <=3D minimum) { + closest =3D i; + minimum =3D distances[i]; + } + } + + return closest; +} + +/** + * Calculate the shortest distances from the source node using Dijkstra me= thod. + * @param (out) distances An array of shortest distances from the source = node. + * @param (out) via An array of nodes needed to be taken as the the last + * before destination, for each destination. + * @param (out) eccent Eccentricity of the source node. + * @param (in) count The number of nodes. + * @param (in) source Source node. + * @param (in) matrix Distance matrix. + */ +void find_shortest_distances(double *distances, int32_t *via, double *ecce= nt, + int32_t count, int32_t source, double **matri= x) +{ + bool *flags; + + flags =3D (bool *)malloc(count * sizeof(bool)); + + for (size_t i =3D 0; i < count; i++) { + distances[i] =3D DBL_MAX; + flags[i] =3D false; + } + + distances[source] =3D 0.0; + via[source] =3D source; + + for (size_t i =3D 0; i < count - 1; i++) { + int32_t closest =3D closest_index(count, distances, flags); + flags[closest] =3D true; + for (size_t j =3D 0; j < count; j++) { + if ((!flags[j]) && + (matrix[closest][j]) && + (distances[closest] !=3D DBL_MAX) && + (distances[j] > distances[closest] + matrix[closest][j= ])) { + distances[j] =3D distances[closest] + matrix[closest][j]; + via[j] =3D closest; + } + } + } + + *eccent =3D 0; + for (size_t i =3D 0; i < count; i++) { + if (*eccent < distances[i]) { + *eccent =3D distances[i]; + } + } + + free(flags); +} + + +void main(int argc, char *argv[]) +{ + double **distance_matrix; + double *shortest_distances; + int32_t *via_node; + int32_t node_count =3D DEFAULT_NODE_COUNT; + int32_t source_node =3D 0; + double node_eccentricity =3D 0.0; + double range_factor =3D 999.0 / (double)(RAND_MAX); + int32_t option; + + /* Parse command line options */ + while ((option =3D getopt(argc, argv, "n:")) !=3D -1) { + if (option =3D=3D 'n') { + int32_t user_node_count =3D atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_node_count =3D=3D 0) { + fprintf(stderr, "Error ... Invalid value for option '-n'.\= n"); + exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_node_count < MIN_NODE_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannot be= a " + "number less than %d.\n", MIN_NODE_COUNT); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_node_count > MAX_NODE_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannot be= " + "more than %d.\n", MAX_NODE_COUNT); + exit(EXIT_FAILURE); + } + node_count =3D user_node_count; + } else { + exit(EXIT_FAILURE); + } + } + + /* Allocate the memory space for all matrixes */ + distance_matrix =3D (double **)malloc(node_count * sizeof(double *)); + for (size_t i =3D 0; i < node_count; i++) { + distance_matrix[i] =3D (double *)malloc(node_count * sizeof(double= )); + } + shortest_distances =3D (double *)malloc(node_count * sizeof(double)); + via_node =3D (int32_t *)malloc(node_count * sizeof(int32_t)); + + /* Initialize helper arrays and populate distance_matrix */ + srand(1); + for (size_t i =3D 0; i < node_count; i++) { + shortest_distances[i] =3D 0.0; + via_node[i] =3D -1; + distance_matrix[i][i] =3D 0.0; + } + for (size_t i =3D 0; i < node_count; i++) { + for (size_t j =3D i + 1; j < node_count; j++) { + distance_matrix[i][j] =3D 1.0 + range_factor * (double)rand(); + distance_matrix[j][i] =3D distance_matrix[i][j]; + } + } + + find_shortest_distances(shortest_distances, via_node, &node_eccentrici= ty, + node_count, source_node, distance_matrix); + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf(" Distance matrix (top left part):\n"); + for (size_t i =3D 0; i < 3; i++) { + for (size_t j =3D 0; j < 3; j++) { + printf(" %7.2f", distance_matrix[i][j]); + } + printf("\n"); + } + printf(" Source: %d (eccentricity: %f)\n", + source_node, node_eccentricity); + printf(" Destination Distance Via Node\n"); + for (size_t i =3D 0; i < 3; i++) { + printf(" %5d %7.2f %4d\n", + i, shortest_distances[i], via_node[i]); + } + + /* Free all previously allocated space */ + for (size_t i =3D 0; i < node_count; i++) { + free(distance_matrix[i]); + } + free(distance_matrix); + free(shortest_distances); + free(via_node); +} diff --git a/tests/performance/nightly-tests/benchmarks/source/dijkstra_int= 32/dijkstra_int32.c b/tests/performance/nightly-tests/benchmarks/source/dij= kstra_int32/dijkstra_int32.c new file mode 100644 index 0000000000..2663cde943 --- /dev/null +++ b/tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijk= stra_int32.c @@ -0,0 +1,192 @@ +/* + * Source file of a benchmark program involving calculations of the + * shortest distances between a source node and all other nodes in a + * graph of n nodes in which all nxn distances are defined as "int32". + * The number n can be given via command line, and the default is 2000. + * The algorithm used is Dijsktra's. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman + * Copyright (C) 2020 Aleksandar Markovic + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + */ + +#include +#include +#include +#include +#include + +/* Number of columns and rows in all matrixes*/ +#define DEFAULT_NODE_COUNT 2000 +#define MIN_NODE_COUNT 3 +#define MAX_NODE_COUNT 10000 + + +int32_t closest_index(int32_t count, int32_t *distances, bool *flags) +{ + int32_t closest; + int32_t minimum =3D INT_MAX; + + for (size_t i =3D 0; i < count; i++) { + if (flags[i] =3D=3D false && distances[i] <=3D minimum) { + closest =3D i; + minimum =3D distances[i]; + } + } + + return closest; +} + +/** + * Calculate the shortest distances from the source node using Dijkstra me= thod. + * @param (out) distances An array of shortest distances from the source = node. + * @param (out) via An array of nodes needed to be taken as the the last + * before destination, for each destination. + * @param (out) eccent Eccentricity of the source node. + * @param (in) count The number of nodes. + * @param (in) source Source node. + * @param (in) matrix Distance matrix. + */ +void find_shortest_distances(int32_t *distances, int32_t *via, int32_t *ec= cent, + int32_t count, int32_t source, int32_t **matr= ix) +{ + bool *flags; + + flags =3D (bool *)malloc(count * sizeof(bool)); + + for (size_t i =3D 0; i < count; i++) { + distances[i] =3D INT_MAX; + flags[i] =3D false; + } + + distances[source] =3D 0; + via[source] =3D source; + + for (size_t i =3D 0; i < count - 1; i++) { + int32_t closest =3D closest_index(count, distances, flags); + flags[closest] =3D true; + for (size_t j =3D 0; j < count; j++) { + if ((!flags[j]) && + (matrix[closest][j]) && + (distances[closest] !=3D INT_MAX) && + (distances[j] > distances[closest] + matrix[closest][j= ])) { + distances[j] =3D distances[closest] + matrix[closest][j]; + via[j] =3D closest; + } + } + } + + *eccent =3D 0; + for (size_t i =3D 0; i < count; i++) { + if (*eccent < distances[i]) { + *eccent =3D distances[i]; + } + } + + free(flags); +} + + +void main(int argc, char *argv[]) +{ + int32_t **distance_matrix; + int32_t *shortest_distances; + int32_t *via_node; + int32_t node_count =3D DEFAULT_NODE_COUNT; + int32_t source_node =3D 0; + int32_t node_eccentricity =3D 0; + int32_t option; + + /* Parse command line options */ + while ((option =3D getopt(argc, argv, "n:")) !=3D -1) { + if (option =3D=3D 'n') { + int32_t user_node_count =3D atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_node_count =3D=3D 0) { + fprintf(stderr, "Error ... Invalid value for option '-n'.\= n"); + exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_node_count < MIN_NODE_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannot be= a " + "number less than %d.\n", MIN_NODE_COUNT); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_node_count > MAX_NODE_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannot be= " + "more than %d.\n", MAX_NODE_COUNT); + exit(EXIT_FAILURE); + } + node_count =3D user_node_count; + } else { + exit(EXIT_FAILURE); + } + } + + /* Allocate the memory space for all matrixes */ + distance_matrix =3D (int32_t **)malloc(node_count * sizeof(int32_t *)); + for (size_t i =3D 0; i < node_count; i++) { + distance_matrix[i] =3D (int32_t *)malloc(node_count * sizeof(int32= _t)); + } + shortest_distances =3D (int32_t *)malloc(node_count * sizeof(int32_t)); + via_node =3D (int32_t *)malloc(node_count * sizeof(int32_t)); + + /* Initialize helper arrays and populate distance_matrix */ + srand(1); + for (size_t i =3D 0; i < node_count; i++) { + shortest_distances[i] =3D 0; + via_node[i] =3D -1; + distance_matrix[i][i] =3D 0; + } + for (size_t i =3D 0; i < node_count; i++) { + for (size_t j =3D i + 1; j < node_count; j++) { + distance_matrix[i][j] =3D 1 + (rand()) / (RAND_MAX / 999); + distance_matrix[j][i] =3D distance_matrix[i][j]; + } + } + + find_shortest_distances(shortest_distances, via_node, &node_eccentrici= ty, + node_count, source_node, distance_matrix); + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf(" Distance matrix (top left part):\n"); + for (size_t i =3D 0; i < 3; i++) { + for (size_t j =3D 0; j < 3; j++) { + printf(" %6d", distance_matrix[i][j]); + } + printf("\n"); + } + printf(" Source: %d (eccentricity: %d)\n", + source_node, node_eccentricity); + printf(" Destination Distance Via Node\n"); + for (size_t i =3D 0; i < 3; i++) { + printf(" %5d %3d %4d\n", + i, shortest_distances[i], via_node[i]); + } + + /* Free all previously allocated space */ + for (size_t i =3D 0; i < node_count; i++) { + free(distance_matrix[i]); + } + free(distance_matrix); + free(shortest_distances); + free(via_node); +} diff --git a/tests/performance/nightly-tests/benchmarks/source/matmult_doub= le/matmult_double.c b/tests/performance/nightly-tests/benchmarks/source/mat= mult_double/matmult_double.c new file mode 100644 index 0000000000..42bbb4717a --- /dev/null +++ b/tests/performance/nightly-tests/benchmarks/source/matmult_double/matm= ult_double.c @@ -0,0 +1,123 @@ +/* + * Source file of a benchmark program involving calculations of + * a product of two matrixes nxn whose elements are "double". The + * number n can be given via command line, and the default is 200. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman + * Copyright (C) 2020 Aleksandar Markovic + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + */ + +#include +#include +#include + +/* Number of columns and rows in all matrixes*/ +#define DEFAULT_MATRIX_SIZE 200 +#define MIN_MATRIX_SIZE 2 +#define MAX_MATRIX_SIZE 200000 + +void main(int argc, char *argv[]) +{ + double **matrix_a; + double **matrix_b; + double **matrix_res; + size_t i; + size_t j; + size_t k; + int32_t matrix_size =3D DEFAULT_MATRIX_SIZE; + int32_t option; + double range_factor =3D 100.0 / (double)(RAND_MAX); + + + /* Parse command line options */ + while ((option =3D getopt(argc, argv, "n:")) !=3D -1) { + if (option =3D=3D 'n') { + int32_t user_matrix_size =3D atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_matrix_size =3D=3D 0) { + fprintf(stderr, "Error ... Invalid value for option '-n'.\= n"); + exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_matrix_size < MIN_MATRIX_SIZE) { + fprintf(stderr, "Error ... Value for option '-n' cannot be= a " + "number less than %d.\n", MIN_MATRIX_SIZE); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_matrix_size > MAX_MATRIX_SIZE) { + fprintf(stderr, "Error ... Value for option '-n' cannot be= " + "more than %d.\n", MAX_MATRIX_SIZE); + exit(EXIT_FAILURE); + } + matrix_size =3D user_matrix_size; + } else { + exit(EXIT_FAILURE); + } + } + + /* Allocate the memory space for all matrixes */ + matrix_a =3D (double **)malloc(matrix_size * sizeof(double *)); + for (i =3D 0; i < matrix_size; i++) { + matrix_a[i] =3D (double *)malloc(matrix_size * sizeof(double)); + } + matrix_b =3D (double **)malloc(matrix_size * sizeof(double *)); + for (i =3D 0; i < matrix_size; i++) { + matrix_b[i] =3D (double *)malloc(matrix_size * sizeof(double)); + } + matrix_res =3D (double **)malloc(matrix_size * sizeof(double *)); + for (i =3D 0; i < matrix_size; i++) { + matrix_res[i] =3D (double *)malloc(matrix_size * sizeof(double)); + } + + /* Populate matrix_a and matrix_b with random numbers */ + srand(1); + for (i =3D 0; i < matrix_size; i++) { + for (j =3D 0; j < matrix_size; j++) { + matrix_a[i][j] =3D range_factor * (double)rand(); + matrix_b[i][j] =3D range_factor * (double)rand(); + } + } + + /* Calculate the product of two matrixes */ + for (i =3D 0; i < matrix_size; i++) { + for (j =3D 0; j < matrix_size; j++) { + matrix_res[i][j] =3D 0.0; + for (k =3D 0; k < matrix_size; k++) { + matrix_res[i][j] +=3D matrix_a[i][k] * matrix_b[k][j]; + } + } + } + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf(" %f %f\n", matrix_res[0][0], matrix_res[0][1]); + printf(" %f %f\n", matrix_res[1][0], matrix_res[1][1]); + + /* Free all previously allocated space */ + for (i =3D 0; i < matrix_size; i++) { + free(matrix_a[i]); + free(matrix_b[i]); + free(matrix_res[i]); + } + free(matrix_a); + free(matrix_b); + free(matrix_res); +} diff --git a/tests/performance/nightly-tests/benchmarks/source/matmult_int3= 2/matmult_int32.c b/tests/performance/nightly-tests/benchmarks/source/matmu= lt_int32/matmult_int32.c new file mode 100644 index 0000000000..29a6eb000d --- /dev/null +++ b/tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmu= lt_int32.c @@ -0,0 +1,121 @@ +/* + * Source file of a benchmark program involving calculations of + * a product of two matrixes nxn whose elements are "int32_t". The + * number n can be given via command line, and the default is 200. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman + * Copyright (C) 2020 Aleksandar Markovic + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + */ + +#include +#include +#include + +/* Number of columns and rows in all matrixes*/ +#define DEFAULT_MATRIX_SIZE 200 +#define MIN_MATRIX_SIZE 2 +#define MAX_MATRIX_SIZE 200000 + +void main(int argc, char *argv[]) +{ + int32_t **matrix_a; + int32_t **matrix_b; + int32_t **matrix_res; + size_t i; + size_t j; + size_t k; + int32_t matrix_size =3D DEFAULT_MATRIX_SIZE; + int32_t option; + + /* Parse command line options */ + while ((option =3D getopt(argc, argv, "n:")) !=3D -1) { + if (option =3D=3D 'n') { + int32_t user_matrix_size =3D atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_matrix_size =3D=3D 0) { + fprintf(stderr, "Error ... Invalid value for option '-n'.\= n"); + exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_matrix_size < MIN_MATRIX_SIZE) { + fprintf(stderr, "Error ... Value for option '-n' cannot be= a " + "number less than %d.\n", MIN_MATRIX_SIZE); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_matrix_size > MAX_MATRIX_SIZE) { + fprintf(stderr, "Error ... Value for option '-n' cannot be= " + "more than %d.\n", MAX_MATRIX_SIZE); + exit(EXIT_FAILURE); + } + matrix_size =3D user_matrix_size; + } else { + exit(EXIT_FAILURE); + } + } + + /* Allocate the memory space for all matrixes */ + matrix_a =3D (int32_t **)malloc(matrix_size * sizeof(int32_t *)); + for (i =3D 0; i < matrix_size; i++) { + matrix_a[i] =3D (int32_t *)malloc(matrix_size * sizeof(int32_t)); + } + matrix_b =3D (int32_t **)malloc(matrix_size * sizeof(int32_t *)); + for (i =3D 0; i < matrix_size; i++) { + matrix_b[i] =3D (int32_t *)malloc(matrix_size * sizeof(int32_t)); + } + matrix_res =3D (int32_t **)malloc(matrix_size * sizeof(int32_t *)); + for (i =3D 0; i < matrix_size; i++) { + matrix_res[i] =3D (int32_t *)malloc(matrix_size * sizeof(int32_t)); + } + + /* Populate matrix_a and matrix_b with random numbers */ + srand(1); + for (i =3D 0; i < matrix_size; i++) { + for (j =3D 0; j < matrix_size; j++) { + matrix_a[i][j] =3D (rand()) / (RAND_MAX / 100); + matrix_b[i][j] =3D (rand()) / (RAND_MAX / 100); + } + } + + /* Calculate the product of two matrixes */ + for (i =3D 0; i < matrix_size; i++) { + for (j =3D 0; j < matrix_size; j++) { + matrix_res[i][j] =3D 0; + for (k =3D 0; k < matrix_size; k++) { + matrix_res[i][j] +=3D matrix_a[i][k] * matrix_b[k][j]; + } + } + } + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf(" %d %d\n", matrix_res[0][0], matrix_res[0][1]); + printf(" %d %d\n", matrix_res[1][0], matrix_res[1][1]); + + /* Free all previously allocated space */ + for (i =3D 0; i < matrix_size; i++) { + free(matrix_a[i]); + free(matrix_b[i]); + free(matrix_res[i]); + } + free(matrix_a); + free(matrix_b); + free(matrix_res); +} diff --git a/tests/performance/nightly-tests/benchmarks/source/qsort_double= /qsort_double.c b/tests/performance/nightly-tests/benchmarks/source/qsort_d= ouble/qsort_double.c new file mode 100644 index 0000000000..efc1b2eee1 --- /dev/null +++ b/tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_= double.c @@ -0,0 +1,104 @@ +/* + * Source file of a benchmark program involving sorting of an array + * of length n whose elements are "double". The default value for n + * is 300000, and it can be set via command line as well. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman + * Copyright (C) 2020 Aleksandar Markovic + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + */ + +#include +#include +#include + +/* Number of elements in the array to be sorted */ +#define DEFAULT_ARRAY_LEN 300000 +#define MIN_ARRAY_LEN 3 +#define MAX_ARRAY_LEN 30000000 + +/* Upper limit used for generation of random numbers */ +#define UPPER_LIMIT 1000.0 + +/* Comparison function passed to qsort() */ +static int compare(const void *a, const void *b) +{ + if (*(const double *)a > *(const double *)b) { + return 1; + } else if (*(const double *)a < *(const double *)b) { + return -1; + } + return 0; +} + +void main(int argc, char *argv[]) +{ + double *array_to_be_sorted; + int32_t array_len =3D DEFAULT_ARRAY_LEN; + int32_t option; + double range_factor =3D UPPER_LIMIT / (double)(RAND_MAX); + + /* Parse command line options */ + while ((option =3D getopt(argc, argv, "n:")) !=3D -1) { + if (option =3D=3D 'n') { + int32_t user_array_len =3D atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_array_len =3D=3D 0) { + fprintf(stderr, "Error ... Invalid value for option '-n'.\= n"); + exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_array_len < MIN_ARRAY_LEN) { + fprintf(stderr, "Error ... Value for option '-n' cannot be= a " + "number less than %d.\n", MIN_ARRAY_LEN); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_array_len > MAX_ARRAY_LEN) { + fprintf(stderr, "Error ... Value for option '-n' cannot be= " + "more than %d.\n", MAX_ARRAY_LEN); + exit(EXIT_FAILURE); + } + array_len =3D user_array_len; + } else { + exit(EXIT_FAILURE); + } + } + + /* Allocate the memory space for the array */ + array_to_be_sorted =3D (double *) malloc(array_len * sizeof(double)); + + /* Populate the_array with random numbers */ + srand(1); + for (size_t i =3D 0; i < array_len; i++) { + array_to_be_sorted[i] =3D range_factor * (double)rand(); + } + + /* Sort the_array using qsort() */ + qsort(array_to_be_sorted, array_len, sizeof(array_to_be_sorted[0]), + compare); + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf("%14.10f %14.10f %14.10f\n", + array_to_be_sorted[0], array_to_be_sorted[1], array_to_be_sorte= d[2]); + + /* Free all previously allocated space */ + free(array_to_be_sorted); +} diff --git a/tests/performance/nightly-tests/benchmarks/source/qsort_int32/= qsort_int32.c b/tests/performance/nightly-tests/benchmarks/source/qsort_int= 32/qsort_int32.c new file mode 100644 index 0000000000..76ca9c3490 --- /dev/null +++ b/tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_i= nt32.c @@ -0,0 +1,103 @@ +/* + * Source file of a benchmark program involving sorting of an array + * of length n whose elements are "int32_t". The default value for n + * is 300000, and it can be set via command line as well. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman + * Copyright (C) 2020 Aleksandar Markovic + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + */ + +#include +#include +#include + +/* Number of elements in the array to be sorted */ +#define DEFAULT_ARRAY_LEN 300000 +#define MIN_ARRAY_LEN 3 +#define MAX_ARRAY_LEN 30000000 + +/* Upper limit used for generation of random numbers */ +#define UPPER_LIMIT 50000000 + +/* Comparison function passed to qsort() */ +static int compare(const void *a, const void *b) +{ + if (*(const int32_t *)a > *(const int32_t *)b) { + return 1; + } else if (*(const int32_t *)a < *(const int32_t *)b) { + return -1; + } + return 0; +} + +void main(int argc, char *argv[]) +{ + int32_t *array_to_be_sorted; + int32_t array_len =3D DEFAULT_ARRAY_LEN; + int32_t option; + + /* Parse command line options */ + while ((option =3D getopt(argc, argv, "n:")) !=3D -1) { + if (option =3D=3D 'n') { + int32_t user_array_len =3D atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_array_len =3D=3D 0) { + fprintf(stderr, "Error ... Invalid value for option '-n'.\= n"); + exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_array_len < MIN_ARRAY_LEN) { + fprintf(stderr, "Error ... Value for option '-n' cannot be= a " + "number less than %d.\n", MIN_ARRAY_LEN); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_array_len > MAX_ARRAY_LEN) { + fprintf(stderr, "Error ... Value for option '-n' cannot be= " + "more than %d.\n", MAX_ARRAY_LEN); + exit(EXIT_FAILURE); + } + array_len =3D user_array_len; + } else { + exit(EXIT_FAILURE); + } + } + + /* Allocate the memory space for the array */ + array_to_be_sorted =3D (int32_t *) malloc(array_len * sizeof(int32_t)); + + /* Populate the_array with random numbers */ + srand(1); + for (size_t i =3D 0; i < array_len; i++) { + array_to_be_sorted[i] =3D (rand()) / (RAND_MAX / UPPER_LIMIT); + } + + /* Sort the_array using qsort() */ + qsort(array_to_be_sorted, array_len, sizeof(array_to_be_sorted[0]), + compare); + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf("%d %d %d\n", + array_to_be_sorted[0], array_to_be_sorted[1], array_to_be_sorte= d[2]); + + /* Free all previously allocated space */ + free(array_to_be_sorted); +} diff --git a/tests/performance/nightly-tests/benchmarks/source/qsort_string= /qsort_string.c b/tests/performance/nightly-tests/benchmarks/source/qsort_s= tring/qsort_string.c new file mode 100644 index 0000000000..7d582b2dd0 --- /dev/null +++ b/tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_= string.c @@ -0,0 +1,122 @@ +/* + * Source file of a benchmark program involving sorting of an array + * of 10000 random strings of length 8 (including terminating zero). + * That sorting is repeated a number of times (default is 20 times), + * and each time a different array of random strings is generated. + * The number of repetitions can be set via command line. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman + * Copyright (C) 2020 Aleksandar Markovic + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + */ + +#include +#include +#include +#include + +/* Length of an individual random string (including terminating zero) */ +#define RANDOM_STRING_LEN 8 +/* Number of elements of the array of random strings */ +#define NUMBER_OF_RANDOM_STRINGS 10000 + +/* Number of repetitions to be performed each with different input */ +#define DEFAULT_REPETITION_COUNT 20 +#define MIN_REPETITION_COUNT 1 +#define MAX_REPETITION_COUNT 1000 + +/* Structure that keeps an array of random strings to be sorted */ +struct StringStruct { + char chars[RANDOM_STRING_LEN]; +}; + +/* Comparison function passed to qsort() */ +int compare_strings(const void *element1, const void *element2) +{ + int result; + + result =3D strcmp((*((struct StringStruct *)element1)).chars, + (*((struct StringStruct *)element2)).chars); + + return (result < 0) ? -1 : ((result =3D=3D 0) ? 0 : 1); +} + +/* Generate a random string of given length and containing only small lett= ers */ +static void gen_random_string(char *s, const int len) +{ + static const char letters[] =3D "abcdefghijklmnopqrstuvwxyz"; + + for (size_t i =3D 0; i < (len - 1); i++) { + s[i] =3D letters[rand() % (sizeof(letters) - 1)]; + } + + s[len - 1] =3D 0; +} + +void main(int argc, char *argv[]) +{ + struct StringStruct strings_to_be_sorted[NUMBER_OF_RANDOM_STRINGS]; + int32_t repetition_count =3D DEFAULT_REPETITION_COUNT; + int32_t option; + + /* Parse command line options */ + while ((option =3D getopt(argc, argv, "n:")) !=3D -1) { + if (option =3D=3D 'n') { + int32_t user_repetition_count =3D atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_repetition_count =3D=3D 0) { + fprintf(stderr, "Error ... Invalid value for option '-n'.\= n"); + exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_repetition_count < MIN_REPETITION_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannot be= a " + "number less than %d.\n", MIN_REPETITION_C= OUNT); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_repetition_count > MAX_REPETITION_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannot be= " + "more than %d.\n", MAX_REPETITION_COUNT); + exit(EXIT_FAILURE); + } + repetition_count =3D user_repetition_count; + } else { + exit(EXIT_FAILURE); + } + } + + srand(1); + + for (size_t i =3D 0; i < repetition_count; ++i) { + /* Generate random strings, and, in turn, sort them */ + for (size_t i =3D 0; i < NUMBER_OF_RANDOM_STRINGS; ++i) { + gen_random_string(strings_to_be_sorted[i].chars, RANDOM_STRING= _LEN); + } + qsort(strings_to_be_sorted, NUMBER_OF_RANDOM_STRINGS, + sizeof(struct StringStruct), compare_strings); + } + + /* Control printing */ + printf("CONTROL RESULT:\n"); + for (size_t i =3D 0; i < 2; ++i) { + printf(" %s", strings_to_be_sorted[i].chars); + } + printf("\n"); +} diff --git a/tests/performance/nightly-tests/benchmarks/source/search_strin= g/search_string.c b/tests/performance/nightly-tests/benchmarks/source/searc= h_string/search_string.c new file mode 100644 index 0000000000..2827ea032e --- /dev/null +++ b/tests/performance/nightly-tests/benchmarks/source/search_string/searc= h_string.c @@ -0,0 +1,110 @@ +/* + * Source file of a benchmark program that searches for the occurrence + * of a small string in a much larger random string ("needle in a hay"). + * That searching is repeated a number of times (default is 20 times), + * and each time a different large random string ("hay") is generated. + * The number of repetitions can be set via command line. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman + * Copyright (C) 2020 Aleksandar Markovic + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + */ + +#include +#include +#include +#include + +/* Length of a long string to be searched (including terminating zero) */ +#define HAYSTACK_LEN 30000 + +/* Number of repetitions to be performed each with different input */ +#define DEFAULT_REPETITION_COUNT 100 +#define MIN_REPETITION_COUNT 1 +#define MAX_REPETITION_COUNT 10000 + + +/* Generate a random string of given length and containing only small lett= ers */ +static void gen_random_string(char *s, const int len) +{ + static const char letters[] =3D "abcdefghijklmnopqrstuvwxyz"; + + for (size_t i =3D 0; i < (len - 1); i++) { + s[i] =3D letters[rand() % (sizeof(letters) - 1)]; + } + + s[len - 1] =3D 0; +} + +void main(int argc, char *argv[]) +{ + char haystack[HAYSTACK_LEN]; + const char needle[] =3D "aaa "; + char *found_needle; + int32_t found_cnt =3D 0; + int32_t not_found_cnt =3D 0; + int32_t repetition_count =3D DEFAULT_REPETITION_COUNT; + int32_t option; + + printf("needle is %s, size %d\n", needle, sizeof(needle)); + + /* Parse command line options */ + while ((option =3D getopt(argc, argv, "n:")) !=3D -1) { + if (option =3D=3D 'n') { + int32_t user_repetition_count =3D atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_repetition_count =3D=3D 0) { + fprintf(stderr, "Error ... Invalid value for option '-n'.\= n"); + exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_repetition_count < MIN_REPETITION_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannot be= a " + "number less than %d.\n", MIN_REPETITION_C= OUNT); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_repetition_count > MAX_REPETITION_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannot be= " + "more than %d.\n", MAX_REPETITION_COUNT); + exit(EXIT_FAILURE); + } + repetition_count =3D user_repetition_count; + } else { + exit(EXIT_FAILURE); + } + } + + srand(1); + + for (size_t i =3D 0; i < repetition_count; ++i) { + /* Generate random hay, and, in turn, find a needle */ + gen_random_string(haystack, HAYSTACK_LEN); + found_needle =3D strstr(haystack, needle); + if (found_needle !=3D NULL) { + found_cnt++; + } else { + not_found_cnt++; + } + } + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf(" Found %d times. Not found %d times.\n", found_cnt, not_found_= cnt); +} diff --git a/tests/performance/nightly-tests/scripts/nightly_tests_core.py = b/tests/performance/nightly-tests/scripts/nightly_tests_core.py new file mode 100755 index 0000000000..da192c704a --- /dev/null +++ b/tests/performance/nightly-tests/scripts/nightly_tests_core.py @@ -0,0 +1,920 @@ +#!/usr/bin/env python3 + +""" +Core script for performing nightly performance tests on QEMU. + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman +Copyright (C) 2020 Aleksandar Markovic + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" + +import argparse +import csv +import datetime +import glob +import multiprocessing +import os +import pathlib +import shutil +import subprocess +import sys +import tempfile +import time +from typing import Dict, List, Optional, Union + + +def get_benchmark_name(benchmark_path: str) -> str: + """ + Return the benchmark name given its path. + + Parameters: + benchmarks_path (str): Absolute path to benchmark + + Return: + (str): Benchmark name + """ + benchmark_source_file =3D os.path.split(benchmark_path)[1] + return os.path.splitext(benchmark_source_file)[0] + + +def get_benchmark_parent_dir(benchmark_path: str) -> str: + """ + Return the benchmark parent directory name given the benchmark path. + + Parameters: + benchmarks_path (str): Absolute path to benchmark + + Return: + (str): Benchmark parent directory name + """ + benchmark_parent_dir_path =3D os.path.split(benchmark_path)[0] + benchmark_parent_dir =3D os.path.split(benchmark_parent_dir_path)[1] + + return benchmark_parent_dir + + +def get_executable_parent_dir_path( + benchmark_path: str, benchmarks_executables_dir_path: str) -> str: + """ + Return the executables parent directory of a benchmark given its path. + This is the directory that includes all compiled executables for the + benchmark. + + Parameters: + benchmarks_path (str): Absolute path to benchmark + benchmarks_executables_dir_path (str): Absolute path to the executables + + Return: + (str): Executables parent directory path + """ + benchmark_parent_dir_path =3D os.path.split(benchmark_path)[0] + benchmark_parent_dir =3D os.path.split(benchmark_parent_dir_path)[1] + executable_parent_dir_path =3D os.path.join(benchmarks_executables_dir= _path, + benchmark_parent_dir) + + return executable_parent_dir_path + + +def get_commit_hash(commit_tag: str, qemu_path: str) -> str: + """ + Find commit hash given the Git commit tag. + + Parameters: + commit_tag (str): Commit tag + qemu_path (str): Absolute path to QEMU + + Returns: + (str): 8 digit commit hash + """ + + commit_hash =3D subprocess.run(["git", + "rev-parse", + commit_tag], + cwd=3Dqemu_path, + stdout=3Dsubprocess.PIPE, + check=3DFalse) + if commit_hash.returncode: + clean_exit(qemu_path, + "Failed to find the commit hash of {}.".format(commit_t= ag)) + + return commit_hash.stdout.decode("utf-8")[:8] + + +def git_checkout(commit: str, qemu_path: str) -> None: + """ + Checkout a given Git commit. + Also pull the latest changes from origin/master if the commit is "mast= er". + + Parameters: + commit (str): Commit hash or tag + qemu_path (str): Absolute path to QEMU + """ + print(datetime.datetime.utcnow().isoformat(), + "- Checking out {}".format(commit), file=3Dsys.stderr, flush=3DT= rue) + + checkout_commit =3D subprocess.run(["git", + "checkout", + commit], + cwd=3Dqemu_path, + stdout=3Dsubprocess.DEVNULL, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if checkout_commit.returncode: + clean_exit(qemu_path, checkout_commit.stderr.decode("utf-8")) + + if commit =3D=3D "master": + print(datetime.datetime.utcnow().isoformat(), + "- Pulling the latest changes from QEMU master", + file=3Dsys.stderr, flush=3DTrue) + # Try pulling the latest changes. + # Limit the number of failed trials to 10. + failure_count, failure_limit =3D 0, 10 + while True: + pull_latest =3D subprocess.run(["git", + "pull", + "origin", + "master"], + cwd=3Dqemu_path, + stdout=3Dsubprocess.DEVNULL, + check=3DFalse) + if pull_latest.returncode: + failure_count +=3D 1 + if failure_count =3D=3D failure_limit: + print(datetime.datetime.utcnow().isoformat(), + "- Trial {}/{}: Failed to pull QEMU".format( + failure_count, failure_limit), + file=3Dsys.stderr, flush=3DTrue) + clean_exit(qemu_path, "") + else: + print(datetime.datetime.utcnow().isoformat(), + "- Trial {}/{}: Failed to pull QEMU" + " ... retrying again in a minute!".format( + failure_count, failure_limit), + file=3Dsys.stderr, flush=3DTrue) + time.sleep(60) + else: + break + + +def git_clone(qemu_path: str) -> None: + """ + Clone QEMU from Git. + + Parameters: + qemu_path (str): Absolute path to clone the QEMU repo to + """ + # Try cloning QEMU. + # Limit the number of failed trials to 10. + failure_count, failure_limit =3D 0, 10 + while True: + clone_qemu =3D subprocess.run(["git", + "clone", + "https://git.qemu.org/git/qemu.git", + qemu_path], + check=3DFalse) + if clone_qemu.returncode: + failure_count +=3D 1 + if failure_count =3D=3D failure_limit: + print(datetime.datetime.utcnow().isoformat(), + "- Trial {}/{}: Failed to clone QEMU".format( + failure_count, failure_limit), + file=3Dsys.stderr, flush=3DTrue) + clean_exit(qemu_path, "") + else: + print(datetime.datetime.utcnow().isoformat(), + "- Trial {}/{}: Failed to clone QEMU" + " ... retrying again in a minute!".format( + failure_count, failure_limit), + file=3Dsys.stderr, flush=3DTrue) + time.sleep(60) + else: + break + + +def build_qemu(qemu_path: str, git_tag: str, targets: List[str]) -> None: + """ + Checkout the Git tag then configure and build QEMU. + + Parameters: + qemu_path (str): Absolute path to QEMU + git_tag (str): Git tag to checkout before building + targets (List[str]): List of targets to configure QEMU for + """ + + # Clean the QEMU build path + qemu_build_path =3D os.path.join(qemu_path, "build-gcc") + if os.path.isdir(qemu_build_path): + shutil.rmtree(qemu_build_path) + os.mkdir(qemu_build_path) + + git_checkout(git_tag, qemu_path) + + # Specify target list for configuring QEMU + target_list =3D ["{}-linux-user".format(target) for target in targets] + + # Configure QEMU + print(datetime.datetime.utcnow().isoformat(), + "- Running 'configure' for {}".format(git_tag), + file=3Dsys.stderr, flush=3DTrue) + configure =3D subprocess.run(["../configure", + "--disable-system", + "--disable-tools", + "--target-list=3D{}". + format(",".join(target_list))], + cwd=3Dqemu_build_path, + stdout=3Dsubprocess.DEVNULL, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if configure.returncode: + clean_exit(qemu_path, configure.stderr.decode("utf-8")) + + # Run "make -j$(nproc)" + print(datetime.datetime.utcnow().isoformat(), + "- Running 'make' for {}".format(git_tag), file=3Dsys.stderr, + flush=3DTrue) + make =3D subprocess.run(["make", + "-j", + str(multiprocessing.cpu_count())], + cwd=3Dqemu_build_path, + stdout=3Dsubprocess.DEVNULL, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if make.returncode: + clean_exit(qemu_path, make.stderr.decode("utf-8")) + + +def compile_target(benchmark_path: str, compiled_benchmark_path: str, + target_compiler: str) -> None: + """ + Compile a benchmark using the provided cross compiler. + + Parameters: + benchmarks_path (str): Absolute path to benchmark + compiled_benchmark_path (str): Path to the output executable + target_compiler (str): Cross compiler + """ + compile_benchmark =3D subprocess.run([target_compiler, + "-O2", + "-static", + "-w", + benchmark_path, + "-o", + compiled_benchmark_path], + check=3DFalse) + if compile_benchmark.returncode: + sys.exit("Compilation of {} failed".format( + os.path.split(compiled_benchmark_path)[1])) + + +def measure_instructions( + benchmark_path: str, benchmarks_executables_dir_path: str, + qemu_path: str, targets: List[str]) -> List[List[Union[str, int]]]: + """ + Measure the number of instructions when running an program with QEMU. + + Parameters: + benchmarks_path (str): Absolute path to benchmark + benchmarks_executables_dir_path (str): Absolute path to the executables + qemu_path (str): Absolute path to QEMU + targets (List[str]): List of QEMU targets + + Returns: + (List[List[Union[str, int]]]): [[target_name, instructions],[...],...] + """ + + benchmark_name =3D get_benchmark_name(benchmark_path) + executable_parent_dir_path =3D get_executable_parent_dir_path( + benchmark_path, benchmarks_executables_dir_path) + qemu_build_path =3D os.path.join(qemu_path, "build-gcc") + + instructions: List[List[Union[str, int]]] =3D [] + + for target in targets: + executable_path =3D os.path.join( + executable_parent_dir_path, "{}-{}".format(benchmark_name, tar= get)) + + qemu_exe_path =3D os.path.join(qemu_build_path, + "{}-linux-user".format(target), + "qemu-{}".format(target)) + + with tempfile.NamedTemporaryFile() as tmpfile: + run_callgrind =3D subprocess.run(["valgrind", + "--tool=3Dcallgrind", + "--callgrind-out-file=3D{}".fo= rmat( + tmpfile.name), + qemu_exe_path, + executable_path], + stdout=3Dsubprocess.DEVNULL, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if run_callgrind.returncode =3D=3D 1: + clean_exit(qemu_path, run_callgrind.stderr.decode("utf-8")) + + callgrind_output =3D run_callgrind.stderr.decode("utf-8").split("\= n") + instructions.append([target, int(callgrind_output[8].split(" ")[-1= ])]) + + return instructions + + +def measure_master_instructions( + reference_version_path: str, reference_commit_hash: str, + latest_version_path: str, benchmark_path: str, + benchmarks_executables_dir_path: str, qemu_path: str, + targets: List[str]) -> List[List[Union[str, int]]]: + """ + Measure the latest QEMU "master" instructions and also append the late= st + instructions and reference version instructions to the instructions li= st. + + Parameters: + reference_version_path (str): Absolute path to reference version resul= ts + reference_commit_hash (str): Git hash of the reference version + latest_version_path (str): Absolute path to the latest version results + benchmark_path (str): Absolute path to benchmark + benchmarks_executables_dir_path (str): + Absolute path to the executables of the benchm= ark + qemu_path (str): Absolute path to QEMU + targets (List[str]): List of QEMU targets + + + Return: + (List[List[Union[str, int]]]): + [[target_name, instructions, comparison_instructions],[...],..= .] + comparsion_instructions: *[latest, reference] + If latest is not available, + then comparsion_instructions =3D reference + """ + benchmark_name =3D get_benchmark_name(benchmark_path) + + print(datetime.datetime.utcnow().isoformat(), + "- Measuring instructions for master - {}".format(benchmark_name= ), + file=3Dsys.stderr, flush=3DTrue) + + instructions =3D measure_instructions( + benchmark_path, benchmarks_executables_dir_path, qemu_path, target= s) + + reference_result =3D "{}-{}-results.csv".format( + reference_commit_hash, benchmark_name) + reference_result_path =3D os.path.join( + reference_version_path, reference_result) + + # Find if this benchmark has a record in the latest results + latest_result =3D "" + latest_results =3D os.listdir(latest_version_path) + for result in latest_results: + if result.split("-")[1] =3D=3D benchmark_name: + latest_result =3D result + break + + # Append instructions from latest version if available + if latest_result !=3D "": + latest_result_path =3D os.path.join(latest_version_path, latest_re= sult) + with open(latest_result_path, "r") as file: + file.readline() + for target_instructions in instructions: + target_instructions.append( + int(file.readline().split('"')[1].replace(",", ""))) + # Delete the latest results. The directory will contain the new la= test + # when the new "master" results are stored later. + os.unlink(latest_result_path) + + # Append instructions from reference version + with open(reference_result_path, "r") as file: + file.readline() + for target_instructions in instructions: + target_instructions.append( + int(file.readline().split('"')[1].replace(",", ""))) + + return instructions + + +def calculate_percentage(old_instructions: int, new_instructions: int) -> = str: + """ + Calculate the change in percentage between two instruction counts + + Parameters: + old_instructions (int): Old number + new_instructions (int): New number + + Return: + (str): [+|-][change][%] or "-----" in case of 0.01% change + """ + percentage =3D round(((new_instructions - old_instructions) / + old_instructions) * 100, 3) + return format_percentage(percentage) + + +def format_percentage(percentage: float) -> str: + """ + Format the percentage value to add +|- and %. + + Parameters: + percentage (float): Percentage + + Returns: + (str): Formatted percentage string + """ + if abs(percentage) <=3D 0.01: + return "-----" + return "+" + str(percentage) + "%" if percentage > 0 \ + else str(percentage) + "%" + + +def calculate_change(instructions: List[List[Union[str, int]]]) -> None: + """ + Calculate the change in the recorded instructions for master compared = to + latest results and reference version results. + + Parameters: + instructions (List[List[Union[str, int]]]): + [[target_name, instructions, comparison_instructions],[...],..= .] + comparsion_instructions: *[latest, reference] + If latest is not available, + then comparsion_instructions =3D reference + """ + for target_instructions in instructions: + target_instructions[-1] =3D calculate_percentage( + int(target_instructions[-1]), int(target_instructions[1])) + # If latest instructions exists + if len(target_instructions) =3D=3D 4: + target_instructions[-2] =3D calculate_percentage( + int(target_instructions[-2]), int(target_instructions[1])) + + +def calculate_average(results: List[List[List[Union[str, int]]]], + targets: List[str], + num_benchmarks: int) -> List[List[Union[str, int]]]: + """ + Calculate the average results for each target for all benchmarks. + + Parameters: + results (List[List[List[Union[str, int]]]]): + [[target_name, instructions, comparison_instructions],[...],..= .] + comparsion_instructions: *[latest, reference] + If latest is not available, + then comparsion_instructions =3D reference + targets (List[str]): List of target names + num_benchmarks (int): Number of benchmarks + + Return: + (List[List[Union[str, int]]]): + [[target_name, average_instructions, \ + comparison_instructions],[...],...] + comparsion_instructions: *[average_latest, average_reference] + If latest is not available, + then comparsion_instructions =3D reference + """ + average_instructions: List[List[Union[str, int]]] =3D [] + + for i, target in enumerate(targets): + average_instructions.append([target]) + + total_instructions =3D 0 + total_latest_percentages: Optional[float] =3D 0.0 + total_reference_percentages =3D 0.0 + + for instructions in results: + total_instructions +=3D int(instructions[i][1]) + if instructions[i][3] !=3D "-----": + total_reference_percentages +=3D float( + str(instructions[i][3])[:-1]) + if total_latest_percentages is not None: + if instructions[i][2] !=3D "N/A": + if instructions[i][2] !=3D "-----": + total_latest_percentages +=3D float( + str(instructions[i][2])[:-1]) + else: + total_latest_percentages =3D None + + avg_instructions =3D total_instructions // num_benchmarks + avg_reference_percentages =3D format_percentage( + round(total_reference_percentages / num_benchmarks, 3)) + avg_latest_percentages =3D format_percentage( + round(total_latest_percentages / num_benchmarks, 3)) \ + if total_latest_percentages is not None else "N/A" + + average_instructions[-1].extend([avg_instructions, + avg_latest_percentages, + avg_reference_percentages]) + + return average_instructions + + +def write_to_csv(instructions: List[List[Union[str, int]]], + output_csv_path: str, percentages: bool =3D False, + reference_version: str =3D "") -> None: + """ + Write the [Target, Instructions] for each target in a CSV file. + comparison_instructions are ignored. + + Parameters: + instructions (List[List[Union[str, int]]]): + [[target_name, instructions, comparison_instructions],[...],..= .] + comparsion_instructions: *[latest, reference] + If latest is not available, + then comparsion_instructions =3D reference + output_csv_path (str): Absolute path to output CSV file + percentages (bool): Add percentages to the output CSV file + """ + with open(output_csv_path, "w") as file: + writer =3D csv.writer(file) + header =3D ["Target", "Instructions"] + if percentages: + header.extend(["Latest", reference_version]) + writer.writerow(header) + for target_instructions in instructions: + row =3D [] + row.extend([target_instructions[0], format( + target_instructions[1], ",")]) + if percentages: + row.extend(target_instructions[2:]) + writer.writerow(row) + + +def print_table(instructions: str, text: str, reference_version: str) -> N= one: + """ + Print the results in a tabular form + + Parameters: + instructions (List[List[Union[str, int]]]): + [[target_name, instructions, comparison_instructions],[...],..= .] + comparsion_instructions: *[latest, reference] + If latest is not available, + then comparsion_instructions =3D reference + text (str): Text be added to the table header + reference_version (str): Reference version used in these results + """ + print("{}\n{}\n{}". + format("-" * 56, text, "-" * 56)) + + print('{:<10} {:>20} {:>10} {:>10}\n{} {} {} {}'. + format('Target', + 'Instructions', + 'Latest', + reference_version, + '-' * 10, + '-' * 20, + '-' * 10, + '-' * 10)) + + for target_change in instructions: + # Replace commas with spaces in instruction count + # for easier readability. + formatted_instructions =3D format( + target_change[1], ",").replace(",", " ") + print('{:<10} {:>20} {:>10} {:>10}'.format( + target_change[0], formatted_instructions, *target_change[2:])) + + print("-" * 56) + + +def clean_exit(qemu_path: str, error_message: str) -> None: + """ + Clean up intermediate files and exit. + + Parameters: + qemu_path (str): Absolute path to QEMU + error_message (str): Error message to display after exiting + """ + # Clean the QEMU build path + qemu_build_path =3D os.path.join(qemu_path, "build-gcc") + if os.path.isdir(qemu_build_path): + shutil.rmtree(qemu_build_path) + sys.exit(error_message) + + +def verify_executables(benchmark_paths: List[str], targets: Dict[str, str], + benchmarks: List[Dict[str, str]], + benchmarks_executables_dir_path: str) -> None: + """ + Verify that all executables exist for each benchmark. + + Parameters: + benchmark_paths (List[str]): List of all paths to benchmarks + targets (Dict[str, str]): Dictionary the contains for each target, + target_name: target_compiler + benchmarks (List[Dict[str, str]]): Benchmarks data (name, parent_dir, = path) + benchmarks_executables_dir_path (str): Absolute path to the executable= s dir + """ + print(datetime.datetime.utcnow().isoformat(), + "- Verifying executables of {} benchmarks for {} targets". + format(len(benchmark_paths), len(targets)), + file=3Dsys.stderr, flush=3DTrue) + + for benchmark in benchmarks: + executable_parent_dir_path =3D get_executable_parent_dir_path( + benchmark["path"], benchmarks_executables_dir_path) + + # Verify that the exists for this benchmark executables, if not, + # create it + if not os.path.isdir(executable_parent_dir_path): + os.mkdir(executable_parent_dir_path) + + for target_name, target_compiler in targets.items(): + compiled_benchmark =3D "{}-{}".format( + benchmark["name"], target_name) + compiled_benchmark_path =3D os.path.join( + executable_parent_dir_path, compiled_benchmark) + # Verify that the the executable for this target is available, + # if not, compile it + if not os.path.isfile(compiled_benchmark_path): + compile_target(benchmark["path"], + compiled_benchmark_path, + target_compiler) + + +def verify_reference_results(reference_version: str, qemu_path: str, + benchmarks: List[Dict[str, str]], + reference_version_results_dir_path: str, + targets: List[str], + benchmarks_executables_dir_path: str) -> None: + """ + Verify that results are available for reference version. + If results are missing, build QEMU for the reference version then perf= orm + the measurements. + + Paramters: + reference_version (str): Reference QEMU version + qemu_path (str): Absolute path to QEMU + benchmark_paths (List[str]): List of all paths to benchmarks + reference_version_results_dir_path (str): Absolute path to the referen= ce + version results dir + targets (List[str]): Target names + benchmarks (List[Dict[str, str]]): Benchmarks data (name, parent_dir, = path) + benchmarks_executables_dir_path (str): Path to the root executables dir + """ + print(datetime.datetime.utcnow().isoformat(), + "- Verifying results of reference version {}". + format(reference_version), file=3Dsys.stderr, flush=3DTrue) + + # Set flag to know if QEMU was built for reference version before + did_build_reference =3D False + + latest_commit_hash =3D get_commit_hash(reference_version, qemu_path) + + for benchmark in benchmarks: + benchmark_results_dir_path =3D os.path.join( + reference_version_results_dir_path, benchmark["parent_dir"]) + + # Verify that the results directory for the benchmark exists, if n= ot, + # create it + if not os.path.isdir(benchmark_results_dir_path): + os.mkdir(benchmark_results_dir_path) + + # Verify that the the results.csv file for the benchmark exits, if= not, + # create it + results_path =3D os.path.join(benchmark_results_dir_path, + "{}-{}-results.csv". + format(latest_commit_hash, + benchmark["name"])) + if not os.path.isfile(results_path): + # Only build qemu if reference version wasn't built before + if not did_build_reference: + build_qemu(qemu_path, reference_version, targets) + did_build_reference =3D True + print(datetime.datetime.utcnow().isoformat(), + "- Measuring instructions for reference version {} - {}". + format(reference_version, benchmark["name"]), + file=3Dsys.stderr, flush=3DTrue) + instructions =3D measure_instructions( + benchmark["path"], + benchmarks_executables_dir_path, + qemu_path, + targets) + write_to_csv(instructions, results_path) + + +def verify_requirements() -> None: + """ + Verify that all script requirements are installed (valgrind & git). + """ + # Insure that valgrind is installed + check_valgrind_installation =3D subprocess.run(["which", "valgrind"], + stdout=3Dsubprocess.DEVNU= LL, + check=3DFalse) + if check_valgrind_installation.returncode: + sys.exit("Please install valgrind before running the script.") + + # Insure that git is installed + check_git_installation =3D subprocess.run(["which", "git"], + stdout=3Dsubprocess.DEVNULL, + check=3DFalse) + if check_git_installation.returncode: + sys.exit("Please install git before running the script.") + + +def main(): + """ + Parse the command line arguments then start the execution. + Output on STDOUT represents the nightly test results. + Output on STDERR represents the execution log and errors if any. + Output on STDERR must be redirected to either /dev/null or to a log fi= le. + + Syntax: + nightly_tests_core.py [-h] [-r REF] + Optional arguments: + -h, --help Show this help message and exit + -r REF, --reference REF + Reference QEMU version - Default is v5.1.0 + Example of usage: + nightly_tests_core.py -r v5.1.0 2>log.txt + """ + parser =3D argparse.ArgumentParser() + parser.add_argument("-r", "--reference", dest=3D"ref", + default=3D"v5.1.0", + help=3D"Reference QEMU version - Default is v5.1.0= ") + reference_version =3D parser.parse_args().ref + + targets =3D { + "aarch64": "aarch64-linux-gnu-gcc", + "alpha": "alpha-linux-gnu-gcc", + "arm": "arm-linux-gnueabi-gcc", + "hppa": "hppa-linux-gnu-gcc", + "m68k": "m68k-linux-gnu-gcc", + "mips": "mips-linux-gnu-gcc", + "mipsel": "mipsel-linux-gnu-gcc", + "mips64": "mips64-linux-gnuabi64-gcc", + "mips64el": "mips64el-linux-gnuabi64-gcc", + "ppc": "powerpc-linux-gnu-gcc", + "ppc64": "powerpc64-linux-gnu-gcc", + "ppc64le": "powerpc64le-linux-gnu-gcc", + "riscv64": "riscv64-linux-gnu-gcc", + "s390x": "s390x-linux-gnu-gcc", + "sh4": "sh4-linux-gnu-gcc", + "sparc64": "sparc64-linux-gnu-gcc", + "x86_64": "gcc" + } + + # Verify that the script requirements are installed + verify_requirements() + + # Get required paths + nightly_tests_dir_path =3D pathlib.Path(__file__).parent.parent.absolu= te() + benchmarks_dir_path =3D os.path.join(nightly_tests_dir_path, "benchmar= ks") + benchmarks_source_dir_path =3D os.path.join(benchmarks_dir_path, "sour= ce") + benchmarks_executables_dir_path =3D os.path.join( + benchmarks_dir_path, "executables") + + # Verify that If the executables directory exists, if not, create it + if not os.path.isdir(benchmarks_executables_dir_path): + os.mkdir(benchmarks_executables_dir_path) + + # Get absolute path to all available benchmarks + benchmark_paths =3D sorted([y for x in os.walk(benchmarks_source_dir_p= ath) + for y in glob.glob(os.path.join(x[0], '*.c')= )]) + + benchmarks =3D [{ + "name": get_benchmark_name(benchmark_path), + "parent_dir": get_benchmark_parent_dir(benchmark_path), + "path": benchmark_path} for benchmark_path in benchmark_paths] + + # Verify that all executables exist for each benchmark + verify_executables(benchmark_paths, targets, benchmarks, + benchmarks_executables_dir_path) + + # Set QEMU path and clone from Git if the path doesn't exist + qemu_path =3D os.path.join(nightly_tests_dir_path, "qemu-nightly") + if not os.path.isdir(qemu_path): + # Clone QEMU into the temporary directory + print(datetime.datetime.utcnow().isoformat(), + "- Fetching QEMU: ", end=3D"", flush=3DTrue, file=3Dsys.stde= rr) + git_clone(qemu_path) + print("\n", file=3Dsys.stderr, flush=3DTrue) + + # Verify that the results directory exists, if not, create it + results_dir_path =3D os.path.join(nightly_tests_dir_path, "results") + if not os.path.isdir(results_dir_path): + os.mkdir(results_dir_path) + + # Verify that the reference version results directory exists, if not, + # create it + reference_version_results_dir_path =3D os.path.join( + results_dir_path, reference_version) + if not os.path.isdir(reference_version_results_dir_path): + os.mkdir(reference_version_results_dir_path) + + # Verify that previous results are available for reference version + verify_reference_results(reference_version, qemu_path, benchmarks, + reference_version_results_dir_path, + targets.keys(), benchmarks_executables_dir_pa= th) + + # Compare results with the latest QEMU master + # --------------------------------------------------------------------= ----- + # Verify that the "latest" results directory exists, if not, + # create it + latest_version_results_dir_path =3D os.path.join(results_dir_path, "la= test") + if not os.path.isdir(latest_version_results_dir_path): + os.mkdir(latest_version_results_dir_path) + + # Verify that the "history" results directory exists, if not, create it + history_results_dir_path =3D os.path.join(results_dir_path, "history") + if not os.path.isdir(history_results_dir_path): + os.mkdir(history_results_dir_path) + + # Build QEMU for master + build_qemu(qemu_path, "master", targets.keys()) + + # Get the commit hash for the top commit at master + master_commit_hash =3D get_commit_hash("master", qemu_path) + + # Print report summary header + print("{}\n{}".format("-" * 56, "{} SUMMARY REPORT - COMMIT {}". + format(" "*11, master_commit_hash))) + + # For each benchmark, compare the current master results with + # latest and reference version + results =3D [] + for benchmark in benchmarks: + reference_version_benchmark_results_dir_path =3D os.path.join( + reference_version_results_dir_path, benchmark["parent_dir"]) + latest_version_benchmark_results_dir_path =3D os.path.join( + latest_version_results_dir_path, benchmark["parent_dir"]) + history_benchmark_results_dir_path =3D os.path.join( + history_results_dir_path, benchmark["parent_dir"]) + + # Verify that the the benchmark directory exists in the "latest" + # directory, if not, create it + if not os.path.isdir(latest_version_benchmark_results_dir_path): + os.mkdir(latest_version_benchmark_results_dir_path) + + # Verify that the the benchmark directory exists in the "history" + # directory, if not, create it + if not os.path.isdir(history_benchmark_results_dir_path): + os.mkdir(history_benchmark_results_dir_path) + + # Obtain the instructions array which will contain for each target, + # the target name, the number of "master" instructions, + # "latest" instructions if available, + # and "reference version" instructions + instructions =3D measure_master_instructions( + reference_version_benchmark_results_dir_path, + get_commit_hash(reference_version, qemu_path), + latest_version_benchmark_results_dir_path, + benchmark["path"], + benchmarks_executables_dir_path, + qemu_path, + targets.keys()) + + # Update the "latest" directory with the new results form master + updated_latest_version_benchmark_results =3D os.path.join( + latest_version_benchmark_results_dir_path, + "{}-{}-results.csv".format(master_commit_hash, benchmark["name= "])) + write_to_csv(instructions, updated_latest_version_benchmark_result= s) + + calculate_change(instructions) + + # Insert "N/A" for targets that don't have "latest" instructions + for target_instructions in instructions: + if len(target_instructions) =3D=3D 3: + target_instructions.insert(2, "N/A") + + history_benchmark_results =3D os.path.join( + history_benchmark_results_dir_path, + "{}-{}-results.csv".format(master_commit_hash, benchmark["name= "])) + write_to_csv(instructions, history_benchmark_results, + True, reference_version) + + # Store the results + results.append([benchmark["name"], instructions]) + + # Calculate the average instructions for each target + # Only send the instructions as results without the benchmark names + average_instructions =3D calculate_average( + [result[1] for result in results], targets.keys(), len(benchmarks)) + + # Save average results to results/history directory + average_results =3D os.path.join( + history_results_dir_path, + "{}-average-results.csv".format(master_commit_hash)) + write_to_csv(average_instructions, average_results, + True, reference_version) + + # Print results + print_table(average_instructions, " "*20 + + "AVERAGE RESULTS", reference_version) + print("\n", " "*17, "DETAILED RESULTS") + for [benchmark_name, instructions] in results: + print_table(instructions, + "Test Program: " + benchmark_name, + reference_version) + + # Cleanup (delete the build directory) + shutil.rmtree(os.path.join(qemu_path, "build-gcc")) + # --------------------------------------------------------------------= ----- + + +if __name__ =3D=3D "__main__": + main() diff --git a/tests/performance/nightly-tests/scripts/run_nightly_tests.py b= /tests/performance/nightly-tests/scripts/run_nightly_tests.py new file mode 100755 index 0000000000..ea65be60dc --- /dev/null +++ b/tests/performance/nightly-tests/scripts/run_nightly_tests.py @@ -0,0 +1,135 @@ +#!/usr/bin/env python3 + +""" +Entry point script for running nightly performance tests on QEMU. + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman +Copyright (C) 2020 Aleksandar Markovic + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" + + +import datetime +import io +import os +import subprocess +import time +import sys +from send_email import send_email + + +# Record system hardware information +with open('/proc/cpuinfo', 'r') as cpuinfo: + for line in cpuinfo: + if line.startswith('model name'): + HOST_CPU =3D line.rstrip('\n').split(':')[1].strip() + break +with open('/proc/meminfo', 'r') as meminfo: + for line in meminfo: + if line.startswith('MemTotal'): + HOST_MEMORY_KB =3D int(line.rstrip('\n').split(':')[1]. + strip().split(' ')[0]) + HOST_MEMORY =3D str(round(HOST_MEMORY_KB / (1024 * 1024), 2)) = + " GB" + break + +# Find path for the "nightly_tests_core.py" script +NIGHTLY_TESTS_CORE_PATH =3D os.path.join( + os.path.dirname(os.path.abspath(__file__)), + "nightly_tests_core.py") + +NIGHTLY_TESTS_ARGUMENTS =3D sys.argv[1:] + +# Start the nightly test +START_EPOCH =3D time.time() +RUN_NIGHTLY_TESTS =3D subprocess.run([NIGHTLY_TESTS_CORE_PATH, + *NIGHTLY_TESTS_ARGUMENTS], + stdout=3Dsubprocess.PIPE, + stderr=3Dsubprocess.PIPE, + check=3DFalse) +END_EPOCH =3D time.time() + +# Perform time calculations +EXECUTION_TIME =3D datetime.timedelta(seconds=3DEND_EPOCH - START_EPOCH) +TEST_DATE =3D datetime.datetime.utcfromtimestamp( + START_EPOCH).strftime('%A, %B %-d, %Y') +START_TIME =3D datetime.datetime.utcfromtimestamp( + START_EPOCH).strftime('%Y-%m-%d %H:%M:%S') +END_TIME =3D datetime.datetime.utcfromtimestamp( + END_EPOCH).strftime('%Y-%m-%d %H:%M:%S') + +# Get nightly test status +if RUN_NIGHTLY_TESTS.returncode: + STATUS =3D "FAILURE" +else: + STATUS =3D "SUCCESS" + +# Initialize a StringIO to print all the output into +OUTPUT =3D io.StringIO() + +# Print the nightly test statistical information +print("{:<17}: {}\n{:<17}: {}\n". + format("Host CPU", + HOST_CPU, + "Host Memory", + HOST_MEMORY), file=3DOUTPUT) +print("{:<17}: {}\n{:<17}: {}\n{:<17}: {}\n". + format("Start Time (UTC)", + START_TIME, + "End Time (UTC)", + END_TIME, + "Execution Time", + EXECUTION_TIME), file=3DOUTPUT) +print("{:<17}: {}\n".format("Status", STATUS), file=3DOUTPUT) + +if STATUS =3D=3D "SUCCESS": + print("Note:\nChanges denoted by '-----' are less than 0.01%.\n", + file=3DOUTPUT) + +# Print the nightly test stdout (main output) +print(RUN_NIGHTLY_TESTS.stdout.decode("utf-8"), file=3DOUTPUT) + +# If the nightly test failed, print the stderr (error logs) +if STATUS =3D=3D "FAILURE": + print("{}\n{}\n{}".format("-" * 56, + " " * 18 + "ERROR LOGS", + "-" * 56), file=3DOUTPUT) + print(RUN_NIGHTLY_TESTS.stderr.decode("utf-8"), file=3DOUTPUT) + + +# Temp file to store the output in case sending the email failed +# with open("temp.txt", "w") as file: +# file.write(OUTPUT.getvalue()) + +# Use an HTML message to preserve monospace formatting +HTML_MESSAGE =3D """\ +
+{body}
+
+""".format(body=3DOUTPUT.getvalue()) +OUTPUT.close() + +# Send the nightly test results email to the QEMU mailing list +while True: + try: + send_email("[REPORT] Nightly Performance Tests - {}".format(TEST_D= ATE), + ["qemu-devel@nongnu.org"], HTML_MESSAGE) + except Exception: # pylint: disable=3DW0703 + # Wait for a minute then retry sending + time.sleep(60) + continue + else: + break diff --git a/tests/performance/nightly-tests/scripts/send_email.py b/tests/= performance/nightly-tests/scripts/send_email.py new file mode 100644 index 0000000000..e24e244f51 --- /dev/null +++ b/tests/performance/nightly-tests/scripts/send_email.py @@ -0,0 +1,56 @@ +""" +Helper script for sending emails with the nightly performance test results. + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman +Copyright (C) 2020 Aleksandar Markovic + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" + + +import smtplib +from typing import List +from email.mime.multipart import MIMEMultipart +from email.mime.text import MIMEText +from email.utils import COMMASPACE, formatdate + + +GMAIL_USER =3D {"name": "", + "email": "", + "pass": ""} + + +def send_email(subject: str, send_to: List[str], html: str) -> None: + """ + Send an HTML email. + + Parameters: + subject (str): Email subject + send_to (List(str)): List of recipients + html (str): HTML message + """ + msg =3D MIMEMultipart('alternative') + msg['From'] =3D "{} <{}>".format(GMAIL_USER["name"], GMAIL_USER["email= "]) + msg['To'] =3D COMMASPACE.join(send_to) + msg['Date'] =3D formatdate(localtime=3DTrue) + msg['Subject'] =3D subject + + msg.attach(MIMEText(html, 'html')) + + server =3D smtplib.SMTP_SSL('smtp.gmail.com', 465) + server.login(GMAIL_USER["email"], GMAIL_USER["pass"]) + server.sendmail(msg['From'], send_to, msg.as_string()) + server.quit() --=20 2.17.1 From nobody Sun Nov 16 04:05:48 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1598611643; cv=none; d=zohomail.com; s=zohoarc; b=EKpvu2TEDy+a7S3yYbSLzRXtGSiT2h4Cq14GgHWEiwdmA9/z0rAl0OtJF6wiz4WyffjI8KuRq5dwA0fmwYxdFwjvHW/xXNYAlmm1C+gxYOz0OXOiKFdEhFuyHP6o+kx8NrhwAf9mLM2dktSVzyYnWOlu6y7vayRrJ8oWkjKFgfk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1598611643; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=ocs60WDznaDScA/gvXj53ZTFJX4+fFayK+ZVchaf5to=; b=IX+PlGaj1XyHM7QsfR9lhbBqTYlxtPLawKcSc8cFa1sPMRyyeONM7S+NG/u+gGdKBSGvBoRh6ADgLdYGttTCRj9xbyNrgIHvQ5RPWf1HcVPAPAi9lRP5if/dQf9oolCRyPs4go+oyQaJJRVAZpQrnJUHNhNsHhZCxYiczpRdhWU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1598611643548376.67421109720556; Fri, 28 Aug 2020 03:47:23 -0700 (PDT) Received: from localhost ([::1]:43516 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kBbuk-0002mR-Ca for importer@patchew.org; Fri, 28 Aug 2020 06:47:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34534) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kBbpQ-0003fM-Ah for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:52 -0400 Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]:45773) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kBbpO-0005jD-C2 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:51 -0400 Received: by mail-wr1-x442.google.com with SMTP id h15so831313wrt.12 for ; Fri, 28 Aug 2020 03:41:49 -0700 (PDT) Received: from localhost.localdomain ([197.58.77.158]) by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:41:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ocs60WDznaDScA/gvXj53ZTFJX4+fFayK+ZVchaf5to=; b=sFkgZ2boQgWhq4eXz6d09Ypkx8IqsyVoGLyWlG08+NeYR0k2QrTXXwqqVN0smG09Sb o89f2Vs0Upy7/Gm/+odj5QYCMICXmj2m8tTkvLXpIORj8cmv/QRO/oor7daXMgMiZw56 JJZZS4tH9NQNzGc4AA6GtJhkVWwwKexihmyv512GrORDoe6+r/rf8EwwDjhEyziNpvh7 L0rX+mp0WG+zTMyhuC+X8WWoGEwb8Ua6SqwwVz/GrndNXBQqmuweQgeeLcF2p45HQcL1 Jq5Ra5zk7uAabEvpMUSrCOQzWqN2ryd+2SoIRJRS3h30PI5qdJ4FIdLMgNVSlmEiw23W V0fA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ocs60WDznaDScA/gvXj53ZTFJX4+fFayK+ZVchaf5to=; b=UB2qNRDP9qKTfrFddQHDOrYSim/mODDD1PeGAxY5aHZhCZYV0HePEYX/8LHbKtXyUl +Fq0ENfkHVicElI7PJyVHtwJxI+mEXWlG2p0flchXNxBrZXJazd+1HW3sZHQ5LHCM70F 1LATI/Dl/kEgAzHzgX5Ez+Ki3xL/43A0NsPxmZdySNHMdDX4lHmagGDUC1puBTlTDu7A yh54kCT3xnb0IxXOqqjdHAH/H+9R4rCF00QrbnWdpxTNRm+J4KuVVs4PBkZIIU8KGLwp KCaT9//7YImOngue6Lcm16QiMTxuzuHLJus148gsa/AtKPdCiqn0tfgfMu0QyfZPnzwi Vyag== X-Gm-Message-State: AOAM530uDkDY6fCqjdBtQoJuiL4dPemHSEZ9qIwHjmye6JQUfiXn9xXI iF9Z9QeR+cMQx0JC75vO/XY0VY+WW493Xg== X-Google-Smtp-Source: ABdhPJzC+ez05Uou+atjjrhL2u5gcFrd8SM0IZ3ReNieo/RoGnrU8Gt8HLHuFj2S1Aw6BCQL6c88KA== X-Received: by 2002:a5d:674c:: with SMTP id l12mr890586wrw.325.1598611308896; Fri, 28 Aug 2020 03:41:48 -0700 (PDT) From: Ahmed Karaman To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com, alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com, jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com Subject: [PATCH 8/9] MAINTAINERS: Add 'tests/performance' to 'Performance Tools and Tests' subsection Date: Fri, 28 Aug 2020 12:41:01 +0200 Message-Id: <20200828104102.4490-9-ahmedkhaledkaraman@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::442; envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x442.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ahmed Karaman Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @gmail.com) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add a new 'tests/performance' directory under the 'Performance Tools and Tests' subsection. Signed-off-by: Ahmed Karaman Reviewed-by: Aleksandar Markovic --- MAINTAINERS | 1 + 1 file changed, 1 insertion(+) diff --git a/MAINTAINERS b/MAINTAINERS index 5a22c8be42..8923307642 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3124,3 +3124,4 @@ Performance Tools and Tests M: Ahmed Karaman S: Maintained F: scripts/performance/ +F: tests/performance/ --=20 2.17.1 From nobody Sun Nov 16 04:05:48 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1598611470; cv=none; d=zohomail.com; s=zohoarc; b=mQvGEIF5gXEGu3Vz6Gs8thZrYXFdqhc6u82+oypnSlEM3rdkmPH6HEvYezTZ/QPyqQ1lOFSSiDlnVTqYCaQ6SFdMae6AbojmzG0+zHL5ytYWAIBH3gt8AhaJBVvOmFfBHrFIhTmU5XUwX6BUIO3Dc/t+lI0TY65rUylHi1VsQ78= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1598611470; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=NKjrR/1jJBl3PMLAgxGFOGP+xi3KNmymEVWR9lFRmRk=; b=gsADlp3UpDfPtOgZ6DZIlunhe2NrtrC0id8oGGxQgugxAPHj6kksXnbH8TkWn1rtSs3/RlRPQkD77mCBAQNj7lscs/WpfKim3HcEXNEgziT7GfIPT/15Cwazt8NoM6RHQ42b5LNMOKQXO+Uhtyo57Ab+xXK5zrP/Jj1RXQBrHy0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 159861147064022.175306563296886; Fri, 28 Aug 2020 03:44:30 -0700 (PDT) Received: from localhost ([::1]:36472 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kBbrx-0008Cb-AG for importer@patchew.org; Fri, 28 Aug 2020 06:44:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34582) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kBbpT-0003pL-V1 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:55 -0400 Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]:43367) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kBbpQ-0005kD-Jh for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:55 -0400 Received: by mail-wr1-x442.google.com with SMTP id k15so834510wrn.10 for ; Fri, 28 Aug 2020 03:41:52 -0700 (PDT) Received: from localhost.localdomain ([197.58.77.158]) by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:41:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NKjrR/1jJBl3PMLAgxGFOGP+xi3KNmymEVWR9lFRmRk=; b=b7Lp06PIVaYJOGD/iWKmSUt1QjjX41g5c9VeXHZhk3l7qeTyJropoduGwPmAMNJ0RC smQxmVX7ivvXTDaMNzWBiWCiIGajG5y1jonWPU/YKLQrwB20wBvHWUFYlM1nDdf6CFWQ zph5CvKoUrduoVOz2a4hk4BnRW2Is0uABwxVPA/rNsRGAeVQ3QlKQSO1N0z5Q15EDFVY M4YWJRHHYTQs+yVy1Ipyf+ObGU5d0UL0D4BVqw2I9t4jITTLakphE5OVbHZja7km9GsB /kHtzlJlRWExC9iPT7/DmO9A2oNZApQi2TB1Q/Olw8kK53bwhuf8LFVDdvnCoP39Q7P9 t51A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NKjrR/1jJBl3PMLAgxGFOGP+xi3KNmymEVWR9lFRmRk=; b=JFw0lDw6e2fU7+ukZcqxus70HcXSlGlRRSdkTjJQHQXESD1FkbQ6A73ZAx5vd9Ag3v oNRwYFDRje0Rrin+c3fRPoOf5N94x2lHfs2PV89kuX7hKlvtvNTLT1b64CVyJsB9wTBm BcH2po1J4gsFYsVO5pRqBOoSVnujgNHN84nWwzZ4hsbjDbbYjuEPvPwJfu8KxYRfLICE 7walVArnLs5iSWo74WBC1H0vvTWO0J6mt//f+gs1RiUdO6yMx3dvoRqrWxIjqVr6giZW EzqPPzxK2QXdAumDunKQ6f2x/qWshrJX5U6dZqjoLtzAB0/bSaTnxc4dS7mMf/3iEbXN tbXg== X-Gm-Message-State: AOAM530pseFF1o4KXG7up1h2MihWyrvstAz3IF76meNuFzRmtjluOPW7 968P62rRIM98xbrircy8sCNwydAr/BXfKg== X-Google-Smtp-Source: ABdhPJwsM0EBllpTnbeUyiaGJcaEEvoElZtbRzP6mYUFKdp3uqyWztCdJ+EosrbuJFcOdtYbGs7lKg== X-Received: by 2002:adf:828e:: with SMTP id 14mr984753wrc.217.1598611310859; Fri, 28 Aug 2020 03:41:50 -0700 (PDT) From: Ahmed Karaman To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com, alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com, jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com Subject: [PATCH 9/9] scripts/performance: Add topN_system.py script Date: Fri, 28 Aug 2020 12:41:02 +0200 Message-Id: <20200828104102.4490-10-ahmedkhaledkaraman@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::442; envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x442.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_PDS_OTHER_BAD_TLD=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ahmed Karaman Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @gmail.com) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Python script for listing the topN executed QEMU functions in system mode. Syntax: topN_system.py [-h] [-n ] -- \ [-h] - Print the script arguments help message. [-n] - Specify the number of top functions to print. - If this flag is not specified, the tool defaults to 25. Example of usage: topN_system.py -n 20 -- qemu-system-x86_64 -m 1024 -kernel \ -initrd Example output: Number of instructions: 150,991,381,071 No. Percentage Name --- ---------- ------------------------------ 1 11.30% helper_lookup_tb_ptr 2 7.01% liveness_pass_1 3 4.48% tcg_gen_code 4 3.41% tcg_optimize 5 1.84% tcg_out_opc.isra.13 6 1.78% helper_pcmpeqb_xmm 7 1.20% object_dynamic_cast_assert 8 1.00% cpu_exec 9 0.99% tcg_temp_new_internal 10 0.88% tb_htable_lookup 11 0.84% object_class_dynamic_cast_assert 12 0.81% init_ts_info 13 0.80% tlb_set_page_with_attrs 14 0.77% victim_tlb_hit 15 0.75% tcg_out_sib_offset 16 0.62% tcg_op_alloc 17 0.61% helper_pmovmskb_xmm 18 0.58% disas_insn.isra.50 19 0.56% helper_pcmpgtb_xmm 20 0.56% address_space_ldq Signed-off-by: Ahmed Karaman --- scripts/performance/topN_system.py | 158 +++++++++++++++++++++++++++++ 1 file changed, 158 insertions(+) create mode 100755 scripts/performance/topN_system.py diff --git a/scripts/performance/topN_system.py b/scripts/performance/topN_= system.py new file mode 100755 index 0000000000..9b1f1a66c7 --- /dev/null +++ b/scripts/performance/topN_system.py @@ -0,0 +1,158 @@ +#!/usr/bin/env python3 + +""" +Print the top N most executed functions in QEMU system mode emulation. + +Syntax: +topN_system.py [-h] [-n ] -- \ + + +[-h] - Print the script arguments help message. +[-n] - Specify the number of top functions to print. + - If this flag is not specified, the tool defaults to 25. + +Example of usage: +topN_system.py -n 20 -- qemu-system-x86_64 -m 1024 -kernel \ + -initrd + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman +Copyright (C) 2020 Aleksandar Markovic + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" + +import argparse +import os +import subprocess +import sys +import tempfile + + +# Parse the command line arguments +PARSER =3D argparse.ArgumentParser( + usage=3D"usage: topN_system.py [-h] [-n ]" + " -- ") + +PARSER.add_argument("-n", dest=3D"top", type=3Dint, default=3D25, + help=3D"Specify the number of top functions to print.") + +PARSER.add_argument("command", type=3Dstr, nargs=3D"+", help=3Dargparse.SU= PPRESS) + +ARGS =3D PARSER.parse_args() + +# Extract the needed variables from the args +COMMAND =3D ARGS.command +TOP =3D ARGS.top + +# Ensure that perf is installed +CHECK_PERF_PRESENCE =3D subprocess.run(["which", "perf"], + stdout=3Dsubprocess.DEVNULL, + check=3DFalse) +if CHECK_PERF_PRESENCE.returncode: + sys.exit("Please install perf before running the script!") + +# Ensure user has previllage to run perf +CHECK_PERF_EXECUTABILITY =3D subprocess.run(["perf", "stat", "ls", "/"], + stdout=3Dsubprocess.DEVNULL, + stderr=3Dsubprocess.DEVNULL, + check=3DFalse) +if CHECK_PERF_EXECUTABILITY.returncode: + sys.exit(""" +Error: +You may not have permission to collect stats. +Consider tweaking /proc/sys/kernel/perf_event_paranoid, +which controls use of the performance events system by +unprivileged users (without CAP_SYS_ADMIN). + -1: Allow use of (almost) all events by all users + Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK + 0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN + Disallow raw tracepoint access by users without CAP_SYS_ADMIN + 1: Disallow CPU event access by users without CAP_SYS_ADMIN + 2: Disallow kernel profiling by users without CAP_SYS_ADMIN +To make this setting permanent, edit /etc/sysctl.conf too, e.g.: + kernel.perf_event_paranoid =3D -1 + +* Alternatively, you can run this script under sudo privileges. +""") + +# Run perf and save all intermediate files in a temporary directory +with tempfile.TemporaryDirectory() as tmpdir: + RECORD_PATH =3D os.path.join(tmpdir, "record.data") + REPORT_PATH =3D os.path.join(tmpdir, "report.txt") + + PERF_RECORD =3D subprocess.run((["perf", + "record", + "-e", + "instructions", + "--output=3D"+RECORD_PATH] + + COMMAND), + stdout=3Dsubprocess.DEVNULL, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if PERF_RECORD.returncode: + sys.exit(PERF_RECORD.stderr.decode("utf-8")) + + with open(REPORT_PATH, "w") as output: + PERF_REPORT =3D subprocess.run(["perf", + "report", + "--input=3D" + RECORD_PATH, + "--stdio"], + stdout=3Doutput, + stderr=3Dsubprocess.PIPE, + check=3DFalse) + if PERF_REPORT.returncode: + sys.exit(PERF_REPORT.stderr.decode("utf-8")) + + # Save the reported data to FUNCTIONS[] and INSTRUCTIONS + with open(REPORT_PATH, "r") as data: + LINES =3D data.readlines() + # Read the number of instructions + INSTRUCTIONS =3D int(LINES[6].split()[-1]) + # Continue reading: + # Only read lines that are not empty + # Only read lines that are not comments (comments start with #) + # Only read functions executed by qemu-system + FUNCTIONS =3D [line for line in LINES if line + and line[0] !=3D "\n" + and line[0] !=3D "#" + and "qemu-system" in line.split()[2]] + + +# Limit the number of top functions to "TOP" +NO_TOP_FUNCTIONS =3D TOP if len(FUNCTIONS) > TOP else len(FUNCTIONS) + +# Store the data of the top functions in TOP_FUNCTIONS[] +TOP_FUNCTIONS =3D FUNCTIONS[:NO_TOP_FUNCTIONS] + +# Print total instructions +print("\nNumber of instructions:", format(INSTRUCTIONS, ",")) +# Print table header +print("\n{:>4} {:>10} {}\n{} {} {}".format("No.", + "Percentage", + "Name", + "-" * 4, + "-" * 10, + "-" * 30)) + +# Print top N functions +for (index, function) in enumerate(TOP_FUNCTIONS, start=3D1): + function_data =3D function.split() + function_percentage =3D function_data[0] + function_name =3D function_data[-1] + function_invoker =3D " ".join(function_data[2:-2]) + print("{:>4} {:>10} {}".format(index, + function_percentage, + function_name)) --=20 2.17.1