From nobody Fri Sep 5 22:32:30 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1603707834; cv=none; d=zohomail.com; s=zohoarc; b=FBYRPziqmfY/Rv/HRQ/9cXB/QWC05kjYocfLclMs42Q9rUczfFBxGtxElRwjWKCQTo7Y2O7atJUq5GYGSgsB433MAjsZbNgcEMsRQZrWwwiyzC6L9Z//wYF6miVgL/rxnXvZM4qy389UWqitSWaOALhaKBSEfh4DFW9KySihe4E= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1603707834; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=+nM0kYcQajA1fWMWAAseVtGUdQwwcwGeyDGj1H0tRms=; b=l3KKKcev1M99y0fKtcgVxo+Nv5D5Vwsk3cgVrZGaoPqg9PU8NI8oUMrt7q5GYO5qGAKpH99tOagP/yMzT4xeDVDnwk1egXqb4PY/9x3lsyMSvaCRGbse2y97x6qjl/uvzPQUvUoyYJhWWvcq63nJd7mFS2SyKJ+MYhcPgqjRBjY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1603707834732217.95792858816867; Mon, 26 Oct 2020 03:23:54 -0700 (PDT) Received: from localhost ([::1]:55520 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kWzfN-0000LI-Ls for importer@patchew.org; Mon, 26 Oct 2020 06:23:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38950) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kWzPH-0002lc-9Q for qemu-devel@nongnu.org; Mon, 26 Oct 2020 06:07:15 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:27022) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kWzPC-0006k2-DH for qemu-devel@nongnu.org; Mon, 26 Oct 2020 06:07:14 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-385-HH44Z7QFMj2iWGkANlxdnA-1; Mon, 26 Oct 2020 06:07:06 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2324110E2184; Mon, 26 Oct 2020 10:07:05 +0000 (UTC) Received: from thuth.com (ovpn-112-104.ams2.redhat.com [10.36.112.104]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0E9528B842; Mon, 26 Oct 2020 10:07:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1603706828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:content-type:content-type:in-reply-to:in-reply-to: references:references; bh=+nM0kYcQajA1fWMWAAseVtGUdQwwcwGeyDGj1H0tRms=; b=G1650yNwErlMwmZV4FWbPAOo6/uEJP5Vo6w+QgL2ko3mZrwxEvv8WxyfkMpAavXSLT8wME f8SNA2XacUKUwMjx1yhHW6AphwazCNXnWy2Z8JwnSU6E2CmqVlJ4u4fA2egkHzUpGryo1t Eoetr1SuhAHO95OrwSeRmv4FZrcKPTI= X-MC-Unique: HH44Z7QFMj2iWGkANlxdnA-1 From: Thomas Huth To: qemu-devel@nongnu.org, Peter Maydell Subject: [PULL 19/31] scripts/oss-fuzz: Add crash trace minimization script Date: Mon, 26 Oct 2020 11:06:20 +0100 Message-Id: <20201026100632.212530-20-thuth@redhat.com> In-Reply-To: <20201026100632.212530-1-thuth@redhat.com> References: <20201026100632.212530-1-thuth@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=thuth@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=thuth@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/26 02:39:09 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alexander Bulekov Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Alexander Bulekov Once we find a crash, we can convert it into a QTest trace. Usually this trace will contain many operations that are unneeded to reproduce the crash. This script tries to minimize the crashing trace, by removing operations and trimming QTest bufwrite(write addr len data...) commands. Signed-off-by: Alexander Bulekov Reviewed-by: Darren Kenny Message-Id: <20201023150746.107063-12-alxndr@bu.edu> Signed-off-by: Thomas Huth --- scripts/oss-fuzz/minimize_qtest_trace.py | 157 +++++++++++++++++++++++ 1 file changed, 157 insertions(+) create mode 100755 scripts/oss-fuzz/minimize_qtest_trace.py diff --git a/scripts/oss-fuzz/minimize_qtest_trace.py b/scripts/oss-fuzz/mi= nimize_qtest_trace.py new file mode 100755 index 0000000000..5e405a0d5f --- /dev/null +++ b/scripts/oss-fuzz/minimize_qtest_trace.py @@ -0,0 +1,157 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- + +""" +This takes a crashing qtest trace and tries to remove superflous operations +""" + +import sys +import os +import subprocess +import time +import struct + +QEMU_ARGS =3D None +QEMU_PATH =3D None +TIMEOUT =3D 5 +CRASH_TOKEN =3D None + +write_suffix_lookup =3D {"b": (1, "B"), + "w": (2, "H"), + "l": (4, "L"), + "q": (8, "Q")} + +def usage(): + sys.exit("""\ +Usage: QEMU_PATH=3D"/path/to/qemu" QEMU_ARGS=3D"args" {} input_trace outpu= t_trace +By default, will try to use the second-to-last line in the output to ident= ify +whether the crash occred. Optionally, manually set a string that idenitife= s the +crash by setting CRASH_TOKEN=3D +""".format((sys.argv[0]))) + +def check_if_trace_crashes(trace, path): + global CRASH_TOKEN + with open(path, "w") as tracefile: + tracefile.write("".join(trace)) + + rc =3D subprocess.Popen("timeout -s 9 {timeout}s {qemu_path} {qemu_arg= s} 2>&1\ + < {trace_path}".format(timeout=3DTIMEOUT, + qemu_path=3DQEMU_PATH, + qemu_args=3DQEMU_ARGS, + trace_path=3Dpath), + shell=3DTrue, + stdin=3Dsubprocess.PIPE, + stdout=3Dsubprocess.PIPE) + stdo =3D rc.communicate()[0] + output =3D stdo.decode('unicode_escape') + if rc.returncode =3D=3D 137: # Timed Out + return False + if len(output.splitlines()) < 2: + return False + + if CRASH_TOKEN is None: + CRASH_TOKEN =3D output.splitlines()[-2] + + return CRASH_TOKEN in output + + +def minimize_trace(inpath, outpath): + global TIMEOUT + with open(inpath) as f: + trace =3D f.readlines() + start =3D time.time() + if not check_if_trace_crashes(trace, outpath): + sys.exit("The input qtest trace didn't cause a crash...") + end =3D time.time() + print("Crashed in {} seconds".format(end-start)) + TIMEOUT =3D (end-start)*5 + print("Setting the timeout for {} seconds".format(TIMEOUT)) + print("Identifying Crashes by this string: {}".format(CRASH_TOKEN)) + + i =3D 0 + newtrace =3D trace[:] + # For each line + while i < len(newtrace): + # 1.) Try to remove it completely and reproduce the crash. If it w= orks, + # we're done. + prior =3D newtrace[i] + print("Trying to remove {}".format(newtrace[i])) + # Try to remove the line completely + newtrace[i] =3D "" + if check_if_trace_crashes(newtrace, outpath): + i +=3D 1 + continue + newtrace[i] =3D prior + + # 2.) Try to replace write{bwlq} commands with a write addr, len + # command. Since this can require swapping endianness, try both LE= and + # BE options. We do this, so we can "trim" the writes in (3) + if (newtrace[i].startswith("write") and not + newtrace[i].startswith("write ")): + suffix =3D newtrace[i].split()[0][-1] + assert(suffix in write_suffix_lookup) + addr =3D int(newtrace[i].split()[1], 16) + value =3D int(newtrace[i].split()[2], 16) + for endianness in ['<', '>']: + data =3D struct.pack("{end}{size}".format(end=3Dendianness, + size=3Dwrite_suffix_lookup[suffix][1]), + value) + newtrace[i] =3D "write {addr} {size} 0x{data}\n".format( + addr=3Dhex(addr), + size=3Dhex(write_suffix_lookup[suffix][0]), + data=3Ddata.hex()) + if(check_if_trace_crashes(newtrace, outpath)): + break + else: + newtrace[i] =3D prior + + # 3.) If it is a qtest write command: write addr len data, try to = split + # it into two separate write commands. If splitting the write down= the + # middle does not work, try to move the pivot "left" and retry, un= til + # there is no space left. The idea is to prune unneccessary bytes = from + # long writes, while accommodating arbitrary MemoryRegion access s= izes + # and alignments. + if newtrace[i].startswith("write "): + addr =3D int(newtrace[i].split()[1], 16) + length =3D int(newtrace[i].split()[2], 16) + data =3D newtrace[i].split()[3][2:] + if length > 1: + leftlength =3D int(length/2) + rightlength =3D length - leftlength + newtrace.insert(i+1, "") + while leftlength > 0: + newtrace[i] =3D "write {addr} {size} 0x{data}\n".forma= t( + addr=3Dhex(addr), + size=3Dhex(leftlength), + data=3Ddata[:leftlength*2]) + newtrace[i+1] =3D "write {addr} {size} 0x{data}\n".for= mat( + addr=3Dhex(addr+leftlength), + size=3Dhex(rightlength), + data=3Ddata[leftlength*2:]) + if check_if_trace_crashes(newtrace, outpath): + break + else: + leftlength -=3D 1 + rightlength +=3D 1 + if check_if_trace_crashes(newtrace, outpath): + i -=3D 1 + else: + newtrace[i] =3D prior + del newtrace[i+1] + i +=3D 1 + check_if_trace_crashes(newtrace, outpath) + + +if __name__ =3D=3D '__main__': + if len(sys.argv) < 3: + usage() + + QEMU_PATH =3D os.getenv("QEMU_PATH") + QEMU_ARGS =3D os.getenv("QEMU_ARGS") + if QEMU_PATH is None or QEMU_ARGS is None: + usage() + # if "accel" not in QEMU_ARGS: + # QEMU_ARGS +=3D " -accel qtest" + CRASH_TOKEN =3D os.getenv("CRASH_TOKEN") + QEMU_ARGS +=3D " -qtest stdio -monitor none -serial none " + minimize_trace(sys.argv[1], sys.argv[2]) --=20 2.18.2