From nobody Thu May 2 07:11:50 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1506467370999829.141354054964; Tue, 26 Sep 2017 16:09:30 -0700 (PDT) Received: from localhost ([::1]:51406 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwyyc-0000V3-MS for importer@patchew.org; Tue, 26 Sep 2017 19:09:18 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50810) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwywe-0007ii-9b for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwywc-0005ES-Ap for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:16 -0400 Received: from mx1.redhat.com ([209.132.183.28]:35834) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwywc-0005E0-4T for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:14 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 489704E34A for ; Tue, 26 Sep 2017 23:07:13 +0000 (UTC) Received: from localhost (ovpn-116-64.gru2.redhat.com [10.97.116.64]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9A62A784BC; Tue, 26 Sep 2017 23:07:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 489704E34A Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=ehabkost@redhat.com From: Eduardo Habkost To: qemu-devel@nongnu.org Date: Tue, 26 Sep 2017 20:07:04 -0300 Message-Id: <20170926230709.22958-2-ehabkost@redhat.com> In-Reply-To: <20170926230709.22958-1-ehabkost@redhat.com> References: <20170926230709.22958-1-ehabkost@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Tue, 26 Sep 2017 23:07:13 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC 1/6] device-crash-test: Make whitelist code a bit more flexible X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Huth , "Dr. David Alan Gilbert" , Cleber Rosa Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Allow test results without the 'log' and 'exitcode' keys. The new device_add code won't report exitcode/log because multiple devices will be tested on a single QEMU run. Signed-off-by: Eduardo Habkost --- scripts/device-crash-test | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/scripts/device-crash-test b/scripts/device-crash-test index ddc8f72eca..598720ffa9 100755 --- a/scripts/device-crash-test +++ b/scripts/device-crash-test @@ -284,9 +284,10 @@ def whitelistResultMatch(wl, r): """ assert whitelistTestCaseMatch(wl, r['testcase']) return ((wl.get('exitcode', 1) is None or - r['exitcode'] =3D=3D wl.get('exitcode', 1)) and + r.get('exitcode') =3D=3D wl.get('exitcode', 1)) and ('log' not in wl or - re.search(wl['log'], r['log'], re.MULTILINE))) + ('log' in r and + re.search(wl['log'], r['log'], re.MULTILINE)))) =20 =20 def checkResultWhitelist(r): --=20 2.13.5 From nobody Thu May 2 07:11:50 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1506467371039810.9468937983341; Tue, 26 Sep 2017 16:09:31 -0700 (PDT) Received: from localhost ([::1]:51407 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwyyc-0000VE-1V for importer@patchew.org; Tue, 26 Sep 2017 19:09:18 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50833) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwywf-0007it-90 for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:18 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwywe-0005Gh-7w for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:17 -0400 Received: from mx1.redhat.com ([209.132.183.28]:35970) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwywd-0005F7-Uf for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:16 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 19DA5C047B66 for ; Tue, 26 Sep 2017 23:07:15 +0000 (UTC) Received: from localhost (ovpn-116-64.gru2.redhat.com [10.97.116.64]) by smtp.corp.redhat.com (Postfix) with ESMTP id A488171CAD; Tue, 26 Sep 2017 23:07:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 19DA5C047B66 Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=ehabkost@redhat.com From: Eduardo Habkost To: qemu-devel@nongnu.org Date: Tue, 26 Sep 2017 20:07:05 -0300 Message-Id: <20170926230709.22958-3-ehabkost@redhat.com> In-Reply-To: <20170926230709.22958-1-ehabkost@redhat.com> References: <20170926230709.22958-1-ehabkost@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Tue, 26 Sep 2017 23:07:15 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC 2/6] device-crash-test: Log detailed info on success too X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Huth , "Dr. David Alan Gilbert" , Cleber Rosa Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Instead of making checkOneCase() return None on success, return detailed info even if no problems were found. This will make handling of device_add errors simpler, and improve debug output. As logFailure() won't handle only failures, rename it to logResult() and make it a bit more flexible about missing fields. Signed-off-by: Eduardo Habkost --- scripts/device-crash-test | 61 ++++++++++++++++++++++---------------------= ---- 1 file changed, 29 insertions(+), 32 deletions(-) diff --git a/scripts/device-crash-test b/scripts/device-crash-test index 598720ffa9..782d7fd6c2 100755 --- a/scripts/device-crash-test +++ b/scripts/device-crash-test @@ -234,6 +234,9 @@ ERROR_WHITELIST =3D [ {'exitcode':-11, 'device':'arm-gicv3', 'loglevel':logging.ERROR, 'expe= cted':True}, {'exitcode':-11, 'machine':'isapc', 'device':'.*-iommu', 'loglevel':lo= gging.ERROR, 'expected':True}, =20 + # 0 exit code means test succeeded, so we just log it as logging.DEBUG: + {'exitcode':0}, + # everything else (including SIGABRT and SIGSEGV) will be a fatal erro= r: {'exitcode':None, 'fatal':True, 'loglevel':logging.FATAL}, ] @@ -381,8 +384,7 @@ def getBinaryInfo(args, binary): def checkOneCase(args, testcase): """Check one specific case =20 - Returns a dictionary containing failure information on error, - or None on success + Returns a dictionary containing results of test case. """ binary =3D testcase['binary'] accel =3D testcase['accel'] @@ -397,26 +399,21 @@ def checkOneCase(args, testcase): dbg("will launch QEMU: %s", cmdline) vm =3D QEMUMachine(binary=3Dbinary, args=3Dargs) =20 - exc_traceback =3D None + r =3D {'testcase':testcase, + 'cmdline':cmdline} + try: vm.launch() except KeyboardInterrupt: raise except: - exc_traceback =3D traceback.format_exc() + r['exc_traceback'] =3D traceback.format_exc() dbg("Exception while running test case") - finally: - vm.shutdown() - ec =3D vm.exitcode() - log =3D vm.get_log() - - if exc_traceback is not None or ec !=3D 0: - return {'exc_traceback':exc_traceback, - 'exitcode':ec, - 'log':log, - 'testcase':testcase, - 'cmdline':cmdline} =20 + vm.shutdown() + r['exitcode'] =3D vm.exitcode() + r['log'] =3D vm.get_log() + return r =20 def binariesToTest(args, testcase): if args.qemu: @@ -490,17 +487,18 @@ def casesToTest(args, testcase): return cases =20 =20 -def logFailure(f, level): +def logResult(f, level): t =3D f['testcase'] - logger.log(level, "failed: %s", formatTestCase(t)) + logger.log(level, "result: %s", formatTestCase(t)) logger.log(level, "cmdline: %s", f['cmdline']) - for l in f['log'].strip().split('\n'): + for l in f.get('log', '').strip().split('\n'): logger.log(level, "log: %s", l) - logger.log(level, "exit code: %r", f['exitcode']) - if f['exc_traceback']: + logger.log(level, "exit code: %r", f.get('exitcode')) + if 'exc_traceback' in f: logger.log(level, "exception:") for l in f['exc_traceback'].split('\n'): logger.log(level, " %s", l.rstrip('\n')) + dbg('raw result dictionary: %r', f) =20 =20 def main(): @@ -579,18 +577,17 @@ def main(): except KeyboardInterrupt: break =20 - if f: - i, wl =3D checkResultWhitelist(f) - dbg("testcase: %r, whitelist match: %r", t, wl) - wl_stats.setdefault(i, []).append(f) - level =3D wl.get('loglevel', logging.DEBUG) - logFailure(f, level) - if wl.get('fatal') or (args.strict and level >=3D logging.WARN= ): - fatal_failures.append(f) - else: - dbg("success: %s", formatTestCase(t)) - if expected_match: - logger.warn("Didn't fail as expected: %s", formatTestCase(= t)) + i, wl =3D checkResultWhitelist(f) + dbg("testcase: %r, whitelist match: %r", t, wl) + wl_stats.setdefault(i, []).append(f) + level =3D wl.get('loglevel', logging.DEBUG) + logResult(f, level) + + if wl.get('fatal') or (args.strict and level >=3D logging.WARN): + fatal_failures.append(f) + + if expected_match and expected_match[0] !=3D i: + logger.warn("Didn't fail as expected: %s", formatTestCase(t)) =20 logger.info("Total: %d test cases", total) if skipped: --=20 2.13.5 From nobody Thu May 2 07:11:51 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1506467504216393.95637975298644; Tue, 26 Sep 2017 16:11:44 -0700 (PDT) Received: from localhost ([::1]:51418 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwz0s-0002qO-GL for importer@patchew.org; Tue, 26 Sep 2017 19:11:38 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50853) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwywg-0007jU-Kl for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwywf-0005KE-SF for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:18 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52880) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwywf-0005Ig-Lh for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:17 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D4F46883AE for ; Tue, 26 Sep 2017 23:07:16 +0000 (UTC) Received: from localhost (ovpn-116-64.gru2.redhat.com [10.97.116.64]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6D5EB702FA; Tue, 26 Sep 2017 23:07:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com D4F46883AE Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=ehabkost@redhat.com From: Eduardo Habkost To: qemu-devel@nongnu.org Date: Tue, 26 Sep 2017 20:07:06 -0300 Message-Id: <20170926230709.22958-4-ehabkost@redhat.com> In-Reply-To: <20170926230709.22958-1-ehabkost@redhat.com> References: <20170926230709.22958-1-ehabkost@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Tue, 26 Sep 2017 23:07:16 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC 3/6] device-crash-test: Allow checkOneCase() to report multiple results X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Huth , "Dr. David Alan Gilbert" , Cleber Rosa Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This will allow the test code to be improved to test multiple devices in a single run. Signed-off-by: Eduardo Habkost --- scripts/device-crash-test | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/scripts/device-crash-test b/scripts/device-crash-test index 782d7fd6c2..0bd599d395 100755 --- a/scripts/device-crash-test +++ b/scripts/device-crash-test @@ -384,7 +384,8 @@ def getBinaryInfo(args, binary): def checkOneCase(args, testcase): """Check one specific case =20 - Returns a dictionary containing results of test case. + Generates a list of dictionaries containing results of testing. If + multiple devices are being tested, multiple entries can be returned. """ binary =3D testcase['binary'] accel =3D testcase['accel'] @@ -413,7 +414,7 @@ def checkOneCase(args, testcase): vm.shutdown() r['exitcode'] =3D vm.exitcode() r['log'] =3D vm.get_log() - return r + yield r =20 def binariesToTest(args, testcase): if args.qemu: @@ -572,22 +573,20 @@ def main(): if args.dry_run: continue =20 - try: - f =3D checkOneCase(args, t) - except KeyboardInterrupt: - break =20 - i, wl =3D checkResultWhitelist(f) - dbg("testcase: %r, whitelist match: %r", t, wl) - wl_stats.setdefault(i, []).append(f) - level =3D wl.get('loglevel', logging.DEBUG) - logResult(f, level) + for r in checkOneCase(args, t): + i, wl =3D checkResultWhitelist(r) + dbg("testcase: %r, whitelist match: %r", t, wl) + wl_stats.setdefault(i, []).append(r) + level =3D wl.get('loglevel', logging.DEBUG) + logResult(r, level) + + if wl.get('fatal') or (args.strict and level >=3D logging.WARN= ): + fatal_failures.append(r) =20 - if wl.get('fatal') or (args.strict and level >=3D logging.WARN): - fatal_failures.append(f) + if expected_match and expected_match[0] !=3D i: + logger.warn("Didn't fail as expected: %s", formatTestCase(= t)) =20 - if expected_match and expected_match[0] !=3D i: - logger.warn("Didn't fail as expected: %s", formatTestCase(t)) =20 logger.info("Total: %d test cases", total) if skipped: --=20 2.13.5 From nobody Thu May 2 07:11:51 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 150646737541847.51007304316033; Tue, 26 Sep 2017 16:09:35 -0700 (PDT) Received: from localhost ([::1]:51408 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwyyd-0000VZ-73 for importer@patchew.org; Tue, 26 Sep 2017 19:09:19 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50882) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwywi-0007ji-Fp for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwywh-0005MB-LL for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:20 -0400 Received: from mx1.redhat.com ([209.132.183.28]:34160) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwywh-0005LV-Fo for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:19 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 96DAD81E0A for ; Tue, 26 Sep 2017 23:07:18 +0000 (UTC) Received: from localhost (ovpn-116-64.gru2.redhat.com [10.97.116.64]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2D61F7E47F; Tue, 26 Sep 2017 23:07:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 96DAD81E0A Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=ehabkost@redhat.com From: Eduardo Habkost To: qemu-devel@nongnu.org Date: Tue, 26 Sep 2017 20:07:07 -0300 Message-Id: <20170926230709.22958-5-ehabkost@redhat.com> In-Reply-To: <20170926230709.22958-1-ehabkost@redhat.com> References: <20170926230709.22958-1-ehabkost@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Tue, 26 Sep 2017 23:07:18 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC 4/6] device-crash-test: Exit immediately on fatal failures on quick mode X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Huth , "Dr. David Alan Gilbert" , Cleber Rosa Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" If we're running on quick mode, we won't wait for the full test run, and will exit immediately if a fatal failure is found. Signed-off-by: Eduardo Habkost --- scripts/device-crash-test | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/scripts/device-crash-test b/scripts/device-crash-test index 0bd599d395..4c23ffa449 100755 --- a/scripts/device-crash-test +++ b/scripts/device-crash-test @@ -583,10 +583,14 @@ def main(): =20 if wl.get('fatal') or (args.strict and level >=3D logging.WARN= ): fatal_failures.append(r) + if args.quick: + break =20 if expected_match and expected_match[0] !=3D i: logger.warn("Didn't fail as expected: %s", formatTestCase(= t)) =20 + if fatal_failures and args.quick: + break =20 logger.info("Total: %d test cases", total) if skipped: --=20 2.13.5 From nobody Thu May 2 07:11:51 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1506467505621807.1702178644737; Tue, 26 Sep 2017 16:11:45 -0700 (PDT) Received: from localhost ([::1]:51420 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwz0w-0002tD-TS for importer@patchew.org; Tue, 26 Sep 2017 19:11:42 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50919) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwywn-0007nm-3I for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwywj-0005OB-SE for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:25 -0400 Received: from mx1.redhat.com ([209.132.183.28]:34256) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwywj-0005NP-Jy for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:21 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B81C981DFC for ; Tue, 26 Sep 2017 23:07:20 +0000 (UTC) Received: from localhost (ovpn-116-64.gru2.redhat.com [10.97.116.64]) by smtp.corp.redhat.com (Postfix) with ESMTP id 03D4F6B6CD; Tue, 26 Sep 2017 23:07:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com B81C981DFC Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=ehabkost@redhat.com From: Eduardo Habkost To: qemu-devel@nongnu.org Date: Tue, 26 Sep 2017 20:07:08 -0300 Message-Id: <20170926230709.22958-6-ehabkost@redhat.com> In-Reply-To: <20170926230709.22958-1-ehabkost@redhat.com> References: <20170926230709.22958-1-ehabkost@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Tue, 26 Sep 2017 23:07:20 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC 5/6] device-crash-test: Basic device_add support X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Huth , "Dr. David Alan Gilbert" , Cleber Rosa Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add a 'method' testcase argument that will test the device using device_add instead of -device. A new device_add_error whitelist key is now supported, to catch device_add errors. Signed-off-by: Eduardo Habkost --- scripts/device-crash-test | 38 +++++++++++++++++++++++++++++++++----- 1 file changed, 33 insertions(+), 5 deletions(-) diff --git a/scripts/device-crash-test b/scripts/device-crash-test index 4c23ffa449..1245e214a0 100755 --- a/scripts/device-crash-test +++ b/scripts/device-crash-test @@ -35,7 +35,7 @@ import argparse from itertools import chain =20 sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'scripts')) -from qemu import QEMUMachine +from qemu import QEMUMachine, MonitorResponseError =20 logger =3D logging.getLogger('device-crash-test') dbg =3D logger.debug @@ -203,6 +203,11 @@ ERROR_WHITELIST =3D [ {'log':r"core 0 already populated"}, {'log':r"could not find stage1 bootloader"}, =20 + {'exitcode':None, 'device_add_error':r"^Bus '[\w.-]+' does not support= hotplugging$"}, + {'exitcode':None, 'device_add_error':r"^No '[\w-]+' bus found for devi= ce '[\w-]+'$"}, + # There are too many ways device_add can fail, ignore all of them as l= ong as QEMU doesn't crash: + {'exitcode':None, 'device_add_error':r".+"}, + # other exitcode=3D1 failures not listed above will just generate INFO= messages: {'exitcode':1, 'loglevel':logging.INFO}, =20 @@ -290,7 +295,10 @@ def whitelistResultMatch(wl, r): r.get('exitcode') =3D=3D wl.get('exitcode', 1)) and ('log' not in wl or ('log' in r and - re.search(wl['log'], r['log'], re.MULTILINE)))) + re.search(wl['log'], r['log'], re.MULTILINE))) and + ('device_add_error' not in wl or + ('device_add_error' in r and + re.search(wl['device_add_error'], r['device_add_error'])))) =20 =20 def checkResultWhitelist(r): @@ -339,6 +347,7 @@ class QemuBinaryInfo(object): self.alldevs =3D set(qomListTypeNames(vm, implements=3Ddevtype= , abstract=3DFalse)) self.dev_info =3D dict((t, vm.command('query-device-type', typ= ename=3Dt)) for t in self.alldevs) self.user_devs =3D set(d for d in self.alldevs if self.dev_inf= o[d]['user-creatable']) + self.hotpluggable_devs =3D set(d for d in self.alldevs if self= .dev_info[d]['hotpluggable']) self.machines =3D list(m['name'] for m in vm.command('query-ma= chines')) self.kvm_available =3D vm.command('query-kvm')['enabled'] finally: @@ -391,11 +400,14 @@ def checkOneCase(args, testcase): accel =3D testcase['accel'] machine =3D testcase['machine'] device =3D testcase['device'] + method =3D testcase['method'] + assert method in ['-device', 'device_add'] =20 dbg("will test: %r", testcase) =20 - args =3D ['-S', '-machine', '%s,accel=3D%s' % (machine, accel), - '-device', qemuOptsEscape(device)] + args =3D ['-S', '-machine', '%s,accel=3D%s' % (machine, accel)] + if method =3D=3D '-device': + args.extend(['-device', qemuOptsEscape(device)]) cmdline =3D ' '.join([binary] + args) dbg("will launch QEMU: %s", cmdline) vm =3D QEMUMachine(binary=3Dbinary, args=3Dargs) @@ -405,6 +417,12 @@ def checkOneCase(args, testcase): =20 try: vm.launch() + if method =3D=3D 'device_add': + dbg('running device_add %s', device) + try: + vm.command('device_add', driver=3Ddevice) + except MonitorResponseError, e: + r['device_add_error'] =3D e.reply['error']['desc'] except KeyboardInterrupt: raise except: @@ -434,14 +452,22 @@ def machinesToTest(args, testcase): return getBinaryInfo(args, testcase['binary']).machines =20 =20 +def plugMethods(args, testcase): + return ['-device', 'device_add'] + + def devicesToTest(args, testcase): - return getBinaryInfo(args, testcase['binary']).user_devs + if testcase['method'] =3D=3D 'device_add': + return getBinaryInfo(args, testcase['binary']).hotpluggable_devs + else: + return getBinaryInfo(args, testcase['binary']).user_devs =20 =20 TESTCASE_VARIABLES =3D [ ('binary', binariesToTest), ('accel', accelsToTest), ('machine', machinesToTest), + ('method', plugMethods), ('device', devicesToTest), ] =20 @@ -494,6 +520,8 @@ def logResult(f, level): logger.log(level, "cmdline: %s", f['cmdline']) for l in f.get('log', '').strip().split('\n'): logger.log(level, "log: %s", l) + if 'device_add_error' in f: + logger.log(level, "device_add error: %s", f['device_add_error']) logger.log(level, "exit code: %r", f.get('exitcode')) if 'exc_traceback' in f: logger.log(level, "exception:") --=20 2.13.5 From nobody Thu May 2 07:11:51 2024 Delivered-To: importer@patchew.org Received-SPF: temperror (zoho.com: Error in retrieving data from DNS) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=temperror (zoho.com: Error in retrieving data from DNS) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1506467502983980.0540132440835; Tue, 26 Sep 2017 16:11:42 -0700 (PDT) Received: from localhost ([::1]:51419 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwz0w-0002t8-6a for importer@patchew.org; Tue, 26 Sep 2017 19:11:42 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50929) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dwywo-0007qV-RN for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dwywl-0005PI-KN for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:26 -0400 Received: from mx1.redhat.com ([209.132.183.28]:41342) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dwywl-0005Oi-BK for qemu-devel@nongnu.org; Tue, 26 Sep 2017 19:07:23 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 740B07E42E for ; Tue, 26 Sep 2017 23:07:22 +0000 (UTC) Received: from localhost (ovpn-116-64.gru2.redhat.com [10.97.116.64]) by smtp.corp.redhat.com (Postfix) with ESMTP id CC7A5779E9; Tue, 26 Sep 2017 23:07:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 740B07E42E Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=ehabkost@redhat.com From: Eduardo Habkost To: qemu-devel@nongnu.org Date: Tue, 26 Sep 2017 20:07:09 -0300 Message-Id: <20170926230709.22958-7-ehabkost@redhat.com> In-Reply-To: <20170926230709.22958-1-ehabkost@redhat.com> References: <20170926230709.22958-1-ehabkost@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Tue, 26 Sep 2017 23:07:22 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC 6/6] device-crash-test: Multi-device device_add test X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Huth , "Dr. David Alan Gilbert" , Cleber Rosa Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_6 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" When running device_add tests, test all devices in a single run instead of restarting QEMU every time. There's a plug_all testcase argument that can be used to make the test code plug all devices at once. I'm adding this mode because there's a crash that was detected while testing the script without any device_del command, but the crash goes away if we device_del every device immediately: $ ../scripts/device-crash-test ./x86_64-softmmu/qemu-system-x86_64 --quic= k -t method=3Ddevice_add --strict -t plug_all=3D1 INFO: running test case: binary=3D./x86_64-softmmu/qemu-system-x86_64 plu= g_all=3D1 accel=3Dkvm machine=3Dpc-0.12 device=3D* method=3Ddevice_add WARNING: qemu received signal -6: ./x86_64-softmmu/qemu-system-x86_64 -ch= ardev socket,id=3Dmon,path=3D/var/tmp/qemu-23578-monitor.sock -mon chardev= =3Dmon,mode=3Dcontrol -display none -vga none -S -machine pc-0.12,accel=3Dk= vm ERROR: result: binary=3D./x86_64-softmmu/qemu-system-x86_64 plug_all=3D1 = accel=3Dkvm machine=3Dpc-0.12 device=3D* method=3Ddevice_add ERROR: cmdline: ./x86_64-softmmu/qemu-system-x86_64 -S -machine pc-0.12,a= ccel=3Dkvm ERROR: log: audio: Could not init `oss' audio driver ERROR: log: qemu-system-x86_64: .../qemu/migration/savevm.c:721: vmstate_= register_with_alias_id: Assertion `!se->compat || se->instance_id =3D=3D 0'= failed. ERROR: last device_add device: usb-net ERROR: exit code: -6 ERROR: exception: ERROR: Traceback (most recent call last): ERROR: File "../scripts/device-crash-test", line 437, in checkOneCase ERROR: vm.command('device_add', driver=3Ddevice, id=3Ddevid) ERROR: File ".../qemu/scripts/qemu.py", line 269, in command ERROR: raise qmp.qmp.QMPError("Monitor is closed") ERROR: QMPError: Monitor is closed ERROR: INFO: Total: 1 test cases ERROR: Fatal failure: binary=3D./x86_64-softmmu/qemu-system-x86_64 plug_a= ll=3D1 accel=3Dkvm machine=3Dpc-0.12 device=3D* method=3Ddevice_add ERROR: Fatal failures on some machine/device combinations Signed-off-by: Eduardo Habkost --- scripts/device-crash-test | 41 ++++++++++++++++++++++++++++++++++------- 1 file changed, 34 insertions(+), 7 deletions(-) diff --git a/scripts/device-crash-test b/scripts/device-crash-test index 1245e214a0..43110fed15 100755 --- a/scripts/device-crash-test +++ b/scripts/device-crash-test @@ -404,10 +404,19 @@ def checkOneCase(args, testcase): assert method in ['-device', 'device_add'] =20 dbg("will test: %r", testcase) + if device =3D=3D '*': + bi =3D getBinaryInfo(args, binary) + if method =3D=3D '-device': + devices =3D list(bi.user_devs) + else: + devices =3D list(bi.hotpluggable_devs) + else: + devices =3D [device] =20 args =3D ['-S', '-machine', '%s,accel=3D%s' % (machine, accel)] if method =3D=3D '-device': - args.extend(['-device', qemuOptsEscape(device)]) + for device in devices: + args.extend(['-device', qemuOptsEscape(device)]) cmdline =3D ' '.join([binary] + args) dbg("will launch QEMU: %s", cmdline) vm =3D QEMUMachine(binary=3Dbinary, args=3Dargs) @@ -418,11 +427,27 @@ def checkOneCase(args, testcase): try: vm.launch() if method =3D=3D 'device_add': - dbg('running device_add %s', device) - try: - vm.command('device_add', driver=3Ddevice) - except MonitorResponseError, e: - r['device_add_error'] =3D e.reply['error']['desc'] + plug_all =3D int(testcase.get('plug_all', 0)) + for device in devices: + r['device_add_device'] =3D device + dbg('running device_add %s', device) + devid =3D 'test-%s' % (device) + try: + vm.command('device_add', driver=3Ddevice, id=3Ddevid) + if not plug_all: + vm.command('device_del', id=3Ddevid) + except MonitorResponseError, e: + r2 =3D r.copy() + r2['device_add_error'] =3D e.reply['error']['desc'] + yield r2 + + if plug_all: + for device in devices: + devid =3D 'test-%s' % (device) + vm.command('device_del', id=3Ddevid) + + except GeneratorExit: + raise except KeyboardInterrupt: raise except: @@ -458,7 +483,7 @@ def plugMethods(args, testcase): =20 def devicesToTest(args, testcase): if testcase['method'] =3D=3D 'device_add': - return getBinaryInfo(args, testcase['binary']).hotpluggable_devs + return '*' # will test all devices in a single checkOneCase() call else: return getBinaryInfo(args, testcase['binary']).user_devs =20 @@ -520,6 +545,8 @@ def logResult(f, level): logger.log(level, "cmdline: %s", f['cmdline']) for l in f.get('log', '').strip().split('\n'): logger.log(level, "log: %s", l) + if 'device_add_device' in f: + logger.log(level, 'last device_add device: %s', f['device_add_devi= ce']) if 'device_add_error' in f: logger.log(level, "device_add error: %s", f['device_add_error']) logger.log(level, "exit code: %r", f.get('exitcode')) --=20 2.13.5