From nobody Mon Apr 29 12:48:14 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1501519015475978.9532792359172; Mon, 31 Jul 2017 09:36:55 -0700 (PDT) Received: from localhost ([::1]:60606 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dcDgb-0007nd-Py for importer@patchew.org; Mon, 31 Jul 2017 12:36:53 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:40603) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dcDWz-0008Ui-1V for qemu-devel@nongnu.org; Mon, 31 Jul 2017 12:26:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dcDWx-0000r0-IV for qemu-devel@nongnu.org; Mon, 31 Jul 2017 12:26:57 -0400 Received: from mx1.redhat.com ([209.132.183.28]:50774) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dcDWs-0000fF-9X; Mon, 31 Jul 2017 12:26:50 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7BCE0AB91F; Mon, 31 Jul 2017 16:26:48 +0000 (UTC) Received: from localhost (ovpn-117-146.ams2.redhat.com [10.36.117.146]) by smtp.corp.redhat.com (Postfix) with ESMTP id 041A960F83; Mon, 31 Jul 2017 16:26:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 7BCE0AB91F Authentication-Results: ext-mx04.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx04.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=stefanha@redhat.com From: Stefan Hajnoczi To: Date: Mon, 31 Jul 2017 17:26:41 +0100 Message-Id: <20170731162641.24160-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Mon, 31 Jul 2017 16:26:49 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v3] qemu-iotests: add a "how to" to ./README X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-block@nongnu.org, Jeff Cody , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Stefan Hajnoczi , John Snow , Ishani Chugh Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" There is not much getting started documentation for qemu-iotests. This patch explains how to create a new test and covers the overall testing approach. Cc: Ishani Chugh Reviewed-by: Eric Blake Reviewed-by: Philippe Mathieu-Daud=C3=A9 Signed-off-by: Stefan Hajnoczi Reviewed-by: Jeff Cody Reviewed-by: John Snow --- v3: * Explicitly mention source and build trees so that out-of-tree builds work [Peter] * Mention common.qemu library for bash tests [Jeff] v2: * Added missing "touch .out" step [Kevin] * Added reference to SubmitAPatch wiki page [Eric & Philippe] --- tests/qemu-iotests/README | 109 ++++++++++++++++++++++++++++++++++++++++++= ++-- 1 file changed, 106 insertions(+), 3 deletions(-) diff --git a/tests/qemu-iotests/README b/tests/qemu-iotests/README index 6079b401ae..6c71f9005c 100644 --- a/tests/qemu-iotests/README +++ b/tests/qemu-iotests/README @@ -10,12 +10,115 @@ but no actual block drivers like ide, scsi or virtio. =20 * Usage =20 -Just run ./check to run all tests for the raw image format, or ./check --qcow2 to test the qcow2 image format. The output of ./check -h explains -additional options to test further image formats or I/O methods. +Just run ./check from the build tree to run all tests for the raw image fo= rmat, +or ./check -qcow2 to test the qcow2 image format. The output of ./check -h +explains additional options to test further image formats or I/O methods. + +* Testing approach + +Each test is an executable file (usually a bash script) that is run by the +./check test harness. Standard out and standard error are captured to an +output file. If the output file differs from the "golden master" output f= ile +for the test then it fails. + +Tests are simply a sequence of commands that produce output and the test i= tself +does not judge whether it passed or failed. If you find yourself writing +checks to determine success or failure then you should rethink the test and +rely on output diffing instead. + +** Filtering volatile output + +When output contains absolute file paths, timestamps, process IDs, hostnam= es, +or other volatile strings, the diff against golden master output will fail. +Such output must be filtered to replace volatile strings with fixed +placeholders. + +For example, the path to the temporary working directory changes between t= est +runs so it must be filtered: + + sed -e "s#$TEST_DIR/#TEST_DIR/#g" + +Commonly needed filters are available in ./common.filter. + +** Bash tests + +Tests are usually bash scripts that perform a sequence of qemu-img and qem= u-io +commands. + +If you wish to create a test in Bash that interacts with the QMP monitor, +'common.qemu' provides functions for interacting with multiple QEMU +processes. + +** Python tests + +Most tests are implemented in bash but it is difficult to interact with th= e QMP +monitor. A Python module called 'iotests' is available for tests that req= uire +JSON and interacting with QEMU. + +* How to create a test + +1. Choose an unused test number + +Tests are identified by a unique number. Look for the highest test case n= umber +by looking at the test files. Then search the qemu-devel@nongnu.org maili= ng +list to check if anyone has already sent patches using the next available +number. You may need to increment the number a few times to reach an unus= ed +number. + +2. Create the test file + +Copy an existing test (one that most closely resembles what you wish to te= st) +to the new test number in the source tree: + + cp 001 + +3. Assign groups to the test + +Add your test to the ./group file. This file is the index of tests and as= signs +them to functional groups like "rw" for read-write tests. Most tests belo= ng to +the "rw" and "auto" groups. "auto" means the test runs when ./check is in= voked +without a -g argument. + +Consider adding your test to the "quick" group if it executes quickly (<1s= ). +This group is run by "make check-block" and is often included as part of b= uild +tests in continuous integration systems. + +4. Write the test + +Edit the test script. Look at existing tests for examples. + +5. Generate the golden master file + +Create an empty golden master file in the source tree: + + touch .out + +Then run your test from the build tree: + + ./check + +You may need additional command-line options to use an image format or +protocol like -qcow2. + +The test will fail because there is no golden master yet. Inspect the out= put +that your test generated with "cat .out.bad". + +Verify that the output is as expected and contains no volatile strings like +timestamps. You may need to add filters to your test to remove volatile +strings. + +Once you are happy with the test output it can be used as the golden master +with "mv .out.bad .out". Rerun the test to veri= fy +that it passes. + +Congratulations, you've created a new test! + +To contribute your test to qemu.git please follow the guidelines here: +http://wiki.qemu.org/Contribute/SubmitAPatch =20 * Feedback and patches =20 Please send improvements to the test suite, general feedback or just reports of failing tests cases to qemu-devel@nongnu.org with a CC: to qemu-block@nongnu.org. + --=20 2.13.3