There is not much getting started documentation for qemu-iotests. This
patch explains how to create a new test and covers the overall testing
approach.
Cc: Ishani Chugh <chugh.ishani@research.iiit.ac.in>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
v3:
* Explicitly mention source and build trees so that out-of-tree builds
work [Peter]
* Mention common.qemu library for bash tests [Jeff]
v2:
* Added missing "touch <test-number>.out" step [Kevin]
* Added reference to SubmitAPatch wiki page [Eric & Philippe]
---
tests/qemu-iotests/README | 109 ++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 106 insertions(+), 3 deletions(-)
diff --git a/tests/qemu-iotests/README b/tests/qemu-iotests/README
index 6079b401ae..6c71f9005c 100644
--- a/tests/qemu-iotests/README
+++ b/tests/qemu-iotests/README
@@ -10,12 +10,115 @@ but no actual block drivers like ide, scsi or virtio.
* Usage
-Just run ./check to run all tests for the raw image format, or ./check
--qcow2 to test the qcow2 image format. The output of ./check -h explains
-additional options to test further image formats or I/O methods.
+Just run ./check from the build tree to run all tests for the raw image format,
+or ./check -qcow2 to test the qcow2 image format. The output of ./check -h
+explains additional options to test further image formats or I/O methods.
+
+* Testing approach
+
+Each test is an executable file (usually a bash script) that is run by the
+./check test harness. Standard out and standard error are captured to an
+output file. If the output file differs from the "golden master" output file
+for the test then it fails.
+
+Tests are simply a sequence of commands that produce output and the test itself
+does not judge whether it passed or failed. If you find yourself writing
+checks to determine success or failure then you should rethink the test and
+rely on output diffing instead.
+
+** Filtering volatile output
+
+When output contains absolute file paths, timestamps, process IDs, hostnames,
+or other volatile strings, the diff against golden master output will fail.
+Such output must be filtered to replace volatile strings with fixed
+placeholders.
+
+For example, the path to the temporary working directory changes between test
+runs so it must be filtered:
+
+ sed -e "s#$TEST_DIR/#TEST_DIR/#g"
+
+Commonly needed filters are available in ./common.filter.
+
+** Bash tests
+
+Tests are usually bash scripts that perform a sequence of qemu-img and qemu-io
+commands.
+
+If you wish to create a test in Bash that interacts with the QMP monitor,
+'common.qemu' provides functions for interacting with multiple QEMU
+processes.
+
+** Python tests
+
+Most tests are implemented in bash but it is difficult to interact with the QMP
+monitor. A Python module called 'iotests' is available for tests that require
+JSON and interacting with QEMU.
+
+* How to create a test
+
+1. Choose an unused test number
+
+Tests are identified by a unique number. Look for the highest test case number
+by looking at the test files. Then search the qemu-devel@nongnu.org mailing
+list to check if anyone has already sent patches using the next available
+number. You may need to increment the number a few times to reach an unused
+number.
+
+2. Create the test file
+
+Copy an existing test (one that most closely resembles what you wish to test)
+to the new test number in the source tree:
+
+ cp 001 <test-number>
+
+3. Assign groups to the test
+
+Add your test to the ./group file. This file is the index of tests and assigns
+them to functional groups like "rw" for read-write tests. Most tests belong to
+the "rw" and "auto" groups. "auto" means the test runs when ./check is invoked
+without a -g argument.
+
+Consider adding your test to the "quick" group if it executes quickly (<1s).
+This group is run by "make check-block" and is often included as part of build
+tests in continuous integration systems.
+
+4. Write the test
+
+Edit the test script. Look at existing tests for examples.
+
+5. Generate the golden master file
+
+Create an empty golden master file in the source tree:
+
+ touch <test-number>.out
+
+Then run your test from the build tree:
+
+ ./check <test-number>
+
+You may need additional command-line options to use an image format or
+protocol like -qcow2.
+
+The test will fail because there is no golden master yet. Inspect the output
+that your test generated with "cat <test-number>.out.bad".
+
+Verify that the output is as expected and contains no volatile strings like
+timestamps. You may need to add filters to your test to remove volatile
+strings.
+
+Once you are happy with the test output it can be used as the golden master
+with "mv <test-number>.out.bad <test-number>.out". Rerun the test to verify
+that it passes.
+
+Congratulations, you've created a new test!
+
+To contribute your test to qemu.git please follow the guidelines here:
+http://wiki.qemu.org/Contribute/SubmitAPatch
* Feedback and patches
Please send improvements to the test suite, general feedback or just
reports of failing tests cases to qemu-devel@nongnu.org with a CC:
to qemu-block@nongnu.org.
+
--
2.13.3
On Mon, Jul 31, 2017 at 05:26:41PM +0100, Stefan Hajnoczi wrote: > There is not much getting started documentation for qemu-iotests. This > patch explains how to create a new test and covers the overall testing > approach. > > Cc: Ishani Chugh <chugh.ishani@research.iiit.ac.in> > Reviewed-by: Eric Blake <eblake@redhat.com> > Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Reviewed-by: Jeff Cody <jcody@redhat.com> > --- > v3: > * Explicitly mention source and build trees so that out-of-tree builds > work [Peter] > * Mention common.qemu library for bash tests [Jeff] > v2: > * Added missing "touch <test-number>.out" step [Kevin] > * Added reference to SubmitAPatch wiki page [Eric & Philippe] > --- > tests/qemu-iotests/README | 109 ++++++++++++++++++++++++++++++++++++++++++++-- > 1 file changed, 106 insertions(+), 3 deletions(-) > > diff --git a/tests/qemu-iotests/README b/tests/qemu-iotests/README > index 6079b401ae..6c71f9005c 100644 > --- a/tests/qemu-iotests/README > +++ b/tests/qemu-iotests/README > @@ -10,12 +10,115 @@ but no actual block drivers like ide, scsi or virtio. > > * Usage > > -Just run ./check to run all tests for the raw image format, or ./check > --qcow2 to test the qcow2 image format. The output of ./check -h explains > -additional options to test further image formats or I/O methods. > +Just run ./check from the build tree to run all tests for the raw image format, > +or ./check -qcow2 to test the qcow2 image format. The output of ./check -h > +explains additional options to test further image formats or I/O methods. > + > +* Testing approach > + > +Each test is an executable file (usually a bash script) that is run by the > +./check test harness. Standard out and standard error are captured to an > +output file. If the output file differs from the "golden master" output file > +for the test then it fails. > + > +Tests are simply a sequence of commands that produce output and the test itself > +does not judge whether it passed or failed. If you find yourself writing > +checks to determine success or failure then you should rethink the test and > +rely on output diffing instead. > + > +** Filtering volatile output > + > +When output contains absolute file paths, timestamps, process IDs, hostnames, > +or other volatile strings, the diff against golden master output will fail. > +Such output must be filtered to replace volatile strings with fixed > +placeholders. > + > +For example, the path to the temporary working directory changes between test > +runs so it must be filtered: > + > + sed -e "s#$TEST_DIR/#TEST_DIR/#g" > + > +Commonly needed filters are available in ./common.filter. > + > +** Bash tests > + > +Tests are usually bash scripts that perform a sequence of qemu-img and qemu-io > +commands. > + > +If you wish to create a test in Bash that interacts with the QMP monitor, > +'common.qemu' provides functions for interacting with multiple QEMU > +processes. > + > +** Python tests > + > +Most tests are implemented in bash but it is difficult to interact with the QMP > +monitor. A Python module called 'iotests' is available for tests that require > +JSON and interacting with QEMU. > + > +* How to create a test > + > +1. Choose an unused test number > + > +Tests are identified by a unique number. Look for the highest test case number > +by looking at the test files. Then search the qemu-devel@nongnu.org mailing > +list to check if anyone has already sent patches using the next available > +number. You may need to increment the number a few times to reach an unused > +number. > + > +2. Create the test file > + > +Copy an existing test (one that most closely resembles what you wish to test) > +to the new test number in the source tree: > + > + cp 001 <test-number> > + > +3. Assign groups to the test > + > +Add your test to the ./group file. This file is the index of tests and assigns > +them to functional groups like "rw" for read-write tests. Most tests belong to > +the "rw" and "auto" groups. "auto" means the test runs when ./check is invoked > +without a -g argument. > + > +Consider adding your test to the "quick" group if it executes quickly (<1s). > +This group is run by "make check-block" and is often included as part of build > +tests in continuous integration systems. > + > +4. Write the test > + > +Edit the test script. Look at existing tests for examples. > + > +5. Generate the golden master file > + > +Create an empty golden master file in the source tree: > + > + touch <test-number>.out > + > +Then run your test from the build tree: > + > + ./check <test-number> > + > +You may need additional command-line options to use an image format or > +protocol like -qcow2. > + > +The test will fail because there is no golden master yet. Inspect the output > +that your test generated with "cat <test-number>.out.bad". > + > +Verify that the output is as expected and contains no volatile strings like > +timestamps. You may need to add filters to your test to remove volatile > +strings. > + > +Once you are happy with the test output it can be used as the golden master > +with "mv <test-number>.out.bad <test-number>.out". Rerun the test to verify > +that it passes. > + > +Congratulations, you've created a new test! > + > +To contribute your test to qemu.git please follow the guidelines here: > +http://wiki.qemu.org/Contribute/SubmitAPatch > > * Feedback and patches > > Please send improvements to the test suite, general feedback or just > reports of failing tests cases to qemu-devel@nongnu.org with a CC: > to qemu-block@nongnu.org. > + > -- > 2.13.3 >
On 07/31/2017 11:26 AM, Stefan Hajnoczi wrote: > There is not much getting started documentation for qemu-iotests. This > patch explains how to create a new test and covers the overall testing > approach. > > Cc: Ishani Chugh <chugh.ishani@research.iiit.ac.in> > Reviewed-by: Eric Blake <eblake@redhat.com> > Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> > --- > Please send improvements to the test suite, general feedback or just > reports of failing tests cases to qemu-devel@nongnu.org with a CC: > to qemu-block@nongnu.org. > + > Is this adding a spurious blank line at EOF? -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On Mon, Jul 31, 2017 at 12:28:44PM -0500, Eric Blake wrote: > On 07/31/2017 11:26 AM, Stefan Hajnoczi wrote: > > There is not much getting started documentation for qemu-iotests. This > > patch explains how to create a new test and covers the overall testing > > approach. > > > > Cc: Ishani Chugh <chugh.ishani@research.iiit.ac.in> > > Reviewed-by: Eric Blake <eblake@redhat.com> > > Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> > > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> > > --- > > > Please send improvements to the test suite, general feedback or just > > reports of failing tests cases to qemu-devel@nongnu.org with a CC: > > to qemu-block@nongnu.org. > > + > > > > Is this adding a spurious blank line at EOF? Yes. Please remove when merging. Thanks, Stefan
On 07/31/2017 12:26 PM, Stefan Hajnoczi wrote: > There is not much getting started documentation for qemu-iotests. This > patch explains how to create a new test and covers the overall testing > approach. > > Cc: Ishani Chugh <chugh.ishani@research.iiit.ac.in> > Reviewed-by: Eric Blake <eblake@redhat.com> > Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Already overkill, but I read it all anyway; so: Reviewed-by: John Snow <jsnow@redhat.com>
© 2016 - 2024 Red Hat, Inc.