The functional tests currently don't allow a single test to be
selected for execution by dotted name, e.g:

../tests/functional/ppc64/test_pseries.py PseriesMachine.test_ppc64_linux_boot
                                          ^
The issue is that the testcase.py main function passes the test
module's name as the second argument to unittest.main(), which makes
it ignore all other positional arguments (presumably because the
module is already the superset of all tests).

After commit cac08383f0 ("tests/functional: expose sys.argv to
unittest.main"), the situation improves by passing the rest of the
argv from the command line invocation into unittest.main(), but it
still doesn't fix the issue. The short form options are now accepted,
so the -k option could be used to filter for a pattern, which is
useful, but not the same as listing the test names.

Fix this by passing the test module name via the "module" argument to
unittest.main() and stop touching argv. The ways of invoking tests are
now as per unittests documentation (-k still works):

  Examples:
    test_pseries.py                           - run default set of tests
    test_pseries.py MyTestSuite               - run suite 'MyTestSuite'
    test_pseries.py MyTestCase.testSomething  - run MyTestCase.testSomething
    test_pseries.py MyTestCase                - run all 'test*' test methods in MyTestCase

Note that ever since we've been programatically passing the module
name to unittest.main(), the usage 'test_pseries.py test_pseries' was
never valid. It used to "work" just the same as 'test_pseries.py
foobar' would. After this patch, that usage results in an error.

Also note that testcase.py:main() pertains to running the test module
that invoked it via QemuSystemTest.main(), i.e. module == __main__. So
the 'discover' usage of unittest doesn't apply here, the module is
already discovered because that's where this code was called from to
begin with. This patch could just as well call unittest.main() instead
of unittest.main(test_module), but the latter provides nicer error
messages prefixed with the module name.

Tested-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
---
v2: added docs
---
 docs/devel/testing/functional.rst      | 19 +++++++++++++++++--
 tests/functional/qemu_test/testcase.py |  7 +++----
 2 files changed, 20 insertions(+), 6 deletions(-)

diff --git a/docs/devel/testing/functional.rst b/docs/devel/testing/functional.rst
index fdeaebaadc..4ccc7c3940 100644
--- a/docs/devel/testing/functional.rst
+++ b/docs/devel/testing/functional.rst
@@ -52,17 +52,32 @@ the following line will only run the tests for the x86_64 target:
 
   make check-functional-x86_64
 
-To run a single test file without the meson test runner, you can also
+To run the tests from a test file without the meson test runner, you can also
 execute the file directly by specifying two environment variables first,
 the PYTHONPATH that has to include the python folder and the tests/functional
 folder of the source tree, and QEMU_TEST_QEMU_BINARY that has to point
 to the QEMU binary that should be used for the test. The current working
-directory should be your build folder. For example::
+directory should be your build folder. Like so::
 
   $ export PYTHONPATH=../python:../tests/functional
   $ export QEMU_TEST_QEMU_BINARY=$PWD/qemu-system-x86_64
+
+With the above set, run all tests from a test file::
+
   $ pyvenv/bin/python3 ../tests/functional/test_file.py
 
+Run all tests from a test class::
+
+  $ pyvenv/bin/python3 ../tests/functional/test_file.py TestClass
+
+Or a single test::
+
+  $ pyvenv/bin/python3 ../tests/functional/test_file.py TestClass.test_name
+
+Filtering test names also works::
+
+  $ pyvenv/bin/python3 ../tests/functional/test_file.py -k pattern
+
 The test framework will automatically purge any scratch files created during
 the tests. If needing to debug a failed test, it is possible to keep these
 files around on disk by setting ``QEMU_TEST_KEEP_SCRATCH=1`` as an env
diff --git a/tests/functional/qemu_test/testcase.py b/tests/functional/qemu_test/testcase.py
index 58f2740100..fa100d9632 100644
--- a/tests/functional/qemu_test/testcase.py
+++ b/tests/functional/qemu_test/testcase.py
@@ -249,17 +249,16 @@ def main():
         warnings.simplefilter("default")
         os.environ["PYTHONWARNINGS"] = "default"
 
-        path = os.path.basename(sys.argv[0])[:-3]
+        test_module = os.path.basename(sys.argv[0])[:-3]
 
         cache = os.environ.get("QEMU_TEST_PRECACHE", None)
         if cache is not None:
-            Asset.precache_suites(path, cache)
+            Asset.precache_suites(test_module, cache)
             return
 
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                    test_output_log = pycotap.LogMode.LogToError)
-        res = unittest.main(module = None, testRunner = tr, exit = False,
-                            argv=[sys.argv[0], path] + sys.argv[1:])
+        res = unittest.main(test_module, testRunner = tr, exit = False)
         failed = {}
         for (test, _message) in res.result.errors + res.result.failures:
             if hasattr(test, "log_filename") and not test.id() in failed:
-- 
2.51.0

On 18/12/2025 13.33, Fabiano Rosas wrote:
> The functional tests currently don't allow a single test to be
> selected for execution by dotted name, e.g:
> 
> ../tests/functional/ppc64/test_pseries.py PseriesMachine.test_ppc64_linux_boot
>                                            ^
> The issue is that the testcase.py main function passes the test
> module's name as the second argument to unittest.main(), which makes
> it ignore all other positional arguments (presumably because the
> module is already the superset of all tests).
> 
> After commit cac08383f0 ("tests/functional: expose sys.argv to
> unittest.main"), the situation improves by passing the rest of the
> argv from the command line invocation into unittest.main(), but it
> still doesn't fix the issue. The short form options are now accepted,
> so the -k option could be used to filter for a pattern, which is
> useful, but not the same as listing the test names.
> 
> Fix this by passing the test module name via the "module" argument to
> unittest.main() and stop touching argv. The ways of invoking tests are
> now as per unittests documentation (-k still works):
> 
>    Examples:
>      test_pseries.py                           - run default set of tests
>      test_pseries.py MyTestSuite               - run suite 'MyTestSuite'
>      test_pseries.py MyTestCase.testSomething  - run MyTestCase.testSomething
>      test_pseries.py MyTestCase                - run all 'test*' test methods in MyTestCase
> 
> Note that ever since we've been programatically passing the module
> name to unittest.main(), the usage 'test_pseries.py test_pseries' was
> never valid. It used to "work" just the same as 'test_pseries.py
> foobar' would. After this patch, that usage results in an error.
> 
> Also note that testcase.py:main() pertains to running the test module
> that invoked it via QemuSystemTest.main(), i.e. module == __main__. So
> the 'discover' usage of unittest doesn't apply here, the module is
> already discovered because that's where this code was called from to
> begin with. This patch could just as well call unittest.main() instead
> of unittest.main(test_module), but the latter provides nicer error
> messages prefixed with the module name.
> 
> Tested-by: Thomas Huth <thuth@redhat.com>
> Signed-off-by: Fabiano Rosas <farosas@suse.de>
> ---
> v2: added docs
> ---
>   docs/devel/testing/functional.rst      | 19 +++++++++++++++++--
>   tests/functional/qemu_test/testcase.py |  7 +++----
>   2 files changed, 20 insertions(+), 6 deletions(-)
> 
> diff --git a/docs/devel/testing/functional.rst b/docs/devel/testing/functional.rst
> index fdeaebaadc..4ccc7c3940 100644
> --- a/docs/devel/testing/functional.rst
> +++ b/docs/devel/testing/functional.rst
> @@ -52,17 +52,32 @@ the following line will only run the tests for the x86_64 target:
>   
>     make check-functional-x86_64
>   
> -To run a single test file without the meson test runner, you can also
> +To run the tests from a test file without the meson test runner, you can also
>   execute the file directly by specifying two environment variables first,
>   the PYTHONPATH that has to include the python folder and the tests/functional
>   folder of the source tree, and QEMU_TEST_QEMU_BINARY that has to point
>   to the QEMU binary that should be used for the test. The current working
> -directory should be your build folder. For example::
> +directory should be your build folder. Like so::
>   
>     $ export PYTHONPATH=../python:../tests/functional
>     $ export QEMU_TEST_QEMU_BINARY=$PWD/qemu-system-x86_64
> +
> +With the above set, run all tests from a test file::
> +
>     $ pyvenv/bin/python3 ../tests/functional/test_file.py
>   
> +Run all tests from a test class::
> +
> +  $ pyvenv/bin/python3 ../tests/functional/test_file.py TestClass
> +
> +Or a single test::
> +
> +  $ pyvenv/bin/python3 ../tests/functional/test_file.py TestClass.test_name
> +
> +Filtering test names also works::
> +
> +  $ pyvenv/bin/python3 ../tests/functional/test_file.py -k pattern
> +
>   The test framework will automatically purge any scratch files created during
>   the tests. If needing to debug a failed test, it is possible to keep these
>   files around on disk by setting ``QEMU_TEST_KEEP_SCRATCH=1`` as an env


  Hi Fabiano!

I was just about to pick this patch up, but seems like this now has a 
conflict with another update to the functional.rst file that got merged in 
between ... could you please rebase and send a v3?

  Thanks,
   Thomas