Python test scripts that use unittest consist of multiple tests.
unittest.main allows selecting which tests to run, but currently this
is not possible because the iotests wrapper ignores sys.argv.

unittest.main command line options also allow the user to pick the
desired options for verbosity, failfast mode, etc.  While "-d" is
currently translated to "-v", it also enables extra debug output,
and other options are not available at all.

These command line options only work if the unittest.main testRunner
argument is a type, rather than a TestRunner instance.  Therefore, pass
the class name and "verbosity" argument to unittest.main, and adjust for
the different default warnings between TextTestRunner and unittest.main.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 tests/qemu-iotests/iotests.py | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 0521235030..c7915684ba 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -1308,12 +1308,16 @@ def __init__(self, stream: Optional[io.TextIOBase] = None,
                          resultclass=resultclass,
                          *args, **kwargs)
 
-def execute_unittest(debug=False):
+def execute_unittest(argv: List[str], debug: bool= False):
     """Executes unittests within the calling module."""
 
-    verbosity = 2 if debug else 1
-    runner = unittest.ReproducibleTestRunner(verbosity=verbosity)
-    unittest.main(testRunner=runner)
+    # Some tests have warnings, especially ResourceWarnings for unclosed
+    # files and sockets.  Ignore them for now to ensure reproducibility of
+    # the test output.
+    unittest.main(argv=argv,
+                  testRunner=ReproducibleTestRunner,
+                  verbosity=2 if debug else 1,
+                  warnings=None if sys.warnoptions else 'ignore')
 
 def execute_setup_common(supported_fmts: Sequence[str] = (),
                          supported_platforms: Sequence[str] = (),
@@ -1350,7 +1354,7 @@ def execute_test(*args, test_function=None, **kwargs):
 
     debug = execute_setup_common(*args, **kwargs)
     if not test_function:
-        execute_unittest(debug)
+        execute_unittest(sys.argv, debug)
     else:
         test_function()
 
-- 
2.30.1

23.03.2021 21:19, Paolo Bonzini wrote:
> Python test scripts that use unittest consist of multiple tests.
> unittest.main allows selecting which tests to run, but currently this
> is not possible because the iotests wrapper ignores sys.argv.
> 
> unittest.main command line options also allow the user to pick the
> desired options for verbosity, failfast mode, etc.  While "-d" is
> currently translated to "-v", it also enables extra debug output,
> and other options are not available at all.
> 
> These command line options only work if the unittest.main testRunner
> argument is a type, rather than a TestRunner instance.  Therefore, pass
> the class name and "verbosity" argument to unittest.main, and adjust for
> the different default warnings between TextTestRunner and unittest.main.
> 
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>   tests/qemu-iotests/iotests.py | 14 +++++++++-----
>   1 file changed, 9 insertions(+), 5 deletions(-)
> 
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index 0521235030..c7915684ba 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -1308,12 +1308,16 @@ def __init__(self, stream: Optional[io.TextIOBase] = None,
>                            resultclass=resultclass,
>                            *args, **kwargs)
>   
> -def execute_unittest(debug=False):
> +def execute_unittest(argv: List[str], debug: bool= False):
>       """Executes unittests within the calling module."""
>   
> -    verbosity = 2 if debug else 1
> -    runner = unittest.ReproducibleTestRunner(verbosity=verbosity)
> -    unittest.main(testRunner=runner)
> +    # Some tests have warnings, especially ResourceWarnings for unclosed
> +    # files and sockets.  Ignore them for now to ensure reproducibility of
> +    # the test output.
> +    unittest.main(argv=argv,
> +                  testRunner=ReproducibleTestRunner,
> +                  verbosity=2 if debug else 1,
> +                  warnings=None if sys.warnoptions else 'ignore')
>   
>   def execute_setup_common(supported_fmts: Sequence[str] = (),
>                            supported_platforms: Sequence[str] = (),
> @@ -1350,7 +1354,7 @@ def execute_test(*args, test_function=None, **kwargs):
>   
>       debug = execute_setup_common(*args, **kwargs)
>       if not test_function:
> -        execute_unittest(debug)
> +        execute_unittest(sys.argv, debug)
>       else:
>           test_function()
>   
> 

If you decide to resend for some of my comments (or due to another reviewer be more careful), I think it would be nicer to merge part of this commit which moves us from passing object to passing ReproducibleTestRunner to the previous commit, to not remove line that we've added in the previous commit. And here only add argv argument.


-- 
Best regards,
Vladimir

On 23/03/21 20:17, Vladimir Sementsov-Ogievskiy wrote:
>>
>> +    unittest.main(argv=argv,
>> +                  testRunner=ReproducibleTestRunner,
>> +                  verbosity=2 if debug else 1,
>> +                  warnings=None if sys.warnoptions else 'ignore')
>>   def execute_setup_common(supported_fmts: Sequence[str] = (),
>>                            supported_platforms: Sequence[str] = (),
>> @@ -1350,7 +1354,7 @@ def execute_test(*args, test_function=None, 
>> **kwargs):
>>       debug = execute_setup_common(*args, **kwargs)
>>       if not test_function:
>> -        execute_unittest(debug)
>> +        execute_unittest(sys.argv, debug)
>>       else:
>>           test_function()
>>
> 
> If you decide to resend for some of my comments (or due to another 
> reviewer be more careful), I think it would be nicer to merge part of 
> this commit which moves us from passing object to passing 
> ReproducibleTestRunner to the previous commit, to not remove line that 
> we've added in the previous commit. And here only add argv argument.

Well, it's the price to pay to make the previous commit as independent 
as possible.  In particular in the previous patch there's no reason to 
add the complexity of warnings.

I could make it three commits, but at some point too much splitting adds 
clutter, the patches are already pretty small.

Paolo

24.03.2021 00:22, Paolo Bonzini wrote:
> On 23/03/21 20:17, Vladimir Sementsov-Ogievskiy wrote:
>>>
>>> +    unittest.main(argv=argv,
>>> +                  testRunner=ReproducibleTestRunner,
>>> +                  verbosity=2 if debug else 1,
>>> +                  warnings=None if sys.warnoptions else 'ignore')
>>>   def execute_setup_common(supported_fmts: Sequence[str] = (),
>>>                            supported_platforms: Sequence[str] = (),
>>> @@ -1350,7 +1354,7 @@ def execute_test(*args, test_function=None, **kwargs):
>>>       debug = execute_setup_common(*args, **kwargs)
>>>       if not test_function:
>>> -        execute_unittest(debug)
>>> +        execute_unittest(sys.argv, debug)
>>>       else:
>>>           test_function()
>>>
>>
>> If you decide to resend for some of my comments (or due to another reviewer be more careful), I think it would be nicer to merge part of this commit which moves us from passing object to passing ReproducibleTestRunner to the previous commit, to not remove line that we've added in the previous commit. And here only add argv argument.
> 
> Well, it's the price to pay to make the previous commit as independent as possible.  In particular in the previous patch there's no reason to add the complexity of warnings.
> 
> I could make it three commits, but at some point too much splitting adds clutter, the patches are already pretty small.
> 

OK, they are good as is



-- 
Best regards,
Vladimir

23.03.2021 21:19, Paolo Bonzini wrote:
> Python test scripts that use unittest consist of multiple tests.
> unittest.main allows selecting which tests to run, but currently this
> is not possible because the iotests wrapper ignores sys.argv.
> 
> unittest.main command line options also allow the user to pick the
> desired options for verbosity, failfast mode, etc.  While "-d" is
> currently translated to "-v", it also enables extra debug output,
> and other options are not available at all.
> 
> These command line options only work if the unittest.main testRunner
> argument is a type, rather than a TestRunner instance.  Therefore, pass
> the class name and "verbosity" argument to unittest.main, and adjust for
> the different default warnings between TextTestRunner and unittest.main.
> 
> Signed-off-by: Paolo Bonzini<pbonzini@redhat.com>

Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

-- 
Best regards,
Vladimir