[libvirt PATCH] testutils: print a helpful summary of failed tests

Ján Tomko posted 1 patch 4 years, 1 month ago
Test syntax-check failed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/libvirt tags/patchew/8c807eeae7a69e4c488469bd8b08105dd9e9d12e.1581006903.git.jtomko@redhat.com
tests/testutils.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
[libvirt PATCH] testutils: print a helpful summary of failed tests
Posted by Ján Tomko 4 years, 1 month ago
When debugging test failures in seven independent test
cases, it might be helpful to only gather the debug output
of the failing cases.

Record the indexes of the tests that fail and print them
in the VIR_TEST_RANGE of the command line that will result
in only those tests being run.

Signed-off-by: Ján Tomko <jtomko@redhat.com>
---
 tests/testutils.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/tests/testutils.c b/tests/testutils.c
index 7b9a5ea05b..0cf0ac7e5c 100644
--- a/tests/testutils.c
+++ b/tests/testutils.c
@@ -54,6 +54,7 @@ static unsigned int testRegenerate = -1;
 
 static size_t testCounter;
 static virBitmapPtr testBitmap;
+static virBitmapPtr failedTests;
 
 virArch virTestHostArch = VIR_ARCH_X86_64;
 
@@ -172,6 +173,9 @@ virTestRun(const char *title,
             fprintf(stderr, "!");
     }
 
+    if (ret != 0)
+        ignore_value(virBitmapSetBitExpand(failedTests, testCounter));
+
     g_unsetenv("VIR_TEST_MOCK_TESTNAME");
     return ret;
 }
@@ -930,6 +934,9 @@ int virTestMain(int argc,
         }
     }
 
+    if (!(failedTests = virBitmapNew(1)))
+        return EXIT_FAILURE;
+
     ret = (func)();
 
     virResetLastError();
@@ -938,6 +945,11 @@ int virTestMain(int argc,
             fprintf(stderr, "%*s", 40 - (int)(testCounter % 40), "");
         fprintf(stderr, " %-3zu %s\n", testCounter, ret == 0 ? "OK" : "FAIL");
     }
+    if (ret == EXIT_FAILURE && !virBitmapIsAllClear(failedTests)) {
+        g_autofree char *failed = virBitmapFormat(failedTests);
+        fprintf(stderr, "Some tests failed. Run them using:\n");
+        fprintf(stderr, "VIR_TEST_DEBUG=1 VIR_TEST_RANGE=%s %s\n", failed, argv[0]);
+    }
     virLogReset();
     return ret;
 }
-- 
2.21.1

Re: [libvirt PATCH] testutils: print a helpful summary of failed tests
Posted by Daniel P. Berrangé 4 years, 1 month ago
On Thu, Feb 06, 2020 at 05:35:05PM +0100, Ján Tomko wrote:
> When debugging test failures in seven independent test
> cases, it might be helpful to only gather the debug output
> of the failing cases.
> 
> Record the indexes of the tests that fail and print them
> in the VIR_TEST_RANGE of the command line that will result
> in only those tests being run.
> 
> Signed-off-by: Ján Tomko <jtomko@redhat.com>
> ---
>  tests/testutils.c | 12 ++++++++++++
>  1 file changed, 12 insertions(+)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|