From: Anthony PERARD <anthony.perard@vates.tech>
We can't rely on an exit value from `run-tools-tests` since we only
have the console output. `console.exp` only look for success or it
times out. We could parse the console output, but the junit is more
concise. Also check if we have it or fail as well.
Signed-off-by: Anthony PERARD <anthony.perard@vates.tech>
---
automation/scripts/qubes-x86-64.sh | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index 046137a4a6..7a4c5ae489 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -298,6 +298,13 @@ TEST_RESULT=$?
if [ -n "$retrieve_xml" ]; then
nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null
+ # Findout if one of the test failed
+ if ! grep -q '</testsuites>' tests-junit.xml; then
+ echo "ERROR: tests-junit.xml is incomplete or missing."
+ TEST_RESULT=1
+ elif grep -q '</failure>' tests-junit.xml; then
+ TEST_RESULT=1
+ fi
fi
exit "$TEST_RESULT"
--
Anthony PERARDOn 03/06/2025 1:42 pm, Anthony PERARD wrote: > From: Anthony PERARD <anthony.perard@vates.tech> > > We can't rely on an exit value from `run-tools-tests` since we only > have the console output. `console.exp` only look for success or it > times out. We could parse the console output, but the junit is more > concise. Also check if we have it or fail as well. > > Signed-off-by: Anthony PERARD <anthony.perard@vates.tech> > --- > automation/scripts/qubes-x86-64.sh | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh > index 046137a4a6..7a4c5ae489 100755 > --- a/automation/scripts/qubes-x86-64.sh > +++ b/automation/scripts/qubes-x86-64.sh > @@ -298,6 +298,13 @@ TEST_RESULT=$? > > if [ -n "$retrieve_xml" ]; then > nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null > + # Findout if one of the test failed > + if ! grep -q '</testsuites>' tests-junit.xml; then > + echo "ERROR: tests-junit.xml is incomplete or missing." > + TEST_RESULT=1 > + elif grep -q '</failure>' tests-junit.xml; then > + TEST_RESULT=1 > + fi > fi > > exit "$TEST_RESULT" A couple of things. From my experimentation with junit, https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1849342222/test_report?job_name=kbl-xtf-x86-64-gcc-debug we can also use </error> for classification. I'm also very disappointed in Gitlab classifying <warning> as success. Not for this patch, but for XTF I need to be able to express "tolerable failure". (All branches of Xen will run the same tests, and we don't have OSSTest to deem "fail never passed" as non-blocking.) Even if the job passes overall, I want tolerable failures to show up in the UI, so I have to use <failure> in junit.xml. But that means needing to be more selective, and I don't have a good idea of how to do this. (I have one terrible idea, which is </failure type=tolerable"> which will escape that grep, but it feels like (ab)buse of XML.) ~Andrew
On Tue, Jun 03, 2025 at 02:41:50PM +0100, Andrew Cooper wrote:
> On 03/06/2025 1:42 pm, Anthony PERARD wrote:
> > if [ -n "$retrieve_xml" ]; then
> > nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null
> > + # Findout if one of the test failed
> > + if ! grep -q '</testsuites>' tests-junit.xml; then
> > + echo "ERROR: tests-junit.xml is incomplete or missing."
> > + TEST_RESULT=1
> > + elif grep -q '</failure>' tests-junit.xml; then
> > + TEST_RESULT=1
> > + fi
> > fi
> >
> > exit "$TEST_RESULT"
>
> A couple of things.
>
> From my experimentation with junit,
> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1849342222/test_report?job_name=kbl-xtf-x86-64-gcc-debug
> we can also use </error> for classification. I'm also very disappointed
> in Gitlab classifying <warning> as success.
According to the documentation [1] which point to this junit xml format [2]
the only elements (and path) are:
testsuites.testsuite.testcase.failure
"error" or "warning" don't exist.
There's the attributes `type` in <failure> but this isn't explained how
it's used.
But I guess if we follow the link in [2], go through web.archive.org, we
can find [3] which has "skipped", "error", "failure", but still no
"warning".
[1] https://docs.gitlab.com/ci/testing/unit_test_reports/#unit-test-reporting-workflow
[2] https://www.ibm.com/docs/en/developer-for-zos/16.0?topic=formats-junit-xml-format
[3] https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd
> Not for this patch, but for XTF I need to be able to express "tolerable
> failure". (All branches of Xen will run the same tests, and we don't
> have OSSTest to deem "fail never passed" as non-blocking.)
According to [1], there's a notion of "Existing failures", but that
might show up only on merge request.
> Even if the job passes overall, I want tolerable failures to show up in
> the UI, so I have to use <failure> in junit.xml. But that means needing
> to be more selective, and I don't have a good idea of how to do this.
> (I have one terrible idea, which is </failure type=tolerable"> which
> will escape that grep, but it feels like (ab)buse of XML.)
At the moment, `run-tools-tests` write '<failure type="failure"' on
failure, so we could grep on that instead event if it is sligtly more
fragile. I've choosen to grep on '</failure>' at first because that's
much less likely to be written differently, while the attributes in the
tag could be written in a different order.
Then, we can always use `sed` and extract the "type" to check it:
sed -n 's/.*<failure \(\|.* \)type="\([^"]\+\)" .*/\2/p' tests-junit.xml | while read type; do
case $type in
failure) echo fail;;
tolerable) echo ok;;
*) echo error unknwon type $type;;
esac
done
But that maybe going a bit too far :-)
Cheers,
--
Anthony PERARD
On 03/06/2025 4:58 pm, Anthony PERARD wrote: > On Tue, Jun 03, 2025 at 02:41:50PM +0100, Andrew Cooper wrote: >> On 03/06/2025 1:42 pm, Anthony PERARD wrote: >>> if [ -n "$retrieve_xml" ]; then >>> nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null >>> + # Findout if one of the test failed >>> + if ! grep -q '</testsuites>' tests-junit.xml; then >>> + echo "ERROR: tests-junit.xml is incomplete or missing." >>> + TEST_RESULT=1 >>> + elif grep -q '</failure>' tests-junit.xml; then >>> + TEST_RESULT=1 >>> + fi >>> fi >>> >>> exit "$TEST_RESULT" >> A couple of things. >> >> From my experimentation with junit, >> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1849342222/test_report?job_name=kbl-xtf-x86-64-gcc-debug >> we can also use </error> for classification. I'm also very disappointed >> in Gitlab classifying <warning> as success. > According to the documentation [1] which point to this junit xml format [2] > the only elements (and path) are: > testsuites.testsuite.testcase.failure > "error" or "warning" don't exist. > There's the attributes `type` in <failure> but this isn't explained how > it's used. > > But I guess if we follow the link in [2], go through web.archive.org, we > can find [3] which has "skipped", "error", "failure", but still no > "warning". > > > [1] https://docs.gitlab.com/ci/testing/unit_test_reports/#unit-test-reporting-workflow > [2] https://www.ibm.com/docs/en/developer-for-zos/16.0?topic=formats-junit-xml-format > [3] https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd I was also very disappointed with the documentation, which seems to be almost non-existent. But after some source diving: https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/lib/gitlab/ci/parsers/test/junit.rb#L69-83 So anything which isn't recognised is deemed to be success. Lovely :( I'm also still none the wiser as to why Skip's system_out is a formatted json blob, unlike the others. > > >> Not for this patch, but for XTF I need to be able to express "tolerable >> failure". (All branches of Xen will run the same tests, and we don't >> have OSSTest to deem "fail never passed" as non-blocking.) > According to [1], there's a notion of "Existing failures", but that > might show up only on merge request. > >> Even if the job passes overall, I want tolerable failures to show up in >> the UI, so I have to use <failure> in junit.xml. But that means needing >> to be more selective, and I don't have a good idea of how to do this. >> (I have one terrible idea, which is </failure type=tolerable"> which >> will escape that grep, but it feels like (ab)buse of XML.) > At the moment, `run-tools-tests` write '<failure type="failure"' on > failure, so we could grep on that instead event if it is sligtly more > fragile. I've choosen to grep on '</failure>' at first because that's > much less likely to be written differently, while the attributes in the > tag could be written in a different order. > > Then, we can always use `sed` and extract the "type" to check it: > sed -n 's/.*<failure \(\|.* \)type="\([^"]\+\)" .*/\2/p' tests-junit.xml | while read type; do > case $type in > failure) echo fail;; > tolerable) echo ok;; > *) echo error unknwon type $type;; > esac > done > But that maybe going a bit too far :-) There's xq which might be able to do this more nicely, and is packaged in alpine. ~Andrew
On Tue, Jun 03, 2025 at 02:41:50PM +0100, Andrew Cooper wrote: > On 03/06/2025 1:42 pm, Anthony PERARD wrote: > > From: Anthony PERARD <anthony.perard@vates.tech> > > > > We can't rely on an exit value from `run-tools-tests` since we only > > have the console output. `console.exp` only look for success or it > > times out. We could parse the console output, but the junit is more > > concise. Also check if we have it or fail as well. > > > > Signed-off-by: Anthony PERARD <anthony.perard@vates.tech> > > --- > > automation/scripts/qubes-x86-64.sh | 7 +++++++ > > 1 file changed, 7 insertions(+) > > > > diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh > > index 046137a4a6..7a4c5ae489 100755 > > --- a/automation/scripts/qubes-x86-64.sh > > +++ b/automation/scripts/qubes-x86-64.sh > > @@ -298,6 +298,13 @@ TEST_RESULT=$? > > > > if [ -n "$retrieve_xml" ]; then > > nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null > > + # Findout if one of the test failed > > + if ! grep -q '</testsuites>' tests-junit.xml; then > > + echo "ERROR: tests-junit.xml is incomplete or missing." > > + TEST_RESULT=1 > > + elif grep -q '</failure>' tests-junit.xml; then > > + TEST_RESULT=1 > > + fi > > fi > > > > exit "$TEST_RESULT" > > A couple of things. > > From my experimentation with junit, > https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1849342222/test_report?job_name=kbl-xtf-x86-64-gcc-debug > we can also use </error> for classification. I'm also very disappointed > in Gitlab classifying <warning> as success. > > Not for this patch, but for XTF I need to be able to express "tolerable > failure". (All branches of Xen will run the same tests, and we don't > have OSSTest to deem "fail never passed" as non-blocking.) > > Even if the job passes overall, I want tolerable failures to show up in > the UI, so I have to use <failure> in junit.xml. But that means needing > to be more selective, and I don't have a good idea of how to do this. > (I have one terrible idea, which is </failure type=tolerable"> which > will escape that grep, but it feels like (ab)buse of XML.) But that automation/ dir (including the run-tools-tests script) is per-branch, so you can specify there which tests should be considered failure and which just warning, no? It will require few more bits in the script, but fundamentally shouldn't be a problem? -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
On 03/06/2025 2:44 pm, Marek Marczykowski-Górecki wrote:
> On Tue, Jun 03, 2025 at 02:41:50PM +0100, Andrew Cooper wrote:
>> On 03/06/2025 1:42 pm, Anthony PERARD wrote:
>>> From: Anthony PERARD <anthony.perard@vates.tech>
>>>
>>> We can't rely on an exit value from `run-tools-tests` since we only
>>> have the console output. `console.exp` only look for success or it
>>> times out. We could parse the console output, but the junit is more
>>> concise. Also check if we have it or fail as well.
>>>
>>> Signed-off-by: Anthony PERARD <anthony.perard@vates.tech>
>>> ---
>>> automation/scripts/qubes-x86-64.sh | 7 +++++++
>>> 1 file changed, 7 insertions(+)
>>>
>>> diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
>>> index 046137a4a6..7a4c5ae489 100755
>>> --- a/automation/scripts/qubes-x86-64.sh
>>> +++ b/automation/scripts/qubes-x86-64.sh
>>> @@ -298,6 +298,13 @@ TEST_RESULT=$?
>>>
>>> if [ -n "$retrieve_xml" ]; then
>>> nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null
>>> + # Findout if one of the test failed
>>> + if ! grep -q '</testsuites>' tests-junit.xml; then
>>> + echo "ERROR: tests-junit.xml is incomplete or missing."
>>> + TEST_RESULT=1
>>> + elif grep -q '</failure>' tests-junit.xml; then
>>> + TEST_RESULT=1
>>> + fi
>>> fi
>>>
>>> exit "$TEST_RESULT"
>> A couple of things.
>>
>> From my experimentation with junit,
>> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1849342222/test_report?job_name=kbl-xtf-x86-64-gcc-debug
>> we can also use </error> for classification. I'm also very disappointed
>> in Gitlab classifying <warning> as success.
>>
>> Not for this patch, but for XTF I need to be able to express "tolerable
>> failure". (All branches of Xen will run the same tests, and we don't
>> have OSSTest to deem "fail never passed" as non-blocking.)
>>
>> Even if the job passes overall, I want tolerable failures to show up in
>> the UI, so I have to use <failure> in junit.xml. But that means needing
>> to be more selective, and I don't have a good idea of how to do this.
>> (I have one terrible idea, which is </failure type=tolerable"> which
>> will escape that grep, but it feels like (ab)buse of XML.)
> But that automation/ dir (including the run-tools-tests script) is
> per-branch, so you can specify there which tests should be considered
> failure and which just warning, no? It will require few more bits in the
> script, but fundamentally shouldn't be a problem?
>
XTF is in a separate repo and does not have branches.
Now consider
https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=d965e2ee07c56c341d8896852550914d87ea5374,
and testing it.
Anything I put into XTF to test it will pass on staging but fail on the
older stable-* trees. In due course it will get backported to the
bugfix branches, but it likely won't get fixed on the security-only
branches.
Furthermore, I need to not change xen.git to make this work, and older
branches need to pick up newer XTF automatically. (XSA tests *are*
expected to pass everywhere once the issue is public.)
My current plan is to have logic of the form:
if ( xenver >= $STAGING )
xtf_failure(...);
else
xtf_success("Expected Failure:" ...);
where $STAGING moves when backports get done. I still want the failures
to show up in the Tests UI.
~Andrew
On 03/06/2025 3:09 pm, Andrew Cooper wrote:
> On 03/06/2025 2:44 pm, Marek Marczykowski-Górecki wrote:
>> On Tue, Jun 03, 2025 at 02:41:50PM +0100, Andrew Cooper wrote:
>>> On 03/06/2025 1:42 pm, Anthony PERARD wrote:
>>>> From: Anthony PERARD <anthony.perard@vates.tech>
>>>>
>>>> We can't rely on an exit value from `run-tools-tests` since we only
>>>> have the console output. `console.exp` only look for success or it
>>>> times out. We could parse the console output, but the junit is more
>>>> concise. Also check if we have it or fail as well.
>>>>
>>>> Signed-off-by: Anthony PERARD <anthony.perard@vates.tech>
>>>> ---
>>>> automation/scripts/qubes-x86-64.sh | 7 +++++++
>>>> 1 file changed, 7 insertions(+)
>>>>
>>>> diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
>>>> index 046137a4a6..7a4c5ae489 100755
>>>> --- a/automation/scripts/qubes-x86-64.sh
>>>> +++ b/automation/scripts/qubes-x86-64.sh
>>>> @@ -298,6 +298,13 @@ TEST_RESULT=$?
>>>>
>>>> if [ -n "$retrieve_xml" ]; then
>>>> nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null
>>>> + # Findout if one of the test failed
>>>> + if ! grep -q '</testsuites>' tests-junit.xml; then
>>>> + echo "ERROR: tests-junit.xml is incomplete or missing."
>>>> + TEST_RESULT=1
>>>> + elif grep -q '</failure>' tests-junit.xml; then
>>>> + TEST_RESULT=1
>>>> + fi
>>>> fi
>>>>
>>>> exit "$TEST_RESULT"
>>> A couple of things.
>>>
>>> From my experimentation with junit,
>>> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1849342222/test_report?job_name=kbl-xtf-x86-64-gcc-debug
>>> we can also use </error> for classification. I'm also very disappointed
>>> in Gitlab classifying <warning> as success.
>>>
>>> Not for this patch, but for XTF I need to be able to express "tolerable
>>> failure". (All branches of Xen will run the same tests, and we don't
>>> have OSSTest to deem "fail never passed" as non-blocking.)
>>>
>>> Even if the job passes overall, I want tolerable failures to show up in
>>> the UI, so I have to use <failure> in junit.xml. But that means needing
>>> to be more selective, and I don't have a good idea of how to do this.
>>> (I have one terrible idea, which is </failure type=tolerable"> which
>>> will escape that grep, but it feels like (ab)buse of XML.)
>> But that automation/ dir (including the run-tools-tests script) is
>> per-branch, so you can specify there which tests should be considered
>> failure and which just warning, no? It will require few more bits in the
>> script, but fundamentally shouldn't be a problem?
>>
> XTF is in a separate repo and does not have branches.
>
> Now consider
> https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=d965e2ee07c56c341d8896852550914d87ea5374,
> and testing it.
>
> Anything I put into XTF to test it will pass on staging but fail on the
> older stable-* trees. In due course it will get backported to the
> bugfix branches, but it likely won't get fixed on the security-only
> branches.
>
> Furthermore, I need to not change xen.git to make this work, and older
> branches need to pick up newer XTF automatically. (XSA tests *are*
> expected to pass everywhere once the issue is public.)
>
> My current plan is to have logic of the form:
>
> if ( xenver >= $STAGING )
> xtf_failure(...);
> else
> xtf_success("Expected Failure:" ...);
>
> where $STAGING moves when backports get done. I still want the failures
> to show up in the Tests UI.
P.S. I'm planning to teach ./xtf-runner how to write out a junit.xml
directly. I'm not interested in interpreting python's stdout and
writing xml in shell...
~Andrew
© 2016 - 2025 Red Hat, Inc.