... and use them from their installed location.
The full recusive copy of tools/tests brings in all build and intermediate
artefacts. e.g. for test-tsx alone:
./tests/tsx
./tests/tsx/.test-tsx.o.d
./tests/tsx/test-tsx.o
./tests/tsx/.gitignore
./tests/tsx/test-tsx
./tests/tsx/Makefile
./tests/tsx/test-tsx.c
duplicating the test binary which is also in ./usr/lib/xen/tests
Rewrite run-tools-tests to run tests from their installed
location (/usr/lib/xen/tests in alpine), which effectively removes the outer
loop over $dir.
No practical change.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
This doesn't change any tests that run, although in the XML we get two fewer skips.
Both skips can be fixed by giving vpci and x86_emulator some install targets
---
automation/scripts/build | 1 -
automation/scripts/qubes-x86-64.sh | 7 +++--
automation/scripts/run-tools-tests | 43 +++++++++++++-----------------
3 files changed, 22 insertions(+), 29 deletions(-)
diff --git a/automation/scripts/build b/automation/scripts/build
index cdb8cd7c722b..0e7494ff6d87 100755
--- a/automation/scripts/build
+++ b/automation/scripts/build
@@ -109,6 +109,5 @@ else
# even though dist/ contains everything, while some containers don't even
# build Xen
(cd dist/install; find | cpio -o -H newc | gzip) > binaries/xen-tools.cpio.gz
- cp -r tools/tests binaries/
collect_xen_artefacts
fi
diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index aa47ba6bf5c0..577a00238a75 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -136,7 +136,7 @@ done
passed="test passed"
domU_check=""
dom0_check="
-/tests/run-tools-tests /tests /tmp/tests-junit.xml && echo \"${passed}\"
+/root/run-tools-tests /usr/lib/xen/tests /tmp/tests-junit.xml && echo \"${passed}\"
nc -l -p 8080 < /tmp/tests-junit.xml >/dev/null &
"
if [ "${test_variant}" = "tools-tests-pvh" ]; then
@@ -195,9 +195,8 @@ cat binaries/xen-tools.cpio.gz >> binaries/dom0-rootfs.cpio.gz
# test-local configuration
mkdir -p rootfs
cd rootfs
-mkdir -p boot etc/local.d
-cp -ar ../binaries/tests .
-cp -a ../automation/scripts/run-tools-tests tests/
+mkdir -p boot etc/local.d root
+cp -a ../automation/scripts/run-tools-tests root/
echo "#!/bin/bash
diff --git a/automation/scripts/run-tools-tests b/automation/scripts/run-tools-tests
index 770e97c3e943..8d7aa8fa5140 100755
--- a/automation/scripts/run-tools-tests
+++ b/automation/scripts/run-tools-tests
@@ -12,30 +12,25 @@ printf '<?xml version="1.0" encoding="UTF-8"?>\n' > "$xml_out"
printf '<testsuites name="tools.tests">\n' >> "$xml_out"
printf ' <testsuite name="tools.tests">\n' >> "$xml_out"
failed=
-for dir in "$1"/*; do
- [ -d "$dir" ] || continue
- echo "Running test in $dir"
- printf ' <testcase name="%s">\n' "$dir" >> "$xml_out"
- ret=
- for f in "$dir"/*; do
- [ -f "$f" ] || continue
- [ -x "$f" ] || continue
- "$f" 2>&1 | tee /tmp/out
- ret=$?
- if [ "$ret" -ne 0 ]; then
- echo "FAILED: $ret"
- failed+=" $dir"
- printf ' <failure type="failure" message="binary %s exited with code %d">\n' "$f" "$ret" >> "$xml_out"
- # TODO: could use xml escaping... but current tests seems to
- # produce sane output
- cat /tmp/out >> "$xml_out"
- printf ' </failure>\n' >> "$xml_out"
- else
- echo "PASSED"
- fi
- done
- if [ -z "$ret" ]; then
- printf ' <skipped type="skipped" message="no executable test found in %s"/>\n' "$dir" >> "$xml_out"
+for f in "$1"/*; do
+ if [ -x "$f" ]; then
+ echo "SKIP: $f not executable"
+ continue
+ fi
+ echo "Running $f"
+ printf ' <testcase name="%s">\n' "$f" >> "$xml_out"
+ "$f" 2>&1 | tee /tmp/out
+ ret=$?
+ if [ "$ret" -ne 0 ]; then
+ echo "FAILED: $f"
+ failed+=" $f"
+ printf ' <failure type="failure" message="binary %s exited with code %d">\n' "$f" "$ret" >> "$xml_out"
+ # TODO: could use xml escaping... but current tests seems to
+ # produce sane output
+ cat /tmp/out >> "$xml_out"
+ printf ' </failure>\n' >> "$xml_out"
+ else
+ echo "PASSED"
fi
printf ' </testcase>\n' >> "$xml_out"
done
--
2.39.5
On 20/05/2025 9:52 pm, Andrew Cooper wrote: > diff --git a/automation/scripts/run-tools-tests b/automation/scripts/run-tools-tests > index 770e97c3e943..8d7aa8fa5140 100755 > --- a/automation/scripts/run-tools-tests > +++ b/automation/scripts/run-tools-tests > @@ -12,30 +12,25 @@ printf '<?xml version="1.0" encoding="UTF-8"?>\n' > "$xml_out" > printf '<testsuites name="tools.tests">\n' >> "$xml_out" > printf ' <testsuite name="tools.tests">\n' >> "$xml_out" > failed= > -for dir in "$1"/*; do > - [ -d "$dir" ] || continue > - echo "Running test in $dir" > - printf ' <testcase name="%s">\n' "$dir" >> "$xml_out" > - ret= > - for f in "$dir"/*; do > - [ -f "$f" ] || continue > - [ -x "$f" ] || continue > - "$f" 2>&1 | tee /tmp/out > - ret=$? > - if [ "$ret" -ne 0 ]; then > - echo "FAILED: $ret" > - failed+=" $dir" > - printf ' <failure type="failure" message="binary %s exited with code %d">\n' "$f" "$ret" >> "$xml_out" > - # TODO: could use xml escaping... but current tests seems to > - # produce sane output > - cat /tmp/out >> "$xml_out" > - printf ' </failure>\n' >> "$xml_out" > - else > - echo "PASSED" > - fi > - done > - if [ -z "$ret" ]; then > - printf ' <skipped type="skipped" message="no executable test found in %s"/>\n' "$dir" >> "$xml_out" > +for f in "$1"/*; do > + if [ -x "$f" ]; then > + echo "SKIP: $f not executable" > + continue This should be ! -x I had that hunk in the wrong patch when posting this series. ~Andrew > + fi > + echo "Running $f" > + printf ' <testcase name="%s">\n' "$f" >> "$xml_out" > + "$f" 2>&1 | tee /tmp/out > + ret=$? > + if [ "$ret" -ne 0 ]; then > + echo "FAILED: $f" > + failed+=" $f" > + printf ' <failure type="failure" message="binary %s exited with code %d">\n' "$f" "$ret" >> "$xml_out" > + # TODO: could use xml escaping... but current tests seems to > + # produce sane output > + cat /tmp/out >> "$xml_out" > + printf ' </failure>\n' >> "$xml_out" > + else > + echo "PASSED" > fi > printf ' </testcase>\n' >> "$xml_out" > done
On Mon, May 26, 2025 at 05:45:29PM +0100, Andrew Cooper wrote:
> On 20/05/2025 9:52 pm, Andrew Cooper wrote:
> > diff --git a/automation/scripts/run-tools-tests b/automation/scripts/run-tools-tests
> > index 770e97c3e943..8d7aa8fa5140 100755
> > --- a/automation/scripts/run-tools-tests
> > +++ b/automation/scripts/run-tools-tests
> > @@ -12,30 +12,25 @@ printf '<?xml version="1.0" encoding="UTF-8"?>\n' > "$xml_out"
> > printf '<testsuites name="tools.tests">\n' >> "$xml_out"
> > printf ' <testsuite name="tools.tests">\n' >> "$xml_out"
> > failed=
> > -for dir in "$1"/*; do
> > - [ -d "$dir" ] || continue
> > - echo "Running test in $dir"
> > - printf ' <testcase name="%s">\n' "$dir" >> "$xml_out"
> > - ret=
> > - for f in "$dir"/*; do
> > - [ -f "$f" ] || continue
> > - [ -x "$f" ] || continue
> > - "$f" 2>&1 | tee /tmp/out
> > - ret=$?
> > - if [ "$ret" -ne 0 ]; then
> > - echo "FAILED: $ret"
> > - failed+=" $dir"
> > - printf ' <failure type="failure" message="binary %s exited with code %d">\n' "$f" "$ret" >> "$xml_out"
> > - # TODO: could use xml escaping... but current tests seems to
> > - # produce sane output
> > - cat /tmp/out >> "$xml_out"
> > - printf ' </failure>\n' >> "$xml_out"
> > - else
> > - echo "PASSED"
> > - fi
> > - done
> > - if [ -z "$ret" ]; then
> > - printf ' <skipped type="skipped" message="no executable test found in %s"/>\n' "$dir" >> "$xml_out"
> > +for f in "$1"/*; do
> > + if [ -x "$f" ]; then
> > + echo "SKIP: $f not executable"
> > + continue
>
> This should be ! -x
>
> I had that hunk in the wrong patch when posting this series.
With that fixed:
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>
But I think there's an issue with the script...
> > + "$f" 2>&1 | tee /tmp/out
> > + ret=$?
> > + if [ "$ret" -ne 0 ]; then
Is this checking the correct exit value? It seems that without `set -o
pipefail`, we only have the exit value of `tee` which should never fail.
But I think we should grab the value of ${PIPESTATUS[0]} to actually
read the exit value of $f.
Thanks,
--
Anthony PERARD
On 26/05/2025 6:22 pm, Anthony PERARD wrote:
> On Mon, May 26, 2025 at 05:45:29PM +0100, Andrew Cooper wrote:
>> On 20/05/2025 9:52 pm, Andrew Cooper wrote:
>>> diff --git a/automation/scripts/run-tools-tests b/automation/scripts/run-tools-tests
>>> index 770e97c3e943..8d7aa8fa5140 100755
>>> --- a/automation/scripts/run-tools-tests
>>> +++ b/automation/scripts/run-tools-tests
>>> @@ -12,30 +12,25 @@ printf '<?xml version="1.0" encoding="UTF-8"?>\n' > "$xml_out"
>>> printf '<testsuites name="tools.tests">\n' >> "$xml_out"
>>> printf ' <testsuite name="tools.tests">\n' >> "$xml_out"
>>> failed=
>>> -for dir in "$1"/*; do
>>> - [ -d "$dir" ] || continue
>>> - echo "Running test in $dir"
>>> - printf ' <testcase name="%s">\n' "$dir" >> "$xml_out"
>>> - ret=
>>> - for f in "$dir"/*; do
>>> - [ -f "$f" ] || continue
>>> - [ -x "$f" ] || continue
>>> - "$f" 2>&1 | tee /tmp/out
>>> - ret=$?
>>> - if [ "$ret" -ne 0 ]; then
>>> - echo "FAILED: $ret"
>>> - failed+=" $dir"
>>> - printf ' <failure type="failure" message="binary %s exited with code %d">\n' "$f" "$ret" >> "$xml_out"
>>> - # TODO: could use xml escaping... but current tests seems to
>>> - # produce sane output
>>> - cat /tmp/out >> "$xml_out"
>>> - printf ' </failure>\n' >> "$xml_out"
>>> - else
>>> - echo "PASSED"
>>> - fi
>>> - done
>>> - if [ -z "$ret" ]; then
>>> - printf ' <skipped type="skipped" message="no executable test found in %s"/>\n' "$dir" >> "$xml_out"
>>> +for f in "$1"/*; do
>>> + if [ -x "$f" ]; then
>>> + echo "SKIP: $f not executable"
>>> + continue
>> This should be ! -x
>>
>> I had that hunk in the wrong patch when posting this series.
> With that fixed:
> Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>
Thanks.
>
> But I think there's an issue with the script...
>
>>> + "$f" 2>&1 | tee /tmp/out
>>> + ret=$?
>>> + if [ "$ret" -ne 0 ]; then
> Is this checking the correct exit value? It seems that without `set -o
> pipefail`, we only have the exit value of `tee` which should never fail.
> But I think we should grab the value of ${PIPESTATUS[0]} to actually
> read the exit value of $f.
Hmm yes, I think this needs adjusting.
It turns out there are multiple problems with junit, including the fact
that putting failures in here doesn't cause the outer job to fail.
The internet suggest having a 'script: grep "<failure" junit.xml' step
to work around this.
I think that wants to be a separate series. The question is whether to
do this series first or second. I expect I'm going to need to backport
all of this work to eventually get XTF back onto the older trees.
~Andrew
On Mon, 26 May 2025, Andrew Cooper wrote:
> On 26/05/2025 6:22 pm, Anthony PERARD wrote:
> > On Mon, May 26, 2025 at 05:45:29PM +0100, Andrew Cooper wrote:
> >> On 20/05/2025 9:52 pm, Andrew Cooper wrote:
> >>> diff --git a/automation/scripts/run-tools-tests b/automation/scripts/run-tools-tests
> >>> index 770e97c3e943..8d7aa8fa5140 100755
> >>> --- a/automation/scripts/run-tools-tests
> >>> +++ b/automation/scripts/run-tools-tests
> >>> @@ -12,30 +12,25 @@ printf '<?xml version="1.0" encoding="UTF-8"?>\n' > "$xml_out"
> >>> printf '<testsuites name="tools.tests">\n' >> "$xml_out"
> >>> printf ' <testsuite name="tools.tests">\n' >> "$xml_out"
> >>> failed=
> >>> -for dir in "$1"/*; do
> >>> - [ -d "$dir" ] || continue
> >>> - echo "Running test in $dir"
> >>> - printf ' <testcase name="%s">\n' "$dir" >> "$xml_out"
> >>> - ret=
> >>> - for f in "$dir"/*; do
> >>> - [ -f "$f" ] || continue
> >>> - [ -x "$f" ] || continue
> >>> - "$f" 2>&1 | tee /tmp/out
> >>> - ret=$?
> >>> - if [ "$ret" -ne 0 ]; then
> >>> - echo "FAILED: $ret"
> >>> - failed+=" $dir"
> >>> - printf ' <failure type="failure" message="binary %s exited with code %d">\n' "$f" "$ret" >> "$xml_out"
> >>> - # TODO: could use xml escaping... but current tests seems to
> >>> - # produce sane output
> >>> - cat /tmp/out >> "$xml_out"
> >>> - printf ' </failure>\n' >> "$xml_out"
> >>> - else
> >>> - echo "PASSED"
> >>> - fi
> >>> - done
> >>> - if [ -z "$ret" ]; then
> >>> - printf ' <skipped type="skipped" message="no executable test found in %s"/>\n' "$dir" >> "$xml_out"
> >>> +for f in "$1"/*; do
> >>> + if [ -x "$f" ]; then
> >>> + echo "SKIP: $f not executable"
> >>> + continue
> >> This should be ! -x
> >>
> >> I had that hunk in the wrong patch when posting this series.
> > With that fixed:
> > Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> Thanks.
>
> >
> > But I think there's an issue with the script...
> >
> >>> + "$f" 2>&1 | tee /tmp/out
> >>> + ret=$?
> >>> + if [ "$ret" -ne 0 ]; then
> > Is this checking the correct exit value? It seems that without `set -o
> > pipefail`, we only have the exit value of `tee` which should never fail.
> > But I think we should grab the value of ${PIPESTATUS[0]} to actually
> > read the exit value of $f.
>
> Hmm yes, I think this needs adjusting.
>
> It turns out there are multiple problems with junit, including the fact
> that putting failures in here doesn't cause the outer job to fail.
>
> The internet suggest having a 'script: grep "<failure" junit.xml' step
> to work around this.
>
> I think that wants to be a separate series. The question is whether to
> do this series first or second. I expect I'm going to need to backport
> all of this work to eventually get XTF back onto the older trees.
I'll leave the choice to you
On 28/05/2025 1:16 am, Stefano Stabellini wrote:
> On Mon, 26 May 2025, Andrew Cooper wrote:
>> On 26/05/2025 6:22 pm, Anthony PERARD wrote:
>>> On Mon, May 26, 2025 at 05:45:29PM +0100, Andrew Cooper wrote:
>>>> On 20/05/2025 9:52 pm, Andrew Cooper wrote:
>>>>> diff --git a/automation/scripts/run-tools-tests b/automation/scripts/run-tools-tests
>>>>> index 770e97c3e943..8d7aa8fa5140 100755
>>>>> --- a/automation/scripts/run-tools-tests
>>>>> +++ b/automation/scripts/run-tools-tests
>>>>> @@ -12,30 +12,25 @@ printf '<?xml version="1.0" encoding="UTF-8"?>\n' > "$xml_out"
>>>>> printf '<testsuites name="tools.tests">\n' >> "$xml_out"
>>>>> printf ' <testsuite name="tools.tests">\n' >> "$xml_out"
>>>>> failed=
>>>>> -for dir in "$1"/*; do
>>>>> - [ -d "$dir" ] || continue
>>>>> - echo "Running test in $dir"
>>>>> - printf ' <testcase name="%s">\n' "$dir" >> "$xml_out"
>>>>> - ret=
>>>>> - for f in "$dir"/*; do
>>>>> - [ -f "$f" ] || continue
>>>>> - [ -x "$f" ] || continue
>>>>> - "$f" 2>&1 | tee /tmp/out
>>>>> - ret=$?
>>>>> - if [ "$ret" -ne 0 ]; then
>>>>> - echo "FAILED: $ret"
>>>>> - failed+=" $dir"
>>>>> - printf ' <failure type="failure" message="binary %s exited with code %d">\n' "$f" "$ret" >> "$xml_out"
>>>>> - # TODO: could use xml escaping... but current tests seems to
>>>>> - # produce sane output
>>>>> - cat /tmp/out >> "$xml_out"
>>>>> - printf ' </failure>\n' >> "$xml_out"
>>>>> - else
>>>>> - echo "PASSED"
>>>>> - fi
>>>>> - done
>>>>> - if [ -z "$ret" ]; then
>>>>> - printf ' <skipped type="skipped" message="no executable test found in %s"/>\n' "$dir" >> "$xml_out"
>>>>> +for f in "$1"/*; do
>>>>> + if [ -x "$f" ]; then
>>>>> + echo "SKIP: $f not executable"
>>>>> + continue
>>>> This should be ! -x
>>>>
>>>> I had that hunk in the wrong patch when posting this series.
>>> With that fixed:
>>> Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>
> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Thanks.
>
>
>> Thanks.
>>
>>> But I think there's an issue with the script...
>>>
>>>>> + "$f" 2>&1 | tee /tmp/out
>>>>> + ret=$?
>>>>> + if [ "$ret" -ne 0 ]; then
>>> Is this checking the correct exit value? It seems that without `set -o
>>> pipefail`, we only have the exit value of `tee` which should never fail.
>>> But I think we should grab the value of ${PIPESTATUS[0]} to actually
>>> read the exit value of $f.
>> Hmm yes, I think this needs adjusting.
>>
>> It turns out there are multiple problems with junit, including the fact
>> that putting failures in here doesn't cause the outer job to fail.
>>
>> The internet suggest having a 'script: grep "<failure" junit.xml' step
>> to work around this.
>>
>> I think that wants to be a separate series. The question is whether to
>> do this series first or second. I expect I'm going to need to backport
>> all of this work to eventually get XTF back onto the older trees.
>
> I'll leave the choice to you
In which case I'll get this committed now, because it's one useful step
forward and reduces my queue a bit.
~Andrew
© 2016 - 2025 Red Hat, Inc.