[Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure

Alex Bennée posted 1 patch 5 years, 9 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20180621184116.910-1-alex.bennee@linaro.org
Test checkpatch failed
Test docker-mingw@fedora passed
Test docker-quick@centos7 passed
Test s390x passed
_posts/2018-06-21-tcg-testing.md | 129 +++++++++++++++++++++++++++++++
1 file changed, 129 insertions(+)
create mode 100644 _posts/2018-06-21-tcg-testing.md
[Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure
Posted by Alex Bennée 5 years, 9 months ago
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 _posts/2018-06-21-tcg-testing.md | 129 +++++++++++++++++++++++++++++++
 1 file changed, 129 insertions(+)
 create mode 100644 _posts/2018-06-21-tcg-testing.md

diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md
new file mode 100644
index 0000000..85abb7a
--- /dev/null
+++ b/_posts/2018-06-21-tcg-testing.md
@@ -0,0 +1,129 @@
+---
+layout: post
+title:  "QEMU TCG Tests"
+date:   2018-06-21 10:30:00:00 +0000
+last_modified_at: 2018-06-21 10:30:00:00 +0000
+author: Alex Bennée
+categories: [testing, docker, compilation, tcg]
+---
+
+Ever since I started working on QEMU there was a small directory
+called tests/tcg that was in a perpetually broken state. It contains
+tests that exercise QEMU's ability to work across architectures using
+the power of the Tiny Code Generator. However as these tests needed to
+be compiled for the *guest* architectures and not the *host*
+architecture - known as cross-compiling - most developers never ran
+them. As the tests were hardly ever built inevitably a certain amount
+of bit-rot set in.
+
+# Cross Compilers
+
+In the old days cross-compilation setups were almost all hand-crafted
+affairs which involved building versions of binutils, gcc and a basic
+libc. If you couldn't get someone to give you a pre-built tarball it
+was something you laboured through once and hopefully never had to
+touch again. There were even dedicated scripts like crosstool-ng which
+attempted to make the process of patching and configuring your
+toolchain easier.
+
+While the distributions have improved their support for cross
+compilers over the years there are still plenty of variations in how
+they are deployed. It is hard for a project like QEMU which has to
+build on a wide range of operating systems and architectures to
+seamlessly use any given distributions compiler setup. However for
+those with cross compilers to hand `configure` now accepts two
+additional flags:
+
+    --cross-cc-$(ARCH)
+    --cross-cc-flags-$(ARCH)
+
+With a compiler specified for each guest architecture you want to test
+the build system can now build and run the tests. However for
+developers that don't have cross compilers around they can now take
+advantage of QEMU's docker images.
+
+# Enter Docker Containers
+
+If you work in IT you would be hard pressed not to have noticed the
+hype around Docker and the concept of containerisation over the last
+few years. Put simply containers allow you to define a known working
+set of software that gets run in an isolated environment for a given
+task. While this has many uses for QEMU it allows us to define build
+environments that any developer can run without having to mess around
+with their preferred host setup.
+
+Over the last few years QEMU's build system has been expanding the
+number of docker images it supports. Most of this has been in service
+of our CI testing such as [Patchew](https://patchew.org/QEMU/) and
+[Shippable](https://app.shippable.com/github/qemu/qemu/dashboard) but
+any developer with a docker setup can run the exact same images. For
+example if you want to check your patches won't break when compiled on
+a 32 bit ARM system you can run:
+
+    make docker-test-build@debian-armhf-cross J=n
+
+instead of tracking down a piece of ARM hardware to actually build on.
+Run `make docker` in your source tree to see the range of builds and
+tests it can support.
+
+# make check-tcg
+
+With the latest work [merged into
+master](https://git.qemu.org/?p=qemu.git;a=commit;h=TODO) we can now
+take advantage of both hand configured and docker based cross
+compilers to build test cases for TCG again. To run the TCG tests
+after you have built QEMU:
+
+    make check-tcg
+
+and the build system will build and run all the tests it can for your
+configured targets.
+
+# Rules for tests/tcg
+
+So now we have the infrastructure in place to add new tests what rules
+need to be followed to add new tests? 
+
+Well the first thing to note is currently all the tests are for
+linux-user versions of QEMU. This means the tests are all currently
+user-space tests that have access to the Linux syscall ABI.
+
+Another thing to note is the tests are standalone from the rest of the
+QEMU test infrastructure. To keep things simple they are compiled as
+standalone "static" binaries. As the cross-compilation setup can be
+quite rudimentary for some of the rarer architectures we only compile
+against a standard libc. There is no support for linking to other
+libraries like for example glib. Thread and maths support is part of
+glibc so shouldn't be a problem.
+
+Finally when writing new tests consider if it really is architecture
+specific or can be added to `tests/tcg/multiarch`. The multiarch tests
+are re-built for every supported architecture and should be the
+default place for anything explicitly testing syscalls and other
+common parts of the code base.
+
+# What's Next
+
+My hope with this work is we can start adding more tests to
+systematically defend functionality in linux-user. In fact I hope the
+first port of call to reproducing a crash would be writing a test case
+that can be added to our growing library of tests.
+
+Another thing that needs sorting out is getting toolchains for all of
+the less common architectures. The current work relies heavily on the
+excellent work of the Debian toolchain team in making multiarch
+aware cross compilers available in their distribution. However QEMU
+supports a lot more architectures than QEMU, some only as system
+emulations. In principle supporting them is as easy as adding another
+docker recipe but it might be these recipes end up having to compile
+the compilers from source.
+
+The `tests/tcg` directory still contains a number of source files we
+don't build. Notably the cris, lm32, mips, openrisc and xtensa targets have
+a set of tests that need a system emulator. Now we have the
+infrastructure for compiling I hope we can get support for system
+tests added fairly quickly. There will need to be some work to figure
+out a nice common way to pass results back to the build-system. For
+linux-user this is simple as all programs can simply return their exit
+code however for system emulation this is a little more involved.
+
-- 
2.17.1


Re: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure
Posted by no-reply@patchew.org 5 years, 9 months ago
Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20180621184116.910-1-alex.bennee@linaro.org
Subject: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]               patchew/20180621184116.910-1-alex.bennee@linaro.org -> patchew/20180621184116.910-1-alex.bennee@linaro.org
Switched to a new branch 'test'
0a765327ad blogposts: add post about the new check-tcg infrastructure

=== OUTPUT BEGIN ===
Checking PATCH 1/1: blogposts: add post about the new check-tcg infrastructure...
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#13: 
new file mode 100644

ERROR: trailing whitespace
#102: FILE: _posts/2018-06-21-tcg-testing.md:85:
+need to be followed to add new tests? $

total: 1 errors, 1 warnings, 129 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Re: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure
Posted by Philippe Mathieu-Daudé 5 years, 9 months ago
On 06/21/2018 06:12 PM, no-reply@patchew.org wrote:
> This series seems to have some coding style problems. See output below for
> more information:
> 
> Type: series
> Message-id: 20180621184116.910-1-alex.bennee@linaro.org
> Subject: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure
[...]> === OUTPUT BEGIN ===
> Checking PATCH 1/1: blogposts: add post about the new check-tcg infrastructure...
> WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
> #13: 
> new file mode 100644
> 
> ERROR: trailing whitespace
> #102: FILE: _posts/2018-06-21-tcg-testing.md:85:
> +need to be followed to add new tests? $

=)

> 
> total: 1 errors, 1 warnings, 129 lines checked
> 
> Your patch has style problems, please review.  If any of these errors
> are false positives report them to the maintainer, see
> CHECKPATCH in MAINTAINERS.

Re: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure
Posted by Max Filippov 5 years, 9 months ago
On Thu, Jun 21, 2018 at 11:41 AM, Alex Bennée <alex.bennee@linaro.org> wrote:
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  _posts/2018-06-21-tcg-testing.md | 129 +++++++++++++++++++++++++++++++
>  1 file changed, 129 insertions(+)
>  create mode 100644 _posts/2018-06-21-tcg-testing.md
>
> diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md

[...]

> +The `tests/tcg` directory still contains a number of source files we
> +don't build. Notably the cris, lm32, mips, openrisc and xtensa targets have
> +a set of tests that need a system emulator. Now we have the
> +infrastructure for compiling I hope we can get support for system
> +tests added fairly quickly. There will need to be some work to figure
> +out a nice common way to pass results back to the build-system. For
> +linux-user this is simple as all programs can simply return their exit
> +code however for system emulation this is a little more involved.

xtensa tests pass exit codes to the build system through semihosting calls.
If any of them fails make check fails as well.

-- 
Thanks.
-- Max

Re: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure
Posted by Alex Bennée 5 years, 9 months ago
Max Filippov <jcmvbkbc@gmail.com> writes:

> On Thu, Jun 21, 2018 at 11:41 AM, Alex Bennée <alex.bennee@linaro.org> wrote:
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> ---
>>  _posts/2018-06-21-tcg-testing.md | 129 +++++++++++++++++++++++++++++++
>>  1 file changed, 129 insertions(+)
>>  create mode 100644 _posts/2018-06-21-tcg-testing.md
>>
>> diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md
>
> [...]
>
>> +The `tests/tcg` directory still contains a number of source files we
>> +don't build. Notably the cris, lm32, mips, openrisc and xtensa targets have
>> +a set of tests that need a system emulator. Now we have the
>> +infrastructure for compiling I hope we can get support for system
>> +tests added fairly quickly. There will need to be some work to figure
>> +out a nice common way to pass results back to the build-system. For
>> +linux-user this is simple as all programs can simply return their exit
>> +code however for system emulation this is a little more involved.
>
> xtensa tests pass exit codes to the build system through semihosting calls.
> If any of them fails make check fails as well.

I've re-written that section as:

  The `tests/tcg` directory still contains a number of source files we
  don't build.

  The cris and openrisc directories contain user-space tests which just
  need the support of a toolchain and the relevant Makefile plumbing to
  be added.

  The lm32, mips and xtensa targets have a set of tests that need a
  system emulator. Aside from adding the compilers as docker images some
  additional work is needed to handle the differences between plain
  linux-user tests which can simply return an exit code to getting the
  results from a qemu-system emulation. Some architectures have
  semi-hosting support already for this while others report their test
  status over a simple serial link which will need to be parsed and
  handled in the `run-%:` test rule.

How is that?

Any chance you could look into what it would take to package up the
xtensa toolchain in a docker container? Are they simply tarballs of
binaries?

--
Alex Bennée

Re: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure
Posted by Max Filippov 5 years, 9 months ago
On Fri, Jun 22, 2018 at 2:03 AM, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Max Filippov <jcmvbkbc@gmail.com> writes:
>
>> On Thu, Jun 21, 2018 at 11:41 AM, Alex Bennée <alex.bennee@linaro.org> wrote:
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>> ---
>>>  _posts/2018-06-21-tcg-testing.md | 129 +++++++++++++++++++++++++++++++
>>>  1 file changed, 129 insertions(+)
>>>  create mode 100644 _posts/2018-06-21-tcg-testing.md
>>>
>>> diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md
>>
>> [...]
>>
>>> +The `tests/tcg` directory still contains a number of source files we
>>> +don't build. Notably the cris, lm32, mips, openrisc and xtensa targets have
>>> +a set of tests that need a system emulator. Now we have the
>>> +infrastructure for compiling I hope we can get support for system
>>> +tests added fairly quickly. There will need to be some work to figure
>>> +out a nice common way to pass results back to the build-system. For
>>> +linux-user this is simple as all programs can simply return their exit
>>> +code however for system emulation this is a little more involved.
>>
>> xtensa tests pass exit codes to the build system through semihosting calls.
>> If any of them fails make check fails as well.
>
> I've re-written that section as:
>
>   The `tests/tcg` directory still contains a number of source files we
>   don't build.
>
>   The cris and openrisc directories contain user-space tests which just
>   need the support of a toolchain and the relevant Makefile plumbing to
>   be added.
>
>   The lm32, mips and xtensa targets have a set of tests that need a
>   system emulator. Aside from adding the compilers as docker images some
>   additional work is needed to handle the differences between plain
>   linux-user tests which can simply return an exit code to getting the
>   results from a qemu-system emulation. Some architectures have
>   semi-hosting support already for this while others report their test
>   status over a simple serial link which will need to be parsed and
>   handled in the `run-%:` test rule.
>
> How is that?

'run-%' goal is only present in xtensa Makefile, other test suites have
explicit loop in the 'check' goal. Otherwise LGTM.

> Any chance you could look into what it would take to package up the
> xtensa toolchain in a docker container?

Can you point me to an example?

> Are they simply tarballs of binaries?

Yes, we have that option:

  https://github.com/foss-xtensa/toolchain/releases

Or they may be built from source.

-- 
Thanks.
-- Max

Re: [Qemu-devel] [PATCH] blogposts: add post about the new check-tcg infrastructure
Posted by Alex Bennée 5 years, 9 months ago
Max Filippov <jcmvbkbc@gmail.com> writes:

> On Fri, Jun 22, 2018 at 2:03 AM, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> Max Filippov <jcmvbkbc@gmail.com> writes:
>>
>>> On Thu, Jun 21, 2018 at 11:41 AM, Alex Bennée <alex.bennee@linaro.org> wrote:
>>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>>> ---
>>>>  _posts/2018-06-21-tcg-testing.md | 129 +++++++++++++++++++++++++++++++
>>>>  1 file changed, 129 insertions(+)
>>>>  create mode 100644 _posts/2018-06-21-tcg-testing.md
>>>>
>>>> diff --git a/_posts/2018-06-21-tcg-testing.md b/_posts/2018-06-21-tcg-testing.md
>>>
>>> [...]
>>>
>>>> +The `tests/tcg` directory still contains a number of source files we
>>>> +don't build. Notably the cris, lm32, mips, openrisc and xtensa targets have
>>>> +a set of tests that need a system emulator. Now we have the
>>>> +infrastructure for compiling I hope we can get support for system
>>>> +tests added fairly quickly. There will need to be some work to figure
>>>> +out a nice common way to pass results back to the build-system. For
>>>> +linux-user this is simple as all programs can simply return their exit
>>>> +code however for system emulation this is a little more involved.
>>>
>>> xtensa tests pass exit codes to the build system through semihosting calls.
>>> If any of them fails make check fails as well.
>>
>> I've re-written that section as:
>>
>>   The `tests/tcg` directory still contains a number of source files we
>>   don't build.
>>
>>   The cris and openrisc directories contain user-space tests which just
>>   need the support of a toolchain and the relevant Makefile plumbing to
>>   be added.
>>
>>   The lm32, mips and xtensa targets have a set of tests that need a
>>   system emulator. Aside from adding the compilers as docker images some
>>   additional work is needed to handle the differences between plain
>>   linux-user tests which can simply return an exit code to getting the
>>   results from a qemu-system emulation. Some architectures have
>>   semi-hosting support already for this while others report their test
>>   status over a simple serial link which will need to be parsed and
>>   handled in the `run-%:` test rule.
>>
>> How is that?
>
> 'run-%' goal is only present in xtensa Makefile, other test suites have
> explicit loop in the 'check' goal. Otherwise LGTM.

The TCG tests added a generic run-% rule:

  run-%: %
          $(call run-test, $<, $(QEMU) $<, "$< on $(TARGET_NAME)")

Although individual tests can override it.

>
>> Any chance you could look into what it would take to package up the
>> xtensa toolchain in a docker container?
>
> Can you point me to an example?

Well this was the base attempt:

https://github.com/stsquad/qemu/commit/daee9dc013b13b13e81f720176c5e379c05ed0e0

So I guess we need to add the overlay to the image?

>
>> Are they simply tarballs of binaries?
>
> Yes, we have that option:
>
>   https://github.com/foss-xtensa/toolchain/releases
>
> Or they may be built from source.


--
Alex Bennée