[Qemu-devel] [PATCH 4/6] cirrus / travis: Add gnu-sed and bash for macOS and FreeBSD

Thomas Huth posted 6 patches 6 years, 9 months ago
Maintainers: "Alex Bennée" <alex.bennee@linaro.org>, "Philippe Mathieu-Daudé" <philmd@redhat.com>, Li-Wen Hsu <lwhsu@freebsd.org>, Kevin Wolf <kwolf@redhat.com>, Fam Zheng <fam@euphon.net>, Max Reitz <mreitz@redhat.com>, Ed Maste <emaste@freebsd.org>
There is a newer version of this series
[Qemu-devel] [PATCH 4/6] cirrus / travis: Add gnu-sed and bash for macOS and FreeBSD
Posted by Thomas Huth 6 years, 9 months ago
We are going to enable the qemu-iotests during "make check" again,
and for running the iotests, we need bash and gnu-sed.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .cirrus.yml | 4 ++--
 .travis.yml | 1 +
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index 47ef5bc604..8326a3a4b1 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -7,7 +7,7 @@ freebsd_12_task:
     cpu: 8
     memory: 8G
   install_script: pkg install -y
-    bison curl cyrus-sasl git glib gmake gnutls
+    bash bison curl cyrus-sasl git glib gmake gnutls gsed
     nettle perl5 pixman pkgconf png usbredir
   script:
     - mkdir build
@@ -20,7 +20,7 @@ macos_task:
   osx_instance:
     image: mojave-base
   install_script:
-    - brew install pkg-config python glib pixman make sdl2
+    - brew install pkg-config python gnu-sed glib pixman make sdl2
   script:
     - ./configure --python=/usr/local/bin/python3 || { cat config.log; exit 1; }
     - gmake -j$(sysctl -n hw.ncpu)
diff --git a/.travis.yml b/.travis.yml
index 2e06aee9d0..ba94644192 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -42,6 +42,7 @@ addons:
     packages:
       - glib
       - pixman
+      - gnu-sed
 
 
 # The channel name "irc.oftc.net#qemu" is encrypted against qemu/qemu
-- 
2.21.0


Re: [Qemu-devel] [PATCH 4/6] cirrus / travis: Add gnu-sed and bash for macOS and FreeBSD
Posted by Wainer dos Santos Moschetta 6 years, 9 months ago
Hello Thomas,

On 04/24/2019 07:37 AM, Thomas Huth wrote:
> We are going to enable the qemu-iotests during "make check" again,
> and for running the iotests, we need bash and gnu-sed.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   .cirrus.yml | 4 ++--
>   .travis.yml | 1 +
>   2 files changed, 3 insertions(+), 2 deletions(-)

It needs to get the Freebsd image [1] updated too, in order to `make 
BUILD_TARGET=check vm-build-freebsd` passes. Here it failed with:
---
env: bash: No such file or directory
gmake: *** [/var/tmp/qemu-test.6OlDFH/tests/Makefile.include:1101: 
check-tests/qemu-iotests-quick.sh] Error 1
gmake: *** Waiting for unfinished jobs....
---

I'm not sure about the netbsd and openbsd images, they might need bash 
and gnu-sed as well.

[1] http://download.patchew.org/freebsd-11.1-amd64.img.xz

- Wainer

>
> diff --git a/.cirrus.yml b/.cirrus.yml
> index 47ef5bc604..8326a3a4b1 100644
> --- a/.cirrus.yml
> +++ b/.cirrus.yml
> @@ -7,7 +7,7 @@ freebsd_12_task:
>       cpu: 8
>       memory: 8G
>     install_script: pkg install -y
> -    bison curl cyrus-sasl git glib gmake gnutls
> +    bash bison curl cyrus-sasl git glib gmake gnutls gsed
>       nettle perl5 pixman pkgconf png usbredir
>     script:
>       - mkdir build
> @@ -20,7 +20,7 @@ macos_task:
>     osx_instance:
>       image: mojave-base
>     install_script:
> -    - brew install pkg-config python glib pixman make sdl2
> +    - brew install pkg-config python gnu-sed glib pixman make sdl2
>     script:
>       - ./configure --python=/usr/local/bin/python3 || { cat config.log; exit 1; }
>       - gmake -j$(sysctl -n hw.ncpu)
> diff --git a/.travis.yml b/.travis.yml
> index 2e06aee9d0..ba94644192 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -42,6 +42,7 @@ addons:
>       packages:
>         - glib
>         - pixman
> +      - gnu-sed
>   
>   
>   # The channel name "irc.oftc.net#qemu" is encrypted against qemu/qemu


Re: [Qemu-devel] [PATCH 4/6] cirrus / travis: Add gnu-sed and bash for macOS and FreeBSD
Posted by Thomas Huth 6 years, 9 months ago
On 24/04/2019 23.29, Wainer dos Santos Moschetta wrote:
> Hello Thomas,
> 
> On 04/24/2019 07:37 AM, Thomas Huth wrote:
>> We are going to enable the qemu-iotests during "make check" again,
>> and for running the iotests, we need bash and gnu-sed.
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   .cirrus.yml | 4 ++--
>>   .travis.yml | 1 +
>>   2 files changed, 3 insertions(+), 2 deletions(-)
> 
> It needs to get the Freebsd image [1] updated too, in order to `make
> BUILD_TARGET=check vm-build-freebsd` passes. Here it failed with:
> ---
> env: bash: No such file or directory
> gmake: *** [/var/tmp/qemu-test.6OlDFH/tests/Makefile.include:1101:
> check-tests/qemu-iotests-quick.sh] Error 1
> gmake: *** Waiting for unfinished jobs....
> ---
> 
> I'm not sure about the netbsd and openbsd images, they might need bash
> and gnu-sed as well.

D'oh! Does anybody know what are the correct steps to update these images?

 Thomas

Re: [Qemu-devel] [PATCH 4/6] cirrus / travis: Add gnu-sed and bash for macOS and FreeBSD
Posted by Laszlo Ersek 6 years, 9 months ago
(+Dan)

On 04/25/19 08:00, Thomas Huth wrote:
> On 24/04/2019 23.29, Wainer dos Santos Moschetta wrote:
>> Hello Thomas,
>>
>> On 04/24/2019 07:37 AM, Thomas Huth wrote:
>>> We are going to enable the qemu-iotests during "make check" again,
>>> and for running the iotests, we need bash and gnu-sed.
>>>
>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>> ---
>>>   .cirrus.yml | 4 ++--
>>>   .travis.yml | 1 +
>>>   2 files changed, 3 insertions(+), 2 deletions(-)
>>
>> It needs to get the Freebsd image [1] updated too, in order to `make
>> BUILD_TARGET=check vm-build-freebsd` passes. Here it failed with:
>> ---
>> env: bash: No such file or directory
>> gmake: *** [/var/tmp/qemu-test.6OlDFH/tests/Makefile.include:1101:
>> check-tests/qemu-iotests-quick.sh] Error 1
>> gmake: *** Waiting for unfinished jobs....
>> ---
>>
>> I'm not sure about the netbsd and openbsd images, they might need bash
>> and gnu-sed as well.
> 
> D'oh! Does anybody know what are the correct steps to update these images?

(1) make the OPENBSD subsystem maintainer care

(2) update <https://wiki.qemu.org/Hosts/BSD>

(3) download the image from download.patchew.org, boot it and update it

(4) upload the image to download.patchew.org

(5) update the "tests/vm/openbsd" script in the QEMU tree in sync
(checksums, commands etc)


I'm fairly sure (1) is necessary; the rest is more or less guesswork on
my part.

Thanks
Laszlo

[Qemu-devel] Update *BSD images with gnu-sed and bash (was: [PATCH 4/6] cirrus / travis: Add gnu-sed and bash for macOS and FreeBSD)
Posted by Thomas Huth 6 years, 9 months ago
On 26/04/2019 18.43, Laszlo Ersek wrote:
> On 04/25/19 08:00, Thomas Huth wrote:
>> On 24/04/2019 23.29, Wainer dos Santos Moschetta wrote:
>>> Hello Thomas,
>>>
>>> On 04/24/2019 07:37 AM, Thomas Huth wrote:
>>>> We are going to enable the qemu-iotests during "make check" again,
>>>> and for running the iotests, we need bash and gnu-sed.
>>> [...]
>>> It needs to get the Freebsd image [1] updated too, in order to `make
>>> BUILD_TARGET=check vm-build-freebsd` passes. Here it failed with:
>>> ---
>>> env: bash: No such file or directory
>>> gmake: *** [/var/tmp/qemu-test.6OlDFH/tests/Makefile.include:1101:
>>> check-tests/qemu-iotests-quick.sh] Error 1
>>> gmake: *** Waiting for unfinished jobs....
>>> ---
>>>
>>> I'm not sure about the netbsd and openbsd images, they might need bash
>>> and gnu-sed as well.
>>
>> D'oh! Does anybody know what are the correct steps to update these images?
> 
> (1) make the OPENBSD subsystem maintainer care
> 
> (2) update <https://wiki.qemu.org/Hosts/BSD>
> 
> (3) download the image from download.patchew.org, boot it and update it
> 
> (4) upload the image to download.patchew.org
> 
> (5) update the "tests/vm/openbsd" script in the QEMU tree in sync
> (checksums, commands etc)

 Ed, Li-Wen,

any chance you could help with updating the FreeBSD image and the
tests/vm/freebsd script, so that we get more test covarage here?

 Kamil,

could you maybe help with the NetBSD image and the tests/vm/netbsd script?

 Brad,

could you please help with the OpenBSD image and the tests/vm/openbsd
script?

I think it would also be good to update the images to the latest
released versions, too...

And IIRC, there was also a thread recently that it would also be good to
have the "xz" tool in all of these images, too...

 Thanks,
  Thomas

Re: [Qemu-devel] Update *BSD images with gnu-sed and bash (was: [PATCH 4/6] cirrus / travis: Add gnu-sed and bash for macOS and FreeBSD)
Posted by Gerd Hoffmann 6 years, 9 months ago
> >> D'oh! Does anybody know what are the correct steps to update these images?
> > 
> > (1) make the OPENBSD subsystem maintainer care
> > 
> > (2) update <https://wiki.qemu.org/Hosts/BSD>
> > 
> > (3) download the image from download.patchew.org, boot it and update it
> > 
> > (4) upload the image to download.patchew.org
> > 
> > (5) update the "tests/vm/openbsd" script in the QEMU tree in sync
> > (checksums, commands etc)
> 
>  Ed, Li-Wen,
> 
> any chance you could help with updating the FreeBSD image and the
> tests/vm/freebsd script, so that we get more test covarage here?
> 
>  Kamil,
> 
> could you maybe help with the NetBSD image and the tests/vm/netbsd script?
> 
>  Brad,
> 
> could you please help with the OpenBSD image and the tests/vm/openbsd
> script?
> 
> I think it would also be good to update the images to the latest
> released versions, too...

FYI:  I'm working on revamping the tests/vm setup, creating the images
scripted on the developer machine.  That'll take download.patchew.org
out of the loop and makes adding packages as easy as patching a docker
script.

Going slower than I want due to /me being busy with other stuff.
Sneak preview at:
	https://git.kraxel.org/cgit/qemu/log/?h=sirius/test-vm-install

cheers,
  Gerd


Re: [Qemu-devel] Update *BSD images with gnu-sed and bash (was: [PATCH 4/6] cirrus / travis: Add gnu-sed and bash for macOS and FreeBSD)
Posted by Thomas Huth 6 years, 9 months ago
On 07/05/2019 10.32, Gerd Hoffmann wrote:
>>>> D'oh! Does anybody know what are the correct steps to update these images?
>>>
>>> (1) make the OPENBSD subsystem maintainer care
>>>
>>> (2) update <https://wiki.qemu.org/Hosts/BSD>
>>>
>>> (3) download the image from download.patchew.org, boot it and update it
>>>
>>> (4) upload the image to download.patchew.org
>>>
>>> (5) update the "tests/vm/openbsd" script in the QEMU tree in sync
>>> (checksums, commands etc)
>>
>>  Ed, Li-Wen,
>>
>> any chance you could help with updating the FreeBSD image and the
>> tests/vm/freebsd script, so that we get more test covarage here?
>>
>>  Kamil,
>>
>> could you maybe help with the NetBSD image and the tests/vm/netbsd script?
>>
>>  Brad,
>>
>> could you please help with the OpenBSD image and the tests/vm/openbsd
>> script?
>>
>> I think it would also be good to update the images to the latest
>> released versions, too...
> 
> FYI:  I'm working on revamping the tests/vm setup, creating the images
> scripted on the developer machine.  That'll take download.patchew.org
> out of the loop and makes adding packages as easy as patching a docker
> script.

Cool, that sounds like the best solution, indeed!

 Thanks,
  Thomas

Re: [Qemu-devel] Update *BSD images with gnu-sed and bash
Posted by Kamil Rytarowski 6 years, 9 months ago
On 06.05.2019 12:12, Thomas Huth wrote:
> On 26/04/2019 18.43, Laszlo Ersek wrote:
>> On 04/25/19 08:00, Thomas Huth wrote:
>>> On 24/04/2019 23.29, Wainer dos Santos Moschetta wrote:
>>>> Hello Thomas,
>>>>
>>>> On 04/24/2019 07:37 AM, Thomas Huth wrote:
>>>>> We are going to enable the qemu-iotests during "make check" again,
>>>>> and for running the iotests, we need bash and gnu-sed.
>>>> [...]
>>>> It needs to get the Freebsd image [1] updated too, in order to `make
>>>> BUILD_TARGET=check vm-build-freebsd` passes. Here it failed with:
>>>> ---
>>>> env: bash: No such file or directory
>>>> gmake: *** [/var/tmp/qemu-test.6OlDFH/tests/Makefile.include:1101:
>>>> check-tests/qemu-iotests-quick.sh] Error 1
>>>> gmake: *** Waiting for unfinished jobs....
>>>> ---
>>>>
>>>> I'm not sure about the netbsd and openbsd images, they might need bash
>>>> and gnu-sed as well.
>>>
>>> D'oh! Does anybody know what are the correct steps to update these images?
>>
>> (1) make the OPENBSD subsystem maintainer care
>>
>> (2) update <https://wiki.qemu.org/Hosts/BSD>
>>
>> (3) download the image from download.patchew.org, boot it and update it
>>
>> (4) upload the image to download.patchew.org
>>
>> (5) update the "tests/vm/openbsd" script in the QEMU tree in sync
>> (checksums, commands etc)
> 
>  Ed, Li-Wen,
> 
> any chance you could help with updating the FreeBSD image and the
> tests/vm/freebsd script, so that we get more test covarage here?
> 
>  Kamil,
> 
> could you maybe help with the NetBSD image and the tests/vm/netbsd script?
> 

Please be more specific what am I expected to do.

>  Brad,
> 
> could you please help with the OpenBSD image and the tests/vm/openbsd
> script?
> 
> I think it would also be good to update the images to the latest
> released versions, too...
> 
> And IIRC, there was also a thread recently that it would also be good to
> have the "xz" tool in all of these images, too...
> 
>  Thanks,
>   Thomas
> 


Re: [Qemu-devel] Update *BSD images with gnu-sed and bash
Posted by Thomas Huth 6 years, 9 months ago
On 08/05/2019 09.06, Kamil Rytarowski wrote:
> On 06.05.2019 12:12, Thomas Huth wrote:
[...]
>>  Kamil,
>>
>> could you maybe help with the NetBSD image and the tests/vm/netbsd script?
>>
> 
> Please be more specific what am I expected to do.

We have some VMs (including NetBSD) available that are used during
Peter's regression tests when somebody sends him a PULL requests. You
can run them also locally with:

 make BUILD_TARGET=check vm-build-netbsd

From time to time, we've got to update these images, either to a newer
version or to add some missing packages (like bash and gnu-sed in this
case).
However, many people (including me) don't have a clue about the various
*BSD flavours, so also no clue about how to update these images easily.
That's why I was hoping you could help here.

But looks like Gerd is already working on a way to generate these images
in a more automated way, so let's hope that he'll find some spare time
to finish that work soon.

 Thomas

Re: [Qemu-devel] Update *BSD images with gnu-sed and bash
Posted by Kamil Rytarowski 6 years, 9 months ago
On 08.05.2019 10:07, Thomas Huth wrote:
> On 08/05/2019 09.06, Kamil Rytarowski wrote:
>> On 06.05.2019 12:12, Thomas Huth wrote:
> [...]
>>>  Kamil,
>>>
>>> could you maybe help with the NetBSD image and the tests/vm/netbsd script?
>>>
>>
>> Please be more specific what am I expected to do.
> 
> We have some VMs (including NetBSD) available that are used during
> Peter's regression tests when somebody sends him a PULL requests. You
> can run them also locally with:
> 
>  make BUILD_TARGET=check vm-build-netbsd
> 
> From time to time, we've got to update these images, either to a newer
> version or to add some missing packages (like bash and gnu-sed in this
> case).

The process has been documented on wiki.

https://wiki.qemu.org/Hosts/BSD#NetBSD

> However, many people (including me) don't have a clue about the various
> *BSD flavours, so also no clue about how to update these images easily.
> That's why I was hoping you could help here.
> 
> But looks like Gerd is already working on a way to generate these images
> in a more automated way, so let's hope that he'll find some spare time
> to finish that work soon.
> 

I recommend to upgrade to 8.0.

One extra step is to disable PaX MPROTECT (tcg violates W^X):

This should be addressed in qemu with an extension flag to
mmap(2)/mremap(2). I still have this on my TODO list.

sysctl -w security.pax.mprotect.enabled=1

NetBSD 9.0 will be released sooner than later and soon after that
NetBSD-7.x will be EOL.

>  Thomas
>