To run basic jobs on custom runners, the environment needs to be
properly set up. The most common requirement is having the right
packages installed.
The playbook introduced here covers a number of different Linux
distributions and FreeBSD, and are intended to provide a reproducible
environment.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
docs/devel/ci.rst | 32 ++++
scripts/ci/setup/build-environment.yml | 220 +++++++++++++++++++++++++
scripts/ci/setup/inventory | 2 +
3 files changed, 254 insertions(+)
create mode 100644 scripts/ci/setup/build-environment.yml
create mode 100644 scripts/ci/setup/inventory
diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
index 41a4bbddad..208b5e399b 100644
--- a/docs/devel/ci.rst
+++ b/docs/devel/ci.rst
@@ -52,3 +52,35 @@ As a general rule, those newly added contributed jobs should run as
The precise minimum requirements and exact rules for machine
configuration documentation/scripts, and the success rate of jobs are
still to be defined.
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks. Start by adding your machines to
+the ``inventory`` file under ``scripts/ci/setup``, such as this::
+
+ [local]
+ fully.qualified.domain
+ other.machine.hostname
+
+You may need to set some variables in the inventory file itself. One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts. This would look like::
+
+ [local]
+ fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+ other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests. It covers a number of different Linux distributions and
+FreeBSD.
+
+To run the playbook, execute::
+
+ cd scripts/ci/setup
+ ansible-playbook -i inventory build-environment.yml
diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
new file mode 100644
index 0000000000..0e8894bca9
--- /dev/null
+++ b/scripts/ci/setup/build-environment.yml
@@ -0,0 +1,220 @@
+---
+- name: Installation of basic packages to build QEMU
+ hosts: all
+ tasks:
+ - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
+ apt:
+ update_cache: yes
+ # Originally from tests/docker/dockerfiles/ubuntu1804.docker
+ pkg:
+ - ccache
+ - clang
+ - gcc
+ - gettext
+ - git
+ - glusterfs-common
+ - libaio-dev
+ - libattr1-dev
+ - libbrlapi-dev
+ - libbz2-dev
+ - libcacard-dev
+ - libcap-ng-dev
+ - libcurl4-gnutls-dev
+ - libdrm-dev
+ - libepoxy-dev
+ - libfdt-dev
+ - libgbm-dev
+ - libgtk-3-dev
+ - libibverbs-dev
+ - libiscsi-dev
+ - libjemalloc-dev
+ - libjpeg-turbo8-dev
+ - liblzo2-dev
+ - libncurses5-dev
+ - libncursesw5-dev
+ - libnfs-dev
+ - libnss3-dev
+ - libnuma-dev
+ - libpixman-1-dev
+ - librados-dev
+ - librbd-dev
+ - librdmacm-dev
+ - libsasl2-dev
+ - libsdl2-dev
+ - libseccomp-dev
+ - libsnappy-dev
+ - libspice-protocol-dev
+ - libssh-dev
+ - libusb-1.0-0-dev
+ - libusbredirhost-dev
+ - libvdeplug-dev
+ - libvte-2.91-dev
+ - libzstd-dev
+ - make
+ - python3-yaml
+ - python3-sphinx
+ - sparse
+ - xfslibs-dev
+ state: present
+ when: "ansible_facts['distribution'] == 'Ubuntu'"
+
+ - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
+ apt:
+ update_cache: yes
+ pkg:
+ - libspice-server-dev
+ - libxen-dev
+ state: present
+ when:
+ - "ansible_facts['distribution'] == 'Ubuntu'"
+ - "ansible_facts['architecture'] != 's390x'"
+
+ - name: Install basic packages to build QEMU on FreeBSD 12.x
+ pkgng:
+ # This matches the packages on .cirrus.yml under the freebsd_12_task
+ name: bash,bison,curl,cyrus-sasl,git,glib,gmake,gnutls,gsed,nettle,perl5,pixman,pkgconf,png,usbredir
+ state: present
+ when: "ansible_facts['os_family'] == 'FreeBSD'"
+
+ - name: Install basic packages to build QEMU on Fedora 30, 31 and 32
+ dnf:
+ # Originally from tests/docker/dockerfiles/fedora.docker
+ name:
+ - SDL2-devel
+ - bc
+ - brlapi-devel
+ - bzip2
+ - bzip2-devel
+ - ccache
+ - clang
+ - cyrus-sasl-devel
+ - dbus-daemon
+ - device-mapper-multipath-devel
+ - diffutils
+ - findutils
+ - gcc
+ - gcc-c++
+ - genisoimage
+ - gettext
+ - git
+ - glib2-devel
+ - glusterfs-api-devel
+ - gnutls-devel
+ - gtk3-devel
+ - hostname
+ - libaio-devel
+ - libasan
+ - libattr-devel
+ - libblockdev-mpath-devel
+ - libcap-ng-devel
+ - libcurl-devel
+ - libepoxy-devel
+ - libfdt-devel
+ - libiscsi-devel
+ - libjpeg-devel
+ - libpmem-devel
+ - libpng-devel
+ - librbd-devel
+ - libseccomp-devel
+ - libssh-devel
+ - libubsan
+ - libudev-devel
+ - libusbx-devel
+ - libxml2-devel
+ - libzstd-devel
+ - llvm
+ - lzo-devel
+ - make
+ - mingw32-SDL2
+ - mingw32-bzip2
+ - mingw32-curl
+ - mingw32-glib2
+ - mingw32-gmp
+ - mingw32-gnutls
+ - mingw32-gtk3
+ - mingw32-libjpeg-turbo
+ - mingw32-libpng
+ - mingw32-libtasn1
+ - mingw32-nettle
+ - mingw32-nsis
+ - mingw32-pixman
+ - mingw32-pkg-config
+ - mingw64-SDL2
+ - mingw64-bzip2
+ - mingw64-curl
+ - mingw64-glib2
+ - mingw64-gmp
+ - mingw64-gnutls
+ - mingw64-gtk3
+ - mingw64-libjpeg-turbo
+ - mingw64-libpng
+ - mingw64-libtasn1
+ - mingw64-nettle
+ - mingw64-pixman
+ - mingw64-pkg-config
+ - ncurses-devel
+ - nettle-devel
+ - nss-devel
+ - numactl-devel
+ - perl
+ - perl-Test-Harness
+ - pixman-devel
+ - python3
+ - python3-PyYAML
+ - python3-numpy
+ - python3-opencv
+ - python3-pillow
+ - python3-pip
+ - python3-sphinx
+ - python3-virtualenv
+ - rdma-core-devel
+ - snappy-devel
+ - sparse
+ - spice-server-devel
+ - systemd-devel
+ - systemtap-sdt-devel
+ - tar
+ - tesseract
+ - tesseract-langpack-eng
+ - usbredir-devel
+ - virglrenderer-devel
+ - vte291-devel
+ - which
+ - xen-devel
+ - zlib-devel
+ state: present
+ when: "ansible_facts['distribution'] == 'Fedora'"
+
+ - name: Install basic packages to build QEMU on CentOS 8
+ dnf:
+ # Originally from tests/docker/dockerfiles/centos8.docker
+ name:
+ - SDL-devel
+ - bzip2
+ - bzip2-devel
+ - dbus-daemon
+ - gcc
+ - gcc-c++
+ - genisoimage
+ - gettext
+ - git
+ - glib2-devel
+ - libaio-devel
+ - libepoxy-devel
+ - libgcrypt-devel
+ - lzo-devel
+ - make
+ - mesa-libEGL-devel
+ - nettle-devel
+ - perl-Test-Harness
+ - pixman-devel
+ - python36
+ - rdma-core-devel
+ - spice-glib-devel
+ - spice-server
+ - tar
+ - zlib-devel
+ state: present
+ when:
+ - "ansible_facts['distribution'] == 'CentOS'"
+ - "ansible_facts['distribution_major_version'] == '8'"
diff --git a/scripts/ci/setup/inventory b/scripts/ci/setup/inventory
new file mode 100644
index 0000000000..8bb7ba6b33
--- /dev/null
+++ b/scripts/ci/setup/inventory
@@ -0,0 +1,2 @@
+[local]
+localhost
--
2.25.4
On Wed, Oct 14, 2020 at 01:21:38AM -0400, Cleber Rosa wrote: > To run basic jobs on custom runners, the environment needs to be > properly set up. The most common requirement is having the right > packages installed. > > The playbook introduced here covers a number of different Linux > distributions and FreeBSD, and are intended to provide a reproducible > environment. > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > --- > docs/devel/ci.rst | 32 ++++ > scripts/ci/setup/build-environment.yml | 220 +++++++++++++++++++++++++ > scripts/ci/setup/inventory | 2 + > 3 files changed, 254 insertions(+) > create mode 100644 scripts/ci/setup/build-environment.yml > create mode 100644 scripts/ci/setup/inventory > > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst > index 41a4bbddad..208b5e399b 100644 > --- a/docs/devel/ci.rst > +++ b/docs/devel/ci.rst > @@ -52,3 +52,35 @@ As a general rule, those newly added contributed jobs should run as > The precise minimum requirements and exact rules for machine > configuration documentation/scripts, and the success rate of jobs are > still to be defined. > + > +Machine Setup Howto > +------------------- > + > +For all Linux based systems, the setup can be mostly automated by the > +execution of two Ansible playbooks. Start by adding your machines to > +the ``inventory`` file under ``scripts/ci/setup``, such as this:: > + > + [local] > + fully.qualified.domain > + other.machine.hostname > + > +You may need to set some variables in the inventory file itself. One > +very common need is to tell Ansible to use a Python 3 interpreter on > +those hosts. This would look like:: > + > + [local] > + fully.qualified.domain ansible_python_interpreter=/usr/bin/python3 > + other.machine.hostname ansible_python_interpreter=/usr/bin/python3 > + > +Build environment > +~~~~~~~~~~~~~~~~~ > + > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will > +set up machines with the environment needed to perform builds and run > +QEMU tests. It covers a number of different Linux distributions and > +FreeBSD. > + > +To run the playbook, execute:: > + > + cd scripts/ci/setup > + ansible-playbook -i inventory build-environment.yml > diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml > new file mode 100644 > index 0000000000..0e8894bca9 > --- /dev/null > +++ b/scripts/ci/setup/build-environment.yml > @@ -0,0 +1,220 @@ > +--- > +- name: Installation of basic packages to build QEMU > + hosts: all > + tasks: > + - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04 > + apt: > + update_cache: yes > + # Originally from tests/docker/dockerfiles/ubuntu1804.docker > + pkg: > + - ccache > + - clang > + - gcc > + - gettext > + - git > + - glusterfs-common > + - libaio-dev > + - libattr1-dev > + - libbrlapi-dev > + - libbz2-dev > + - libcacard-dev > + - libcap-ng-dev > + - libcurl4-gnutls-dev > + - libdrm-dev > + - libepoxy-dev > + - libfdt-dev > + - libgbm-dev > + - libgtk-3-dev > + - libibverbs-dev > + - libiscsi-dev > + - libjemalloc-dev > + - libjpeg-turbo8-dev > + - liblzo2-dev > + - libncurses5-dev > + - libncursesw5-dev > + - libnfs-dev > + - libnss3-dev > + - libnuma-dev > + - libpixman-1-dev > + - librados-dev > + - librbd-dev > + - librdmacm-dev > + - libsasl2-dev > + - libsdl2-dev > + - libseccomp-dev > + - libsnappy-dev > + - libspice-protocol-dev > + - libssh-dev > + - libusb-1.0-0-dev > + - libusbredirhost-dev > + - libvdeplug-dev > + - libvte-2.91-dev > + - libzstd-dev > + - make > + - python3-yaml > + - python3-sphinx > + - sparse > + - xfslibs-dev This needs updating to add meson, and with Paolo's series today you might as well go ahead and add ninja-build immediately too https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg04025.html Same for all the other distro package lists. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Oct 14, 2020 at 06:30:09PM +0100, Daniel P. Berrangé wrote: > On Wed, Oct 14, 2020 at 01:21:38AM -0400, Cleber Rosa wrote: > > To run basic jobs on custom runners, the environment needs to be > > properly set up. The most common requirement is having the right > > packages installed. > > > > The playbook introduced here covers a number of different Linux > > distributions and FreeBSD, and are intended to provide a reproducible > > environment. > > > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > > --- > > docs/devel/ci.rst | 32 ++++ > > scripts/ci/setup/build-environment.yml | 220 +++++++++++++++++++++++++ > > scripts/ci/setup/inventory | 2 + > > 3 files changed, 254 insertions(+) > > create mode 100644 scripts/ci/setup/build-environment.yml > > create mode 100644 scripts/ci/setup/inventory > > > > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst > > index 41a4bbddad..208b5e399b 100644 > > --- a/docs/devel/ci.rst > > +++ b/docs/devel/ci.rst > > @@ -52,3 +52,35 @@ As a general rule, those newly added contributed jobs should run as > > The precise minimum requirements and exact rules for machine > > configuration documentation/scripts, and the success rate of jobs are > > still to be defined. > > + > > +Machine Setup Howto > > +------------------- > > + > > +For all Linux based systems, the setup can be mostly automated by the > > +execution of two Ansible playbooks. Start by adding your machines to > > +the ``inventory`` file under ``scripts/ci/setup``, such as this:: > > + > > + [local] > > + fully.qualified.domain > > + other.machine.hostname > > + > > +You may need to set some variables in the inventory file itself. One > > +very common need is to tell Ansible to use a Python 3 interpreter on > > +those hosts. This would look like:: > > + > > + [local] > > + fully.qualified.domain ansible_python_interpreter=/usr/bin/python3 > > + other.machine.hostname ansible_python_interpreter=/usr/bin/python3 > > + > > +Build environment > > +~~~~~~~~~~~~~~~~~ > > + > > +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will > > +set up machines with the environment needed to perform builds and run > > +QEMU tests. It covers a number of different Linux distributions and > > +FreeBSD. > > + > > +To run the playbook, execute:: > > + > > + cd scripts/ci/setup > > + ansible-playbook -i inventory build-environment.yml > > diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml > > new file mode 100644 > > index 0000000000..0e8894bca9 > > --- /dev/null > > +++ b/scripts/ci/setup/build-environment.yml > > @@ -0,0 +1,220 @@ > > +--- > > +- name: Installation of basic packages to build QEMU > > + hosts: all > > + tasks: > > + - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04 > > + apt: > > + update_cache: yes > > + # Originally from tests/docker/dockerfiles/ubuntu1804.docker > > + pkg: > > + - ccache > > + - clang > > + - gcc > > + - gettext > > + - git > > + - glusterfs-common > > + - libaio-dev > > + - libattr1-dev > > + - libbrlapi-dev > > + - libbz2-dev > > + - libcacard-dev > > + - libcap-ng-dev > > + - libcurl4-gnutls-dev > > + - libdrm-dev > > + - libepoxy-dev > > + - libfdt-dev > > + - libgbm-dev > > + - libgtk-3-dev > > + - libibverbs-dev > > + - libiscsi-dev > > + - libjemalloc-dev > > + - libjpeg-turbo8-dev > > + - liblzo2-dev > > + - libncurses5-dev > > + - libncursesw5-dev > > + - libnfs-dev > > + - libnss3-dev > > + - libnuma-dev > > + - libpixman-1-dev > > + - librados-dev > > + - librbd-dev > > + - librdmacm-dev > > + - libsasl2-dev > > + - libsdl2-dev > > + - libseccomp-dev > > + - libsnappy-dev > > + - libspice-protocol-dev > > + - libssh-dev > > + - libusb-1.0-0-dev > > + - libusbredirhost-dev > > + - libvdeplug-dev > > + - libvte-2.91-dev > > + - libzstd-dev > > + - make > > + - python3-yaml > > + - python3-sphinx > > + - sparse > > + - xfslibs-dev > > This needs updating to add meson, and with Paolo's series today you > might as well go ahead and add ninja-build immediately too > > https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg04025.html > > Same for all the other distro package lists. > > Good point, will do. Thanks! - Cleber. > Regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > >
On Wed, Oct 14, 2020 at 02:59:58PM -0400, Cleber Rosa wrote: > On Wed, Oct 14, 2020 at 06:30:09PM +0100, Daniel P. Berrangé wrote: > > > > This needs updating to add meson, and with Paolo's series today you > > might as well go ahead and add ninja-build immediately too > > I replied too quickly, but allow me to get this right: meson is *not* included in the dockerfiles (and other similar configurations), and all setups I found rely on the submodule. Are suggesting to add meson and diverge from the dockerfiles? > > https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg04025.html > > ^ I'll add meson according to this, of course. Thanks, - Cleber. > > Same for all the other distro package lists. > > > >
On Wed, Oct 14, 2020 at 03:19:47PM -0400, Cleber Rosa wrote: > On Wed, Oct 14, 2020 at 02:59:58PM -0400, Cleber Rosa wrote: > > On Wed, Oct 14, 2020 at 06:30:09PM +0100, Daniel P. Berrangé wrote: > > > > > > This needs updating to add meson, and with Paolo's series today you > > > might as well go ahead and add ninja-build immediately too > > > > > I replied too quickly, but allow me to get this right: meson is *not* > included in the dockerfiles (and other similar configurations), and > all setups I found rely on the submodule. Are suggesting to add meson > and diverge from the dockerfiles? Doh, right, I forgot that we use the submodule for now, since we need such a new meson. So ignore this... > > > https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg04025.html > > > > > ^ I'll add meson according to this, of course. Just ninja is needed Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Oct 15, 2020 at 09:29:40AM +0100, Daniel P. Berrangé wrote: > On Wed, Oct 14, 2020 at 03:19:47PM -0400, Cleber Rosa wrote: > > On Wed, Oct 14, 2020 at 02:59:58PM -0400, Cleber Rosa wrote: > > > On Wed, Oct 14, 2020 at 06:30:09PM +0100, Daniel P. Berrangé wrote: > > > > > > > > This needs updating to add meson, and with Paolo's series today you > > > > might as well go ahead and add ninja-build immediately too > > > > > > > > I replied too quickly, but allow me to get this right: meson is *not* > > included in the dockerfiles (and other similar configurations), and > > all setups I found rely on the submodule. Are suggesting to add meson > > and diverge from the dockerfiles? > > Doh, right, I forgot that we use the submodule for now, since we need > such a new meson. So ignore this... > > > > > https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg04025.html > > > > > > > > ^ I'll add meson according to this, of course. > > Just ninja is needed > Right, I meant ninja there! Thanks, - Cleber.
© 2016 - 2026 Red Hat, Inc.