1. Patchew
  2. QEMU
  3. [PATCH 0/3] ci: use cirrus-run to utilize Cirrus CI from GitLab CI
  4. View patch

[PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run

Daniel P. Berrangé posted 3 patches 4 years, 7 months ago
[PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run
Posted by Daniel P. Berrangé 4 years, 7 months ago 
This adds support for running 4 jobs via Cirrus CI runners:

 * FreeBSD 12
 * FreeBSD 13
 * macOS 11 with default XCode
 * macOS 11 with latest XCode

The gitlab job uses a container published by the libvirt-ci
project (https://gitlab.com/libvirt/libvirt-ci) that contains
the 'cirrus-run' command. This accepts a short yaml file that
describes a single Cirrus CI job, runs it using the Cirrus CI
REST API, and reports any output to the console.

In this way Cirrus CI is effectively working as an indirect
custom runner for GitLab CI pipelines. The key benefit is that
Cirrus CI job results affect the GitLab CI pipeline result and
so the user only has look at one CI dashboard.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/cirrus.yml             | 103 ++++++++++++++++++++++++++++
 .gitlab-ci.d/cirrus/README.rst      |  54 +++++++++++++++
 .gitlab-ci.d/cirrus/build.yml       |  35 ++++++++++
 .gitlab-ci.d/cirrus/freebsd-12.vars |  13 ++++
 .gitlab-ci.d/cirrus/freebsd-13.vars |  13 ++++
 .gitlab-ci.d/cirrus/macos-11.vars   |  15 ++++
 .gitlab-ci.d/qemu-project.yml       |   1 +
 7 files changed, 234 insertions(+)
 create mode 100644 .gitlab-ci.d/cirrus.yml
 create mode 100644 .gitlab-ci.d/cirrus/README.rst
 create mode 100644 .gitlab-ci.d/cirrus/build.yml
 create mode 100644 .gitlab-ci.d/cirrus/freebsd-12.vars
 create mode 100644 .gitlab-ci.d/cirrus/freebsd-13.vars
 create mode 100644 .gitlab-ci.d/cirrus/macos-11.vars

diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
new file mode 100644
index 0000000000..d7b4cce79b
--- /dev/null
+++ b/.gitlab-ci.d/cirrus.yml
@@ -0,0 +1,103 @@
+# Jobs that we delegate to Cirrus CI because they require an operating
+# system other than Linux. These jobs will only run if the required
+# setup has been performed on the GitLab account.
+#
+# The Cirrus CI configuration is generated by replacing target-specific
+# variables in a generic template: some of these variables are provided
+# when the GitLab CI job is defined, others are taken from a shell
+# snippet generated using lcitool.
+#
+# Note that the $PATH environment variable has to be treated with
+# special care, because we can't just override it at the GitLab CI job
+# definition level or we risk breaking it completely.
+.cirrus_build_job:
+  stage: build
+  image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
+  needs: []
+  script:
+    - source .gitlab-ci.d/cirrus/$NAME.vars
+    - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
+          -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
+          -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
+          -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
+          -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
+          -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
+          -e "s|[@]CIRRUS_VM_CPUS@|$CIRRUS_VM_CPUS|g"
+          -e "s|[@]CIRRUS_VM_RAM@|$CIRRUS_VM_RAM|g"
+          -e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g"
+          -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
+          -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g"
+          -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g"
+          -e "s|[@]PKGS@|$PKGS|g"
+          -e "s|[@]MAKE@|$MAKE|g"
+          -e "s|[@]PYTHON@|$PYTHON|g"
+          -e "s|[@]PIP3@|$PIP3|g"
+          -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g"
+          -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g"
+          -e "s|[@]TEST_TARGETSS@|$TEST_TARGETSS|g"
+      <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
+    - cat .gitlab-ci.d/cirrus/$NAME.yml
+    - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
+  rules:
+    - if: "$TEMPORARILY_DISABLED"
+      allow_failure: true
+    - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
+
+x64-freebsd-12-build:
+  extends: .cirrus_build_job
+  variables:
+    NAME: freebsd-12
+    CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
+    CIRRUS_VM_IMAGE_SELECTOR: image_family
+    CIRRUS_VM_IMAGE_NAME: freebsd-12-2
+    CIRRUS_VM_CPUS: 8
+    CIRRUS_VM_RAM: 8G
+    UPDATE_COMMAND: pkg update
+    INSTALL_COMMAND: pkg install -y
+    # TODO: Enable gnutls again once FreeBSD's libtasn1 got fixed
+    # See: https://gitlab.com/gnutls/libtasn1/-/merge_requests/71
+    CONFIGURE_ARGS: --disable-gnutls
+    TEST_TARGETS: check
+
+x64-freebsd-13-build:
+  extends: .cirrus_build_job
+  variables:
+    NAME: freebsd-13
+    CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
+    CIRRUS_VM_IMAGE_SELECTOR: image_family
+    CIRRUS_VM_IMAGE_NAME: freebsd-13-0
+    CIRRUS_VM_CPUS: 8
+    CIRRUS_VM_RAM: 8G
+    UPDATE_COMMAND: pkg update
+    INSTALL_COMMAND: pkg install -y
+    TEST_TARGETS: check
+
+x64-macos-11-base-build:
+  extends: .cirrus_build_job
+  variables:
+    NAME: macos-11
+    CIRRUS_VM_INSTANCE_TYPE: osx_instance
+    CIRRUS_VM_IMAGE_SELECTOR: image
+    CIRRUS_VM_IMAGE_NAME: big-sur-base
+    CIRRUS_VM_CPUS: 12
+    CIRRUS_VM_RAM: 24G
+    UPDATE_COMMAND: brew update
+    INSTALL_COMMAND: brew install
+    PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
+    PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
+    TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
+
+x64-macos-11-xcode-build:
+  extends: .cirrus_build_job
+  variables:
+    NAME: macos-11
+    CIRRUS_VM_INSTANCE_TYPE: osx_instance
+    CIRRUS_VM_IMAGE_SELECTOR: image
+    CIRRUS_VM_IMAGE_NAME: big-sur-xcode
+    CIRRUS_VM_CPUS: 12
+    CIRRUS_VM_RAM: 24G
+    UPDATE_COMMAND: brew update
+    INSTALL_COMMAND: brew install
+    PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
+    PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
+    TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
diff --git a/.gitlab-ci.d/cirrus/README.rst b/.gitlab-ci.d/cirrus/README.rst
new file mode 100644
index 0000000000..657b0706d7
--- /dev/null
+++ b/.gitlab-ci.d/cirrus/README.rst
@@ -0,0 +1,54 @@
+Cirrus CI integration
+=====================
+
+GitLab CI shared runners only provide a docker environment running on Linux.
+While it is possible to provide private runners for non-Linux platforms this
+is not something most contributors/maintainers will wish to do.
+
+To work around this limitation, we take advantage of `Cirrus CI`_'s free
+offering: more specifically, we use the `cirrus-run`_ script to trigger Cirrus
+CI jobs from GitLab CI jobs so that Cirrus CI job output is integrated into
+the main GitLab CI pipeline dashboard.
+
+There is, however, some one-time setup required. If you want FreeBSD and macOS
+builds to happen when you push to your GitLab repository, you need to
+
+* set up a GitHub repository for the project, eg. ``yourusername/qemu``.
+  This repository needs to exist for cirrus-run to work, but it doesn't need to
+  be kept up to date, so you can create it and then forget about it;
+
+* enable the `Cirrus CI GitHub app`_  for your GitHub account;
+
+* sign up for Cirrus CI. It's enough to log into the website using your GitHub
+  account;
+
+* grab an API token from the `Cirrus CI settings`_ page;
+
+* it may be necessary to push an empty ``.cirrus.yml`` file to your github fork
+  for Cirrus CI to properly recognize the project. You can check whether
+  Cirrus CI knows about your project by navigating to:
+
+  ``https://cirrus-ci.com/yourusername/qemu``
+
+* in the *CI/CD / Variables* section of the settings page for your GitLab
+  repository, create two new variables:
+
+  * ``CIRRUS_GITHUB_REPO``, containing the name of the GitHub repository
+    created earlier, eg. ``yourusername/qemu``;
+
+  * ``CIRRUS_API_TOKEN``, containing the Cirrus CI API token generated earlier.
+    This variable **must** be marked as *Masked*, because anyone with knowledge
+    of it can impersonate you as far as Cirrus CI is concerned.
+
+  Neither of these variables should be marked as *Protected*, because in
+  general you'll want to be able to trigger Cirrus CI builds from non-protected
+  branches.
+
+Once this one-time setup is complete, you can just keep pushing to your GitLab
+repository as usual and you'll automatically get the additional CI coverage.
+
+
+.. _Cirrus CI GitHub app: https://github.com/marketplace/cirrus-ci
+.. _Cirrus CI settings: https://cirrus-ci.com/settings/profile/
+.. _Cirrus CI: https://cirrus-ci.com/
+.. _cirrus-run: https://github.com/sio/cirrus-run/
diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
new file mode 100644
index 0000000000..857bdc5536
--- /dev/null
+++ b/.gitlab-ci.d/cirrus/build.yml
@@ -0,0 +1,35 @@
+@CIRRUS_VM_INSTANCE_TYPE@:
+  @CIRRUS_VM_IMAGE_SELECTOR@: @CIRRUS_VM_IMAGE_NAME@
+  cpu: @CIRRUS_VM_CPUS@
+  memory: @CIRRUS_VM_RAM@
+
+env:
+  CIRRUS_CLONE_DEPTH: 1
+  CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@"
+  CI_COMMIT_REF_NAME: "@CI_COMMIT_REF_NAME@"
+  CI_COMMIT_SHA: "@CI_COMMIT_SHA@"
+  PATH: "@PATH@"
+  PKG_CONFIG_PATH: "@PKG_CONFIG_PATH@"
+  PYTHON: "@PYTHON@"
+  MAKE: "@MAKE@"
+  CONFIGURE_ARGS: "@CONFIGURE_ARGS@"
+
+build_task:
+  install_script:
+    - @UPDATE_COMMAND@
+    - @INSTALL_COMMAND@ @PKGS@
+    - if test -n "@PYPI_PKGS@" ; then @PIP3@ install @PYPI_PKGS@ ; fi
+  clone_script:
+    - git clone --depth 100 "$CI_REPOSITORY_URL" .
+    - git fetch origin "$CI_COMMIT_REF_NAME"
+    - git reset --hard "$CI_COMMIT_SHA"
+  build_script:
+    - mkdir build
+    - cd build
+    - ../configure --enable-werror $CONFIGURE_ARGS
+      || { cat config.log meson-logs/meson-log.txt; exit 1; }
+    - $MAKE -j$(sysctl -n hw.ncpu)
+    - for TARGET in $TEST_TARGETS ;
+      do
+        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
+      done
diff --git a/.gitlab-ci.d/cirrus/freebsd-12.vars b/.gitlab-ci.d/cirrus/freebsd-12.vars
new file mode 100644
index 0000000000..5e51e50ea9
--- /dev/null
+++ b/.gitlab-ci.d/cirrus/freebsd-12.vars
@@ -0,0 +1,13 @@
+# THIS FILE WAS AUTO-GENERATED
+#
+#  $ lcitool variables freebsd-12 qemu
+#
+# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1
+
+PACKAGING_COMMAND='pkg'
+CCACHE='/usr/local/bin/ccache'
+MAKE='/usr/local/bin/gmake'
+NINJA='/usr/local/bin/ninja'
+PYTHON='/usr/local/bin/python3'
+PIP3='/usr/local/bin/pip-3.7'
+PKGS='alsa-lib bash bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage ctags curl cyrus-sasl dbus diffutils gettext git glib gmake gnutls gsed gtk3 libepoxy libffi libgcrypt libjpeg-turbo libnfs libspice-server libssh libtasn1 libxml2 llvm lttng-ust lzo2 meson ncurses nettle ninja opencv p5-Test-Harness perl5 pixman pkgconf png py37-numpy py37-pillow py37-pip py37-sphinx py37-sphinx_rtd_theme py37-virtualenv py37-yaml python3 rpm2cpio sdl2 sdl2_image snappy spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
diff --git a/.gitlab-ci.d/cirrus/freebsd-13.vars b/.gitlab-ci.d/cirrus/freebsd-13.vars
new file mode 100644
index 0000000000..07716833d9
--- /dev/null
+++ b/.gitlab-ci.d/cirrus/freebsd-13.vars
@@ -0,0 +1,13 @@
+# THIS FILE WAS AUTO-GENERATED
+#
+#  $ lcitool variables freebsd-13 qemu
+#
+# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1
+
+PACKAGING_COMMAND='pkg'
+CCACHE='/usr/local/bin/ccache'
+MAKE='/usr/local/bin/gmake'
+NINJA='/usr/local/bin/ninja'
+PYTHON='/usr/local/bin/python3'
+PIP3='/usr/local/bin/pip-3.7'
+PKGS='alsa-lib bash bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage ctags curl cyrus-sasl dbus diffutils gettext git glib gmake gnutls gsed gtk3 libepoxy libffi libgcrypt libjpeg-turbo libnfs libspice-server libssh libtasn1 libxml2 llvm lttng-ust lzo2 meson ncurses nettle ninja opencv p5-Test-Harness perl5 pixman pkgconf png py37-numpy py37-pillow py37-pip py37-sphinx py37-sphinx_rtd_theme py37-virtualenv py37-yaml python3 rpm2cpio sdl2 sdl2_image snappy spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
diff --git a/.gitlab-ci.d/cirrus/macos-11.vars b/.gitlab-ci.d/cirrus/macos-11.vars
new file mode 100644
index 0000000000..cbec8a44a3
--- /dev/null
+++ b/.gitlab-ci.d/cirrus/macos-11.vars
@@ -0,0 +1,15 @@
+# THIS FILE WAS AUTO-GENERATED
+#
+#  $ lcitool variables macos-11 qemu
+#
+# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1
+
+PACKAGING_COMMAND='brew'
+CCACHE='/usr/local/bin/ccache'
+MAKE='/usr/local/bin/gmake'
+NINJA='/usr/local/bin/ninja'
+PYTHON='/usr/local/bin/python3'
+PIP3='/usr/local/bin/pip3'
+PKGS='bash bc bzip2 capstone ccache cpanminus ctags curl dbus diffutils gcovr gettext git glib gnu-sed gnutls gtk+3 jemalloc jpeg-turbo libepoxy libffi libgcrypt libiscsi libnfs libpng libslirp libssh libtasn1 libusb libxml2 llvm lzo make meson ncurses nettle ninja perl pixman pkg-config python3 rpm2cpio sdl2 sdl2_image snappy sparse spice-protocol tesseract texinfo usbredir vde vte3 zlib zstd'
+PYPI_PKGS='PyYAML numpy pillow sphinx sphinx-rtd-theme virtualenv'
+CPAN_PKGS='Test::Harness'
diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
index 64cb2ba1da..35d4e62c12 100644
--- a/.gitlab-ci.d/qemu-project.yml
+++ b/.gitlab-ci.d/qemu-project.yml
@@ -9,3 +9,4 @@ include:
   - local: '/.gitlab-ci.d/crossbuilds.yml'
   - local: '/.gitlab-ci.d/buildtest.yml'
   - local: '/.gitlab-ci.d/static_checks.yml'
+  - local: '/.gitlab-ci.d/cirrus.yml'
-- 
2.31.1
Re: [PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run
Posted by Thomas Huth 4 years, 7 months ago 
On 25/06/2021 19.22, Daniel P. Berrangé wrote:
> This adds support for running 4 jobs via Cirrus CI runners:
> 
>   * FreeBSD 12
>   * FreeBSD 13
>   * macOS 11 with default XCode
>   * macOS 11 with latest XCode
> 
> The gitlab job uses a container published by the libvirt-ci
> project (https://gitlab.com/libvirt/libvirt-ci) that contains
> the 'cirrus-run' command. This accepts a short yaml file that
> describes a single Cirrus CI job, runs it using the Cirrus CI
> REST API, and reports any output to the console.
> 
> In this way Cirrus CI is effectively working as an indirect
> custom runner for GitLab CI pipelines. The key benefit is that
> Cirrus CI job results affect the GitLab CI pipeline result and
> so the user only has look at one CI dashboard.

Cool, thanks for tackling this!

> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/cirrus.yml             | 103 ++++++++++++++++++++++++++++
>   .gitlab-ci.d/cirrus/README.rst      |  54 +++++++++++++++
>   .gitlab-ci.d/cirrus/build.yml       |  35 ++++++++++
>   .gitlab-ci.d/cirrus/freebsd-12.vars |  13 ++++
>   .gitlab-ci.d/cirrus/freebsd-13.vars |  13 ++++
>   .gitlab-ci.d/cirrus/macos-11.vars   |  15 ++++
>   .gitlab-ci.d/qemu-project.yml       |   1 +
>   7 files changed, 234 insertions(+)
>   create mode 100644 .gitlab-ci.d/cirrus.yml
>   create mode 100644 .gitlab-ci.d/cirrus/README.rst
>   create mode 100644 .gitlab-ci.d/cirrus/build.yml
>   create mode 100644 .gitlab-ci.d/cirrus/freebsd-12.vars
>   create mode 100644 .gitlab-ci.d/cirrus/freebsd-13.vars
>   create mode 100644 .gitlab-ci.d/cirrus/macos-11.vars
> 
> diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> new file mode 100644
> index 0000000000..d7b4cce79b
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus.yml
> @@ -0,0 +1,103 @@
> +# Jobs that we delegate to Cirrus CI because they require an operating
> +# system other than Linux. These jobs will only run if the required
> +# setup has been performed on the GitLab account.
> +#
> +# The Cirrus CI configuration is generated by replacing target-specific
> +# variables in a generic template: some of these variables are provided
> +# when the GitLab CI job is defined, others are taken from a shell
> +# snippet generated using lcitool.
> +#
> +# Note that the $PATH environment variable has to be treated with
> +# special care, because we can't just override it at the GitLab CI job
> +# definition level or we risk breaking it completely.
> +.cirrus_build_job:
> +  stage: build
> +  image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
> +  needs: []
> +  script:
> +    - source .gitlab-ci.d/cirrus/$NAME.vars
> +    - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
> +          -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
> +          -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
> +          -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
> +          -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
> +          -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
> +          -e "s|[@]CIRRUS_VM_CPUS@|$CIRRUS_VM_CPUS|g"
> +          -e "s|[@]CIRRUS_VM_RAM@|$CIRRUS_VM_RAM|g"
> +          -e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g"
> +          -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
> +          -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g"
> +          -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g"
> +          -e "s|[@]PKGS@|$PKGS|g"
> +          -e "s|[@]MAKE@|$MAKE|g"
> +          -e "s|[@]PYTHON@|$PYTHON|g"
> +          -e "s|[@]PIP3@|$PIP3|g"
> +          -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g"
> +          -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g"
> +          -e "s|[@]TEST_TARGETSS@|$TEST_TARGETSS|g"
> +      <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
> +    - cat .gitlab-ci.d/cirrus/$NAME.yml
> +    - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
> +  rules:
> +    - if: "$TEMPORARILY_DISABLED"
> +      allow_failure: true
> +    - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
> +
> +x64-freebsd-12-build:
> +  extends: .cirrus_build_job
> +  variables:
> +    NAME: freebsd-12
> +    CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
> +    CIRRUS_VM_IMAGE_SELECTOR: image_family
> +    CIRRUS_VM_IMAGE_NAME: freebsd-12-2
> +    CIRRUS_VM_CPUS: 8
> +    CIRRUS_VM_RAM: 8G
> +    UPDATE_COMMAND: pkg update
> +    INSTALL_COMMAND: pkg install -y
> +    # TODO: Enable gnutls again once FreeBSD's libtasn1 got fixed
> +    # See: https://gitlab.com/gnutls/libtasn1/-/merge_requests/71
> +    CONFIGURE_ARGS: --disable-gnutls
> +    TEST_TARGETS: check
> +
> +x64-freebsd-13-build:
> +  extends: .cirrus_build_job
> +  variables:
> +    NAME: freebsd-13
> +    CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
> +    CIRRUS_VM_IMAGE_SELECTOR: image_family
> +    CIRRUS_VM_IMAGE_NAME: freebsd-13-0
> +    CIRRUS_VM_CPUS: 8
> +    CIRRUS_VM_RAM: 8G
> +    UPDATE_COMMAND: pkg update
> +    INSTALL_COMMAND: pkg install -y
> +    TEST_TARGETS: check
> +
> +x64-macos-11-base-build:
> +  extends: .cirrus_build_job
> +  variables:
> +    NAME: macos-11
> +    CIRRUS_VM_INSTANCE_TYPE: osx_instance
> +    CIRRUS_VM_IMAGE_SELECTOR: image
> +    CIRRUS_VM_IMAGE_NAME: big-sur-base
> +    CIRRUS_VM_CPUS: 12
> +    CIRRUS_VM_RAM: 24G
> +    UPDATE_COMMAND: brew update
> +    INSTALL_COMMAND: brew install
> +    PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
> +    PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
> +    TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
> +
> +x64-macos-11-xcode-build:
> +  extends: .cirrus_build_job
> +  variables:
> +    NAME: macos-11
> +    CIRRUS_VM_INSTANCE_TYPE: osx_instance
> +    CIRRUS_VM_IMAGE_SELECTOR: image
> +    CIRRUS_VM_IMAGE_NAME: big-sur-xcode
> +    CIRRUS_VM_CPUS: 12
> +    CIRRUS_VM_RAM: 24G
> +    UPDATE_COMMAND: brew update
> +    INSTALL_COMMAND: brew install
> +    PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
> +    PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
> +    TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64

Not directly related to your patch, but I wonder whether we really gain much 
by having two macos build jobs in our CI ... they seem to be very similar, 
e.g. compare the output of the "configure" step of the "macos" and the 
"macos_xcode" job here:

  https://cirrus-ci.com/build/4919300914937856

The diff basically looks like this:

diff -u /tmp/macos.txt /tmp/macos_xcode.txt
--- /tmp/macos.txt	2021-06-28 09:18:48.218698482 +0200
+++ /tmp/macos_xcode.txt	2021-06-28 09:21:01.811348076 +0200
@@ -1,6 +1,6 @@
  mkdir build
  cd build
-../configure --python=/usr/local/bin/python3 --enable-werror 
--extra-cflags='-Wno-error=deprecated-declarations' || { cat config.log 
meson-logs/meson-log.txt; exit 1; }
+../configure --extra-cflags='-Wno-error=deprecated-declarations' 
--enable-modules --enable-werror --cc=clang || { cat config.log 
meson-logs/meson-log.txt; exit 1; }
  Disabling PIE due to missing toolchain support
  Submodule 'capstone' (https://gitlab.com/qemu-project/capstone.git) 
registered for path 'capstone'
  Submodule 'dtc' (https://gitlab.com/qemu-project/dtc.git) registered for 
path 'dtc'
@@ -23,18 +23,18 @@
  Build type: native build
  Project name: qemu
  Project version: 6.0.50
-C compiler for the host machine: cc (clang 12.0.0 "Apple clang version 
12.0.0 (clang-1200.0.32.28)")
-C linker for the host machine: cc ld64 609.8
+C compiler for the host machine: clang (clang 12.0.0 "Apple clang version 
12.0.0 (clang-1200.0.32.27)")
+C linker for the host machine: clang ld64 609.7
  Host machine cpu family: x86_64
  Host machine cpu: x86_64
  ../meson.build:10: WARNING: Module unstable-keyval has no backwards or 
forwards compatibility and might not exist in future releases.
  Program sh found: YES
  Program python3 found: YES (/usr/local/opt/python@3.9/bin/python3.9)
  Program bzip2 found: YES
-C++ compiler for the host machine: c++ (clang 12.0.0 "Apple clang version 
12.0.0 (clang-1200.0.32.28)")
-C++ linker for the host machine: c++ ld64 609.8
+C++ compiler for the host machine: c++ (clang 12.0.0 "Apple clang version 
12.0.0 (clang-1200.0.32.27)")
+C++ linker for the host machine: c++ ld64 609.7
  Objective-C compiler for the host machine: clang (clang 12.0.0)
-Objective-C linker for the host machine: clang ld64 609.8
+Objective-C linker for the host machine: clang ld64 609.7
  Program cgcc found: NO
  Library m found: YES
  Library util found: YES
@@ -238,7 +238,7 @@
  Program diff found: YES
  Program dbus-daemon found: NO
  Program initrd-stress.sh found: YES
-Build targets in project: 534
+Build targets in project: 556

  qemu 6.0.50

@@ -273,7 +273,8 @@
                user-mode emulation: NO
                        block layer: YES
                      Install blobs: YES
-                   module support: NO
+                   module support: YES
+          alternative module path: NO
                     plugin support: NO
                    fuzzing support: NO
                      Audio drivers: coreaudio sdl
@@ -293,8 +294,8 @@
    Compilation
                           host CPU: x86_64
                    host endianness: little
-                       C compiler: cc
-                  Host C compiler: cc
+                       C compiler: clang
+                  Host C compiler: clang
                       C++ compiler: c++
               Objective-C compiler: clang
                            ARFLAGS: rv
@@ -376,7 +377,7 @@
                             pixman: YES
                        VTE support: NO
                      slirp support: internal
-                         libtasn1: NO
+                         libtasn1: YES
                                PAM: YES
                      iconv support: YES
                     curses support: NO
@@ -403,7 +404,7 @@
                     xfsctl support: NO
                  smartcard support: NO
                        U2F support: NO
-                           libusb: NO
+                           libusb: YES
                      usb net redir: NO
                     OpenGL support: NO
                                GBM: NO

... IMHO it's not worth wasting precious CI minutes here just because of 
those small differences, and I'd like to suggest that we continue with only 
on macos job in the future.

> diff --git a/.gitlab-ci.d/cirrus/README.rst b/.gitlab-ci.d/cirrus/README.rst
> new file mode 100644
> index 0000000000..657b0706d7
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus/README.rst
> @@ -0,0 +1,54 @@
> +Cirrus CI integration
> +=====================
> +
> +GitLab CI shared runners only provide a docker environment running on Linux.
> +While it is possible to provide private runners for non-Linux platforms this
> +is not something most contributors/maintainers will wish to do.
> +
> +To work around this limitation, we take advantage of `Cirrus CI`_'s free
> +offering: more specifically, we use the `cirrus-run`_ script to trigger Cirrus
> +CI jobs from GitLab CI jobs so that Cirrus CI job output is integrated into
> +the main GitLab CI pipeline dashboard.
> +
> +There is, however, some one-time setup required. If you want FreeBSD and macOS
> +builds to happen when you push to your GitLab repository, you need to
> +
> +* set up a GitHub repository for the project, eg. ``yourusername/qemu``.
> +  This repository needs to exist for cirrus-run to work, but it doesn't need to
> +  be kept up to date, so you can create it and then forget about it;
> +
> +* enable the `Cirrus CI GitHub app`_  for your GitHub account;
> +
> +* sign up for Cirrus CI. It's enough to log into the website using your GitHub
> +  account;
> +
> +* grab an API token from the `Cirrus CI settings`_ page;
> +
> +* it may be necessary to push an empty ``.cirrus.yml`` file to your github fork
> +  for Cirrus CI to properly recognize the project. You can check whether
> +  Cirrus CI knows about your project by navigating to:
> +
> +  ``https://cirrus-ci.com/yourusername/qemu``
> +
> +* in the *CI/CD / Variables* section of the settings page for your GitLab
> +  repository, create two new variables:
> +
> +  * ``CIRRUS_GITHUB_REPO``, containing the name of the GitHub repository
> +    created earlier, eg. ``yourusername/qemu``;
> +
> +  * ``CIRRUS_API_TOKEN``, containing the Cirrus CI API token generated earlier.
> +    This variable **must** be marked as *Masked*, because anyone with knowledge
> +    of it can impersonate you as far as Cirrus CI is concerned.
> +
> +  Neither of these variables should be marked as *Protected*, because in
> +  general you'll want to be able to trigger Cirrus CI builds from non-protected
> +  branches.
> +
> +Once this one-time setup is complete, you can just keep pushing to your GitLab
> +repository as usual and you'll automatically get the additional CI coverage.
> +
> +
> +.. _Cirrus CI GitHub app: https://github.com/marketplace/cirrus-ci
> +.. _Cirrus CI settings: https://cirrus-ci.com/settings/profile/
> +.. _Cirrus CI: https://cirrus-ci.com/
> +.. _cirrus-run: https://github.com/sio/cirrus-run/
> diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
> new file mode 100644
> index 0000000000..857bdc5536
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus/build.yml
> @@ -0,0 +1,35 @@
> +@CIRRUS_VM_INSTANCE_TYPE@:
> +  @CIRRUS_VM_IMAGE_SELECTOR@: @CIRRUS_VM_IMAGE_NAME@
> +  cpu: @CIRRUS_VM_CPUS@
> +  memory: @CIRRUS_VM_RAM@
> +
> +env:
> +  CIRRUS_CLONE_DEPTH: 1
> +  CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@"
> +  CI_COMMIT_REF_NAME: "@CI_COMMIT_REF_NAME@"
> +  CI_COMMIT_SHA: "@CI_COMMIT_SHA@"
> +  PATH: "@PATH@"
> +  PKG_CONFIG_PATH: "@PKG_CONFIG_PATH@"
> +  PYTHON: "@PYTHON@"
> +  MAKE: "@MAKE@"
> +  CONFIGURE_ARGS: "@CONFIGURE_ARGS@"
> +
> +build_task:
> +  install_script:
> +    - @UPDATE_COMMAND@
> +    - @INSTALL_COMMAND@ @PKGS@
> +    - if test -n "@PYPI_PKGS@" ; then @PIP3@ install @PYPI_PKGS@ ; fi
> +  clone_script:
> +    - git clone --depth 100 "$CI_REPOSITORY_URL" .
> +    - git fetch origin "$CI_COMMIT_REF_NAME"
> +    - git reset --hard "$CI_COMMIT_SHA"
> +  build_script:
> +    - mkdir build
> +    - cd build
> +    - ../configure --enable-werror $CONFIGURE_ARGS
> +      || { cat config.log meson-logs/meson-log.txt; exit 1; }
> +    - $MAKE -j$(sysctl -n hw.ncpu)
> +    - for TARGET in $TEST_TARGETS ;
> +      do
> +        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
> +      done

You seem to try to enable ccache in the other files (e.g. by extending the 
PATH) ... however, I don't see where you try to save the ccache directory 
between the runs... so I guess that ccache won't be working with this setup 
yet? In that case, I'd recommend to rather drop the other ccache changes 
again, since the initial run with ccache (where the cache gets populated) is 
rather slower than compiling without ccache.

  Thomas
Re: [PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run
Posted by Daniel P. Berrangé 4 years, 7 months ago 
On Mon, Jun 28, 2021 at 09:28:18AM +0200, Thomas Huth wrote:
> On 25/06/2021 19.22, Daniel P. Berrangé wrote:
> > This adds support for running 4 jobs via Cirrus CI runners:
> > 
> >   * FreeBSD 12
> >   * FreeBSD 13
> >   * macOS 11 with default XCode
> >   * macOS 11 with latest XCode
> > 
> > The gitlab job uses a container published by the libvirt-ci
> > project (https://gitlab.com/libvirt/libvirt-ci) that contains
> > the 'cirrus-run' command. This accepts a short yaml file that
> > describes a single Cirrus CI job, runs it using the Cirrus CI
> > REST API, and reports any output to the console.
> > 
> > In this way Cirrus CI is effectively working as an indirect
> > custom runner for GitLab CI pipelines. The key benefit is that
> > Cirrus CI job results affect the GitLab CI pipeline result and
> > so the user only has look at one CI dashboard.
> 
> Cool, thanks for tackling this!
> 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >   .gitlab-ci.d/cirrus.yml             | 103 ++++++++++++++++++++++++++++
> >   .gitlab-ci.d/cirrus/README.rst      |  54 +++++++++++++++
> >   .gitlab-ci.d/cirrus/build.yml       |  35 ++++++++++
> >   .gitlab-ci.d/cirrus/freebsd-12.vars |  13 ++++
> >   .gitlab-ci.d/cirrus/freebsd-13.vars |  13 ++++
> >   .gitlab-ci.d/cirrus/macos-11.vars   |  15 ++++
> >   .gitlab-ci.d/qemu-project.yml       |   1 +
> >   7 files changed, 234 insertions(+)
> >   create mode 100644 .gitlab-ci.d/cirrus.yml
> >   create mode 100644 .gitlab-ci.d/cirrus/README.rst
> >   create mode 100644 .gitlab-ci.d/cirrus/build.yml
> >   create mode 100644 .gitlab-ci.d/cirrus/freebsd-12.vars
> >   create mode 100644 .gitlab-ci.d/cirrus/freebsd-13.vars
> >   create mode 100644 .gitlab-ci.d/cirrus/macos-11.vars
> > 
> > diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> > new file mode 100644
> > index 0000000000..d7b4cce79b
> > --- /dev/null
> > +++ b/.gitlab-ci.d/cirrus.yml
> > @@ -0,0 +1,103 @@
> > +# Jobs that we delegate to Cirrus CI because they require an operating
> > +# system other than Linux. These jobs will only run if the required
> > +# setup has been performed on the GitLab account.
> > +#
> > +# The Cirrus CI configuration is generated by replacing target-specific
> > +# variables in a generic template: some of these variables are provided
> > +# when the GitLab CI job is defined, others are taken from a shell
> > +# snippet generated using lcitool.
> > +#
> > +# Note that the $PATH environment variable has to be treated with
> > +# special care, because we can't just override it at the GitLab CI job
> > +# definition level or we risk breaking it completely.
> > +.cirrus_build_job:
> > +  stage: build
> > +  image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
> > +  needs: []
> > +  script:
> > +    - source .gitlab-ci.d/cirrus/$NAME.vars
> > +    - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
> > +          -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
> > +          -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
> > +          -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
> > +          -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
> > +          -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
> > +          -e "s|[@]CIRRUS_VM_CPUS@|$CIRRUS_VM_CPUS|g"
> > +          -e "s|[@]CIRRUS_VM_RAM@|$CIRRUS_VM_RAM|g"
> > +          -e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g"
> > +          -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
> > +          -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g"
> > +          -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g"
> > +          -e "s|[@]PKGS@|$PKGS|g"
> > +          -e "s|[@]MAKE@|$MAKE|g"
> > +          -e "s|[@]PYTHON@|$PYTHON|g"
> > +          -e "s|[@]PIP3@|$PIP3|g"
> > +          -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g"
> > +          -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g"
> > +          -e "s|[@]TEST_TARGETSS@|$TEST_TARGETSS|g"
> > +      <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
> > +    - cat .gitlab-ci.d/cirrus/$NAME.yml
> > +    - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
> > +  rules:
> > +    - if: "$TEMPORARILY_DISABLED"
> > +      allow_failure: true
> > +    - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
> > +
> > +x64-freebsd-12-build:
> > +  extends: .cirrus_build_job
> > +  variables:
> > +    NAME: freebsd-12
> > +    CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
> > +    CIRRUS_VM_IMAGE_SELECTOR: image_family
> > +    CIRRUS_VM_IMAGE_NAME: freebsd-12-2
> > +    CIRRUS_VM_CPUS: 8
> > +    CIRRUS_VM_RAM: 8G
> > +    UPDATE_COMMAND: pkg update
> > +    INSTALL_COMMAND: pkg install -y
> > +    # TODO: Enable gnutls again once FreeBSD's libtasn1 got fixed
> > +    # See: https://gitlab.com/gnutls/libtasn1/-/merge_requests/71
> > +    CONFIGURE_ARGS: --disable-gnutls
> > +    TEST_TARGETS: check
> > +
> > +x64-freebsd-13-build:
> > +  extends: .cirrus_build_job
> > +  variables:
> > +    NAME: freebsd-13
> > +    CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
> > +    CIRRUS_VM_IMAGE_SELECTOR: image_family
> > +    CIRRUS_VM_IMAGE_NAME: freebsd-13-0
> > +    CIRRUS_VM_CPUS: 8
> > +    CIRRUS_VM_RAM: 8G
> > +    UPDATE_COMMAND: pkg update
> > +    INSTALL_COMMAND: pkg install -y
> > +    TEST_TARGETS: check
> > +
> > +x64-macos-11-base-build:
> > +  extends: .cirrus_build_job
> > +  variables:
> > +    NAME: macos-11
> > +    CIRRUS_VM_INSTANCE_TYPE: osx_instance
> > +    CIRRUS_VM_IMAGE_SELECTOR: image
> > +    CIRRUS_VM_IMAGE_NAME: big-sur-base
> > +    CIRRUS_VM_CPUS: 12
> > +    CIRRUS_VM_RAM: 24G
> > +    UPDATE_COMMAND: brew update
> > +    INSTALL_COMMAND: brew install
> > +    PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
> > +    PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
> > +    TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
> > +
> > +x64-macos-11-xcode-build:
> > +  extends: .cirrus_build_job
> > +  variables:
> > +    NAME: macos-11
> > +    CIRRUS_VM_INSTANCE_TYPE: osx_instance
> > +    CIRRUS_VM_IMAGE_SELECTOR: image
> > +    CIRRUS_VM_IMAGE_NAME: big-sur-xcode
> > +    CIRRUS_VM_CPUS: 12
> > +    CIRRUS_VM_RAM: 24G
> > +    UPDATE_COMMAND: brew update
> > +    INSTALL_COMMAND: brew install
> > +    PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
> > +    PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
> > +    TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
> 
> Not directly related to your patch, but I wonder whether we really gain much
> by having two macos build jobs in our CI ... they seem to be very similar,
> e.g. compare the output of the "configure" step of the "macos" and the
> "macos_xcode" job here:
> 
>  https://cirrus-ci.com/build/4919300914937856
> 
> The diff basically looks like this:

snip


Looking back, the 2nd job was added by


commit fc84471ae2867823f56b1ad1705de324c2d8b725
Author: Alex Bennée <alex.bennee@linaro.org>
Date:   Wed Oct 16 19:27:13 2019 +0100

    cirrus.yml: add latest Xcode build target
    
    CirrusCI provides a mojave-xcode alias for the latest Xcode available.
    Let's use it to make sure we track the latest releases.
    

At the time this was added there was likely a significant difference
in XCode versions.

Since that time, both images were upgraded to catalina, instead of
mojave, and that appears to have eliminated the differences. My
changes upgrade again to big-sur.

So yes, we can drop the 2nd job at this time,


> > diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
> > new file mode 100644
> > index 0000000000..857bdc5536
> > --- /dev/null
> > +++ b/.gitlab-ci.d/cirrus/build.yml
> > @@ -0,0 +1,35 @@
> > +@CIRRUS_VM_INSTANCE_TYPE@:
> > +  @CIRRUS_VM_IMAGE_SELECTOR@: @CIRRUS_VM_IMAGE_NAME@
> > +  cpu: @CIRRUS_VM_CPUS@
> > +  memory: @CIRRUS_VM_RAM@
> > +
> > +env:
> > +  CIRRUS_CLONE_DEPTH: 1
> > +  CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@"
> > +  CI_COMMIT_REF_NAME: "@CI_COMMIT_REF_NAME@"
> > +  CI_COMMIT_SHA: "@CI_COMMIT_SHA@"
> > +  PATH: "@PATH@"
> > +  PKG_CONFIG_PATH: "@PKG_CONFIG_PATH@"
> > +  PYTHON: "@PYTHON@"
> > +  MAKE: "@MAKE@"
> > +  CONFIGURE_ARGS: "@CONFIGURE_ARGS@"
> > +
> > +build_task:
> > +  install_script:
> > +    - @UPDATE_COMMAND@
> > +    - @INSTALL_COMMAND@ @PKGS@
> > +    - if test -n "@PYPI_PKGS@" ; then @PIP3@ install @PYPI_PKGS@ ; fi
> > +  clone_script:
> > +    - git clone --depth 100 "$CI_REPOSITORY_URL" .
> > +    - git fetch origin "$CI_COMMIT_REF_NAME"
> > +    - git reset --hard "$CI_COMMIT_SHA"
> > +  build_script:
> > +    - mkdir build
> > +    - cd build
> > +    - ../configure --enable-werror $CONFIGURE_ARGS
> > +      || { cat config.log meson-logs/meson-log.txt; exit 1; }
> > +    - $MAKE -j$(sysctl -n hw.ncpu)
> > +    - for TARGET in $TEST_TARGETS ;
> > +      do
> > +        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
> > +      done
> 
> You seem to try to enable ccache in the other files (e.g. by extending the
> PATH) ... however, I don't see where you try to save the ccache directory
> between the runs... so I guess that ccache won't be working with this setup
> yet? In that case, I'd recommend to rather drop the other ccache changes
> again, since the initial run with ccache (where the cache gets populated) is
> rather slower than compiling without ccache.

I'll examine that. I think the path change is harmless though, because it
points to a location that I don't think exists. 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Re: [PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run
Posted by Wainer dos Santos Moschetta 4 years, 7 months ago 
Hi,

On 6/25/21 2:22 PM, Daniel P. Berrangé wrote:
> This adds support for running 4 jobs via Cirrus CI runners:
>
>   * FreeBSD 12
>   * FreeBSD 13
>   * macOS 11 with default XCode
>   * macOS 11 with latest XCode
>
> The gitlab job uses a container published by the libvirt-ci
> project (https://gitlab.com/libvirt/libvirt-ci) that contains
> the 'cirrus-run' command. This accepts a short yaml file that
> describes a single Cirrus CI job, runs it using the Cirrus CI
> REST API, and reports any output to the console.
>
> In this way Cirrus CI is effectively working as an indirect
> custom runner for GitLab CI pipelines. The key benefit is that
> Cirrus CI job results affect the GitLab CI pipeline result and
> so the user only has look at one CI dashboard.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/cirrus.yml             | 103 ++++++++++++++++++++++++++++
>   .gitlab-ci.d/cirrus/README.rst      |  54 +++++++++++++++
>   .gitlab-ci.d/cirrus/build.yml       |  35 ++++++++++
>   .gitlab-ci.d/cirrus/freebsd-12.vars |  13 ++++
>   .gitlab-ci.d/cirrus/freebsd-13.vars |  13 ++++
>   .gitlab-ci.d/cirrus/macos-11.vars   |  15 ++++
>   .gitlab-ci.d/qemu-project.yml       |   1 +
>   7 files changed, 234 insertions(+)
>   create mode 100644 .gitlab-ci.d/cirrus.yml
>   create mode 100644 .gitlab-ci.d/cirrus/README.rst
>   create mode 100644 .gitlab-ci.d/cirrus/build.yml
>   create mode 100644 .gitlab-ci.d/cirrus/freebsd-12.vars
>   create mode 100644 .gitlab-ci.d/cirrus/freebsd-13.vars
>   create mode 100644 .gitlab-ci.d/cirrus/macos-11.vars
>
> diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> new file mode 100644
> index 0000000000..d7b4cce79b
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus.yml
> @@ -0,0 +1,103 @@
> +# Jobs that we delegate to Cirrus CI because they require an operating
> +# system other than Linux. These jobs will only run if the required
> +# setup has been performed on the GitLab account.
> +#
> +# The Cirrus CI configuration is generated by replacing target-specific
> +# variables in a generic template: some of these variables are provided
> +# when the GitLab CI job is defined, others are taken from a shell
> +# snippet generated using lcitool.
> +#
> +# Note that the $PATH environment variable has to be treated with
> +# special care, because we can't just override it at the GitLab CI job
> +# definition level or we risk breaking it completely.
> +.cirrus_build_job:
> +  stage: build
> +  image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
> +  needs: []
> +  script:
> +    - source .gitlab-ci.d/cirrus/$NAME.vars
> +    - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
> +          -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
> +          -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
> +          -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
> +          -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
> +          -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
> +          -e "s|[@]CIRRUS_VM_CPUS@|$CIRRUS_VM_CPUS|g"
> +          -e "s|[@]CIRRUS_VM_RAM@|$CIRRUS_VM_RAM|g"
> +          -e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g"
> +          -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
> +          -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g"
> +          -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g"
> +          -e "s|[@]PKGS@|$PKGS|g"
> +          -e "s|[@]MAKE@|$MAKE|g"
> +          -e "s|[@]PYTHON@|$PYTHON|g"
> +          -e "s|[@]PIP3@|$PIP3|g"
> +          -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g"
> +          -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g"
> +          -e "s|[@]TEST_TARGETSS@|$TEST_TARGETSS|g"
> +      <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
> +    - cat .gitlab-ci.d/cirrus/$NAME.yml
> +    - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
> +  rules:
> +    - if: "$TEMPORARILY_DISABLED"

Reading 'TEMPORARILY_DISABLED' I immediately think the job is 
malfunctioning or under maintenance.

But since the plan is to keep it running as 'non-gate' until it proves 
reliable, so maybe you could rename the variable to 'NON_GATE' or 
'STAGING_JOB' (i.e. some words to better express the intent).

Thanks!

- Wainer

> +      allow_failure: true
> +    - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
> +
> +x64-freebsd-12-build:
> +  extends: .cirrus_build_job
> +  variables:
> +    NAME: freebsd-12
> +    CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
> +    CIRRUS_VM_IMAGE_SELECTOR: image_family
> +    CIRRUS_VM_IMAGE_NAME: freebsd-12-2
> +    CIRRUS_VM_CPUS: 8
> +    CIRRUS_VM_RAM: 8G
> +    UPDATE_COMMAND: pkg update
> +    INSTALL_COMMAND: pkg install -y
> +    # TODO: Enable gnutls again once FreeBSD's libtasn1 got fixed
> +    # See: https://gitlab.com/gnutls/libtasn1/-/merge_requests/71
> +    CONFIGURE_ARGS: --disable-gnutls
> +    TEST_TARGETS: check
> +
> +x64-freebsd-13-build:
> +  extends: .cirrus_build_job
> +  variables:
> +    NAME: freebsd-13
> +    CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
> +    CIRRUS_VM_IMAGE_SELECTOR: image_family
> +    CIRRUS_VM_IMAGE_NAME: freebsd-13-0
> +    CIRRUS_VM_CPUS: 8
> +    CIRRUS_VM_RAM: 8G
> +    UPDATE_COMMAND: pkg update
> +    INSTALL_COMMAND: pkg install -y
> +    TEST_TARGETS: check
> +
> +x64-macos-11-base-build:
> +  extends: .cirrus_build_job
> +  variables:
> +    NAME: macos-11
> +    CIRRUS_VM_INSTANCE_TYPE: osx_instance
> +    CIRRUS_VM_IMAGE_SELECTOR: image
> +    CIRRUS_VM_IMAGE_NAME: big-sur-base
> +    CIRRUS_VM_CPUS: 12
> +    CIRRUS_VM_RAM: 24G
> +    UPDATE_COMMAND: brew update
> +    INSTALL_COMMAND: brew install
> +    PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
> +    PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
> +    TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
> +
> +x64-macos-11-xcode-build:
> +  extends: .cirrus_build_job
> +  variables:
> +    NAME: macos-11
> +    CIRRUS_VM_INSTANCE_TYPE: osx_instance
> +    CIRRUS_VM_IMAGE_SELECTOR: image
> +    CIRRUS_VM_IMAGE_NAME: big-sur-xcode
> +    CIRRUS_VM_CPUS: 12
> +    CIRRUS_VM_RAM: 24G
> +    UPDATE_COMMAND: brew update
> +    INSTALL_COMMAND: brew install
> +    PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
> +    PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
> +    TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
> diff --git a/.gitlab-ci.d/cirrus/README.rst b/.gitlab-ci.d/cirrus/README.rst
> new file mode 100644
> index 0000000000..657b0706d7
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus/README.rst
> @@ -0,0 +1,54 @@
> +Cirrus CI integration
> +=====================
> +
> +GitLab CI shared runners only provide a docker environment running on Linux.
> +While it is possible to provide private runners for non-Linux platforms this
> +is not something most contributors/maintainers will wish to do.
> +
> +To work around this limitation, we take advantage of `Cirrus CI`_'s free
> +offering: more specifically, we use the `cirrus-run`_ script to trigger Cirrus
> +CI jobs from GitLab CI jobs so that Cirrus CI job output is integrated into
> +the main GitLab CI pipeline dashboard.
> +
> +There is, however, some one-time setup required. If you want FreeBSD and macOS
> +builds to happen when you push to your GitLab repository, you need to
> +
> +* set up a GitHub repository for the project, eg. ``yourusername/qemu``.
> +  This repository needs to exist for cirrus-run to work, but it doesn't need to
> +  be kept up to date, so you can create it and then forget about it;
> +
> +* enable the `Cirrus CI GitHub app`_  for your GitHub account;
> +
> +* sign up for Cirrus CI. It's enough to log into the website using your GitHub
> +  account;
> +
> +* grab an API token from the `Cirrus CI settings`_ page;
> +
> +* it may be necessary to push an empty ``.cirrus.yml`` file to your github fork
> +  for Cirrus CI to properly recognize the project. You can check whether
> +  Cirrus CI knows about your project by navigating to:
> +
> +  ``https://cirrus-ci.com/yourusername/qemu``
> +
> +* in the *CI/CD / Variables* section of the settings page for your GitLab
> +  repository, create two new variables:
> +
> +  * ``CIRRUS_GITHUB_REPO``, containing the name of the GitHub repository
> +    created earlier, eg. ``yourusername/qemu``;
> +
> +  * ``CIRRUS_API_TOKEN``, containing the Cirrus CI API token generated earlier.
> +    This variable **must** be marked as *Masked*, because anyone with knowledge
> +    of it can impersonate you as far as Cirrus CI is concerned.
> +
> +  Neither of these variables should be marked as *Protected*, because in
> +  general you'll want to be able to trigger Cirrus CI builds from non-protected
> +  branches.
> +
> +Once this one-time setup is complete, you can just keep pushing to your GitLab
> +repository as usual and you'll automatically get the additional CI coverage.
> +
> +
> +.. _Cirrus CI GitHub app: https://github.com/marketplace/cirrus-ci
> +.. _Cirrus CI settings: https://cirrus-ci.com/settings/profile/
> +.. _Cirrus CI: https://cirrus-ci.com/
> +.. _cirrus-run: https://github.com/sio/cirrus-run/
> diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
> new file mode 100644
> index 0000000000..857bdc5536
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus/build.yml
> @@ -0,0 +1,35 @@
> +@CIRRUS_VM_INSTANCE_TYPE@:
> +  @CIRRUS_VM_IMAGE_SELECTOR@: @CIRRUS_VM_IMAGE_NAME@
> +  cpu: @CIRRUS_VM_CPUS@
> +  memory: @CIRRUS_VM_RAM@
> +
> +env:
> +  CIRRUS_CLONE_DEPTH: 1
> +  CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@"
> +  CI_COMMIT_REF_NAME: "@CI_COMMIT_REF_NAME@"
> +  CI_COMMIT_SHA: "@CI_COMMIT_SHA@"
> +  PATH: "@PATH@"
> +  PKG_CONFIG_PATH: "@PKG_CONFIG_PATH@"
> +  PYTHON: "@PYTHON@"
> +  MAKE: "@MAKE@"
> +  CONFIGURE_ARGS: "@CONFIGURE_ARGS@"
> +
> +build_task:
> +  install_script:
> +    - @UPDATE_COMMAND@
> +    - @INSTALL_COMMAND@ @PKGS@
> +    - if test -n "@PYPI_PKGS@" ; then @PIP3@ install @PYPI_PKGS@ ; fi
> +  clone_script:
> +    - git clone --depth 100 "$CI_REPOSITORY_URL" .
> +    - git fetch origin "$CI_COMMIT_REF_NAME"
> +    - git reset --hard "$CI_COMMIT_SHA"
> +  build_script:
> +    - mkdir build
> +    - cd build
> +    - ../configure --enable-werror $CONFIGURE_ARGS
> +      || { cat config.log meson-logs/meson-log.txt; exit 1; }
> +    - $MAKE -j$(sysctl -n hw.ncpu)
> +    - for TARGET in $TEST_TARGETS ;
> +      do
> +        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
> +      done
> diff --git a/.gitlab-ci.d/cirrus/freebsd-12.vars b/.gitlab-ci.d/cirrus/freebsd-12.vars
> new file mode 100644
> index 0000000000..5e51e50ea9
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus/freebsd-12.vars
> @@ -0,0 +1,13 @@
> +# THIS FILE WAS AUTO-GENERATED
> +#
> +#  $ lcitool variables freebsd-12 qemu
> +#
> +# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1
> +
> +PACKAGING_COMMAND='pkg'
> +CCACHE='/usr/local/bin/ccache'
> +MAKE='/usr/local/bin/gmake'
> +NINJA='/usr/local/bin/ninja'
> +PYTHON='/usr/local/bin/python3'
> +PIP3='/usr/local/bin/pip-3.7'
> +PKGS='alsa-lib bash bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage ctags curl cyrus-sasl dbus diffutils gettext git glib gmake gnutls gsed gtk3 libepoxy libffi libgcrypt libjpeg-turbo libnfs libspice-server libssh libtasn1 libxml2 llvm lttng-ust lzo2 meson ncurses nettle ninja opencv p5-Test-Harness perl5 pixman pkgconf png py37-numpy py37-pillow py37-pip py37-sphinx py37-sphinx_rtd_theme py37-virtualenv py37-yaml python3 rpm2cpio sdl2 sdl2_image snappy spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
> diff --git a/.gitlab-ci.d/cirrus/freebsd-13.vars b/.gitlab-ci.d/cirrus/freebsd-13.vars
> new file mode 100644
> index 0000000000..07716833d9
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus/freebsd-13.vars
> @@ -0,0 +1,13 @@
> +# THIS FILE WAS AUTO-GENERATED
> +#
> +#  $ lcitool variables freebsd-13 qemu
> +#
> +# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1
> +
> +PACKAGING_COMMAND='pkg'
> +CCACHE='/usr/local/bin/ccache'
> +MAKE='/usr/local/bin/gmake'
> +NINJA='/usr/local/bin/ninja'
> +PYTHON='/usr/local/bin/python3'
> +PIP3='/usr/local/bin/pip-3.7'
> +PKGS='alsa-lib bash bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage ctags curl cyrus-sasl dbus diffutils gettext git glib gmake gnutls gsed gtk3 libepoxy libffi libgcrypt libjpeg-turbo libnfs libspice-server libssh libtasn1 libxml2 llvm lttng-ust lzo2 meson ncurses nettle ninja opencv p5-Test-Harness perl5 pixman pkgconf png py37-numpy py37-pillow py37-pip py37-sphinx py37-sphinx_rtd_theme py37-virtualenv py37-yaml python3 rpm2cpio sdl2 sdl2_image snappy spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
> diff --git a/.gitlab-ci.d/cirrus/macos-11.vars b/.gitlab-ci.d/cirrus/macos-11.vars
> new file mode 100644
> index 0000000000..cbec8a44a3
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus/macos-11.vars
> @@ -0,0 +1,15 @@
> +# THIS FILE WAS AUTO-GENERATED
> +#
> +#  $ lcitool variables macos-11 qemu
> +#
> +# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1
> +
> +PACKAGING_COMMAND='brew'
> +CCACHE='/usr/local/bin/ccache'
> +MAKE='/usr/local/bin/gmake'
> +NINJA='/usr/local/bin/ninja'
> +PYTHON='/usr/local/bin/python3'
> +PIP3='/usr/local/bin/pip3'
> +PKGS='bash bc bzip2 capstone ccache cpanminus ctags curl dbus diffutils gcovr gettext git glib gnu-sed gnutls gtk+3 jemalloc jpeg-turbo libepoxy libffi libgcrypt libiscsi libnfs libpng libslirp libssh libtasn1 libusb libxml2 llvm lzo make meson ncurses nettle ninja perl pixman pkg-config python3 rpm2cpio sdl2 sdl2_image snappy sparse spice-protocol tesseract texinfo usbredir vde vte3 zlib zstd'
> +PYPI_PKGS='PyYAML numpy pillow sphinx sphinx-rtd-theme virtualenv'
> +CPAN_PKGS='Test::Harness'
> diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
> index 64cb2ba1da..35d4e62c12 100644
> --- a/.gitlab-ci.d/qemu-project.yml
> +++ b/.gitlab-ci.d/qemu-project.yml
> @@ -9,3 +9,4 @@ include:
>     - local: '/.gitlab-ci.d/crossbuilds.yml'
>     - local: '/.gitlab-ci.d/buildtest.yml'
>     - local: '/.gitlab-ci.d/static_checks.yml'
> +  - local: '/.gitlab-ci.d/cirrus.yml'
Re: [PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run
Posted by Daniel P. Berrangé 4 years, 7 months ago 
On Wed, Jun 30, 2021 at 03:58:57PM -0300, Wainer dos Santos Moschetta wrote:
> Hi,
> 
> On 6/25/21 2:22 PM, Daniel P. Berrangé wrote:
> > This adds support for running 4 jobs via Cirrus CI runners:
> > 
> >   * FreeBSD 12
> >   * FreeBSD 13
> >   * macOS 11 with default XCode
> >   * macOS 11 with latest XCode
> > 
> > The gitlab job uses a container published by the libvirt-ci
> > project (https://gitlab.com/libvirt/libvirt-ci) that contains
> > the 'cirrus-run' command. This accepts a short yaml file that
> > describes a single Cirrus CI job, runs it using the Cirrus CI
> > REST API, and reports any output to the console.
> > 
> > In this way Cirrus CI is effectively working as an indirect
> > custom runner for GitLab CI pipelines. The key benefit is that
> > Cirrus CI job results affect the GitLab CI pipeline result and
> > so the user only has look at one CI dashboard.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >   .gitlab-ci.d/cirrus.yml             | 103 ++++++++++++++++++++++++++++
> >   .gitlab-ci.d/cirrus/README.rst      |  54 +++++++++++++++
> >   .gitlab-ci.d/cirrus/build.yml       |  35 ++++++++++
> >   .gitlab-ci.d/cirrus/freebsd-12.vars |  13 ++++
> >   .gitlab-ci.d/cirrus/freebsd-13.vars |  13 ++++
> >   .gitlab-ci.d/cirrus/macos-11.vars   |  15 ++++
> >   .gitlab-ci.d/qemu-project.yml       |   1 +
> >   7 files changed, 234 insertions(+)
> >   create mode 100644 .gitlab-ci.d/cirrus.yml
> >   create mode 100644 .gitlab-ci.d/cirrus/README.rst
> >   create mode 100644 .gitlab-ci.d/cirrus/build.yml
> >   create mode 100644 .gitlab-ci.d/cirrus/freebsd-12.vars
> >   create mode 100644 .gitlab-ci.d/cirrus/freebsd-13.vars
> >   create mode 100644 .gitlab-ci.d/cirrus/macos-11.vars
> > 
> > diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> > new file mode 100644
> > index 0000000000..d7b4cce79b
> > --- /dev/null
> > +++ b/.gitlab-ci.d/cirrus.yml
> > @@ -0,0 +1,103 @@
> > +# Jobs that we delegate to Cirrus CI because they require an operating
> > +# system other than Linux. These jobs will only run if the required
> > +# setup has been performed on the GitLab account.
> > +#
> > +# The Cirrus CI configuration is generated by replacing target-specific
> > +# variables in a generic template: some of these variables are provided
> > +# when the GitLab CI job is defined, others are taken from a shell
> > +# snippet generated using lcitool.
> > +#
> > +# Note that the $PATH environment variable has to be treated with
> > +# special care, because we can't just override it at the GitLab CI job
> > +# definition level or we risk breaking it completely.
> > +.cirrus_build_job:
> > +  stage: build
> > +  image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
> > +  needs: []
> > +  script:
> > +    - source .gitlab-ci.d/cirrus/$NAME.vars
> > +    - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
> > +          -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
> > +          -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
> > +          -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
> > +          -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
> > +          -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
> > +          -e "s|[@]CIRRUS_VM_CPUS@|$CIRRUS_VM_CPUS|g"
> > +          -e "s|[@]CIRRUS_VM_RAM@|$CIRRUS_VM_RAM|g"
> > +          -e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g"
> > +          -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
> > +          -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g"
> > +          -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g"
> > +          -e "s|[@]PKGS@|$PKGS|g"
> > +          -e "s|[@]MAKE@|$MAKE|g"
> > +          -e "s|[@]PYTHON@|$PYTHON|g"
> > +          -e "s|[@]PIP3@|$PIP3|g"
> > +          -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g"
> > +          -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g"
> > +          -e "s|[@]TEST_TARGETSS@|$TEST_TARGETSS|g"
> > +      <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
> > +    - cat .gitlab-ci.d/cirrus/$NAME.yml
> > +    - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
> > +  rules:
> > +    - if: "$TEMPORARILY_DISABLED"
> 
> Reading 'TEMPORARILY_DISABLED' I immediately think the job is malfunctioning
> or under maintenance.

Actually this is cruft that I mistakenly copied from libvirt's rules.

> But since the plan is to keep it running as 'non-gate' until it proves
> reliable, so maybe you could rename the variable to 'NON_GATE' or
> 'STAGING_JOB' (i.e. some words to better express the intent).

We can just remove the 'if $TEMPORARILY_DISABLED' bit and
have only the 'allow_failure: true' bit


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Re: [PATCH 2/3] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run
Posted by Alex Bennée 4 years, 7 months ago 
Daniel P. Berrangé <berrange@redhat.com> writes:

> On Wed, Jun 30, 2021 at 03:58:57PM -0300, Wainer dos Santos Moschetta wrote:
>> Hi,
>> 
>> On 6/25/21 2:22 PM, Daniel P. Berrangé wrote:
>> > This adds support for running 4 jobs via Cirrus CI runners:
>> > 
>> >   * FreeBSD 12
>> >   * FreeBSD 13
>> >   * macOS 11 with default XCode
>> >   * macOS 11 with latest XCode
>> > 
>> > The gitlab job uses a container published by the libvirt-ci
>> > project (https://gitlab.com/libvirt/libvirt-ci) that contains
>> > the 'cirrus-run' command. This accepts a short yaml file that
>> > describes a single Cirrus CI job, runs it using the Cirrus CI
>> > REST API, and reports any output to the console.
>> > 
>> > In this way Cirrus CI is effectively working as an indirect
>> > custom runner for GitLab CI pipelines. The key benefit is that
>> > Cirrus CI job results affect the GitLab CI pipeline result and
>> > so the user only has look at one CI dashboard.
>> > 
>> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>> > ---
>> >   .gitlab-ci.d/cirrus.yml             | 103 ++++++++++++++++++++++++++++
>> >   .gitlab-ci.d/cirrus/README.rst      |  54 +++++++++++++++
>> >   .gitlab-ci.d/cirrus/build.yml       |  35 ++++++++++
>> >   .gitlab-ci.d/cirrus/freebsd-12.vars |  13 ++++
>> >   .gitlab-ci.d/cirrus/freebsd-13.vars |  13 ++++
>> >   .gitlab-ci.d/cirrus/macos-11.vars   |  15 ++++
>> >   .gitlab-ci.d/qemu-project.yml       |   1 +
>> >   7 files changed, 234 insertions(+)
>> >   create mode 100644 .gitlab-ci.d/cirrus.yml
>> >   create mode 100644 .gitlab-ci.d/cirrus/README.rst
>> >   create mode 100644 .gitlab-ci.d/cirrus/build.yml
>> >   create mode 100644 .gitlab-ci.d/cirrus/freebsd-12.vars
>> >   create mode 100644 .gitlab-ci.d/cirrus/freebsd-13.vars
>> >   create mode 100644 .gitlab-ci.d/cirrus/macos-11.vars
>> > 
>> > diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
>> > new file mode 100644
>> > index 0000000000..d7b4cce79b
>> > --- /dev/null
>> > +++ b/.gitlab-ci.d/cirrus.yml
>> > @@ -0,0 +1,103 @@
>> > +# Jobs that we delegate to Cirrus CI because they require an operating
>> > +# system other than Linux. These jobs will only run if the required
>> > +# setup has been performed on the GitLab account.
>> > +#
>> > +# The Cirrus CI configuration is generated by replacing target-specific
>> > +# variables in a generic template: some of these variables are provided
>> > +# when the GitLab CI job is defined, others are taken from a shell
>> > +# snippet generated using lcitool.
>> > +#
>> > +# Note that the $PATH environment variable has to be treated with
>> > +# special care, because we can't just override it at the GitLab CI job
>> > +# definition level or we risk breaking it completely.
>> > +.cirrus_build_job:
>> > +  stage: build
>> > +  image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
>> > +  needs: []
>> > +  script:
>> > +    - source .gitlab-ci.d/cirrus/$NAME.vars
>> > +    - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
>> > +          -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
>> > +          -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
>> > +          -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
>> > +          -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
>> > +          -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
>> > +          -e "s|[@]CIRRUS_VM_CPUS@|$CIRRUS_VM_CPUS|g"
>> > +          -e "s|[@]CIRRUS_VM_RAM@|$CIRRUS_VM_RAM|g"
>> > +          -e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g"
>> > +          -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
>> > +          -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g"
>> > +          -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g"
>> > +          -e "s|[@]PKGS@|$PKGS|g"
>> > +          -e "s|[@]MAKE@|$MAKE|g"
>> > +          -e "s|[@]PYTHON@|$PYTHON|g"
>> > +          -e "s|[@]PIP3@|$PIP3|g"
>> > +          -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g"
>> > +          -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g"
>> > +          -e "s|[@]TEST_TARGETSS@|$TEST_TARGETSS|g"
>> > +      <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
>> > +    - cat .gitlab-ci.d/cirrus/$NAME.yml
>> > +    - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
>> > +  rules:
>> > +    - if: "$TEMPORARILY_DISABLED"
>> 
>> Reading 'TEMPORARILY_DISABLED' I immediately think the job is malfunctioning
>> or under maintenance.
>
> Actually this is cruft that I mistakenly copied from libvirt's rules.
>
>> But since the plan is to keep it running as 'non-gate' until it proves
>> reliable, so maybe you could rename the variable to 'NON_GATE' or
>> 'STAGING_JOB' (i.e. some words to better express the intent).
>
> We can just remove the 'if $TEMPORARILY_DISABLED' bit and
> have only the 'allow_failure: true' bit

I've cleaned that up on testing/next



-- 
Alex Bennée

Patchew 2.1-devel-b67e4c2

© 2016 - 2026 Red Hat, Inc.