1. Patchew
  2. Libvirt
  3. [libvirt PATCH 0/7] Extract the integration job commands to a shell scripts
  4. View patch

[libvirt PATCH 1/7] ci: integration: Extract several hidden job definitions to a script

Erik Skultety posted 7 patches 3 years ago
[libvirt PATCH 1/7] ci: integration: Extract several hidden job definitions to a script
Posted by Erik Skultety 3 years ago 
This will allow us to re-use the script in other scenarios like local
executions where up until now one had to copy-paste the recipe
line-by-line.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
---
 ci/integration-template.yml | 36 +-----------------------------------
 ci/integration.sh           | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 33 insertions(+), 35 deletions(-)
 create mode 100644 ci/integration.sh

diff --git a/ci/integration-template.yml b/ci/integration-template.yml
index 9141d5cedf..f9903cdbb4 100644
--- a/ci/integration-template.yml
+++ b/ci/integration-template.yml
@@ -27,34 +27,6 @@
 .install-deps: &install-deps
   - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* libvirt-python-rpms/*
 
-  # Avocado >98.0 fails with the nwfilter TCK tests, so stick with 98.0 for now
-  - sudo pip3 install --prefix=/usr avocado-framework==98.0
-
-
-.enable-core-dumps: &enable-core-dumps
-  - sudo sh -c "echo DefaultLimitCORE=infinity >> /etc/systemd/system.conf" # Explicitly allow storing cores globally
-  - sudo systemctl daemon-reexec # need to reexec systemd after changing config
-
-
-.enable-libvirt-debugging: &enable-libvirt-debugging
-  - source /etc/os-release  # in order to query the vendor-provided variables
-  - if test "$ID" = "centos" && test "$VERSION_ID" -lt 9 ||
-       test "$ID" = "fedora" && test "$VERSION_ID" -lt 35;
-    then
-      DAEMONS="libvirtd virtlockd virtlogd";
-    else
-      DAEMONS="virtinterfaced virtlockd virtlogd virtnetworkd virtnodedevd virtnwfilterd virtproxyd virtqemud virtsecretd virtstoraged";
-    fi
-  - for daemon in $DAEMONS;
-    do
-      LOG_OUTPUTS="1:file:/var/log/libvirt/${daemon}.log";
-      LOG_FILTERS="3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 1:*";
-      sudo augtool set /files/etc/libvirt/${daemon}.conf/log_filters "'$LOG_FILTERS'" &>/dev/null;
-      sudo augtool set /files/etc/libvirt/${daemon}.conf/log_outputs "'$LOG_OUTPUTS'" &>/dev/null;
-      sudo systemctl --quiet stop ${daemon}.service;
-      sudo systemctl restart ${daemon}.socket;
-    done
-
 
 .collect-logs: &collect-logs
   - set +e
@@ -79,14 +51,8 @@
   before_script:
     - mkdir "$SCRATCH_DIR"
     - *install-deps
-    - *enable-core-dumps
-    - *enable-libvirt-debugging
-    - sudo virsh net-start default &>/dev/null || true;
   script:
-    - cd "$SCRATCH_DIR"
-    - git clone --depth 1 https://gitlab.com/libvirt/libvirt-tck.git
-    - cd libvirt-tck
-    - sudo avocado --config avocado.config run --job-results-dir "$SCRATCH_DIR"/avocado
+    - chmod +x ci/integration.sh && ci/integration.sh
   after_script:
     - test "$CI_JOB_STATUS" = "success" && exit 0;
     - *collect-logs
diff --git a/ci/integration.sh b/ci/integration.sh
new file mode 100644
index 0000000000..21795518c3
--- /dev/null
+++ b/ci/integration.sh
@@ -0,0 +1,32 @@
+#!/bin/sh
+
+# Avocado >98.0 fails with the nwfilter TCK tests, so stick with 98.0 for now
+sudo pip3 install --prefix=/usr avocado-framework==98.0
+
+sudo sh -c "echo DefaultLimitCORE=infinity >> /etc/systemd/system.conf" # Explicitly allow storing cores globally
+sudo systemctl daemon-reexec # need to reexec systemd after changing config
+
+source /etc/os-release  # in order to query the vendor-provided variables
+if test "$ID" = "centos" && test "$VERSION_ID" -lt 9 ||
+   test "$ID" = "fedora" && test "$VERSION_ID" -lt 35
+then
+    DAEMONS="libvirtd virtlockd virtlogd"
+else
+    DAEMONS="virtinterfaced virtlockd virtlogd virtnetworkd virtnodedevd virtnwfilterd virtproxyd virtqemud virtsecretd virtstoraged"
+fi
+for daemon in $DAEMONS
+do
+    LOG_OUTPUTS="1:file:/var/log/libvirt/${daemon}.log"
+    LOG_FILTERS="3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 1:*"
+    sudo augtool set /files/etc/libvirt/${daemon}.conf/log_filters "'$LOG_FILTERS'" &>/dev/null
+    sudo augtool set /files/etc/libvirt/${daemon}.conf/log_outputs "'$LOG_OUTPUTS'" &>/dev/null
+    sudo systemctl --quiet stop ${daemon}.service
+    sudo systemctl restart ${daemon}.socket
+done
+
+sudo virsh net-start default &>/dev/null || true
+
+cd "$SCRATCH_DIR"
+git clone --depth 1 https://gitlab.com/libvirt/libvirt-tck.git
+cd libvirt-tck
+sudo avocado --config avocado.config run --job-results-dir "$SCRATCH_DIR"/avocado
-- 
2.39.0
Re: [libvirt PATCH 1/7] ci: integration: Extract several hidden job definitions to a script
Posted by Daniel P. Berrangé 3 years ago 
On Fri, Jan 20, 2023 at 04:05:57PM +0100, Erik Skultety wrote:
> This will allow us to re-use the script in other scenarios like local
> executions where up until now one had to copy-paste the recipe
> line-by-line.
> 
> Signed-off-by: Erik Skultety <eskultet@redhat.com>
> ---
>  ci/integration-template.yml | 36 +-----------------------------------
>  ci/integration.sh           | 32 ++++++++++++++++++++++++++++++++
>  2 files changed, 33 insertions(+), 35 deletions(-)
>  create mode 100644 ci/integration.sh
> 
> diff --git a/ci/integration-template.yml b/ci/integration-template.yml
> index 9141d5cedf..f9903cdbb4 100644
> --- a/ci/integration-template.yml
> +++ b/ci/integration-template.yml
> @@ -27,34 +27,6 @@
>  .install-deps: &install-deps
>    - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* libvirt-python-rpms/*
>  
> -  # Avocado >98.0 fails with the nwfilter TCK tests, so stick with 98.0 for now
> -  - sudo pip3 install --prefix=/usr avocado-framework==98.0
> -
> -
> -.enable-core-dumps: &enable-core-dumps
> -  - sudo sh -c "echo DefaultLimitCORE=infinity >> /etc/systemd/system.conf" # Explicitly allow storing cores globally
> -  - sudo systemctl daemon-reexec # need to reexec systemd after changing config
> -
> -
> -.enable-libvirt-debugging: &enable-libvirt-debugging
> -  - source /etc/os-release  # in order to query the vendor-provided variables
> -  - if test "$ID" = "centos" && test "$VERSION_ID" -lt 9 ||
> -       test "$ID" = "fedora" && test "$VERSION_ID" -lt 35;
> -    then
> -      DAEMONS="libvirtd virtlockd virtlogd";
> -    else
> -      DAEMONS="virtinterfaced virtlockd virtlogd virtnetworkd virtnodedevd virtnwfilterd virtproxyd virtqemud virtsecretd virtstoraged";
> -    fi
> -  - for daemon in $DAEMONS;
> -    do
> -      LOG_OUTPUTS="1:file:/var/log/libvirt/${daemon}.log";
> -      LOG_FILTERS="3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 1:*";
> -      sudo augtool set /files/etc/libvirt/${daemon}.conf/log_filters "'$LOG_FILTERS'" &>/dev/null;
> -      sudo augtool set /files/etc/libvirt/${daemon}.conf/log_outputs "'$LOG_OUTPUTS'" &>/dev/null;
> -      sudo systemctl --quiet stop ${daemon}.service;
> -      sudo systemctl restart ${daemon}.socket;
> -    done
> -
>  
>  .collect-logs: &collect-logs
>    - set +e
> @@ -79,14 +51,8 @@
>    before_script:
>      - mkdir "$SCRATCH_DIR"
>      - *install-deps
> -    - *enable-core-dumps
> -    - *enable-libvirt-debugging
> -    - sudo virsh net-start default &>/dev/null || true;
>    script:
> -    - cd "$SCRATCH_DIR"
> -    - git clone --depth 1 https://gitlab.com/libvirt/libvirt-tck.git
> -    - cd libvirt-tck
> -    - sudo avocado --config avocado.config run --job-results-dir "$SCRATCH_DIR"/avocado
> +    - chmod +x ci/integration.sh && ci/integration.sh
>    after_script:
>      - test "$CI_JOB_STATUS" = "success" && exit 0;
>      - *collect-logs

Kind of makes me wish for a command that could take a gitlab job
name, and extract the shell commands from the YAML and run them.
I guess that's a gitlab runner really, but taking YML directly
instead of getting jobs sent from gitlab

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Re: [libvirt PATCH 1/7] ci: integration: Extract several hidden job definitions to a script
Posted by Erik Skultety 3 years ago 
On Mon, Jan 23, 2023 at 12:06:21PM +0000, Daniel P. Berrangé wrote:
> On Fri, Jan 20, 2023 at 04:05:57PM +0100, Erik Skultety wrote:
> > This will allow us to re-use the script in other scenarios like local
> > executions where up until now one had to copy-paste the recipe
> > line-by-line.
> > 
> > Signed-off-by: Erik Skultety <eskultet@redhat.com>
> > ---
> >  ci/integration-template.yml | 36 +-----------------------------------
> >  ci/integration.sh           | 32 ++++++++++++++++++++++++++++++++
> >  2 files changed, 33 insertions(+), 35 deletions(-)
> >  create mode 100644 ci/integration.sh
> > 
> > diff --git a/ci/integration-template.yml b/ci/integration-template.yml
> > index 9141d5cedf..f9903cdbb4 100644
> > --- a/ci/integration-template.yml
> > +++ b/ci/integration-template.yml
> > @@ -27,34 +27,6 @@
> >  .install-deps: &install-deps
> >    - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* libvirt-python-rpms/*
> >  
> > -  # Avocado >98.0 fails with the nwfilter TCK tests, so stick with 98.0 for now
> > -  - sudo pip3 install --prefix=/usr avocado-framework==98.0
> > -
> > -
> > -.enable-core-dumps: &enable-core-dumps
> > -  - sudo sh -c "echo DefaultLimitCORE=infinity >> /etc/systemd/system.conf" # Explicitly allow storing cores globally
> > -  - sudo systemctl daemon-reexec # need to reexec systemd after changing config
> > -
> > -
> > -.enable-libvirt-debugging: &enable-libvirt-debugging
> > -  - source /etc/os-release  # in order to query the vendor-provided variables
> > -  - if test "$ID" = "centos" && test "$VERSION_ID" -lt 9 ||
> > -       test "$ID" = "fedora" && test "$VERSION_ID" -lt 35;
> > -    then
> > -      DAEMONS="libvirtd virtlockd virtlogd";
> > -    else
> > -      DAEMONS="virtinterfaced virtlockd virtlogd virtnetworkd virtnodedevd virtnwfilterd virtproxyd virtqemud virtsecretd virtstoraged";
> > -    fi
> > -  - for daemon in $DAEMONS;
> > -    do
> > -      LOG_OUTPUTS="1:file:/var/log/libvirt/${daemon}.log";
> > -      LOG_FILTERS="3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 1:*";
> > -      sudo augtool set /files/etc/libvirt/${daemon}.conf/log_filters "'$LOG_FILTERS'" &>/dev/null;
> > -      sudo augtool set /files/etc/libvirt/${daemon}.conf/log_outputs "'$LOG_OUTPUTS'" &>/dev/null;
> > -      sudo systemctl --quiet stop ${daemon}.service;
> > -      sudo systemctl restart ${daemon}.socket;
> > -    done
> > -
> >  
> >  .collect-logs: &collect-logs
> >    - set +e
> > @@ -79,14 +51,8 @@
> >    before_script:
> >      - mkdir "$SCRATCH_DIR"
> >      - *install-deps
> > -    - *enable-core-dumps
> > -    - *enable-libvirt-debugging
> > -    - sudo virsh net-start default &>/dev/null || true;
> >    script:
> > -    - cd "$SCRATCH_DIR"
> > -    - git clone --depth 1 https://gitlab.com/libvirt/libvirt-tck.git
> > -    - cd libvirt-tck
> > -    - sudo avocado --config avocado.config run --job-results-dir "$SCRATCH_DIR"/avocado
> > +    - chmod +x ci/integration.sh && ci/integration.sh
> >    after_script:
> >      - test "$CI_JOB_STATUS" = "success" && exit 0;
> >      - *collect-logs
> 
> Kind of makes me wish for a command that could take a gitlab job
> name, and extract the shell commands from the YAML and run them.
> I guess that's a gitlab runner really, but taking YML directly
> instead of getting jobs sent from gitlab

Yeah, I don't think that's going to happen. Then again, we'd be probably the
only consumers of such thing by far, so adding an explicit script which
everyone can read and run  without anything else is for the best.

Regards,
Erik

Patchew 2.1-devel-b67e4c2

© 2016 - 2026 Red Hat, Inc.