From nobody Sun May 5 16:15:18 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 207.211.31.81 as permitted sender) client-ip=207.211.31.81; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-1.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 207.211.31.81 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1591788800; cv=none; d=zohomail.com; s=zohoarc; b=A6IL+j/T9IythDBVuEVdtlGoVS1mVhf62YylHVUqVK2biR3Ed8sVOqcTD877P4Pp/3hfJd8f8uGvDaAOIKMZovL3uYxZVZyF9oe52ZolP3wMww6Hb28pMFQQX5mnafYjXflBgWSIj4RJgM30FF5yhdS33kI3MBDOUyPf25IWT8o= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1591788800; h=Content-Type:Content-Transfer-Encoding:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=uNAXPN//pKUMZR7yYhaIcNEqTBCs6iY6OpTP1oDUAzg=; b=afyMRRg78EDY79qJ+sxtNuWhbxGbuFN+JLI+Q6X1EHm3NkrS3ObCKX28YYY7qCM9ieTU51MyaLe02ZnYwtrqHdEYrte64WQgw22O3mW0xOgAC9xxnGA7WbZjuGYLXw6AALYuilg+w1A8QcLCRSycxvsCMQufFMBWXkjlT8HBNME= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 207.211.31.81 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-delivery-1.mimecast.com (us-smtp-1.mimecast.com [207.211.31.81]) by mx.zohomail.com with SMTPS id 1591788800260124.72402164433856; Wed, 10 Jun 2020 04:33:20 -0700 (PDT) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-153-F0uiVqdOObie9ZTAfNNeFA-1; Wed, 10 Jun 2020 07:33:16 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 84602EC1A1; Wed, 10 Jun 2020 11:33:10 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 46C825C1BD; Wed, 10 Jun 2020 11:33:10 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 0B1BF1809543; Wed, 10 Jun 2020 11:33:08 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 05ABX7xv030580 for ; Wed, 10 Jun 2020 07:33:07 -0400 Received: by smtp.corp.redhat.com (Postfix) id DC55919D82; Wed, 10 Jun 2020 11:33:07 +0000 (UTC) Received: from kinshicho.usersys.redhat.com (unknown [10.40.194.85]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 365091A4D9 for ; Wed, 10 Jun 2020 11:33:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591788799; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=uNAXPN//pKUMZR7yYhaIcNEqTBCs6iY6OpTP1oDUAzg=; b=Tx3aKsSBoOMDNM3hTju2dng2s4Ogig3UF25nFb3oSwPr42XRAVEnlE681V6pVqcSPxRc+W NJJHpi5tUCohNSwFM5pxkNca4+eUqwKw2ggXvyse+kOnd1NWaQ95A3lB7JDUgtHR0biYHk uwyoTCWgdIOLbBW2WmRgY8CFAPjeTM8= X-MC-Unique: F0uiVqdOObie9ZTAfNNeFA-1 From: Andrea Bolognani To: libvir-list@redhat.com Subject: [libvirt PATCH] ci: Reduce number of stages Date: Wed, 10 Jun 2020 13:33:01 +0200 Message-Id: <20200610113301.20446-1-abologna@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" Right now we're dividing the jobs into three stages: prebuild, which includes DCO checking as well as building artifacts such as the website, and native_build/cross_build, which do exactly what you'd expect based on their names. This organization is nice from the logical point of view, but results in poor utilization of the available CI resources: in particular, the fact that cross_build jobs can only start after all native_build jobs have finished means that if even a single one of the latter takes a bit longer the pipeline will stall, and with native builds taking anywhere from less than 10 minutes to more than 20, this happens all the time. Building artifacts in a separate pipeline stage also doesn't have any advantages, and only delays further stages by a couple of minutes. The only job that really makes sense in its own stage is the DCO check, because it's extremely fast (less than 1 minute) and, if that fails, we can avoid kicking off all other jobs. Reducing the number of stages results in significant speedups: specifically, going from three stages to two stages reduces the overall completion time for a full CI pipeline from ~45 minutes[1] to ~30 minutes[2]. [1] https://gitlab.com/abologna/libvirt/-/pipelines/154751893 [2] https://gitlab.com/abologna/libvirt/-/pipelines/154771173 Signed-off-by: Andrea Bolognani Reviewed-by: Daniel P. Berrang=C3=A9 --- .gitlab-ci.yml | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 7a8142b506..8d9313e415 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -2,9 +2,8 @@ variables: GIT_DEPTH: 100 =20 stages: - - prebuild - - native_build - - cross_build + - sanity_checks + - builds =20 .script_variables: &script_variables | export MAKEFLAGS=3D"-j$(getconf _NPROCESSORS_ONLN)" @@ -17,7 +16,7 @@ stages: =20 # Default native build jobs that are always run .native_build_default_job_template: &native_build_default_job_definition - stage: native_build + stage: builds cache: paths: - ccache/ @@ -42,7 +41,7 @@ stages: # system other than Linux. These jobs will only run if the required # setup has been performed on the GitLab account (see ci/README.rst). .cirrus_build_default_job_template: &cirrus_build_default_job_definition - stage: native_build + stage: builds image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master script: - cirrus-run ci/cirrus/$NAME.yml.j2 @@ -64,7 +63,7 @@ stages: =20 # Default cross build jobs that are always run .cross_build_default_job_template: &cross_build_default_job_definition - stage: cross_build + stage: builds cache: paths: - ccache/ @@ -194,7 +193,7 @@ mingw64-fedora-rawhide: # be deployed to the web root: # https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?j= ob=3Dwebsite website: - stage: prebuild + stage: builds before_script: - *script_variables script: @@ -216,7 +215,7 @@ website: =20 =20 codestyle: - stage: prebuild + stage: builds before_script: - *script_variables script: @@ -231,7 +230,7 @@ codestyle: # for translation usage: # https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?j= ob=3Dpotfile potfile: - stage: prebuild + stage: builds only: - master before_script: @@ -259,7 +258,7 @@ potfile: # this test on developer's personal forks from which # merge requests are submitted check-dco: - stage: prebuild + stage: sanity_checks image: registry.gitlab.com/libvirt/libvirt-ci/check-dco:master script: - /check-dco --=20 2.25.4