guests/playbooks/build/jobs/prepare.yml | 1 + jenkins/jobs/autotools.yaml | 1 + jenkins/jobs/generic.yaml | 1 + jenkins/jobs/go.yaml | 1 + jenkins/jobs/perl-modulebuild.yaml | 1 + jenkins/jobs/python-distutils.yaml | 1 + 6 files changed, 6 insertions(+)
First of all, we don't need the full history for builds and we can save
some time during the prepare phase. The disk footprint is about 5x
smaller, but the most important thing is the repo clone duration
difference e.g. in libvirt's case is quite significant:
full history:
real 3m45.236s
user 1m53.074s
sys 0m15.577s
depth 1:
real 0m14.868s
user 0m2.215s
sys 0m0.906s
---
guests/playbooks/build/jobs/prepare.yml | 1 +
jenkins/jobs/autotools.yaml | 1 +
jenkins/jobs/generic.yaml | 1 +
jenkins/jobs/go.yaml | 1 +
jenkins/jobs/perl-modulebuild.yaml | 1 +
jenkins/jobs/python-distutils.yaml | 1 +
6 files changed, 6 insertions(+)
diff --git a/guests/playbooks/build/jobs/prepare.yml b/guests/playbooks/build/jobs/prepare.yml
index 576afbc..35ca9e4 100644
--- a/guests/playbooks/build/jobs/prepare.yml
+++ b/guests/playbooks/build/jobs/prepare.yml
@@ -5,6 +5,7 @@
version: '{{ git_branch }}'
dest: '{{ name }}'
force: yes
+ depth: 1
when:
- inventory_hostname in machines
diff --git a/jenkins/jobs/autotools.yaml b/jenkins/jobs/autotools.yaml
index f04ec17..1a85781 100644
--- a/jenkins/jobs/autotools.yaml
+++ b/jenkins/jobs/autotools.yaml
@@ -27,6 +27,7 @@
after: true
skip-tag: true
wipe-workspace: false
+ shallow-clone: true
triggers:
- reverse:
jobs: '{obj:parent_jobs}'
diff --git a/jenkins/jobs/generic.yaml b/jenkins/jobs/generic.yaml
index ca69513..f58945e 100644
--- a/jenkins/jobs/generic.yaml
+++ b/jenkins/jobs/generic.yaml
@@ -27,6 +27,7 @@
after: true
skip-tag: true
wipe-workspace: false
+ shallow-clone: true
triggers:
- reverse:
jobs: '{obj:parent_jobs}'
diff --git a/jenkins/jobs/go.yaml b/jenkins/jobs/go.yaml
index 1bdcb5f..dd3c5d4 100644
--- a/jenkins/jobs/go.yaml
+++ b/jenkins/jobs/go.yaml
@@ -27,6 +27,7 @@
after: true
skip-tag: true
wipe-workspace: false
+ shallow-clone: true
triggers:
- reverse:
jobs: '{obj:parent_jobs}'
diff --git a/jenkins/jobs/perl-modulebuild.yaml b/jenkins/jobs/perl-modulebuild.yaml
index 7173493..0ea271e 100644
--- a/jenkins/jobs/perl-modulebuild.yaml
+++ b/jenkins/jobs/perl-modulebuild.yaml
@@ -27,6 +27,7 @@
after: true
skip-tag: true
wipe-workspace: false
+ shallow-clone: true
triggers:
- reverse:
jobs: '{obj:parent_jobs}'
diff --git a/jenkins/jobs/python-distutils.yaml b/jenkins/jobs/python-distutils.yaml
index f56794c..a3eb076 100644
--- a/jenkins/jobs/python-distutils.yaml
+++ b/jenkins/jobs/python-distutils.yaml
@@ -27,6 +27,7 @@
after: true
skip-tag: true
wipe-workspace: false
+ shallow-clone: true
triggers:
- reverse:
jobs: '{obj:parent_jobs}'
--
2.20.1
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
On Tue, Aug 27, 2019 at 02:56:02PM +0200, Erik Skultety wrote:
> First of all, we don't need the full history for builds and we can save
> some time during the prepare phase. The disk footprint is about 5x
> smaller, but the most important thing is the repo clone duration
> difference e.g. in libvirt's case is quite significant:
>
> full history:
> real 3m45.236s
> user 1m53.074s
> sys 0m15.577s
>
> depth 1:
> real 0m14.868s
> user 0m2.215s
> sys 0m0.906s
On Jenkins, the perf diff should be negligible. The CI build host
already has a full git clone with entire history. So all the worker
VMs are effectively cloning from localhost, not hitting the WAN.
They also are never doing a full git clone, just a git pull to get
the delta from the previous run.
Personally I find it useful to have the history when I'm debugging
problems on my build host/VM, so I'm not really in favour of doing
this change. It would make the VMs setup less usable for me, so I
prefer to accept the time penalty.
Is there something we can do to make the standalone VMs pull from
a repo on the build host by default, instead of fetching from
libvirt.org. Or to get their initial clone from the host, but
then get subsequent incremental updates from libvirt.org ?
> ---
> guests/playbooks/build/jobs/prepare.yml | 1 +
> jenkins/jobs/autotools.yaml | 1 +
> jenkins/jobs/generic.yaml | 1 +
> jenkins/jobs/go.yaml | 1 +
> jenkins/jobs/perl-modulebuild.yaml | 1 +
> jenkins/jobs/python-distutils.yaml | 1 +
> 6 files changed, 6 insertions(+)
>
> diff --git a/guests/playbooks/build/jobs/prepare.yml b/guests/playbooks/build/jobs/prepare.yml
> index 576afbc..35ca9e4 100644
> --- a/guests/playbooks/build/jobs/prepare.yml
> +++ b/guests/playbooks/build/jobs/prepare.yml
> @@ -5,6 +5,7 @@
> version: '{{ git_branch }}'
> dest: '{{ name }}'
> force: yes
> + depth: 1
> when:
> - inventory_hostname in machines
>
> diff --git a/jenkins/jobs/autotools.yaml b/jenkins/jobs/autotools.yaml
> index f04ec17..1a85781 100644
> --- a/jenkins/jobs/autotools.yaml
> +++ b/jenkins/jobs/autotools.yaml
> @@ -27,6 +27,7 @@
> after: true
> skip-tag: true
> wipe-workspace: false
> + shallow-clone: true
> triggers:
> - reverse:
> jobs: '{obj:parent_jobs}'
> diff --git a/jenkins/jobs/generic.yaml b/jenkins/jobs/generic.yaml
> index ca69513..f58945e 100644
> --- a/jenkins/jobs/generic.yaml
> +++ b/jenkins/jobs/generic.yaml
> @@ -27,6 +27,7 @@
> after: true
> skip-tag: true
> wipe-workspace: false
> + shallow-clone: true
> triggers:
> - reverse:
> jobs: '{obj:parent_jobs}'
> diff --git a/jenkins/jobs/go.yaml b/jenkins/jobs/go.yaml
> index 1bdcb5f..dd3c5d4 100644
> --- a/jenkins/jobs/go.yaml
> +++ b/jenkins/jobs/go.yaml
> @@ -27,6 +27,7 @@
> after: true
> skip-tag: true
> wipe-workspace: false
> + shallow-clone: true
> triggers:
> - reverse:
> jobs: '{obj:parent_jobs}'
> diff --git a/jenkins/jobs/perl-modulebuild.yaml b/jenkins/jobs/perl-modulebuild.yaml
> index 7173493..0ea271e 100644
> --- a/jenkins/jobs/perl-modulebuild.yaml
> +++ b/jenkins/jobs/perl-modulebuild.yaml
> @@ -27,6 +27,7 @@
> after: true
> skip-tag: true
> wipe-workspace: false
> + shallow-clone: true
> triggers:
> - reverse:
> jobs: '{obj:parent_jobs}'
> diff --git a/jenkins/jobs/python-distutils.yaml b/jenkins/jobs/python-distutils.yaml
> index f56794c..a3eb076 100644
> --- a/jenkins/jobs/python-distutils.yaml
> +++ b/jenkins/jobs/python-distutils.yaml
> @@ -27,6 +27,7 @@
> after: true
> skip-tag: true
> wipe-workspace: false
> + shallow-clone: true
> triggers:
> - reverse:
> jobs: '{obj:parent_jobs}'
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
On Tue, Aug 27, 2019 at 02:12:12PM +0100, Daniel P. Berrangé wrote: > On Tue, Aug 27, 2019 at 02:56:02PM +0200, Erik Skultety wrote: > > First of all, we don't need the full history for builds and we can save > > some time during the prepare phase. The disk footprint is about 5x > > smaller, but the most important thing is the repo clone duration > > difference e.g. in libvirt's case is quite significant: > > > > full history: > > real 3m45.236s > > user 1m53.074s > > sys 0m15.577s > > > > depth 1: > > real 0m14.868s > > user 0m2.215s > > sys 0m0.906s > > On Jenkins, the perf diff should be negligible. The CI build host > already has a full git clone with entire history. So all the worker > VMs are effectively cloning from localhost, not hitting the WAN. > They also are never doing a full git clone, just a git pull to get > the delta from the previous run. > > Personally I find it useful to have the history when I'm debugging > problems on my build host/VM, so I'm not really in favour of doing > this change. It would make the VMs setup less usable for me, so I > prefer to accept the time penalty. Hmm, I have often machines that come and go, could we make this a variable then? If set we'd make a shallow copy and by default leave it 'false' for the local CI build host and drop the jenkins changes? > > Is there something we can do to make the standalone VMs pull from > a repo on the build host by default, instead of fetching from > libvirt.org. Or to get their initial clone from the host, but > then get subsequent incremental updates from libvirt.org ? For that, you need to change the git_url in guests/playbooks/build/jobs/defaults.yml locally, not something we can make universal for everyone. Erik -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On Tue, Aug 27, 2019 at 03:39:53PM +0200, Erik Skultety wrote: > On Tue, Aug 27, 2019 at 02:12:12PM +0100, Daniel P. Berrangé wrote: > > On Tue, Aug 27, 2019 at 02:56:02PM +0200, Erik Skultety wrote: > > > First of all, we don't need the full history for builds and we can save > > > some time during the prepare phase. The disk footprint is about 5x > > > smaller, but the most important thing is the repo clone duration > > > difference e.g. in libvirt's case is quite significant: > > > > > > full history: > > > real 3m45.236s > > > user 1m53.074s > > > sys 0m15.577s > > > > > > depth 1: > > > real 0m14.868s > > > user 0m2.215s > > > sys 0m0.906s > > > > On Jenkins, the perf diff should be negligible. The CI build host > > already has a full git clone with entire history. So all the worker > > VMs are effectively cloning from localhost, not hitting the WAN. > > They also are never doing a full git clone, just a git pull to get > > the delta from the previous run. > > > > Personally I find it useful to have the history when I'm debugging > > problems on my build host/VM, so I'm not really in favour of doing > > this change. It would make the VMs setup less usable for me, so I > > prefer to accept the time penalty. > > Hmm, I have often machines that come and go, could we make this a variable > then? If set we'd make a shallow copy and by default leave it 'false' for the > local CI build host and drop the jenkins changes? Yeah that sounds ok Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
© 2016 - 2024 Red Hat, Inc.