gitlab-ci/cirrus: Increase timeout to 100 minutes

[PATCH] gitlab-ci/cirrus: Increase timeout to 100 minutes

Posted by Stefan Hajnoczi 7 months, 2 weeks ago

The 80m timeout is not enough:

  672/832 qemu:block / io-qcow2-041          OK             39.77s   1 subtests passed
  Timed out!

https://gitlab.com/qemu-project/qemu/-/jobs/5058610599

Buglink: https://gitlab.com/qemu-project/qemu/-/issues/1882
Fixes: d06f3bf92267 ("gitlab-ci/cirrus: Increase timeout to 80 minutes")
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 .gitlab-ci.d/cirrus.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
index 41d64d6680..d19633f758 100644
--- a/.gitlab-ci.d/cirrus.yml
+++ b/.gitlab-ci.d/cirrus.yml
@@ -15,7 +15,7 @@
   stage: build
   image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
   needs: []
-  timeout: 80m
+  timeout: 100m
   allow_failure: true
   script:
     - source .gitlab-ci.d/cirrus/$NAME.vars
-- 
2.41.0

Re: [PATCH] gitlab-ci/cirrus: Increase timeout to 100 minutes

Posted by Daniel P. Berrangé 7 months, 2 weeks ago

On Tue, Sep 12, 2023 at 09:38:29AM -0400, Stefan Hajnoczi wrote:
> The 80m timeout is not enough:
> 
>   672/832 qemu:block / io-qcow2-041          OK             39.77s   1 subtests passed
>   Timed out!

IIUC, that 'timed out' message is coming from Cirrus CI logs, which
we can see over on the cirrus task:

  https://cirrus-ci.com/task/6462328380588032

> https://gitlab.com/qemu-project/qemu/-/jobs/5058610599

This reports duration "64 minutes", vs a GitLab timeout of 1hr20.

IOW, we're not hitting the gitlab timeout, we're hitting hte
Cirrus CI timeout, which defaults to 60 minutes.  The other
4 minuts gitlab reports is likely because Cirrus queued the
job for 4 minutes before starting execution.

> 
> Buglink: https://gitlab.com/qemu-project/qemu/-/issues/1882
> Fixes: d06f3bf92267 ("gitlab-ci/cirrus: Increase timeout to 80 minutes")
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  .gitlab-ci.d/cirrus.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> index 41d64d6680..d19633f758 100644
> --- a/.gitlab-ci.d/cirrus.yml
> +++ b/.gitlab-ci.d/cirrus.yml
> @@ -15,7 +15,7 @@
>    stage: build
>    image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
>    needs: []
> -  timeout: 80m
> +  timeout: 100m
>    allow_failure: true
>    script:
>      - source .gitlab-ci.d/cirrus/$NAME.vars

IIUC, we need to put a 'timeout_in' setting someone in
.gitlab-ci.d/cirrus/build.yml instead, to override
Cirrus 60 minute limit:

https://cirrus-ci.org/faq/#instance-timed-out


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] gitlab-ci/cirrus: Increase timeout to 100 minutes

Posted by Stefan Hajnoczi 7 months, 2 weeks ago

On Tue, 12 Sept 2023 at 09:53, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Sep 12, 2023 at 09:38:29AM -0400, Stefan Hajnoczi wrote:
> > The 80m timeout is not enough:
> >
> >   672/832 qemu:block / io-qcow2-041          OK             39.77s   1 subtests passed
> >   Timed out!
>
> IIUC, that 'timed out' message is coming from Cirrus CI logs, which
> we can see over on the cirrus task:
>
>   https://cirrus-ci.com/task/6462328380588032
>
> > https://gitlab.com/qemu-project/qemu/-/jobs/5058610599
>
> This reports duration "64 minutes", vs a GitLab timeout of 1hr20.
>
> IOW, we're not hitting the gitlab timeout, we're hitting hte
> Cirrus CI timeout, which defaults to 60 minutes.  The other
> 4 minuts gitlab reports is likely because Cirrus queued the
> job for 4 minutes before starting execution.

I'm glad you spotted that. I'm not familiar with Cirrus. Could you
send a patch that sets 'timeout_in'?

Thanks,
Stefan

>
> >
> > Buglink: https://gitlab.com/qemu-project/qemu/-/issues/1882
> > Fixes: d06f3bf92267 ("gitlab-ci/cirrus: Increase timeout to 80 minutes")
> > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> > ---
> >  .gitlab-ci.d/cirrus.yml | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> > index 41d64d6680..d19633f758 100644
> > --- a/.gitlab-ci.d/cirrus.yml
> > +++ b/.gitlab-ci.d/cirrus.yml
> > @@ -15,7 +15,7 @@
> >    stage: build
> >    image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
> >    needs: []
> > -  timeout: 80m
> > +  timeout: 100m
> >    allow_failure: true
> >    script:
> >      - source .gitlab-ci.d/cirrus/$NAME.vars
>
> IIUC, we need to put a 'timeout_in' setting someone in
> .gitlab-ci.d/cirrus/build.yml instead, to override
> Cirrus 60 minute limit:
>
> https://cirrus-ci.org/faq/#instance-timed-out
>
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>
>

Re: [PATCH] gitlab-ci/cirrus: Increase timeout to 100 minutes

Posted by Daniel P. Berrangé 7 months, 2 weeks ago

On Tue, Sep 12, 2023 at 10:02:17AM -0400, Stefan Hajnoczi wrote:
> On Tue, 12 Sept 2023 at 09:53, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Tue, Sep 12, 2023 at 09:38:29AM -0400, Stefan Hajnoczi wrote:
> > > The 80m timeout is not enough:
> > >
> > >   672/832 qemu:block / io-qcow2-041          OK             39.77s   1 subtests passed
> > >   Timed out!
> >
> > IIUC, that 'timed out' message is coming from Cirrus CI logs, which
> > we can see over on the cirrus task:
> >
> >   https://cirrus-ci.com/task/6462328380588032
> >
> > > https://gitlab.com/qemu-project/qemu/-/jobs/5058610599
> >
> > This reports duration "64 minutes", vs a GitLab timeout of 1hr20.
> >
> > IOW, we're not hitting the gitlab timeout, we're hitting hte
> > Cirrus CI timeout, which defaults to 60 minutes.  The other
> > 4 minuts gitlab reports is likely because Cirrus queued the
> > job for 4 minutes before starting execution.
> 
> I'm glad you spotted that. I'm not familiar with Cirrus. Could you
> send a patch that sets 'timeout_in'?

Yes, testing now

  https://gitlab.com/berrange/qemu/-/commit/c15d677de5ed2965464bc6212f049ed9785c4434

  https://gitlab.com/berrange/qemu/-/jobs/5069195895

  https://cirrus-ci.com/task/5135339078025216

The cirrus CI job page looks to be picking up the elevated timeout.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] gitlab-ci/cirrus: Increase timeout to 100 minutes

Posted by Daniel P. Berrangé 7 months, 2 weeks ago

On Tue, Sep 12, 2023 at 03:15:28PM +0100, Daniel P. Berrangé wrote:
> On Tue, Sep 12, 2023 at 10:02:17AM -0400, Stefan Hajnoczi wrote:
> > On Tue, 12 Sept 2023 at 09:53, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > >
> > > On Tue, Sep 12, 2023 at 09:38:29AM -0400, Stefan Hajnoczi wrote:
> > > > The 80m timeout is not enough:
> > > >
> > > >   672/832 qemu:block / io-qcow2-041          OK             39.77s   1 subtests passed
> > > >   Timed out!
> > >
> > > IIUC, that 'timed out' message is coming from Cirrus CI logs, which
> > > we can see over on the cirrus task:
> > >
> > >   https://cirrus-ci.com/task/6462328380588032
> > >
> > > > https://gitlab.com/qemu-project/qemu/-/jobs/5058610599
> > >
> > > This reports duration "64 minutes", vs a GitLab timeout of 1hr20.
> > >
> > > IOW, we're not hitting the gitlab timeout, we're hitting hte
> > > Cirrus CI timeout, which defaults to 60 minutes.  The other
> > > 4 minuts gitlab reports is likely because Cirrus queued the
> > > job for 4 minutes before starting execution.
> > 
> > I'm glad you spotted that. I'm not familiar with Cirrus. Could you
> > send a patch that sets 'timeout_in'?
> 
> Yes, testing now
> 
>   https://gitlab.com/berrange/qemu/-/commit/c15d677de5ed2965464bc6212f049ed9785c4434
> 
>   https://gitlab.com/berrange/qemu/-/jobs/5069195895
> 
>   https://cirrus-ci.com/task/5135339078025216
> 
> The cirrus CI job page looks to be picking up the elevated timeout.

Still fails, and given it previously worked at 35 minutes, needing
a 1h30 timeout would be incredulous. This appears to be a genuine
hang in QEMU's test suite on FreeBSD 13.2.

I'm spinning up a VM to debug further

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] gitlab-ci/cirrus: Increase timeout to 100 minutes

Posted by Stefan Hajnoczi 7 months, 2 weeks ago

Thank you!

Stefan

On Tue, 12 Sept 2023 at 10:15, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Sep 12, 2023 at 10:02:17AM -0400, Stefan Hajnoczi wrote:
> > On Tue, 12 Sept 2023 at 09:53, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > >
> > > On Tue, Sep 12, 2023 at 09:38:29AM -0400, Stefan Hajnoczi wrote:
> > > > The 80m timeout is not enough:
> > > >
> > > >   672/832 qemu:block / io-qcow2-041          OK             39.77s   1 subtests passed
> > > >   Timed out!
> > >
> > > IIUC, that 'timed out' message is coming from Cirrus CI logs, which
> > > we can see over on the cirrus task:
> > >
> > >   https://cirrus-ci.com/task/6462328380588032
> > >
> > > > https://gitlab.com/qemu-project/qemu/-/jobs/5058610599
> > >
> > > This reports duration "64 minutes", vs a GitLab timeout of 1hr20.
> > >
> > > IOW, we're not hitting the gitlab timeout, we're hitting hte
> > > Cirrus CI timeout, which defaults to 60 minutes.  The other
> > > 4 minuts gitlab reports is likely because Cirrus queued the
> > > job for 4 minutes before starting execution.
> >
> > I'm glad you spotted that. I'm not familiar with Cirrus. Could you
> > send a patch that sets 'timeout_in'?
>
> Yes, testing now
>
>   https://gitlab.com/berrange/qemu/-/commit/c15d677de5ed2965464bc6212f049ed9785c4434
>
>   https://gitlab.com/berrange/qemu/-/jobs/5069195895
>
>   https://cirrus-ci.com/task/5135339078025216
>
> The cirrus CI job page looks to be picking up the elevated timeout.
>
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>