1. Patchew
  2. QEMU
  3. View series

[PATCH] ci: add automatic retry

Pierrick Bouvier posted 1 patch 2 days, 21 hours ago 
.gitlab-ci.d/base.yml | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
[PATCH] ci: add automatic retry
Posted by Pierrick Bouvier 2 days, 21 hours ago 
When merging PR on the project, it's possible to observe failures on CI
that can be related to sporadic test failures or infrastructure issues
(pod eviction with Kubernetes). As a result, it adds latency on the
process and workload for our top maintainers.

This patch sets an automatic retry (supported by gitlab out of the box),
so failing jobs are automatically retriggered.

It's important to note that in case there is a real test or build
failure, concerned jobs will run a second time, and fail again. That
said, it happens seldomly since sub maintainers are supposed to test for
this before sending it. Also, the retry is limited to the job that
failed, so it should be cheap in terms of CI minutes.

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
---
 .gitlab-ci.d/base.yml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index 8fe00809357..63ba82e3632 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -31,8 +31,9 @@ variables:
     # Ensure docker.py / tests/docker/Makefile.include always displays stdout
     # from any docker commands to aid debugging of failures
     DOCKER_V: 1
-
   interruptible: true
+  # automatic retry on failure (sporadic test failures or infra issues)
+  retry: 2
 
   rules:
     #############################################################
-- 
2.53.0
Re: [PATCH] ci: add automatic retry
Posted by Pierrick Bouvier 1 day, 3 hours ago 
On 10/20/2026 4:16 PM, Pierrick Bouvier wrote:
> When merging PR on the project, it's possible to observe failures on CI
> that can be related to sporadic test failures or infrastructure issues
> (pod eviction with Kubernetes). As a result, it adds latency on the
> process and workload for our top maintainers.
> 
> This patch sets an automatic retry (supported by gitlab out of the box),
> so failing jobs are automatically retriggered.
> 
> It's important to note that in case there is a real test or build
> failure, concerned jobs will run a second time, and fail again. That
> said, it happens seldomly since sub maintainers are supposed to test for
> this before sending it. Also, the retry is limited to the job that
> failed, so it should be cheap in terms of CI minutes.
> 
> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
> ---
>  .gitlab-ci.d/base.yml | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
> index 8fe00809357..63ba82e3632 100644
> --- a/.gitlab-ci.d/base.yml
> +++ b/.gitlab-ci.d/base.yml
> @@ -31,8 +31,9 @@ variables:
>      # Ensure docker.py / tests/docker/Makefile.include always displays stdout
>      # from any docker commands to aid debugging of failures
>      DOCKER_V: 1
> -
>    interruptible: true
> +  # automatic retry on failure (sporadic test failures or infra issues)
> +  retry: 2
>  
>    rules:
>      #############################################################

This was merged into master (3ad81d3d271cef6ce95b3a3a33bb71faabef56a2).

First pipeline was green without requiring a manual retry:
https://lore.kernel.org/qemu-devel/c0774bbe-dcee-489f-862c-f481e58c1d73@oss.qualcomm.com/T/#md488890f5d439b02b5da24d7770e71f7e6283390

Regards,
Pierrick
Re: [PATCH] ci: add automatic retry
Posted by Richard Henderson 2 days, 12 hours ago 
On 10/21/26 01:16, Pierrick Bouvier wrote:
> When merging PR on the project, it's possible to observe failures on CI
> that can be related to sporadic test failures or infrastructure issues
> (pod eviction with Kubernetes). As a result, it adds latency on the
> process and workload for our top maintainers.
> 
> This patch sets an automatic retry (supported by gitlab out of the box),
> so failing jobs are automatically retriggered.
> 
> It's important to note that in case there is a real test or build
> failure, concerned jobs will run a second time, and fail again. That
> said, it happens seldomly since sub maintainers are supposed to test for
> this before sending it. Also, the retry is limited to the job that
> failed, so it should be cheap in terms of CI minutes.
> 
> Signed-off-by: Pierrick Bouvier<pierrick.bouvier@oss.qualcomm.com>
> ---
>   .gitlab-ci.d/base.yml | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)

Acked-by: Richard Henderson <richard.henderson@linaro.org>

This is effectively what we're doing anyway, modulo looking for real 
failures first.


r~
Re: [PATCH] ci: add automatic retry
Posted by Brian Cain 2 days, 18 hours ago 
On 10/20/2026 6:16 PM, Pierrick Bouvier wrote:
> When merging PR on the project, it's possible to observe failures on CI
> that can be related to sporadic test failures or infrastructure issues
> (pod eviction with Kubernetes). As a result, it adds latency on the
> process and workload for our top maintainers.
>
> This patch sets an automatic retry (supported by gitlab out of the box),
> so failing jobs are automatically retriggered.
>
> It's important to note that in case there is a real test or build
> failure, concerned jobs will run a second time, and fail again. That
> said, it happens seldomly since sub maintainers are supposed to test for
> this before sending it. Also, the retry is limited to the job that
> failed, so it should be cheap in terms of CI minutes.

Will this effectively mask those flaky failures so that no one hears 
about them?  Is there a way to find out about them with something like a 
weekly/monthly digest of failures?


>
> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
> ---
>   .gitlab-ci.d/base.yml | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
> index 8fe00809357..63ba82e3632 100644
> --- a/.gitlab-ci.d/base.yml
> +++ b/.gitlab-ci.d/base.yml
> @@ -31,8 +31,9 @@ variables:
>       # Ensure docker.py / tests/docker/Makefile.include always displays stdout
>       # from any docker commands to aid debugging of failures
>       DOCKER_V: 1
> -
>     interruptible: true
> +  # automatic retry on failure (sporadic test failures or infra issues)
> +  retry: 2
>   
>     rules:
>       #############################################################
Re: [PATCH] ci: add automatic retry
Posted by Peter Maydell 2 days, 11 hours ago 
On Wed, 21 Oct 2026 at 03:41, Brian Cain <brian.cain@oss.qualcomm.com> wrote:
>
>
> On 10/20/2026 6:16 PM, Pierrick Bouvier wrote:
> > When merging PR on the project, it's possible to observe failures on CI
> > that can be related to sporadic test failures or infrastructure issues
> > (pod eviction with Kubernetes). As a result, it adds latency on the
> > process and workload for our top maintainers.
> >
> > This patch sets an automatic retry (supported by gitlab out of the box),
> > so failing jobs are automatically retriggered.
> >
> > It's important to note that in case there is a real test or build
> > failure, concerned jobs will run a second time, and fail again. That
> > said, it happens seldomly since sub maintainers are supposed to test for
> > this before sending it. Also, the retry is limited to the job that
> > failed, so it should be cheap in terms of CI minutes.
>
> Will this effectively mask those flaky failures so that no one hears
> about them?  Is there a way to find out about them with something like a
> weekly/monthly digest of failures?

Broadly speaking, it's already the case that flaky failures are masked.
Random infra-type failures / random timeouts /etc are common enough
that when I'm doing a merge CI run I will just hit the retry button
without much deep investigation.

In an ideal world we'd pull down the gitlab job history and look
through it for cases of "job X failed running against commit hash Y,
but we also saw job X succeed on commit hash Y" (i.e. the retry
worked second time) and then check the failure logs for what
kinds of things are intermittent like that. But that requires
somebody to put in the work to do that analysis. Gitlab's own
CI job history web UI is very bad and doesn't let you do even
basic things like "show me just the history for job X".

thanks
-- PMM
Re: [PATCH] ci: add automatic retry
Posted by Alex Bennée 2 days, 9 hours ago 
Peter Maydell <peter.maydell@linaro.org> writes:

> On Wed, 21 Oct 2026 at 03:41, Brian Cain <brian.cain@oss.qualcomm.com> wrote:
>>
>>
>> On 10/20/2026 6:16 PM, Pierrick Bouvier wrote:
>> > When merging PR on the project, it's possible to observe failures on CI
>> > that can be related to sporadic test failures or infrastructure issues
>> > (pod eviction with Kubernetes). As a result, it adds latency on the
>> > process and workload for our top maintainers.
>> >
>> > This patch sets an automatic retry (supported by gitlab out of the box),
>> > so failing jobs are automatically retriggered.
>> >
>> > It's important to note that in case there is a real test or build
>> > failure, concerned jobs will run a second time, and fail again. That
>> > said, it happens seldomly since sub maintainers are supposed to test for
>> > this before sending it. Also, the retry is limited to the job that
>> > failed, so it should be cheap in terms of CI minutes.
>>
>> Will this effectively mask those flaky failures so that no one hears
>> about them?  Is there a way to find out about them with something like a
>> weekly/monthly digest of failures?
>
> Broadly speaking, it's already the case that flaky failures are masked.
> Random infra-type failures / random timeouts /etc are common enough
> that when I'm doing a merge CI run I will just hit the retry button
> without much deep investigation.
>
> In an ideal world we'd pull down the gitlab job history and look
> through it for cases of "job X failed running against commit hash Y,
> but we also saw job X succeed on commit hash Y" (i.e. the retry
> worked second time) and then check the failure logs for what
> kinds of things are intermittent like that. But that requires
> somebody to put in the work to do that analysis. Gitlab's own
> CI job history web UI is very bad and doesn't let you do even
> basic things like "show me just the history for job X".

The gitlab-failure-analysis script was an attempt to start doing this,
so the last 10 failing staging pipelines where:

  $ ./scripts/ci/gitlab-failure-analysis --count 10
  failed pipeline 2949515618, total jobs 120, skipped 5, failed 8,  45059 tests, 1112 skipped tests, 8 failed tests
    Failed job 17194356901, block: [luks], https://gitlab.com/qemu-project/qemu/-/jobs/17194356901
    Failed job 17192659533, block: [vmdk], https://gitlab.com/qemu-project/qemu/-/jobs/17192659533
    Failed job 17192659440, block: [vhdx], https://gitlab.com/qemu-project/qemu/-/jobs/17192659440
    Failed job 17192659398, block: [vdi], https://gitlab.com/qemu-project/qemu/-/jobs/17192659398
    Failed job 17192659309, block: [qed], https://gitlab.com/qemu-project/qemu/-/jobs/17192659309
    Failed job 17192659287, block: [qcow2], https://gitlab.com/qemu-project/qemu/-/jobs/17192659287
    Failed job 17192659245, block: [parallels], https://gitlab.com/qemu-project/qemu/-/jobs/17192659245
    Failed job 17192659237, block: [nbd], https://gitlab.com/qemu-project/qemu/-/jobs/17192659237
    Failed test qemu.block-nbd-optional+block-thorough - qemu:io-nbd-nbd-stale-connection, block: [nbd], 1 raw nbd-stale-connection
    Failed test qemu.block-parallels-optional+block-thorough - qemu:io-parallels-nbd-stale-connection, block: [parallels], 1 parallels nbd-stale-connection
    Failed test qemu.block-qcow2-optional+block-slow - qemu:io-qcow2-nbd-stale-connection, block: [qcow2], 1 qcow2 nbd-stale-connection
    Failed test qemu.block-qed-optional+block-thorough - qemu:io-qed-nbd-stale-connection, block: [qed], 1 qed nbd-stale-connection
    Failed test qemu.block-vdi-optional+block-thorough - qemu:io-vdi-nbd-stale-connection, block: [vdi], 1 vdi nbd-stale-connection
    Failed test qemu.block-vhdx-optional+block-thorough - qemu:io-vhdx-nbd-stale-connection, block: [vhdx], 1 vhdx nbd-stale-connection
    Failed test qemu.block-vmdk-optional+block-thorough - qemu:io-vmdk-nbd-stale-connection, block: [vmdk], 1 vmdk nbd-stale-connection
    Failed test qemu.block-luks-optional+block-thorough - qemu:io-luks-nbd-stale-connection, block: [luks], 1 luks nbd-stale-connection
  failed pipeline 2945631627, total jobs 120, skipped 5, failed 13,  0 tests, 0 skipped tests, 0 failed tests
    Failed job 17162982778, check-python-minreqs, https://gitlab.com/qemu-project/qemu/-/jobs/17162982778
    Failed job 17162982652, block: [vpc], https://gitlab.com/qemu-project/qemu/-/jobs/17162982652
    Failed job 17162982622, block: [vmdk], https://gitlab.com/qemu-project/qemu/-/jobs/17162982622
    Failed job 17162982605, block: [vhdx], https://gitlab.com/qemu-project/qemu/-/jobs/17162982605
    Failed job 17162982484, block: [vdi], https://gitlab.com/qemu-project/qemu/-/jobs/17162982484
    Failed job 17162982417, block: [raw], https://gitlab.com/qemu-project/qemu/-/jobs/17162982417
    Failed job 17162982413, block: [qed], https://gitlab.com/qemu-project/qemu/-/jobs/17162982413
    Failed job 17162982411, block: [qcow2], https://gitlab.com/qemu-project/qemu/-/jobs/17162982411
    Failed job 17162982380, block: [parallels], https://gitlab.com/qemu-project/qemu/-/jobs/17162982380
    Failed job 17162982361, block: [nbd], https://gitlab.com/qemu-project/qemu/-/jobs/17162982361
    Failed job 17162982339, block: [luks], https://gitlab.com/qemu-project/qemu/-/jobs/17162982339
    Failed job 17162261239, check-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/17162261239
    Failed job 17162261192, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/17162261192
  failed pipeline 2929887766, total jobs 120, skipped 7, failed 18,  0 tests, 0 skipped tests, 0 failed tests
    Failed job 17055360127, check-python-minreqs, https://gitlab.com/qemu-project/qemu/-/jobs/17055360127
    Failed job 17055360119, functional-system-opensuse, https://gitlab.com/qemu-project/qemu/-/jobs/17055360119
    Failed job 17055360117, functional-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/17055360117
    Failed job 17055360116, check-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/17055360116
    Failed job 17055360113, block: [vpc], https://gitlab.com/qemu-project/qemu/-/jobs/17055360113
    Failed job 17055360112, block: [vmdk], https://gitlab.com/qemu-project/qemu/-/jobs/17055360112
    Failed job 17055360111, block: [vhdx], https://gitlab.com/qemu-project/qemu/-/jobs/17055360111
    Failed job 17055360110, block: [vdi], https://gitlab.com/qemu-project/qemu/-/jobs/17055360110
    Failed job 17055360109, block: [raw], https://gitlab.com/qemu-project/qemu/-/jobs/17055360109
    Failed job 17055360108, block: [qed], https://gitlab.com/qemu-project/qemu/-/jobs/17055360108
    Failed job 17055360107, block: [qcow2], https://gitlab.com/qemu-project/qemu/-/jobs/17055360107
    Failed job 17055360106, block: [parallels], https://gitlab.com/qemu-project/qemu/-/jobs/17055360106
    Failed job 17055360105, block: [nbd], https://gitlab.com/qemu-project/qemu/-/jobs/17055360105
    Failed job 17055360104, block: [luks], https://gitlab.com/qemu-project/qemu/-/jobs/17055360104
    Failed job 17055360078, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/17055360078
    Failed job 17055360066, tsan-build, https://gitlab.com/qemu-project/qemu/-/jobs/17055360066
    Failed job 17055360065, build-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/17055360065
    Failed job 17055360061, clang-system, https://gitlab.com/qemu-project/qemu/-/jobs/17055360061
  failed pipeline 2908947622, total jobs 119, skipped 5, failed 5,  0 tests, 0 skipped tests, 0 failed tests
    Failed job 16911683051, functional-system-debian, https://gitlab.com/qemu-project/qemu/-/jobs/16911683051
    Failed job 16911681150, functional-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/16911681150
    Failed job 16911679766, functional-system-opensuse, https://gitlab.com/qemu-project/qemu/-/jobs/16911679766
    Failed job 16911678710, functional-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/16911678710
    Failed job 16911433872, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/16911433872
  failed pipeline 2906503036, total jobs 119, skipped 5, failed 4,  0 tests, 0 skipped tests, 0 failed tests
    Failed job 16896095104, functional-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/16896095104
    Failed job 16896090260, functional-system-opensuse, https://gitlab.com/qemu-project/qemu/-/jobs/16896090260
    Failed job 16896083592, functional-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/16896083592
    Failed job 16895432496, functional-system-debian, https://gitlab.com/qemu-project/qemu/-/jobs/16895432496
  failed pipeline 2876539219, total jobs 119, skipped 5, failed 6,  0 tests, 0 skipped tests, 0 failed tests
    Failed job 16692899473, functional-system-fedora, https://gitlab.com/qemu-project/qemu/-/jobs/16692899473
    Failed job 16692899464, aarch64-macos-26-build, https://gitlab.com/qemu-project/qemu/-/jobs/16692899464
    Failed job 16692899463, aarch64-macos-15-build, https://gitlab.com/qemu-project/qemu/-/jobs/16692899463
    Failed job 16692899432, clang-system, https://gitlab.com/qemu-project/qemu/-/jobs/16692899432
    Failed job 16692899429, build-some-softmmu, https://gitlab.com/qemu-project/qemu/-/jobs/16692899429
    Failed job 16692899413, cross-win64-system, https://gitlab.com/qemu-project/qemu/-/jobs/16692899413
  failed pipeline 2840522620, total jobs 119, skipped 5, failed 7,  0 tests, 0 skipped tests, 0 failed tests
    Failed job 16454091776, functional-system-opensuse, https://gitlab.com/qemu-project/qemu/-/jobs/16454091776
    Failed job 16454091774, functional-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/16454091774
    Failed job 16454091761, check-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/16454091761
    Failed job 16454091760, check-system-opensuse, https://gitlab.com/qemu-project/qemu/-/jobs/16454091760
    Failed job 16454091758, functional-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/16454091758
    Failed job 16454091740, check-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/16454091740
    Failed job 16454091730, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/16454091730
  failed pipeline 2839257817, total jobs 119, skipped 5, failed 2,  0 tests, 0 skipped tests, 0 failed tests
    Failed job 16436167533, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/16436167533
    Failed job 16436167532, ubuntu-24.04-s390x-all-linux, https://gitlab.com/qemu-project/qemu/-/jobs/16436167532
  failed pipeline 2830538304, total jobs 119, skipped 5, failed 3,  0 tests, 0 skipped tests, 0 failed tests
    Failed job 16377263801, check-rust-tools-nightly, https://gitlab.com/qemu-project/qemu/-/jobs/16377263801
    Failed job 16377263686, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/16377263686
    Failed job 16377263684, ubuntu-24.04-s390x-all-linux, https://gitlab.com/qemu-project/qemu/-/jobs/16377263684
  failed pipeline 2820341086, total jobs 119, skipped 5, failed 2,  0 tests, 0 skipped tests, 0 failed tests
    Failed job 16307778927, check-rust-tools-nightly, https://gitlab.com/qemu-project/qemu/-/jobs/16307778927
    Failed job 16307778897, functional-system-debian, https://gitlab.com/qemu-project/qemu/-/jobs/16307778897

It has rudimentary access to the tests but I think the deep dive would
require better retention of the job artefacts. The second pipeline in
the above list lists the "job artefacts" as removed 3hrs ago which is I
guess why we don't get the test breakdown for the earlier jobs.

Anyway perhaps we can expand this to be more useful?

>
> thanks
> -- PMM

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro
Re: [PATCH] ci: add automatic retry
Posted by Pierrick Bouvier 2 days, 4 hours ago 
On 10/21/2026 3:56 AM, Alex Bennée wrote:
> Peter Maydell <peter.maydell@linaro.org> writes:
> 
>> On Wed, 21 Oct 2026 at 03:41, Brian Cain <brian.cain@oss.qualcomm.com> wrote:
>>>
>>>
>>> On 10/20/2026 6:16 PM, Pierrick Bouvier wrote:
>>>> When merging PR on the project, it's possible to observe failures on CI
>>>> that can be related to sporadic test failures or infrastructure issues
>>>> (pod eviction with Kubernetes). As a result, it adds latency on the
>>>> process and workload for our top maintainers.
>>>>
>>>> This patch sets an automatic retry (supported by gitlab out of the box),
>>>> so failing jobs are automatically retriggered.
>>>>
>>>> It's important to note that in case there is a real test or build
>>>> failure, concerned jobs will run a second time, and fail again. That
>>>> said, it happens seldomly since sub maintainers are supposed to test for
>>>> this before sending it. Also, the retry is limited to the job that
>>>> failed, so it should be cheap in terms of CI minutes.
>>>
>>> Will this effectively mask those flaky failures so that no one hears
>>> about them?  Is there a way to find out about them with something like a
>>> weekly/monthly digest of failures?
>>
>> Broadly speaking, it's already the case that flaky failures are masked.
>> Random infra-type failures / random timeouts /etc are common enough
>> that when I'm doing a merge CI run I will just hit the retry button
>> without much deep investigation.
>>
>> In an ideal world we'd pull down the gitlab job history and look
>> through it for cases of "job X failed running against commit hash Y,
>> but we also saw job X succeed on commit hash Y" (i.e. the retry
>> worked second time) and then check the failure logs for what
>> kinds of things are intermittent like that. But that requires
>> somebody to put in the work to do that analysis. Gitlab's own
>> CI job history web UI is very bad and doesn't let you do even
>> basic things like "show me just the history for job X".
> 
> The gitlab-failure-analysis script was an attempt to start doing this,
> so the last 10 failing staging pipelines where:
> 
>   $ ./scripts/ci/gitlab-failure-analysis --count 10
>   failed pipeline 2949515618, total jobs 120, skipped 5, failed 8,  45059 tests, 1112 skipped tests, 8 failed tests
>     Failed job 17194356901, block: [luks], https://gitlab.com/qemu-project/qemu/-/jobs/17194356901
>     Failed job 17192659533, block: [vmdk], https://gitlab.com/qemu-project/qemu/-/jobs/17192659533
>     Failed job 17192659440, block: [vhdx], https://gitlab.com/qemu-project/qemu/-/jobs/17192659440
>     Failed job 17192659398, block: [vdi], https://gitlab.com/qemu-project/qemu/-/jobs/17192659398
>     Failed job 17192659309, block: [qed], https://gitlab.com/qemu-project/qemu/-/jobs/17192659309
>     Failed job 17192659287, block: [qcow2], https://gitlab.com/qemu-project/qemu/-/jobs/17192659287
>     Failed job 17192659245, block: [parallels], https://gitlab.com/qemu-project/qemu/-/jobs/17192659245
>     Failed job 17192659237, block: [nbd], https://gitlab.com/qemu-project/qemu/-/jobs/17192659237
>     Failed test qemu.block-nbd-optional+block-thorough - qemu:io-nbd-nbd-stale-connection, block: [nbd], 1 raw nbd-stale-connection
>     Failed test qemu.block-parallels-optional+block-thorough - qemu:io-parallels-nbd-stale-connection, block: [parallels], 1 parallels nbd-stale-connection
>     Failed test qemu.block-qcow2-optional+block-slow - qemu:io-qcow2-nbd-stale-connection, block: [qcow2], 1 qcow2 nbd-stale-connection
>     Failed test qemu.block-qed-optional+block-thorough - qemu:io-qed-nbd-stale-connection, block: [qed], 1 qed nbd-stale-connection
>     Failed test qemu.block-vdi-optional+block-thorough - qemu:io-vdi-nbd-stale-connection, block: [vdi], 1 vdi nbd-stale-connection
>     Failed test qemu.block-vhdx-optional+block-thorough - qemu:io-vhdx-nbd-stale-connection, block: [vhdx], 1 vhdx nbd-stale-connection
>     Failed test qemu.block-vmdk-optional+block-thorough - qemu:io-vmdk-nbd-stale-connection, block: [vmdk], 1 vmdk nbd-stale-connection
>     Failed test qemu.block-luks-optional+block-thorough - qemu:io-luks-nbd-stale-connection, block: [luks], 1 luks nbd-stale-connection
>   failed pipeline 2945631627, total jobs 120, skipped 5, failed 13,  0 tests, 0 skipped tests, 0 failed tests
>     Failed job 17162982778, check-python-minreqs, https://gitlab.com/qemu-project/qemu/-/jobs/17162982778
>     Failed job 17162982652, block: [vpc], https://gitlab.com/qemu-project/qemu/-/jobs/17162982652
>     Failed job 17162982622, block: [vmdk], https://gitlab.com/qemu-project/qemu/-/jobs/17162982622
>     Failed job 17162982605, block: [vhdx], https://gitlab.com/qemu-project/qemu/-/jobs/17162982605
>     Failed job 17162982484, block: [vdi], https://gitlab.com/qemu-project/qemu/-/jobs/17162982484
>     Failed job 17162982417, block: [raw], https://gitlab.com/qemu-project/qemu/-/jobs/17162982417
>     Failed job 17162982413, block: [qed], https://gitlab.com/qemu-project/qemu/-/jobs/17162982413
>     Failed job 17162982411, block: [qcow2], https://gitlab.com/qemu-project/qemu/-/jobs/17162982411
>     Failed job 17162982380, block: [parallels], https://gitlab.com/qemu-project/qemu/-/jobs/17162982380
>     Failed job 17162982361, block: [nbd], https://gitlab.com/qemu-project/qemu/-/jobs/17162982361
>     Failed job 17162982339, block: [luks], https://gitlab.com/qemu-project/qemu/-/jobs/17162982339
>     Failed job 17162261239, check-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/17162261239
>     Failed job 17162261192, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/17162261192
>   failed pipeline 2929887766, total jobs 120, skipped 7, failed 18,  0 tests, 0 skipped tests, 0 failed tests
>     Failed job 17055360127, check-python-minreqs, https://gitlab.com/qemu-project/qemu/-/jobs/17055360127
>     Failed job 17055360119, functional-system-opensuse, https://gitlab.com/qemu-project/qemu/-/jobs/17055360119
>     Failed job 17055360117, functional-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/17055360117
>     Failed job 17055360116, check-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/17055360116
>     Failed job 17055360113, block: [vpc], https://gitlab.com/qemu-project/qemu/-/jobs/17055360113
>     Failed job 17055360112, block: [vmdk], https://gitlab.com/qemu-project/qemu/-/jobs/17055360112
>     Failed job 17055360111, block: [vhdx], https://gitlab.com/qemu-project/qemu/-/jobs/17055360111
>     Failed job 17055360110, block: [vdi], https://gitlab.com/qemu-project/qemu/-/jobs/17055360110
>     Failed job 17055360109, block: [raw], https://gitlab.com/qemu-project/qemu/-/jobs/17055360109
>     Failed job 17055360108, block: [qed], https://gitlab.com/qemu-project/qemu/-/jobs/17055360108
>     Failed job 17055360107, block: [qcow2], https://gitlab.com/qemu-project/qemu/-/jobs/17055360107
>     Failed job 17055360106, block: [parallels], https://gitlab.com/qemu-project/qemu/-/jobs/17055360106
>     Failed job 17055360105, block: [nbd], https://gitlab.com/qemu-project/qemu/-/jobs/17055360105
>     Failed job 17055360104, block: [luks], https://gitlab.com/qemu-project/qemu/-/jobs/17055360104
>     Failed job 17055360078, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/17055360078
>     Failed job 17055360066, tsan-build, https://gitlab.com/qemu-project/qemu/-/jobs/17055360066
>     Failed job 17055360065, build-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/17055360065
>     Failed job 17055360061, clang-system, https://gitlab.com/qemu-project/qemu/-/jobs/17055360061
>   failed pipeline 2908947622, total jobs 119, skipped 5, failed 5,  0 tests, 0 skipped tests, 0 failed tests
>     Failed job 16911683051, functional-system-debian, https://gitlab.com/qemu-project/qemu/-/jobs/16911683051
>     Failed job 16911681150, functional-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/16911681150
>     Failed job 16911679766, functional-system-opensuse, https://gitlab.com/qemu-project/qemu/-/jobs/16911679766
>     Failed job 16911678710, functional-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/16911678710
>     Failed job 16911433872, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/16911433872
>   failed pipeline 2906503036, total jobs 119, skipped 5, failed 4,  0 tests, 0 skipped tests, 0 failed tests
>     Failed job 16896095104, functional-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/16896095104
>     Failed job 16896090260, functional-system-opensuse, https://gitlab.com/qemu-project/qemu/-/jobs/16896090260
>     Failed job 16896083592, functional-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/16896083592
>     Failed job 16895432496, functional-system-debian, https://gitlab.com/qemu-project/qemu/-/jobs/16895432496
>   failed pipeline 2876539219, total jobs 119, skipped 5, failed 6,  0 tests, 0 skipped tests, 0 failed tests
>     Failed job 16692899473, functional-system-fedora, https://gitlab.com/qemu-project/qemu/-/jobs/16692899473
>     Failed job 16692899464, aarch64-macos-26-build, https://gitlab.com/qemu-project/qemu/-/jobs/16692899464
>     Failed job 16692899463, aarch64-macos-15-build, https://gitlab.com/qemu-project/qemu/-/jobs/16692899463
>     Failed job 16692899432, clang-system, https://gitlab.com/qemu-project/qemu/-/jobs/16692899432
>     Failed job 16692899429, build-some-softmmu, https://gitlab.com/qemu-project/qemu/-/jobs/16692899429
>     Failed job 16692899413, cross-win64-system, https://gitlab.com/qemu-project/qemu/-/jobs/16692899413
>   failed pipeline 2840522620, total jobs 119, skipped 5, failed 7,  0 tests, 0 skipped tests, 0 failed tests
>     Failed job 16454091776, functional-system-opensuse, https://gitlab.com/qemu-project/qemu/-/jobs/16454091776
>     Failed job 16454091774, functional-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/16454091774
>     Failed job 16454091761, check-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/16454091761
>     Failed job 16454091760, check-system-opensuse, https://gitlab.com/qemu-project/qemu/-/jobs/16454091760
>     Failed job 16454091758, functional-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/16454091758
>     Failed job 16454091740, check-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/16454091740
>     Failed job 16454091730, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/16454091730
>   failed pipeline 2839257817, total jobs 119, skipped 5, failed 2,  0 tests, 0 skipped tests, 0 failed tests
>     Failed job 16436167533, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/16436167533
>     Failed job 16436167532, ubuntu-24.04-s390x-all-linux, https://gitlab.com/qemu-project/qemu/-/jobs/16436167532
>   failed pipeline 2830538304, total jobs 119, skipped 5, failed 3,  0 tests, 0 skipped tests, 0 failed tests
>     Failed job 16377263801, check-rust-tools-nightly, https://gitlab.com/qemu-project/qemu/-/jobs/16377263801
>     Failed job 16377263686, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/16377263686
>     Failed job 16377263684, ubuntu-24.04-s390x-all-linux, https://gitlab.com/qemu-project/qemu/-/jobs/16377263684
>   failed pipeline 2820341086, total jobs 119, skipped 5, failed 2,  0 tests, 0 skipped tests, 0 failed tests
>     Failed job 16307778927, check-rust-tools-nightly, https://gitlab.com/qemu-project/qemu/-/jobs/16307778927
>     Failed job 16307778897, functional-system-debian, https://gitlab.com/qemu-project/qemu/-/jobs/16307778897
> 
> It has rudimentary access to the tests but I think the deep dive would
> require better retention of the job artefacts. The second pipeline in
> the above list lists the "job artefacts" as removed 3hrs ago which is I
> guess why we don't get the test breakdown for the earlier jobs.
> 
> Anyway perhaps we can expand this to be more useful?
>

I guess take the latest 1000 runs, report failing jobs, see if there is
a pattern. Then for each run of job failing, see if they have a pattern
inside also.

Running CI on GitHub side, I found there are some recurrent flaky (that
I simply deactivated for now) but some real sporadic that are rarely
triggered. Since adding an automatic retry on individual tests, it's
almost green 100% of the time, which convinced me to propose this here.

>>
>> thanks
>> -- PMM
>
Re: [PATCH] ci: add automatic retry
Posted by Pierrick Bouvier 2 days, 15 hours ago 
On 10/20/26 7:41 PM, Brian Cain wrote:
> 
> On 10/20/2026 6:16 PM, Pierrick Bouvier wrote:
>> When merging PR on the project, it's possible to observe failures on CI
>> that can be related to sporadic test failures or infrastructure issues
>> (pod eviction with Kubernetes). As a result, it adds latency on the
>> process and workload for our top maintainers.
>>
>> This patch sets an automatic retry (supported by gitlab out of the box),
>> so failing jobs are automatically retriggered.
>>
>> It's important to note that in case there is a real test or build
>> failure, concerned jobs will run a second time, and fail again. That
>> said, it happens seldomly since sub maintainers are supposed to test for
>> this before sending it. Also, the retry is limited to the job that
>> failed, so it should be cheap in terms of CI minutes.
> 
> Will this effectively mask those flaky failures so that no one hears
> about them?  Is there a way to find out about them with something like a
> weekly/monthly digest of failures?
>

I don't have the exact measure about how many tries on average make sure 
CI is pass, so retry twice is just a first attempt.
Out of occasional flaky tests, there are also sporadic infra issues 
(related to our usage of Kubernetes).

Regards,
Pierrick

Patchew 2.1-devel-ea86937

© 2016 - 2026 Red Hat, Inc.