On Wed, 22 May 2024, Matthieu Baerts (NGI0) wrote:
> Some subtests can be unstable, failing once every X runs. Fixing them
> can take time: there could be an issue in the kernel or in the subtest,
> and it is then important to do a proper analysis, not to hide real bugs.
>
> To avoid creating noises on the different CIs, some subtests have been
> marked as flaky. As a result, errors with these subtests (if any) are
> ignored.
>
> Note that the MPTCP CI will continue to track these flaky subtests. All
> these unstable subtests are also tracked by our bug tracker.
>
> Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
> ---
> Changes in v3:
> - Patch 1/4:
> - use SELFTESTS_MPTCP_LIB_OVERRIDE_FLAKY env var (Mat)
> - clearly indicate 'flaky' in the output / TAP format
> - Target mptcp-net: these are "fixes" for the selftests. I don't think
> we need to add Fixes tag, right?
> - Link to v2: https://lore.kernel.org/r/20240521-selftests-mptcp-disable-flaky-v2-0-eba143bcbad0@kernel.org
>
Thanks for the v3, this version LGTM:
Reviewed-by: Mat Martineau <martineau@kernel.org>
> Changes in v2:
> - Patch 2/4: only mark "unbalanced bwidth" subtest as flaky (Paolo)
> - Link to v1: https://lore.kernel.org/r/20240520-selftests-mptcp-disable-flaky-v1-0-02e23ba0bc3b@kernel.org
>
> ---
> Matthieu Baerts (NGI0) (4):
> selftests: mptcp: lib: support flaky subtests
> selftests: mptcp: simult flows: mark 'unbalanced' tests as flaky
> selftests: mptcp: join: mark 'fastclose' tests as flaky
> selftests: mptcp: join: mark 'fail' tests as flaky
>
> tools/testing/selftests/net/mptcp/mptcp_join.sh | 10 +++++++-
> tools/testing/selftests/net/mptcp/mptcp_lib.sh | 30 +++++++++++++++++++++--
> tools/testing/selftests/net/mptcp/simult_flows.sh | 6 ++---
> 3 files changed, 40 insertions(+), 6 deletions(-)
> ---
> base-commit: d048726b432ccff9b4e3d8b0012a7328c5c7c2f6
> change-id: 20240520-selftests-mptcp-disable-flaky-380759d6503c
>
> Best regards,
> --
> Matthieu Baerts (NGI0) <matttbe@kernel.org>
>
>
>