From nobody Mon Sep 16 19:18:38 2024 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2847079B9D for ; Wed, 22 May 2024 10:47:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716374874; cv=none; b=GSQupyAa+l+3eYMw8/wlQEawy9kgvxWA/ioy12JALA3e1siAN0KHrGDE9m1W+Rg1uJjTOaEU9hny2+0gwR1LiGhtSPhDB0B4Sqeewzg7ZZMId08SOJBfuh1hH9s6+SMWaYkzmlOpDRQ7J2SjLIFC0l4AdF3fpad1ZfQlX1K7HJM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716374874; c=relaxed/simple; bh=FNpzG5YrNfqfglGKIkGAIvhRVCv3J44sO+WOWiMI968=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=fKvKabNDWpB0VYx4GzeOiddUBHtznir/k6JxXMT4Nmsk6MiOMZRkKQdU7e/fSndNpdOPNmwiRHSUBl+hiJzkYsFCIJ9kA348FL+//kHYQKKwP9NZNpTucr36GSy0aiaW193DnOp6Uc3hWTak3c3UlSTAmtb0D6ALtiZjIpBO+4M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=cxnWKIAQ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="cxnWKIAQ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2E46EC32789; Wed, 22 May 2024 10:47:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1716374873; bh=FNpzG5YrNfqfglGKIkGAIvhRVCv3J44sO+WOWiMI968=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=cxnWKIAQNlKH/cncvYH/WUV1ttwaKnWqfMpL/O9n/HiCaXEz/aDekxEqvPSt5YMtJ mouvwniiqgDoNiYMAvQrOgKSmNDRp0gyCB7Sq11aImHb6xyyolG2cBjK2v+GFFTUo+ DrntuPMlgnj60isdCC0eDBYvEiY8PoBKNwDQCyMHldp/0jidGfkiN2OnxmQacJaRdd fRuizudANkFyRppDOz+vLYob30z03UT8a5xOnoHbo5gmC6QUlrsyT4SVgw4ueNkoJK icqdo+E+3T6vMvqK0I1h0AAK8sny7l6xa9EN9n4uStX3ch6U2iylBrFF+U8sZx5FFh xvxootLBYFqBg== From: "Matthieu Baerts (NGI0)" Date: Wed, 22 May 2024 12:47:43 +0200 Subject: [PATCH mptcp-net v3 1/4] selftests: mptcp: lib: support flaky subtests Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20240522-selftests-mptcp-disable-flaky-v3-1-a10c68bf2680@kernel.org> References: <20240522-selftests-mptcp-disable-flaky-v3-0-a10c68bf2680@kernel.org> In-Reply-To: <20240522-selftests-mptcp-disable-flaky-v3-0-a10c68bf2680@kernel.org> To: mptcp@lists.linux.dev Cc: "Matthieu Baerts (NGI0)" X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2918; i=matttbe@kernel.org; h=from:subject:message-id; bh=FNpzG5YrNfqfglGKIkGAIvhRVCv3J44sO+WOWiMI968=; b=owEBbQKS/ZANAwAIAfa3gk9CaaBzAcsmYgBmTc1XXAL6HkhDH52lBdEJDFepxRR1mVStdWqZS gVCX2xMP3GJAjMEAAEIAB0WIQToy4X3aHcFem4n93r2t4JPQmmgcwUCZk3NVwAKCRD2t4JPQmmg c6PZEADtf4SfVFPMt9pyDYtQUoBw50uUSkrBkf+bh6TQAjtR9Hkm9wWARJC4YspnIazqrbjx7a+ ixCcIpo5xma+iN73wKVHKaSiKj3NLTYUO4hFRtZlJOxBK/xYVLbYqJl3S3yZHm6rdxDkR8+FkTp Mt2xqqXWBHjN6TW75mf6+7tVbfT/2HRv4KX+2Q+N6PR1wBMUXptntiezDcCAJaw784qDv+6NKWN GZC7puZI0JCYLE1xpI/HrzfdU+CMlJA6fL9ygPWkCblvYOZCNptCrLb3kNOr6wiJ+qKE+ztY49m rD1Kw/aBLQUfE4M+IxTHaJOBuTM3Nb8M09gvghbyPOLxspKCjYuroo4Hnk2UgFcSFXQDhSp57a+ eF47KR1X6Y//kcV3HINvYN1GDL3Zr06nLrz42noW4vviSfZOdlAN/vXvIv+CwhcNF7yAylhQacK PD+masa1nic65N2ddNPr2Ld5vJmPv4jnKn7uZVKOxULPaJHLJthlBM8eqfoUvkJkAYu01dfRHto GlV25gcsoMvcZnsArdkYJF0RTDFbbUg1AqcU2Sf4dxD/LiAsV2LfFqmfCYmvBDm6aREkXnFsG4d SZdrz/OP82UMCAWJrVdlavfOAPfjPIP/U9W8B9f6tK/OtRv2aG+c+Haws6GnbkOl8DDNEMD6N8A novXjYeDBBjwe5A== X-Developer-Key: i=matttbe@kernel.org; a=openpgp; fpr=E8CB85F76877057A6E27F77AF6B7824F4269A073 Some subtests can be unstable, failing once every X runs. Fixing them can take time: there could be an issue in the kernel or in the subtest, and it is then important to do a proper analysis, not to hide real bugs. To avoid creating noises on the different CIs, it is important to have a simple way to mark subtests as flaky, and ignore the errors. This is what this patch introduces: subtests can be marked as flaky by setting MPTCP_LIB_SUBTEST_FLAKY env var to 1, e.g. MPTCP_LIB_SUBTEST_FLAKY=3D1 The subtest will be executed, and errors (if any) will be ignored. It is still good to run these subtests, as it exercises code, and the results can still be useful for the on-going investigations. Note that the MPTCP CI will continue to track these flaky subtests by setting SELFTESTS_MPTCP_LIB_OVERRIDE_FLAKY env var to 1, and a ticket has to be created before marking subtests as flaky. Signed-off-by: Matthieu Baerts (NGI0) Reviewed-by: Mat Martineau --- tools/testing/selftests/net/mptcp/mptcp_lib.sh | 30 ++++++++++++++++++++++= ++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/net/mptcp/mptcp_lib.sh b/tools/testing= /selftests/net/mptcp/mptcp_lib.sh index ad2ebda5cb64..6ffa9b7a3260 100644 --- a/tools/testing/selftests/net/mptcp/mptcp_lib.sh +++ b/tools/testing/selftests/net/mptcp/mptcp_lib.sh @@ -21,6 +21,7 @@ declare -rx MPTCP_LIB_AF_INET6=3D10 =20 MPTCP_LIB_SUBTESTS=3D() MPTCP_LIB_SUBTESTS_DUPLICATED=3D0 +MPTCP_LIB_SUBTEST_FLAKY=3D0 MPTCP_LIB_TEST_COUNTER=3D0 MPTCP_LIB_TEST_FORMAT=3D"%02u %-50s" MPTCP_LIB_IP_MPTCP=3D0 @@ -41,6 +42,16 @@ else readonly MPTCP_LIB_COLOR_RESET=3D fi =20 +# SELFTESTS_MPTCP_LIB_OVERRIDE_FLAKY env var can be set not to ignore erro= rs +# from subtests marked as flaky +mptcp_lib_override_flaky() { + [ "${SELFTESTS_MPTCP_LIB_OVERRIDE_FLAKY:-}" =3D 1 ] +} + +mptcp_lib_subtest_is_flaky() { + [ "${MPTCP_LIB_SUBTEST_FLAKY}" =3D 1 ] && ! mptcp_lib_override_flaky +} + # $1: color, $2: text mptcp_lib_print_color() { echo -e "${MPTCP_LIB_START_PRINT:-}${*}${MPTCP_LIB_COLOR_RESET}" @@ -72,7 +83,16 @@ mptcp_lib_pr_skip() { } =20 mptcp_lib_pr_fail() { - mptcp_lib_print_err "[FAIL]${1:+ ${*}}" + local title cmt + + if mptcp_lib_subtest_is_flaky; then + title=3D"IGNO" + cmt=3D" (flaky)" + else + title=3D"FAIL" + fi + + mptcp_lib_print_err "[${title}]${cmt}${1:+ ${*}}" } =20 mptcp_lib_pr_info() { @@ -208,7 +228,13 @@ mptcp_lib_result_pass() { =20 # $1: test name mptcp_lib_result_fail() { - __mptcp_lib_result_add "not ok" "${1}" + if mptcp_lib_subtest_is_flaky; then + # It might sound better to use 'not ok # TODO' or 'ok # SKIP', + # but some CIs don't understand 'TODO' and treat SKIP as errors. + __mptcp_lib_result_add "ok" "${1} # IGNORE Flaky" + else + __mptcp_lib_result_add "not ok" "${1}" + fi } =20 # $1: test name --=20 2.43.0 From nobody Mon Sep 16 19:18:38 2024 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2E03979B9D for ; Wed, 22 May 2024 10:47:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716374875; cv=none; b=cSfedJGzku8TP8QfD0E2kMmsJvLIDnPh3Xx2op2NOVROmt5Pk3dssUrPCcdJ9bX9hzsfcDOt/1SZB40rzfrV0tXjrk7pRiWbuskPeqPcE3nKiIt8g5c0dvpWHyBO8tc7pTf0gWzDi9mvczuIG8ysBeSJhwmAC4cZi0EIZYg3Dc4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716374875; c=relaxed/simple; bh=Ggy7iYS0IsFK7puQFPQ7OdbUwg1pe5OIKJkUZqeKNe0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=It27eQWOUkTb8q2hHYyNc7OSdy3WvOlcCUneLWzjij20FNoPeL6yWDyP57IqUDoO2H/5+0hQrGckFTOjjULXGewCqZvz3vpzc/eHYwhtmNJv4jmcpoRoCvrL0vP3+K8dw0ZX3Y4C+Z9woOpgTEc5WZ442cY9AJMwdb34Hgs78Fs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ja2jj8ok; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ja2jj8ok" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 33770C4AF07; Wed, 22 May 2024 10:47:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1716374874; bh=Ggy7iYS0IsFK7puQFPQ7OdbUwg1pe5OIKJkUZqeKNe0=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=ja2jj8ok1x+6n6mM8mLiXhBGi8ULKN/9/ZgIyBcwW8ceGPt62/SHbO+1LjybKea9q wEWOp4B+rBVJNwEKlcWAPr5F2zkqbI5W+bt90hewxHGodwgC4fKfexYCJ0LagdrXg4 paYn8L2/Q7OS4WUZb46HyJtAPREYsQjgYfM/mR9whb/1pddieNMGGOFyfdPr9eVkEQ w6qEM3TG+CFt22Tw2w5RjQ02Pb7ZEZRYhLYHRQ+14/FyAz+BqyBjYzkCEWN1cbLIIp 3+LwsA5JX1+Crrn28gvbzp3rjtl3cE77ybwNTMV3eYyNu9YYxm0zP6Y9RX4r/oM8XS XaJbfE6x2449w== From: "Matthieu Baerts (NGI0)" Date: Wed, 22 May 2024 12:47:44 +0200 Subject: [PATCH mptcp-net v3 2/4] selftests: mptcp: simult flows: mark 'unbalanced' tests as flaky Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20240522-selftests-mptcp-disable-flaky-v3-2-a10c68bf2680@kernel.org> References: <20240522-selftests-mptcp-disable-flaky-v3-0-a10c68bf2680@kernel.org> In-Reply-To: <20240522-selftests-mptcp-disable-flaky-v3-0-a10c68bf2680@kernel.org> To: mptcp@lists.linux.dev Cc: "Matthieu Baerts (NGI0)" X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2221; i=matttbe@kernel.org; h=from:subject:message-id; bh=Ggy7iYS0IsFK7puQFPQ7OdbUwg1pe5OIKJkUZqeKNe0=; b=owEBbQKS/ZANAwAIAfa3gk9CaaBzAcsmYgBmTc1XSSc2IYMIDUMnD3v8VBf0L265DIAzdxcU6 /6iRvIgNhmJAjMEAAEIAB0WIQToy4X3aHcFem4n93r2t4JPQmmgcwUCZk3NVwAKCRD2t4JPQmmg c3e6D/0ZJtGq2hzY/NZIsXlhD6q6VrJCOmAdHR23dil+pG45dsvwAxfqWkFqxFjOSnyOIfRjMv8 Vux1INXAHdT3duFLNADebTyZCtIMQAg0LFqLioM4ODFw/bC/o6f5VbALg7QIrx/tXj3FzwYMKf9 Z7WiP9IwQ7grFbVySVJuS9j3WKkI2kZrtr9ToEdnVmdvS4QgKNZj8A0AWJsEFUycBdez9rQYXyt gCOENqzya8Ajm5yN2DMeZdstLvRYIAQoNBuuDMdBSr99nOkAnZUCv1LDM5n9b2x8Q7IX60SnVmS BbI5p25WXJHty5YaaHYR4CL9NHK1S0Dnw9iK6d/cgJla2vQQ/7sdzEnmlJxRxOUEfN+VQdMN91w rYlQ5qzAVZWs/+o6YQXygE7/ncKEGMZIigKbfSww9uz2X68UVvptAgkBXYjviCShtOjCp+abwa2 8/jBgz8uuELLRZ53EgjbWdRbL3DZMqgNkC6yMCiEsa0ueHbMhLHS1xvOB4fHzVmvN2rkg7MUFhi clMy2F6G3XUwzlh6NWRMFkX+tt+xHqbnH7AkjWqJCjmQIoZ7kYbWIdfdc6+rHGnf6+TkqqBpVZl odQbXGfQGXi5oj+biwxRP96noBhzKX6x5hXt9xJjgzrEetN98u5D41LgLqdz7TCCRZNbgBjLM8U Bj89iImCOqB6W6Q== X-Developer-Key: i=matttbe@kernel.org; a=openpgp; fpr=E8CB85F76877057A6E27F77AF6B7824F4269A073 These tests are flaky since their introduction. This might be less or not visible depending on the CI running the tests, especially if it is also busy doing other tasks in parallel. A first analysis shown that the transfer can be slowed down when there are some re-injections at the MPTCP level. Such re-injections can of course happen, and disturb the transfer, but it looks strange to have them in this lab. That could be caused by the kernel having access to less CPU cycles -- e.g. when other activities are executed in parallel -- or by a misinterpretation on the MPTCP packet scheduler side. While this is being investigated, the tests are marked as flaky not to create noises in other CIs. Link: https://github.com/multipath-tcp/mptcp_net-next/issues/475 Signed-off-by: Matthieu Baerts (NGI0) Reviewed-by: Mat Martineau --- tools/testing/selftests/net/mptcp/simult_flows.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/net/mptcp/simult_flows.sh b/tools/test= ing/selftests/net/mptcp/simult_flows.sh index d0b39c2e38a3..b42d767246cc 100755 --- a/tools/testing/selftests/net/mptcp/simult_flows.sh +++ b/tools/testing/selftests/net/mptcp/simult_flows.sh @@ -244,7 +244,7 @@ run_test() do_transfer $small $large $time lret=3D$? mptcp_lib_result_code "${lret}" "${msg}" - if [ $lret -ne 0 ]; then + if [ $lret -ne 0 ] && ! mptcp_lib_subtest_is_flaky; then ret=3D$lret [ $bail -eq 0 ] || exit $ret fi @@ -254,7 +254,7 @@ run_test() do_transfer $large $small $time lret=3D$? mptcp_lib_result_code "${lret}" "${msg}" - if [ $lret -ne 0 ]; then + if [ $lret -ne 0 ] && ! mptcp_lib_subtest_is_flaky; then ret=3D$lret [ $bail -eq 0 ] || exit $ret fi @@ -290,7 +290,7 @@ run_test 10 10 0 0 "balanced bwidth" run_test 10 10 1 25 "balanced bwidth with unbalanced delay" =20 # we still need some additional infrastructure to pass the following test-= cases -run_test 15 5 0 0 "unbalanced bwidth" +MPTCP_LIB_SUBTEST_FLAKY=3D1 run_test 15 5 0 0 "unbalanced bwidth" run_test 15 5 1 25 "unbalanced bwidth with unbalanced delay" run_test 15 5 25 1 "unbalanced bwidth with opposed, unbalanced delay" =20 --=20 2.43.0 From nobody Mon Sep 16 19:18:38 2024 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F20879B9D for ; Wed, 22 May 2024 10:47:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716374876; cv=none; b=eh8yLsRIgMstduuYwHmThrJdjef8c5JfFI4J+AviF0rUFRBJkEEvpZ9t8/bRRUKeIsfBk7qscOA7s82Cf68bmaUM+PtX8rAJS4f85cOr+IOirnFt7VEm7bmyj3ydH83CNyERHuHN2PCZ9UPnaICT1JDjHkT7WqqpueAgIwkWbn4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716374876; c=relaxed/simple; bh=NNwISTmCUhySHdqJqAx/XtaM/YAyCnKegpzkvXP4K8I=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=jKcluaQU7ek9WsmIPBCsp/vbS4pvxZ6bKEF+iPQhNW8fALvPc5wc7OROyLhppeenDzy+21hL6idVUiZMQQ0W84gIu+dt80py8k1B65moO/ehy2cCchKcFqOOPzdGLI6RcSoQI9SRU+LdJxLhGx6yNMG3DbW2DH3TEWw00BHBl7w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kyQb5Rz2; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kyQb5Rz2" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 36DB6C32781; Wed, 22 May 2024 10:47:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1716374875; bh=NNwISTmCUhySHdqJqAx/XtaM/YAyCnKegpzkvXP4K8I=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=kyQb5Rz2gl2XcJek0WJQfYqyLcpBTE/9oGejVKiZU6VAq9w4X3vQWOCBZyK4+PDlv zXxq2CLg6+/ezZPkJNS8T1NlomcT46KDrElXnac7YOn+rCYlPWimU7hYArt4wPCHxm Rth+hh84fskEoRHWJ/YqZ3Y95lM2hJXKgvkDsMaqClhyLdiE0LGIAD5ogWjPDyAHnh IjTFZ7KSQutBfjydZ0JitTKRfTtpLREqS9DlPfxURUdxPy6NRyQAfbj5K39+o5J/af uXdGfx3+TmvtgzO0zsq4i6M1sVxRSzkmo+rluDRlkT5CtpzboTj41EF77LIHIVceVC 8kuLxDuc4vMEA== From: "Matthieu Baerts (NGI0)" Date: Wed, 22 May 2024 12:47:45 +0200 Subject: [PATCH mptcp-net v3 3/4] selftests: mptcp: join: mark 'fastclose' tests as flaky Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20240522-selftests-mptcp-disable-flaky-v3-3-a10c68bf2680@kernel.org> References: <20240522-selftests-mptcp-disable-flaky-v3-0-a10c68bf2680@kernel.org> In-Reply-To: <20240522-selftests-mptcp-disable-flaky-v3-0-a10c68bf2680@kernel.org> To: mptcp@lists.linux.dev Cc: "Matthieu Baerts (NGI0)" X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1880; i=matttbe@kernel.org; h=from:subject:message-id; bh=NNwISTmCUhySHdqJqAx/XtaM/YAyCnKegpzkvXP4K8I=; b=owEBbQKS/ZANAwAIAfa3gk9CaaBzAcsmYgBmTc1XBCL51eZqNF/WCzhKTzyqxP55R7BcjfUmr zo/z0AbxTyJAjMEAAEIAB0WIQToy4X3aHcFem4n93r2t4JPQmmgcwUCZk3NVwAKCRD2t4JPQmmg c+hID/wKyMWGTVi6L6sf9SEVJ/cw+GHYANbpEJuqOotsNNLXS3GDo8i5QxeBKqnvEhxaalHf3T5 8AMCTJHvuaU9UNVT0oGrjH1LvOqNXijOf3K2cKROtWg0RaPila4ap+t1XEvKzxP0teCJicbUU0+ oPUCEssKO5TMAyUglXyaAs0nXSWiJVC5uHws9EBIaYDos78hejKcpmA1NXd1F5KMePPLPNNjjmr bdH1U4+10nnvWTDpOExggmtZ4fq2UlivH61r8jfp8DoiiW1Y4BfF+0mSk1ZYR6Ss/tdVhdqGfxY W9CIW8Pk+lhRDWxjP0TwRqP1kkhV/hEO5YvY3SAkRnkJevcxYftYCAxtszvwU3Vn1RDLb8wOZgc ENF9OXUfWmeuc8DoS7tYf7NOO8etm8AkCAl/Xm7LBmHlndV02TcwLcfZjFjZAUVaFgPiBIBFH3y YPyIG6nd2nMRoaN0TgiwflUUqMaMaz/EaEXChELgtY+hkZ3n3+OGJgtvb0Eo3hqUIy65O0f/opV 9f6HaQwpiCVPWpro5t8CB8a6EnKCkSpLGXimUXrW3OO9CxJCDWHp25ieeFQOj1pM7o8uZxykdg7 FKMzSQY7i03JiD0QrJOcI1k8T2P+vtdkC5Oy4oDQjFHAMrND35NmzNjL7zp+W84QbJEts/LOJaY eeRzz7yQUhhLbWA== X-Developer-Key: i=matttbe@kernel.org; a=openpgp; fpr=E8CB85F76877057A6E27F77AF6B7824F4269A073 These tests are flaky since their introduction. This might be less or not visible depending on the CI running the tests, especially if it is also busy doing other tasks in parallel, and if a debug kernel config is being used. It looks like this issue is often present with the NetDev CI. While this is being investigated, the tests are marked as flaky not to create noises on such CIs. Link: https://github.com/multipath-tcp/mptcp_net-next/issues/324 Signed-off-by: Matthieu Baerts (NGI0) Reviewed-by: Mat Martineau --- tools/testing/selftests/net/mptcp/mptcp_join.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/net/mptcp/mptcp_join.sh b/tools/testin= g/selftests/net/mptcp/mptcp_join.sh index fefa9173bdaa..b869b46823d7 100755 --- a/tools/testing/selftests/net/mptcp/mptcp_join.sh +++ b/tools/testing/selftests/net/mptcp/mptcp_join.sh @@ -261,6 +261,8 @@ reset() =20 TEST_NAME=3D"${1}" =20 + MPTCP_LIB_SUBTEST_FLAKY=3D0 # reset if modified + if skip_test; then MPTCP_LIB_TEST_COUNTER=3D$((MPTCP_LIB_TEST_COUNTER+1)) last_test_ignored=3D1 @@ -448,7 +450,9 @@ reset_with_tcp_filter() # $1: err msg fail_test() { - ret=3D${KSFT_FAIL} + if ! mptcp_lib_subtest_is_flaky; then + ret=3D${KSFT_FAIL} + fi =20 if [ ${#} -gt 0 ]; then print_fail "${@}" @@ -3069,6 +3073,7 @@ fullmesh_tests() fastclose_tests() { if reset_check_counter "fastclose test" "MPTcpExtMPFastcloseTx"; then + MPTCP_LIB_SUBTEST_FLAKY=3D1 test_linkfail=3D1024 fastclose=3Dclient \ run_tests $ns1 $ns2 10.0.1.1 chk_join_nr 0 0 0 @@ -3077,6 +3082,7 @@ fastclose_tests() fi =20 if reset_check_counter "fastclose server test" "MPTcpExtMPFastcloseRx"; t= hen + MPTCP_LIB_SUBTEST_FLAKY=3D1 test_linkfail=3D1024 fastclose=3Dserver \ run_tests $ns1 $ns2 10.0.1.1 chk_join_nr 0 0 0 0 0 0 1 --=20 2.43.0 From nobody Mon Sep 16 19:18:38 2024 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46FFA79B9D for ; Wed, 22 May 2024 10:47:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716374877; cv=none; b=jzFQqidRCBmMQOkcxIeqScTJwlmbx9Iivy7xU63b4ENSq7q60udENPkb5HaUpym6mmQYKfbBNYsVBP748crS2URFdSTuyXlQJOnRepCAibyRGolXG3P0eEnQ0eTZt7sVOjh/iZXvreNcx01wNlMVNKwlSmKM6JvAvgEYgeD2oOY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716374877; c=relaxed/simple; bh=8MBb3YGLk1JPv2fvJpk6cJVJlQcDE2+JIiyeeg0Lk68=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=RGsGyxw3pRf4yH4rez4+BenG4R30tSPyQ4zYYDwoP05D4/walCt1gycFKaRyM6FpMeD0o2otLHMQamRNEdlGVVOmmXixHqCXV2LCIcL142k4EbkKMYHfXbuTYAHl8Pu2EBLEXvIdXdm3coMky5JW/JK36IOM+WVT7zTJxjGXxT8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Q17mj77x; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Q17mj77x" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3A470C4AF07; Wed, 22 May 2024 10:47:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1716374876; bh=8MBb3YGLk1JPv2fvJpk6cJVJlQcDE2+JIiyeeg0Lk68=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=Q17mj77xIzz8OmSctEXUF6G3mK0f0IQj7tr3VwfnVJ3XnufAocORvh3e2GF6OYDAD giBETMCoXey30QRaeadgtIh7DUyCRXsss+xoLc6AzNAWap2iLB3lkR6pSj36MaPSgp D7IOS9nGUp7s+z+ODDGs/cKFRD0JSCpKu2op/v/wzcIYWZi80FJ0Est8zWfwU9arUp YO+oX6RqaGBgBJx+9nqasomOMzwCOYZlIunNHghq/9ZBA4kJuj2+8ulbpPbhdfLXuk mayjmZDptEbsgYKRConh++Wj/RyO6AWTNEKp6pV2p+6X8QxniuCMWk8QAvVdoSlquB miNgsN3z7bAkw== From: "Matthieu Baerts (NGI0)" Date: Wed, 22 May 2024 12:47:46 +0200 Subject: [PATCH mptcp-net v3 4/4] selftests: mptcp: join: mark 'fail' tests as flaky Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20240522-selftests-mptcp-disable-flaky-v3-4-a10c68bf2680@kernel.org> References: <20240522-selftests-mptcp-disable-flaky-v3-0-a10c68bf2680@kernel.org> In-Reply-To: <20240522-selftests-mptcp-disable-flaky-v3-0-a10c68bf2680@kernel.org> To: mptcp@lists.linux.dev Cc: "Matthieu Baerts (NGI0)" X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1372; i=matttbe@kernel.org; h=from:subject:message-id; bh=8MBb3YGLk1JPv2fvJpk6cJVJlQcDE2+JIiyeeg0Lk68=; b=owEBbQKS/ZANAwAIAfa3gk9CaaBzAcsmYgBmTc1XywECc8w0CxRsqYSiVYwkocib3T4IWr4TJ EG8rKKjCQWJAjMEAAEIAB0WIQToy4X3aHcFem4n93r2t4JPQmmgcwUCZk3NVwAKCRD2t4JPQmmg c2G3D/0RAMZwl5xOZE/qb0srTh9tc8P/PVEgyE4w3WKEAJnDte8JOGkeaTSSi8483gNhVmAOzkC AUH9FbTRC/H+D4IDIq8wEZQB+T7UnSXHLSUXgv/1VyWg+nJMhg9yVh8zvsr2MAFV5GNbB5oXa0l Vnx7Omf9fgII6Cz5LKlSe24rQaOvPRBJnxfEuxbLzshpaZU7wvhUTQOjPiiejIHU9CIF9as0OYy 8FDgry6jZ2xZNMWz38hUTKOAbirdP71fb4acNV9jeuItc/pv2aVGLFI0JCPMKMfkf0D7qipIXB/ aEb56HCkNCSSVUZKkYLpPyngYh+XR5RKF5gkLWIxXfMFK8xr7uuHfaW7F5GOXee4UGFiqPQ2zIa EL4pJ20GW7sfKVoL9P6wCbmdAnVw6hzM60/q1KM8tJXV7/EZlxGbSM2FeSwthCrY/H3NXm+WQkH tJUXnDwzPNdkbJtmaYAvvp7KeONZQ8zDGqqyVZUVqCdyzIdjplTJVE1Ae42GYZSSqHcsdD5fzfk 7dlfvXTvafGnaJZc1tFiGUO/8AVy1CdWair7hReLP2HZkECh6JRwAC+oQHpFWG+lrcMUzABblTx tnStid3miNgU26oxSGPHvln7X1niGFYiIxtzcjVh2vy6ZWbPyfT6NtL0c9KYcKQEFIpu0ZtjLLb WRVcA9UKk0dbLrA== X-Developer-Key: i=matttbe@kernel.org; a=openpgp; fpr=E8CB85F76877057A6E27F77AF6B7824F4269A073 These tests are rarely unstable. It depends on the CI running the tests, especially if it is also busy doing other tasks in parallel, and if a debug kernel config is being used. It looks like this issue is sometimes present with the NetDev CI. While this is being investigated, the tests are marked as flaky not to create noises on such CIs. Link: https://github.com/multipath-tcp/mptcp_net-next/issues/491 Signed-off-by: Matthieu Baerts (NGI0) Reviewed-by: Mat Martineau --- tools/testing/selftests/net/mptcp/mptcp_join.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tools/testing/selftests/net/mptcp/mptcp_join.sh b/tools/testin= g/selftests/net/mptcp/mptcp_join.sh index b869b46823d7..2b66c5fa71eb 100755 --- a/tools/testing/selftests/net/mptcp/mptcp_join.sh +++ b/tools/testing/selftests/net/mptcp/mptcp_join.sh @@ -3101,6 +3101,7 @@ fail_tests() { # single subflow if reset_with_fail "Infinite map" 1; then + MPTCP_LIB_SUBTEST_FLAKY=3D1 test_linkfail=3D128 \ run_tests $ns1 $ns2 10.0.1.1 chk_join_nr 0 0 0 +1 +0 1 0 1 "$(pedit_action_pkts)" @@ -3109,6 +3110,7 @@ fail_tests() =20 # multiple subflows if reset_with_fail "MP_FAIL MP_RST" 2; then + MPTCP_LIB_SUBTEST_FLAKY=3D1 tc -n $ns2 qdisc add dev ns2eth1 root netem rate 1mbit delay 5ms pm_nl_set_limits $ns1 0 1 pm_nl_set_limits $ns2 0 1 --=20 2.43.0