[v1] selftests: mptcp: format subtests results in TAP (part 2)

[PATCH mptcp-next 2/2] selftests: mptcp: join: colored results

Posted by Matthieu Baerts 2 years, 7 months ago

Thanks to the parent commit, it is easy to change the output and add
some colours to easily spot issues.

The colours are not used if stdout is redirected or if NO_COLOR env var
is set to 1 as specified in https://no-color.org.

Signed-off-by: Matthieu Baerts <matthieu.baerts@tessares.net>
---
 .../testing/selftests/net/mptcp/mptcp_join.sh |  8 ++--
 .../testing/selftests/net/mptcp/mptcp_lib.sh  | 38 +++++++++++++++++++
 2 files changed, 42 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/net/mptcp/mptcp_join.sh b/tools/testing/selftests/net/mptcp/mptcp_join.sh
index 3a0e2fa7754b..e9a3e85970fb 100755
--- a/tools/testing/selftests/net/mptcp/mptcp_join.sh
+++ b/tools/testing/selftests/net/mptcp/mptcp_join.sh
@@ -209,22 +209,22 @@ print_check()
 
 print_info()
 {
-	printf "%-${nr_blank}sInfo: %s\n" " " "${*}"
+	mptcp_lib_print_info "      Info: ${*}"
 }
 
 print_ok()
 {
-	echo "[ ok ]${1:+ ${*}}"
+	mptcp_lib_print_ok "[ ok ]${1:+ ${*}}"
 }
 
 print_fail()
 {
-	echo "[fail]${1:+ ${*}}"
+	mptcp_lib_print_err "[fail]${1:+ ${*}}"
 }
 
 print_skip()
 {
-	echo "[skip]${1:+ ${*}}"
+	mptcp_lib_print_warn "[skip]${1:+ ${*}}"
 }
 
 # [ $1: fail msg ]
diff --git a/tools/testing/selftests/net/mptcp/mptcp_lib.sh b/tools/testing/selftests/net/mptcp/mptcp_lib.sh
index b1a0fdd0408b..1080d5d2430b 100644
--- a/tools/testing/selftests/net/mptcp/mptcp_lib.sh
+++ b/tools/testing/selftests/net/mptcp/mptcp_lib.sh
@@ -4,10 +4,48 @@
 readonly KSFT_PASS=0
 readonly KSFT_FAIL=1
 readonly KSFT_SKIP=4
+
+# shellcheck disable=SC2155 # declare and assign separately
 readonly KSFT_TEST=$(basename "${0}" | sed 's/\.sh$//g')
 
 MPTCP_LIB_SUBTESTS=()
 
+# only if supported and not disabled, see no-color.org
+if [ -t 1 ] && [ "${NO_COLOR:-}" != "1" ]; then
+	readonly MPTCP_LIB_COLOR_RED="\E[1;31m"
+	readonly MPTCP_LIB_COLOR_GREEN="\E[1;32m"
+	readonly MPTCP_LIB_COLOR_YELLOW="\E[1;33m"
+	readonly MPTCP_LIB_COLOR_BLUE="\E[1;34m"
+	readonly MPTCP_LIB_COLOR_RESET="\E[0m"
+else
+	readonly MPTCP_LIB_COLOR_RED=
+	readonly MPTCP_LIB_COLOR_GREEN=
+	readonly MPTCP_LIB_COLOR_YELLOW=
+	readonly MPTCP_LIB_COLOR_BLUE=
+	readonly MPTCP_LIB_COLOR_RESET=
+fi
+
+# $1: color, $2: text
+mptcp_lib_print_color() {
+	echo -e "${MPTCP_LIB_START_PRINT:-}${*}${MPTCP_LIB_COLOR_RESET}"
+}
+
+mptcp_lib_print_ok() {
+	mptcp_lib_print_color "${MPTCP_LIB_COLOR_GREEN}${*}"
+}
+
+mptcp_lib_print_warn() {
+	mptcp_lib_print_color "${MPTCP_LIB_COLOR_YELLOW}${*}"
+}
+
+mptcp_lib_print_info() {
+	mptcp_lib_print_color "${MPTCP_LIB_COLOR_BLUE}${*}"
+}
+
+mptcp_lib_print_err() {
+	mptcp_lib_print_color "${MPTCP_LIB_COLOR_RED}${*}"
+}
+
 # SELFTESTS_MPTCP_LIB_EXPECT_ALL_FEATURES env var can be set when validating all
 # features using the last version of the kernel and the selftests to make sure
 # a test is not being skipped by mistake.
-- 
2.40.1

Re: [PATCH mptcp-next 2/2] selftests: mptcp: join: colored results

Posted by Paolo Abeni 2 years, 7 months ago

On Wed, 2023-06-28 at 16:49 +0200, Matthieu Baerts wrote:
> Thanks to the parent commit, it is easy to change the output and add
> some colours to easily spot issues.
> 
> The colours are not used if stdout is redirected or if NO_COLOR env var
> is set to 1 as specified in https://no-color.org.
> 
> Signed-off-by: Matthieu Baerts <matthieu.baerts@tessares.net>
> ---
>  .../testing/selftests/net/mptcp/mptcp_join.sh |  8 ++--
>  .../testing/selftests/net/mptcp/mptcp_lib.sh  | 38 +++++++++++++++++++
>  2 files changed, 42 insertions(+), 4 deletions(-)
> 
> diff --git a/tools/testing/selftests/net/mptcp/mptcp_join.sh b/tools/testing/selftests/net/mptcp/mptcp_join.sh
> index 3a0e2fa7754b..e9a3e85970fb 100755
> --- a/tools/testing/selftests/net/mptcp/mptcp_join.sh
> +++ b/tools/testing/selftests/net/mptcp/mptcp_join.sh
> @@ -209,22 +209,22 @@ print_check()
>  
>  print_info()
>  {
> -	printf "%-${nr_blank}sInfo: %s\n" " " "${*}"
> +	mptcp_lib_print_info "      Info: ${*}"
>  }
>  
>  print_ok()
>  {
> -	echo "[ ok ]${1:+ ${*}}"
> +	mptcp_lib_print_ok "[ ok ]${1:+ ${*}}"
>  }
>  
>  print_fail()
>  {
> -	echo "[fail]${1:+ ${*}}"
> +	mptcp_lib_print_err "[fail]${1:+ ${*}}"
>  }
>  
>  print_skip()
>  {
> -	echo "[skip]${1:+ ${*}}"
> +	mptcp_lib_print_warn "[skip]${1:+ ${*}}"
>  }
>  
>  # [ $1: fail msg ]
> diff --git a/tools/testing/selftests/net/mptcp/mptcp_lib.sh b/tools/testing/selftests/net/mptcp/mptcp_lib.sh
> index b1a0fdd0408b..1080d5d2430b 100644
> --- a/tools/testing/selftests/net/mptcp/mptcp_lib.sh
> +++ b/tools/testing/selftests/net/mptcp/mptcp_lib.sh
> @@ -4,10 +4,48 @@
>  readonly KSFT_PASS=0
>  readonly KSFT_FAIL=1
>  readonly KSFT_SKIP=4
> +
> +# shellcheck disable=SC2155 # declare and assign separately
>  readonly KSFT_TEST=$(basename "${0}" | sed 's/\.sh$//g')
>  
>  MPTCP_LIB_SUBTESTS=()
>  
> +# only if supported and not disabled, see no-color.org
> +if [ -t 1 ] && [ "${NO_COLOR:-}" != "1" ]; then
> +	readonly MPTCP_LIB_COLOR_RED="\E[1;31m"
> +	readonly MPTCP_LIB_COLOR_GREEN="\E[1;32m"
> +	readonly MPTCP_LIB_COLOR_YELLOW="\E[1;33m"
> +	readonly MPTCP_LIB_COLOR_BLUE="\E[1;34m"
> +	readonly MPTCP_LIB_COLOR_RESET="\E[0m"
> +else
> +	readonly MPTCP_LIB_COLOR_RED=
> +	readonly MPTCP_LIB_COLOR_GREEN=
> +	readonly MPTCP_LIB_COLOR_YELLOW=
> +	readonly MPTCP_LIB_COLOR_BLUE=
> +	readonly MPTCP_LIB_COLOR_RESET=
> +fi
> +
> +# $1: color, $2: text
> +mptcp_lib_print_color() {
> +	echo -e "${MPTCP_LIB_START_PRINT:-}${*}${MPTCP_LIB_COLOR_RESET}"
> +}
> +
> +mptcp_lib_print_ok() {
> +	mptcp_lib_print_color "${MPTCP_LIB_COLOR_GREEN}${*}"
> +}
> +
> +mptcp_lib_print_warn() {
> +	mptcp_lib_print_color "${MPTCP_LIB_COLOR_YELLOW}${*}"
> +}
> +
> +mptcp_lib_print_info() {
> +	mptcp_lib_print_color "${MPTCP_LIB_COLOR_BLUE}${*}"
> +}
> +
> +mptcp_lib_print_err() {
> +	mptcp_lib_print_color "${MPTCP_LIB_COLOR_RED}${*}"
> +}
> +
>  # SELFTESTS_MPTCP_LIB_EXPECT_ALL_FEATURES env var can be set when validating all
>  # features using the last version of the kernel and the selftests to make sure
>  # a test is not being skipped by mistake.

Acked-by: Paolo Abeni <pabeni@redhat.com>

Thanks!

/P

Re: selftests: mptcp: join: colored results: Tests Results

Posted by MPTCP CI 2 years, 7 months ago

Hi Matthieu,

Thank you for your modifications, that's great!

Our CI did some validations and here is its report:

- KVM Validation: normal (except selftest_mptcp_join):
  - Success! ✅:
  - Task: https://cirrus-ci.com/task/4971218193874944
  - Summary: https://api.cirrus-ci.com/v1/artifact/task/4971218193874944/summary/summary.txt

- KVM Validation: normal (only selftest_mptcp_join):
  - Success! ✅:
  - Task: https://cirrus-ci.com/task/6293277847060480
  - Summary: https://api.cirrus-ci.com/v1/artifact/task/6293277847060480/summary/summary.txt

- KVM Validation: debug (except selftest_mptcp_join):
  - Success! ✅:
  - Task: https://cirrus-ci.com/task/6511369239920640
  - Summary: https://api.cirrus-ci.com/v1/artifact/task/6511369239920640/summary/summary.txt

- KVM Validation: debug (only selftest_mptcp_join):
  - Success! ✅:
  - Task: https://cirrus-ci.com/task/5158516046954496
  - Summary: https://api.cirrus-ci.com/v1/artifact/task/5158516046954496/summary/summary.txt

Initiator: Matthieu Baerts
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/07c93e69ca2a


If there are some issues, you can reproduce them using the same environment as
the one used by the CI thanks to a docker image, e.g.:

    $ cd [kernel source code]
    $ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
        --pull always mptcp/mptcp-upstream-virtme-docker:latest \
        auto-debug

For more details:

    https://github.com/multipath-tcp/mptcp-upstream-virtme-docker


Please note that despite all the efforts that have been already done to have a
stable tests suite when executed on a public CI like here, it is possible some
reported issues are not due to your modifications. Still, do not hesitate to
help us improve that ;-)

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (Tessares)

Re: selftests: mptcp: join: colored results: Tests Results

Posted by MPTCP CI 2 years, 7 months ago

Hi Matthieu,

Thank you for your modifications, that's great!

Our CI did some validations and here is its report:

- {"code":404,"message":
  - "Can't find artifacts containing file conclusion.txt"}:
  - Task: https://cirrus-ci.com/task/6585027727720448
  - Summary: https://api.cirrus-ci.com/v1/artifact/task/6585027727720448/summary/summary.txt

- {"code":404,"message":
  - "Can't find artifacts containing file conclusion.txt"}:
  - Task: https://cirrus-ci.com/task/5368007371784192
  - Summary: https://api.cirrus-ci.com/v1/artifact/task/5368007371784192/summary/summary.txt

- {"code":404,"message":
  - "Can't find artifacts containing file conclusion.txt"}:
  - Task: https://cirrus-ci.com/task/5917564174336000
  - Summary: https://api.cirrus-ci.com/v1/artifact/task/5917564174336000/summary/summary.txt

- {"code":404,"message":
  - "Can't find artifacts containing file conclusion.txt"}:
  - Task: https://cirrus-ci.com/task/5842187364073472
  - Summary: https://api.cirrus-ci.com/v1/artifact/task/5842187364073472/summary/summary.txt

Initiator: Matthieu Baerts
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/07c93e69ca2a


If there are some issues, you can reproduce them using the same environment as
the one used by the CI thanks to a docker image, e.g.:

    $ cd [kernel source code]
    $ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
        --pull always mptcp/mptcp-upstream-virtme-docker:latest \
        auto-debug

For more details:

    https://github.com/multipath-tcp/mptcp-upstream-virtme-docker


Please note that despite all the efforts that have been already done to have a
stable tests suite when executed on a public CI like here, it is possible some
reported issues are not due to your modifications. Still, do not hesitate to
help us improve that ;-)

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (Tessares)

Re: selftests: mptcp: join: colored results: Tests Results

Posted by MPTCP CI 2 years, 7 months ago

Hi Matthieu,

Thank you for your modifications, that's great!

Our CI did some validations and here is its report:

- {"code":404,"message":
  - "Can't find artifacts containing file conclusion.txt"}:
  - Task: https://cirrus-ci.com/task/6585027727720448
  - Summary: https://api.cirrus-ci.com/v1/artifact/task/6585027727720448/summary/summary.txt

- {"code":404,"message":
  - "Can't find artifacts containing file conclusion.txt"}:
  - Task: https://cirrus-ci.com/task/5368007371784192
  - Summary: https://api.cirrus-ci.com/v1/artifact/task/5368007371784192/summary/summary.txt

- {"code":404,"message":
  - "Can't find artifacts containing file conclusion.txt"}:
  - Task: https://cirrus-ci.com/task/5917564174336000
  - Summary: https://api.cirrus-ci.com/v1/artifact/task/5917564174336000/summary/summary.txt

- {"code":404,"message":
  - "Can't find artifacts containing file conclusion.txt"}:
  - Task: https://cirrus-ci.com/task/5842187364073472
  - Summary: https://api.cirrus-ci.com/v1/artifact/task/5842187364073472/summary/summary.txt

Initiator: Matthieu Baerts
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/07c93e69ca2a


If there are some issues, you can reproduce them using the same environment as
the one used by the CI thanks to a docker image, e.g.:

    $ cd [kernel source code]
    $ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
        --pull always mptcp/mptcp-upstream-virtme-docker:latest \
        auto-debug

For more details:

    https://github.com/multipath-tcp/mptcp-upstream-virtme-docker


Please note that despite all the efforts that have been already done to have a
stable tests suite when executed on a public CI like here, it is possible some
reported issues are not due to your modifications. Still, do not hesitate to
help us improve that ;-)

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (Tessares)

Re: selftests: mptcp: join: colored results: Tests Results

Posted by MPTCP CI 2 years, 7 months ago

Hi Matthieu,

Thank you for your modifications, that's great!

Our CI did some validations and here is its report:

- KVM Validation: normal (except selftest_mptcp_join):
  - Success! ✅:
  - Task: https://cirrus-ci.com/task/6223547677278208
  - Summary: https://api.cirrus-ci.com/v1/artifact/task/6223547677278208/summary/summary.txt

- KVM Validation: normal (only selftest_mptcp_join):
  - Unstable: 1 failed test(s): selftest_mptcp_join 🔴:
  - Task: https://cirrus-ci.com/task/4816172793724928
  - Summary: https://api.cirrus-ci.com/v1/artifact/task/4816172793724928/summary/summary.txt

- KVM Validation: debug (except selftest_mptcp_join):
  - Unstable: 1 failed test(s): packetdrill_sockopts 🔴:
  - Task: https://cirrus-ci.com/task/5942072700567552
  - Summary: https://api.cirrus-ci.com/v1/artifact/task/5942072700567552/summary/summary.txt

- KVM Validation: debug (only selftest_mptcp_join):
  - Success! ✅:
  - Task: https://cirrus-ci.com/task/5609745210408960
  - Summary: https://api.cirrus-ci.com/v1/artifact/task/5609745210408960/summary/summary.txt

Initiator: Matthieu Baerts
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/07c93e69ca2a


If there are some issues, you can reproduce them using the same environment as
the one used by the CI thanks to a docker image, e.g.:

    $ cd [kernel source code]
    $ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
        --pull always mptcp/mptcp-upstream-virtme-docker:latest \
        auto-debug

For more details:

    https://github.com/multipath-tcp/mptcp-upstream-virtme-docker


Please note that despite all the efforts that have been already done to have a
stable tests suite when executed on a public CI like here, it is possible some
reported issues are not due to your modifications. Still, do not hesitate to
help us improve that ;-)

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (Tessares)