The current selftests infrastructure formats the results in TAP 13. This
version doesn't support subtests and only the end result of each
selftest is taken into account. It means that a single issue in a
subtest of a selftest containing multiple subtests forces the whole
selftest to be marked as failed. It also means that subtests results are
not tracked by CI executing selftests.

MPTCP selftests run hundreds of various subtests. It is then important
to track each of them and not one result per selftest.

It is particularly interesting to do that when validating stable kernels
with the last version of the test suite: tests might fail because a
feature is not supported but the test didn't skip that part. In this
case, if subtests are not tracked, the whole selftest will be marked as
failed making the other subtests useless because their results are
ignored.

Regarding this patch set, the 5 first patches are fixes for -net:
 - connect: issues with nft were not reported and then ignored by the CI
 - sockopt: issues with packet marks were also not reported and ignored
 - userspace_pm: similar fix for "remove" tests + use correct var

Then some various improvements for net-next:
 - connect & userspace_pm: continue executing tests if there is an error
 - userspace_pm: some refactoring to reduce duplicated code and prepare
   subtests' support.

And finally we have one commit adding helpers to support subtests
followed by commits in the different selftests to support this feature.

Signed-off-by: Matthieu Baerts <matthieu.baerts@tessares.net>
---
Matthieu Baerts (18):
      selftests: mptcp: connect: fail if nft supposed to work
      selftests: mptcp: sockopt: use 'iptables-legacy' if available
      selftests: mptcp: sockopt: return error if wrong mark
      selftests: mptcp: userspace_pm: use correct server port
      selftests: mptcp: userspace_pm: report errors with 'remove' tests
      selftests: mptcp: connect: don't stop if error
      selftests: mptcp: userspace pm: don't stop if error
      selftests: mptcp: userspace_pm: fix shellcheck warnings
      selftests: mptcp: userspace_pm: uniform results printing
      selftests: mptcp: userspace_pm: reduce dup code around printf
      selftests: mptcp: lib: format subtests results in TAP
      selftests: mptcp: connect: format subtests results in TAP
      selftests: mptcp: pm_netlink: format subtests results in TAP
      selftests: mptcp: join: format subtests results in TAP
      selftests: mptcp: diag: format subtests results in TAP
      selftests: mptcp: simult flows: format subtests results in TAP
      selftests: mptcp: sockopt: format subtests results in TAP
      selftests: mptcp: userspace_pm: format subtests results in TAP

 tools/testing/selftests/net/mptcp/diag.sh          |   7 +
 tools/testing/selftests/net/mptcp/mptcp_connect.sh |  69 ++++++--
 tools/testing/selftests/net/mptcp/mptcp_join.sh    |  41 ++++-
 tools/testing/selftests/net/mptcp/mptcp_lib.sh     |  66 ++++++++
 tools/testing/selftests/net/mptcp/mptcp_sockopt.sh |  49 ++++--
 tools/testing/selftests/net/mptcp/pm_netlink.sh    |   6 +-
 tools/testing/selftests/net/mptcp/simult_flows.sh  |   4 +
 tools/testing/selftests/net/mptcp/userspace_pm.sh  | 181 +++++++++++++--------
 8 files changed, 319 insertions(+), 104 deletions(-)
---
base-commit: 9b5467a5f7f3f8c37c9fd84acaac3d4a5e2c64c4
change-id: 20230621-mptcp-issue-368-selftests-subtests-3c29a649dcdd

Best regards,
-- 
Matthieu Baerts <matthieu.baerts@tessares.net>

Hi Paolo,

On 21/06/2023 19:18, Matthieu Baerts wrote:
> The current selftests infrastructure formats the results in TAP 13. This
> version doesn't support subtests and only the end result of each
> selftest is taken into account. It means that a single issue in a
> subtest of a selftest containing multiple subtests forces the whole
> selftest to be marked as failed. It also means that subtests results are
> not tracked by CI executing selftests.
> 
> MPTCP selftests run hundreds of various subtests. It is then important
> to track each of them and not one result per selftest.
> 
> It is particularly interesting to do that when validating stable kernels
> with the last version of the test suite: tests might fail because a
> feature is not supported but the test didn't skip that part. In this
> case, if subtests are not tracked, the whole selftest will be marked as
> failed making the other subtests useless because their results are
> ignored.
> 
> Regarding this patch set, the 5 first patches are fixes for -net:
>  - connect: issues with nft were not reported and then ignored by the CI
>  - sockopt: issues with packet marks were also not reported and ignored
>  - userspace_pm: similar fix for "remove" tests + use correct var
> 
> Then some various improvements for net-next:
>  - connect & userspace_pm: continue executing tests if there is an error
>  - userspace_pm: some refactoring to reduce duplicated code and prepare
>    subtests' support.
> 
> And finally we have one commit adding helpers to support subtests
> followed by commits in the different selftests to support this feature.
> 
> Signed-off-by: Matthieu Baerts <matthieu.baerts@tessares.net>
> ---
> Matthieu Baerts (18):
>       selftests: mptcp: connect: fail if nft supposed to work
>       selftests: mptcp: sockopt: use 'iptables-legacy' if available
>       selftests: mptcp: sockopt: return error if wrong mark
>       selftests: mptcp: userspace_pm: use correct server port
>       selftests: mptcp: userspace_pm: report errors with 'remove' tests
>       selftests: mptcp: connect: don't stop if error
>       selftests: mptcp: userspace pm: don't stop if error
>       selftests: mptcp: userspace_pm: fix shellcheck warnings
>       selftests: mptcp: userspace_pm: uniform results printing
>       selftests: mptcp: userspace_pm: reduce dup code around printf
>       selftests: mptcp: lib: format subtests results in TAP
>       selftests: mptcp: connect: format subtests results in TAP
>       selftests: mptcp: pm_netlink: format subtests results in TAP
>       selftests: mptcp: join: format subtests results in TAP
>       selftests: mptcp: diag: format subtests results in TAP
>       selftests: mptcp: simult flows: format subtests results in TAP
>       selftests: mptcp: sockopt: format subtests results in TAP
>       selftests: mptcp: userspace_pm: format subtests results in TAP

Thank you for the review!

I just applied these patches in our tree (1-5: fixes for -net ; 6-18:
feat. for net-next) with your Acked-by and without the fix in patch 14/18:

New patches for t/upstream-net and t/upstream:
- bd32644a1704: selftests: mptcp: connect: fail if nft supposed to work
- e514dd195a47: selftests: mptcp: sockopt: use 'iptables-legacy' if
available
- 239d7db18bb5: selftests: mptcp: sockopt: return error if wrong mark
- 38a841f8e2ba: selftests: mptcp: userspace_pm: use correct server port
- ff68420a15db: selftests: mptcp: userspace_pm: report errors with
'remove' tests
- Results: 045d70ca4742..221a200140cf (export-net)

New patches for t/upstream:
- 7ace02ce8ee8: selftests: mptcp: connect: don't stop if error
- a9f0d3c9ed84: selftests: mptcp: userspace pm: don't stop if error
- 03f8da0a8a4c: selftests: mptcp: userspace_pm: fix shellcheck warnings
- 81c83275036e: selftests: mptcp: userspace_pm: uniform results printing
- 7be13e337f8b: selftests: mptcp: userspace_pm: reduce dup code around
printf
- 544e972c95d9: selftests: mptcp: lib: format subtests results in TAP
- 38a281bb1703: selftests: mptcp: connect: format subtests results in TAP
- aed83c0c6b35: selftests: mptcp: pm_netlink: format subtests results in TAP
- a600e75d6df4: selftests: mptcp: join: format subtests results in TAP
- 8d64331739d6: selftests: mptcp: diag: format subtests results in TAP
- b4c131ff9e9a: selftests: mptcp: simult flows: format subtests results
in TAP
- 9f91056e4349: selftests: mptcp: sockopt: format subtests results in TAP
- 4c4a5aa7ab0d: selftests: mptcp: userspace_pm: format subtests results
in TAP
- Results: 922570c97fcb..619c3c3e5a9b (export)

Tests are now in progress:

https://cirrus-ci.com/github/multipath-tcp/mptcp_net-next/export-net/20230703T163105
https://cirrus-ci.com/github/multipath-tcp/mptcp_net-next/export/20230703T163926

Cheers,
Matt
-- 
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net