[v3] selftests: mptcp: skip tests when features are not supported

[PATCH mptcp-net v3 00/32] selftests: mptcp: skip tests when features are not supported

Posted by Matthieu Baerts 10 months, 3 weeks ago

Selftests are supposed to run on any kernels, including the old ones not
supporting all MPTCP features.

These different patches allow the different MPTCP selftests to be
executed on older kernels.

I tried to minimise the modifications and do the checks from the bash
script not to have to detect the feature when being used in a .c file,
then pass the info to the bash script, skip the test, clean the
environment, etc. As you can see, even with this minimalistic way, a lot
of modifications have been done to support old kernels.

Note that if we expect to have all features available,
SELFTESTS_MPTCP_LIB_EXPECT_ALL_FEATURES env var can be set to 1 to mark
the tests as failed instead of skipped and track regressions just in
case one of these checks is wrong. The public CI already set this env
var to 1.

This is a big series impacting new development. Probably best to apply
it "quickly" in our tree and fix issues later to avoid multiple rebases.

Closes: https://github.com/multipath-tcp/mptcp_net-next/issues/368
Signed-off-by: Matthieu Baerts <matthieu.baerts@tessares.net>
---
Changes in v3:
- Patches 1-7 have been already applied
- check kernel version: new env var that can be set to skip the versions
  check so it works if a new feature is backported to an older kernel.
- get_counter() to get the MIB counters: I can help the stable team with
  the backports that will not be difficult I think but will cause
  conflicts
- check nf/tc issues: looks safer just in case a kconfig is missing or
  an issue with nf/tc but can be removed if you prefer (still I don't
  think we should exit in the middle of a test so I kept it for the
  moment)
- one check per block of tests around the same theme: it would indeed
  avoid re-doing the same check 2 or 3 times but it would break the IDs
  (+maybe less obvious a test might be skipped when reviewing code)
- backports: at the end, we just need all these new checks to be
  backported on the last stable kernel (6.3) because that's what is used
  by the different CIs. It is fine if it is not backported after. (but
  still I can help with the backports, I'm used to :) )
- New patches:
  - selftests: mptcp: join: uniform listener tests
  - Squash to "selftests: mptcp: add MPTCP_FULL_INFO testcase"
- See the changelog on the individual patches
- Link to v2: https://lore.kernel.org/r/20230406-mptcp-issue-368-selftests-old-kernels-v2-0-50313e4f83ab@tessares.net

Changes in v2:
- The first patches are almost the same as in v1, the main modifications
  are in mptcp_lib.sh (and in the commit messages): new helpers but also
  the use of SELFTESTS_MPTCP_LIB_EXPECT_ALL_FEATURES env var.
- Link to v1: https://lore.kernel.org/r/20230406-mptcp-issue-368-selftests-old-kernels-v1-0-246ac567724d@tessares.net

---
Matthieu Baerts (32):
      selftests: mptcp: lib: skip if missing symbol
      selftests: mptcp: lib: skip if not below kernel version
      selftests: mptcp: connect: skip transp tests if not supported
      selftests: mptcp: connect: skip disconnect tests if not supported
      selftests: mptcp: connect: skip TFO tests if not supported
      selftests: mptcp: diag: skip listen tests if not supported
      selftests: mptcp: diag: skip inuse tests if not supported
      selftests: mptcp: pm nl: remove hardcoded default limits
      selftests: mptcp: pm nl: skip fullmesh flag checks if not supported
      selftests: mptcp: sockopt: relax expected returned size
      selftests: mptcp: sockopt: skip getsockopt checks if not supported
      selftests: mptcp: sockopt: skip TCP_INQ checks if not supported
      selftests: mptcp: userspace pm: skip if 'ip' tool is unavailable
      selftests: mptcp: userspace pm: skip if not supported
      selftests: mptcp: userspace pm: skip PM listener events tests if unavailable
      selftests: mptcp: join: use 'iptables-legacy' if available
      selftests: mptcp: join: helpers to skip tests
      selftests: mptcp: join: skip check if MIB counter not supported
      selftests: mptcp: join: skip test if iptables/tc cmds fail
      selftests: mptcp: join: support local endpoint being tracked or not
      selftests: mptcp: join: skip Fastclose tests if not supported
      selftests: mptcp: join: support RM_ADDR for used endpoints or not
      selftests: mptcp: join: skip implicit tests if not supported
      selftests: mptcp: join: skip backup if set flag on ID not supported
      selftests: mptcp: join: skip fullmesh flag tests if not supported
      selftests: mptcp: join: skip userspace PM tests if not supported
      selftests: mptcp: join: skip fail tests if not supported
      selftests: mptcp: join: skip MPC backups tests if not supported
      selftests: mptcp: join: skip PM listener tests if not supported
      selftests: mptcp: join: uniform listener tests
      selftests: mptcp: join: skip mixed tests if not supported
      Squash to "selftests: mptcp: add MPTCP_FULL_INFO testcase"

 tools/testing/selftests/net/mptcp/config           |   1 +
 tools/testing/selftests/net/mptcp/diag.sh          |  42 +-
 tools/testing/selftests/net/mptcp/mptcp_connect.sh |  20 +
 tools/testing/selftests/net/mptcp/mptcp_join.sh    | 536 +++++++++++++--------
 tools/testing/selftests/net/mptcp/mptcp_lib.sh     |  64 +++
 tools/testing/selftests/net/mptcp/mptcp_sockopt.c  |  26 +-
 tools/testing/selftests/net/mptcp/mptcp_sockopt.sh |  20 +-
 tools/testing/selftests/net/mptcp/pm_netlink.sh    |  27 +-
 tools/testing/selftests/net/mptcp/userspace_pm.sh  |  13 +-
 9 files changed, 506 insertions(+), 243 deletions(-)
---
base-commit: 3d6d7b1262115b5ccc87b47a61f0a2db3d1b2fbf
change-id: 20230406-mptcp-issue-368-selftests-old-kernels-fffd69d67d91

Best regards,
-- 
Matthieu Baerts <matthieu.baerts@tessares.net>

Re: [PATCH mptcp-net v3 00/32] selftests: mptcp: skip tests when features are not supported

Posted by Matthieu Baerts 10 months, 2 weeks ago

Hello,

On 31/05/2023 16:16, Matthieu Baerts wrote:
> Selftests are supposed to run on any kernels, including the old ones not
> supporting all MPTCP features.
> 
> These different patches allow the different MPTCP selftests to be
> executed on older kernels.
> 
> I tried to minimise the modifications and do the checks from the bash
> script not to have to detect the feature when being used in a .c file,
> then pass the info to the bash script, skip the test, clean the
> environment, etc. As you can see, even with this minimalistic way, a lot
> of modifications have been done to support old kernels.
> 
> Note that if we expect to have all features available,
> SELFTESTS_MPTCP_LIB_EXPECT_ALL_FEATURES env var can be set to 1 to mark
> the tests as failed instead of skipped and track regressions just in
> case one of these checks is wrong. The public CI already set this env
> var to 1.
> 
> This is a big series impacting new development. Probably best to apply
> it "quickly" in our tree and fix issues later to avoid multiple rebases.
> 
> Closes: https://github.com/multipath-tcp/mptcp_net-next/issues/368
> Signed-off-by: Matthieu Baerts <matthieu.baerts@tessares.net>
> ---
> Changes in v3:
> - Patches 1-7 have been already applied
> - check kernel version: new env var that can be set to skip the versions
>   check so it works if a new feature is backported to an older kernel.
> - get_counter() to get the MIB counters: I can help the stable team with
>   the backports that will not be difficult I think but will cause
>   conflicts
> - check nf/tc issues: looks safer just in case a kconfig is missing or
>   an issue with nf/tc but can be removed if you prefer (still I don't
>   think we should exit in the middle of a test so I kept it for the
>   moment)
> - one check per block of tests around the same theme: it would indeed
>   avoid re-doing the same check 2 or 3 times but it would break the IDs
>   (+maybe less obvious a test might be skipped when reviewing code)
> - backports: at the end, we just need all these new checks to be
>   backported on the last stable kernel (6.3) because that's what is used
>   by the different CIs. It is fine if it is not backported after. (but
>   still I can help with the backports, I'm used to :) )
> - New patches:
>   - selftests: mptcp: join: uniform listener tests
>   - Squash to "selftests: mptcp: add MPTCP_FULL_INFO testcase"
> - See the changelog on the individual patches
> - Link to v2: https://lore.kernel.org/r/20230406-mptcp-issue-368-selftests-old-kernels-v2-0-50313e4f83ab@tessares.net

As discussed at the last weekly meeting, I just applied this series
(without any extra ACK/RvB for the moment, I can always add some later
if needed). Note that there is a conflict between net and net-next:

- a73cbdf0325c: conflict in top-bases/t/DO-NOT-MERGE-git-markup-net-next

I will make sure we pass the information to netdev maintainers when we
will send these patches to netdev for -net.


Applied in our tree (fix for -net + 1 fix for net-next and a squash-to
patch, see below):

New patches for t/upstream-net and t/upstream:
- c8230ff8c611: selftests: mptcp: lib: skip if missing symbol
- d5b45288feab: selftests: mptcp: lib: skip if not below kernel version
- 13c09048a8a9: selftests: mptcp: connect: skip transp tests if not
supported
- da3fbb1c9d55: selftests: mptcp: connect: skip disconnect tests if not
supported
- 17fd0d99c269: selftests: mptcp: connect: skip TFO tests if not supported
- 56f59ccac581: selftests: mptcp: diag: skip listen tests if not supported
- dc15a4faa5d2: selftests: mptcp: diag: skip inuse tests if not supported
- 0a0aedc21e62: selftests: mptcp: pm nl: remove hardcoded default limits
- a4f952fdfd0d: selftests: mptcp: pm nl: skip fullmesh flag checks if
not supported
- c9e5532bbb9a: selftests: mptcp: sockopt: relax expected returned size
- 119bd6b5574d: selftests: mptcp: sockopt: skip getsockopt checks if not
supported
- 5057b6fd18b4: selftests: mptcp: sockopt: skip TCP_INQ checks if not
supported
- b83b318c8d6a: selftests: mptcp: userspace pm: skip if 'ip' tool is
unavailable
- fae0ec9a5799: selftests: mptcp: userspace pm: skip if not supported
- c8763f8313c2: selftests: mptcp: userspace pm: skip PM listener events
tests if unavailable
- 96914defcb81: selftests: mptcp: join: use 'iptables-legacy' if available
- 67a7d697e202: selftests: mptcp: join: helpers to skip tests
- 8bf1ea244e7a: selftests: mptcp: join: skip check if MIB counter not
supported
- da5bce0ddee8: selftests: mptcp: join: skip test if iptables/tc cmds fail
- 5ce49553bff9: selftests: mptcp: join: support local endpoint being
tracked or not
- caa512715f88: selftests: mptcp: join: skip Fastclose tests if not
supported
- 66a74fd06d9e: selftests: mptcp: join: support RM_ADDR for used
endpoints or not
- de48230c2cc5: selftests: mptcp: join: skip implicit tests if not supported
- f17f007808ad: selftests: mptcp: join: skip backup if set flag on ID
not supported
- 4b6f29b3fdee: selftests: mptcp: join: skip fullmesh flag tests if not
supported
- 14545e86df1e: selftests: mptcp: join: skip userspace PM tests if not
supported
- 4085b667634e: selftests: mptcp: join: skip fail tests if not supported
- 93c1d1a9c615: selftests: mptcp: join: skip MPC backups tests if not
supported
- d42fc1b70fb7: selftests: mptcp: join: skip PM listener tests if not
supported
- 8b438e39bbee: selftests: mptcp: join: uniform listener tests
- 356b01bcdc41: selftests: mptcp: join: skip mixed tests if not supported
- 426c2b289431: "squashed" patch 32/32 in "selftests: mptcp: add
MPTCP_FULL_INFO testcase"
- Results: 551314e25c6e..4865fe6804ff (export-net)
- a17f99898f8a: selftests: mptcp: join: skip check if MIB counter not
supported (part 2)
- Results: 81bf7db050d7..7aa34a6db848 (export)

https://cirrus-ci.com/github/multipath-tcp/mptcp_net-next/export-net/20230607T094539
https://cirrus-ci.com/github/multipath-tcp/mptcp_net-next/export/20230607T094151

Cheers,
Matt
-- 
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net