With the increase number of tests, on a slow public CI with kernel debug
settings, it has been observed it can easily and regularly take up to 45
minutes to execute all tests in this very slow environment. The slowest
test, mptcp_join.sh takes ~30 minutes in this case.

In such environments, the selftests timeout set in the 'settings' file
is disabled. But because other similar environments can be used to run
MPTCP selftests, it is certainly better to increase the timeout for
everybody else to 1h.

Most of the time, running all MPTCP selftests takes around 5 to 10
minutes on decent environments and the slowest test takes less than 10
minutes, the current timeout.

The timeout could be disabled but it is always good to have an extra
safeguard, just in case.

Signed-off-by: Matthieu Baerts <matthieu.baerts@tessares.net>
---

Notes:
    On my CI at Tessares, another machine is now used with a different host
    kernel. mptcp_join.sh selftest is now regularly taking more than 10
    minutes when testing with a debug kernel.

 tools/testing/selftests/net/mptcp/settings | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/testing/selftests/net/mptcp/settings b/tools/testing/selftests/net/mptcp/settings
index a62d2fa1275c..ed8418e8217a 100644
--- a/tools/testing/selftests/net/mptcp/settings
+++ b/tools/testing/selftests/net/mptcp/settings
@@ -1 +1 @@
-timeout=600
+timeout=3600
-- 
2.34.1

On Mon, 7 Feb 2022, Matthieu Baerts wrote:

> With the increase number of tests, on a slow public CI with kernel debug
> settings, it has been observed it can easily and regularly take up to 45
> minutes to execute all tests in this very slow environment. The slowest
> test, mptcp_join.sh takes ~30 minutes in this case.
>
> In such environments, the selftests timeout set in the 'settings' file
> is disabled. But because other similar environments can be used to run
> MPTCP selftests, it is certainly better to increase the timeout for
> everybody else to 1h.
>
> Most of the time, running all MPTCP selftests takes around 5 to 10
> minutes on decent environments and the slowest test takes less than 10
> minutes, the current timeout.
>
> The timeout could be disabled but it is always good to have an extra
> safeguard, just in case.
>

I agree that it's good to have some timeout in place for our tests. An 
hour seems like a lot, has there been any feedback from others with CI as 
slow as ours?

The largest existing selftest timeout is 1500 (net/settings), and looking 
at the kbuild logs it takes their infrastructure 28 minutes to run the net 
tests and 11 minutes to run the MPTCP tests. What do you think about 
a smaller timeout increase for the MPTCP tests?

-Mat

> Signed-off-by: Matthieu Baerts <matthieu.baerts@tessares.net>
> ---
>
> Notes:
>    On my CI at Tessares, another machine is now used with a different host
>    kernel. mptcp_join.sh selftest is now regularly taking more than 10
>    minutes when testing with a debug kernel.
>
> tools/testing/selftests/net/mptcp/settings | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tools/testing/selftests/net/mptcp/settings b/tools/testing/selftests/net/mptcp/settings
> index a62d2fa1275c..ed8418e8217a 100644
> --- a/tools/testing/selftests/net/mptcp/settings
> +++ b/tools/testing/selftests/net/mptcp/settings
> @@ -1 +1 @@
> -timeout=600
> +timeout=3600
> -- 
> 2.34.1
>
>
>

--
Mat Martineau
Intel

Hi Mat,

On 07/02/2022 20:49, Mat Martineau wrote:
> On Mon, 7 Feb 2022, Matthieu Baerts wrote:
> 
>> With the increase number of tests, on a slow public CI with kernel debug
>> settings, it has been observed it can easily and regularly take up to 45
>> minutes to execute all tests in this very slow environment. The slowest
>> test, mptcp_join.sh takes ~30 minutes in this case.
>>
>> In such environments, the selftests timeout set in the 'settings' file
>> is disabled. But because other similar environments can be used to run
>> MPTCP selftests, it is certainly better to increase the timeout for
>> everybody else to 1h.
>>
>> Most of the time, running all MPTCP selftests takes around 5 to 10
>> minutes on decent environments and the slowest test takes less than 10
>> minutes, the current timeout.
>>
>> The timeout could be disabled but it is always good to have an extra
>> safeguard, just in case.
>>
> 
> I agree that it's good to have some timeout in place for our tests. An
> hour seems like a lot, has there been any feedback from others with CI
> as slow as ours?
> 
> The largest existing selftest timeout is 1500 (net/settings), and
> looking at the kbuild logs it takes their infrastructure 28 minutes to
> run the net tests and 11 minutes to run the MPTCP tests. What do you
> think about a smaller timeout increase for the MPTCP tests?

I can, it would be enough for my CI. But not for a setup like the public
CI where mptcp_join.sh tests can take up to 30 minutes and it keeps
increasing.

But if you prefer, I can double the current value and that's enough for
my case. What do you prefer?

Cheers,
Matt
-- 
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net

On Mon, 7 Feb 2022, Matthieu Baerts wrote:

> Hi Mat,
>
> On 07/02/2022 20:49, Mat Martineau wrote:
>> On Mon, 7 Feb 2022, Matthieu Baerts wrote:
>>
>>> With the increase number of tests, on a slow public CI with kernel debug
>>> settings, it has been observed it can easily and regularly take up to 45
>>> minutes to execute all tests in this very slow environment. The slowest
>>> test, mptcp_join.sh takes ~30 minutes in this case.
>>>
>>> In such environments, the selftests timeout set in the 'settings' file
>>> is disabled. But because other similar environments can be used to run
>>> MPTCP selftests, it is certainly better to increase the timeout for
>>> everybody else to 1h.
>>>
>>> Most of the time, running all MPTCP selftests takes around 5 to 10
>>> minutes on decent environments and the slowest test takes less than 10
>>> minutes, the current timeout.
>>>
>>> The timeout could be disabled but it is always good to have an extra
>>> safeguard, just in case.
>>>
>>
>> I agree that it's good to have some timeout in place for our tests. An
>> hour seems like a lot, has there been any feedback from others with CI
>> as slow as ours?
>>
>> The largest existing selftest timeout is 1500 (net/settings), and
>> looking at the kbuild logs it takes their infrastructure 28 minutes to
>> run the net tests and 11 minutes to run the MPTCP tests. What do you
>> think about a smaller timeout increase for the MPTCP tests?
>
> I can, it would be enough for my CI. But not for a setup like the public
> CI where mptcp_join.sh tests can take up to 30 minutes and it keeps
> increasing.
>
> But if you prefer, I can double the current value and that's enough for
> my case. What do you prefer?
>

Doubling the current value is my preference for now.

--
Mat Martineau
Intel