Documentation/dev-tools/kselftest.rst | 22 +++++++++++++++++++++ tools/testing/selftests/kselftest/runner.sh | 11 ++++++++++- tools/testing/selftests/run_kselftest.sh | 5 +++++ 3 files changed, 37 insertions(+), 1 deletion(-)
The default timeout for selftests tests is 45 seconds. Although
we already have 13 settings for tests of about 96 sefltests which
use a timeout greater than this, we want to try to avoid encouraging
more tests to forcing a higher test timeout as selftests strives to
run all tests quickly. Selftests also uses the timeout as a non-fatal
error. Only tests runners which have control over a system would know
if to treat a timeout as fatal or not.
To help with all this:
o Enhance documentation to avoid future increases of insane timeouts
o Add the option to allow overriding the default timeout with test
runners with a command line option
Suggested-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
Documentation/dev-tools/kselftest.rst | 22 +++++++++++++++++++++
tools/testing/selftests/kselftest/runner.sh | 11 ++++++++++-
tools/testing/selftests/run_kselftest.sh | 5 +++++
3 files changed, 37 insertions(+), 1 deletion(-)
diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
index 12b575b76b20..dd214af7b7ff 100644
--- a/Documentation/dev-tools/kselftest.rst
+++ b/Documentation/dev-tools/kselftest.rst
@@ -168,6 +168,28 @@ the `-t` option for specific single tests. Either can be used multiple times::
For other features see the script usage output, seen with the `-h` option.
+Timeout for selftests
+=====================
+
+Selftests are designed to be quick and so a default timeout is used of 45
+seconds for each test. Tests can override the default timeout by adding
+a settings file in their directory and set a timeout variable there to the
+configured a desired upper timeout for the test. Only a few tests override
+the timeout with a value higher than 45 seconds, selftests strives to keep
+it that way. Timeouts in selftests are not considered fatal because the
+system under which a test runs may change and this can also modify the
+expected time it takes to run a test. If you have control over the systems
+which will run the tests you can configure a test runner on those systems to
+use a greater or lower timeout on the command line as with the `-o` or
+the `--override-timeout` argument. For example to use 165 seconds instead
+one would use:
+
+ $ ./run_kselftest.sh --override-timeout 165
+
+You can look at the TAP output to see if you ran into the timeout. Test
+runners which know a test must run under a specific time can then optionally
+treat these timeouts then as fatal.
+
Packaging selftests
===================
diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh
index 294619ade49f..1c952d1401d4 100644
--- a/tools/testing/selftests/kselftest/runner.sh
+++ b/tools/testing/selftests/kselftest/runner.sh
@@ -8,7 +8,8 @@ export logfile=/dev/stdout
export per_test_logging=
# Defaults for "settings" file fields:
-# "timeout" how many seconds to let each test run before failing.
+# "timeout" how many seconds to let each test run before running
+# over our soft timeout limit.
export kselftest_default_timeout=45
# There isn't a shell-agnostic way to find the path of a sourced file,
@@ -90,6 +91,14 @@ run_one()
done < "$settings"
fi
+ # Command line timeout overrides the settings file
+ if [ -n "$kselftest_override_timeout" ]; then
+ kselftest_timeout="$kselftest_override_timeout"
+ echo "# overriding timeout to $kselftest_timeout" >> "$logfile"
+ else
+ echo "# timeout set to $kselftest_timeout" >> "$logfile"
+ fi
+
TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST"
echo "# $TEST_HDR_MSG"
if [ ! -e "$TEST" ]; then
diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh
index 97165a83df63..9a981b36bd7f 100755
--- a/tools/testing/selftests/run_kselftest.sh
+++ b/tools/testing/selftests/run_kselftest.sh
@@ -26,6 +26,7 @@ Usage: $0 [OPTIONS]
-l | --list List the available collection:test entries
-d | --dry-run Don't actually run any tests
-h | --help Show this usage info
+ -o | --override-timeout Number of seconds after which we timeout
EOF
exit $1
}
@@ -33,6 +34,7 @@ EOF
COLLECTIONS=""
TESTS=""
dryrun=""
+kselftest_override_timeout=""
while true; do
case "$1" in
-s | --summary)
@@ -51,6 +53,9 @@ while true; do
-d | --dry-run)
dryrun="echo"
shift ;;
+ -o | --override-timeout)
+ kselftest_override_timeout="$2"
+ shift 2 ;;
-h | --help)
usage 0 ;;
"")
--
2.39.2On 4/15/23 12:38 AM, Luis Chamberlain wrote: > The default timeout for selftests tests is 45 seconds. Although > we already have 13 settings for tests of about 96 sefltests which > use a timeout greater than this, we want to try to avoid encouraging > more tests to forcing a higher test timeout as selftests strives to > run all tests quickly. Selftests also uses the timeout as a non-fatal > error. Only tests runners which have control over a system would know > if to treat a timeout as fatal or not. > > To help with all this: > > o Enhance documentation to avoid future increases of insane timeouts > o Add the option to allow overriding the default timeout with test > runners with a command line option > > Suggested-by: Shuah Khan <skhan@linuxfoundation.org> > Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com> Tested-by:Muhammad Usama Anjum <usama.anjum@collabora.com> > --- > Documentation/dev-tools/kselftest.rst | 22 +++++++++++++++++++++ > tools/testing/selftests/kselftest/runner.sh | 11 ++++++++++- > tools/testing/selftests/run_kselftest.sh | 5 +++++ > 3 files changed, 37 insertions(+), 1 deletion(-) > > diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst > index 12b575b76b20..dd214af7b7ff 100644 > --- a/Documentation/dev-tools/kselftest.rst > +++ b/Documentation/dev-tools/kselftest.rst > @@ -168,6 +168,28 @@ the `-t` option for specific single tests. Either can be used multiple times:: > > For other features see the script usage output, seen with the `-h` option. > > +Timeout for selftests > +===================== > + > +Selftests are designed to be quick and so a default timeout is used of 45 > +seconds for each test. Tests can override the default timeout by adding > +a settings file in their directory and set a timeout variable there to the > +configured a desired upper timeout for the test. Only a few tests override > +the timeout with a value higher than 45 seconds, selftests strives to keep > +it that way. Timeouts in selftests are not considered fatal because the > +system under which a test runs may change and this can also modify the > +expected time it takes to run a test. If you have control over the systems > +which will run the tests you can configure a test runner on those systems to > +use a greater or lower timeout on the command line as with the `-o` or > +the `--override-timeout` argument. For example to use 165 seconds instead > +one would use: > + > + $ ./run_kselftest.sh --override-timeout 165 > + > +You can look at the TAP output to see if you ran into the timeout. Test > +runners which know a test must run under a specific time can then optionally > +treat these timeouts then as fatal. > + > Packaging selftests > =================== > > diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh > index 294619ade49f..1c952d1401d4 100644 > --- a/tools/testing/selftests/kselftest/runner.sh > +++ b/tools/testing/selftests/kselftest/runner.sh > @@ -8,7 +8,8 @@ export logfile=/dev/stdout > export per_test_logging= > > # Defaults for "settings" file fields: > -# "timeout" how many seconds to let each test run before failing. > +# "timeout" how many seconds to let each test run before running > +# over our soft timeout limit. > export kselftest_default_timeout=45 > > # There isn't a shell-agnostic way to find the path of a sourced file, > @@ -90,6 +91,14 @@ run_one() > done < "$settings" > fi > > + # Command line timeout overrides the settings file > + if [ -n "$kselftest_override_timeout" ]; then > + kselftest_timeout="$kselftest_override_timeout" > + echo "# overriding timeout to $kselftest_timeout" >> "$logfile" > + else > + echo "# timeout set to $kselftest_timeout" >> "$logfile" > + fi > + > TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST" > echo "# $TEST_HDR_MSG" > if [ ! -e "$TEST" ]; then > diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh > index 97165a83df63..9a981b36bd7f 100755 > --- a/tools/testing/selftests/run_kselftest.sh > +++ b/tools/testing/selftests/run_kselftest.sh > @@ -26,6 +26,7 @@ Usage: $0 [OPTIONS] > -l | --list List the available collection:test entries > -d | --dry-run Don't actually run any tests > -h | --help Show this usage info > + -o | --override-timeout Number of seconds after which we timeout > EOF > exit $1 > } > @@ -33,6 +34,7 @@ EOF > COLLECTIONS="" > TESTS="" > dryrun="" > +kselftest_override_timeout="" > while true; do > case "$1" in > -s | --summary) > @@ -51,6 +53,9 @@ while true; do > -d | --dry-run) > dryrun="echo" > shift ;; > + -o | --override-timeout) > + kselftest_override_timeout="$2" > + shift 2 ;; > -h | --help) > usage 0 ;; > "") -- BR, Muhammad Usama Anjum
On Fri, Apr 28, 2023 at 1:34 AM Muhammad Usama Anjum <usama.anjum@collabora.com> wrote: > > On 4/15/23 12:38 AM, Luis Chamberlain wrote: > > The default timeout for selftests tests is 45 seconds. Although > > we already have 13 settings for tests of about 96 sefltests which > > use a timeout greater than this, we want to try to avoid encouraging > > more tests to forcing a higher test timeout as selftests strives to > > run all tests quickly. Selftests also uses the timeout as a non-fatal > > error. Only tests runners which have control over a system would know > > if to treat a timeout as fatal or not. > > > > To help with all this: > > > > o Enhance documentation to avoid future increases of insane timeouts > > o Add the option to allow overriding the default timeout with test > > runners with a command line option > > > > Suggested-by: Shuah Khan <skhan@linuxfoundation.org> > > Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> > Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com> > Tested-by:Muhammad Usama Anjum <usama.anjum@collabora.com> Shuah, just a friendly poke! This is needed to allow me to enable full automation for kdevops for selftests. Luis
On Thu, May 11, 2023 at 08:26:42AM -0700, Luis Chamberlain wrote: > On Fri, Apr 28, 2023 at 1:34 AM Muhammad Usama Anjum > <usama.anjum@collabora.com> wrote: > > > > On 4/15/23 12:38 AM, Luis Chamberlain wrote: > > > The default timeout for selftests tests is 45 seconds. Although > > > we already have 13 settings for tests of about 96 sefltests which > > > use a timeout greater than this, we want to try to avoid encouraging > > > more tests to forcing a higher test timeout as selftests strives to > > > run all tests quickly. Selftests also uses the timeout as a non-fatal > > > error. Only tests runners which have control over a system would know > > > if to treat a timeout as fatal or not. > > > > > > To help with all this: > > > > > > o Enhance documentation to avoid future increases of insane timeouts > > > o Add the option to allow overriding the default timeout with test > > > runners with a command line option > > > > > > Suggested-by: Shuah Khan <skhan@linuxfoundation.org> > > > Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> > > Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com> > > Tested-by:Muhammad Usama Anjum <usama.anjum@collabora.com> > > Shuah, just a friendly poke! This is needed to allow me to enable full > automation for kdevops for selftests. Shuah, friendly re-poke. Luis
On 5/24/23 16:58, Luis Chamberlain wrote: > On Thu, May 11, 2023 at 08:26:42AM -0700, Luis Chamberlain wrote: >> On Fri, Apr 28, 2023 at 1:34 AM Muhammad Usama Anjum >> <usama.anjum@collabora.com> wrote: >>> >>> On 4/15/23 12:38 AM, Luis Chamberlain wrote: >>>> The default timeout for selftests tests is 45 seconds. Although >>>> we already have 13 settings for tests of about 96 sefltests which >>>> use a timeout greater than this, we want to try to avoid encouraging >>>> more tests to forcing a higher test timeout as selftests strives to >>>> run all tests quickly. Selftests also uses the timeout as a non-fatal >>>> error. Only tests runners which have control over a system would know >>>> if to treat a timeout as fatal or not. >>>> >>>> To help with all this: >>>> >>>> o Enhance documentation to avoid future increases of insane timeouts >>>> o Add the option to allow overriding the default timeout with test >>>> runners with a command line option >>>> >>>> Suggested-by: Shuah Khan <skhan@linuxfoundation.org> >>>> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> >>> Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com> >>> Tested-by:Muhammad Usama Anjum <usama.anjum@collabora.com> >> >> Shuah, just a friendly poke! This is needed to allow me to enable full >> automation for kdevops for selftests. > > Shuah, friendly re-poke. > > Luis Thanks. I will apply this for next. thanks, -- Shuah
© 2016 - 2025 Red Hat, Inc.