perf lock contention: Reject more than 10ms delays for safety

[PATCH] perf lock contention: Reject more than 10ms delays for safety

Posted by Namhyung Kim 7 months, 1 week ago

Delaying kernel operations can be dangerous and the kernel may kill
(non-sleepable) BPF programs running for long in the future.

Limit the max delay to 10ms and update the document about it.

  $ sudo ./perf lock con -abl -J 100000us@cgroup_mutex true
  lock delay is too long: 100000us (> 10ms)

   Usage: perf lock contention [<options>]

      -J, --inject-delay <TIME@FUNC>
                            Inject delays to specific locks

Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/Documentation/perf-lock.txt | 8 ++++++--
 tools/perf/builtin-lock.c              | 5 +++++
 2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt
index 2d9aecf630422aa6..c17b3e318169f9dc 100644
--- a/tools/perf/Documentation/perf-lock.txt
+++ b/tools/perf/Documentation/perf-lock.txt
@@ -224,8 +224,12 @@ CONTENTION OPTIONS
 	only with -b/--use-bpf.
 
 	The 'time' is specified in nsec but it can have a unit suffix.  Available
-	units are "ms" and "us".  Note that it will busy-wait after it gets the
-	lock.  Please use it at your own risk.
+	units are "ms", "us" and "ns".  Currently it accepts up to 10ms of delays
+	for safety reasons.
+
+	Note that it will busy-wait after it gets the lock. Delaying locks can
+	have significant consequences including potential kernel crashes.  Please
+	use it at your own risk.
 
 
 SEE ALSO
diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
index 41f6f3d2b779b986..3b3ade7a39cad01f 100644
--- a/tools/perf/builtin-lock.c
+++ b/tools/perf/builtin-lock.c
@@ -2537,6 +2537,11 @@ static bool add_lock_delay(char *spec)
 		return false;
 	}
 
+	if (duration > 10 * 1000 * 1000) {
+		pr_err("lock delay is too long: %s (> 10ms)\n", spec);
+		return false;
+	}
+
 	tmp = realloc(delays, (nr_delays + 1) * sizeof(*delays));
 	if (tmp == NULL) {
 		pr_err("Memory allocation failure\n");
-- 
2.49.0.1101.gccaa498523-goog

Re: [PATCH] perf lock contention: Reject more than 10ms delays for safety

Posted by Arnaldo Carvalho de Melo 6 months, 3 weeks ago

On Thu, May 15, 2025 at 11:10:42AM -0700, Namhyung Kim wrote:
> Delaying kernel operations can be dangerous and the kernel may kill
> (non-sleepable) BPF programs running for long in the future.
> 
> Limit the max delay to 10ms and update the document about it.
> 
>   $ sudo ./perf lock con -abl -J 100000us@cgroup_mutex true
>   lock delay is too long: 100000us (> 10ms)
> 
>    Usage: perf lock contention [<options>]
> 
>       -J, --inject-delay <TIME@FUNC>
>                             Inject delays to specific locks
> 
> Suggested-by: Alexei Starovoitov <ast@kernel.org>
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/Documentation/perf-lock.txt | 8 ++++++--
>  tools/perf/builtin-lock.c              | 5 +++++
>  2 files changed, 11 insertions(+), 2 deletions(-)
> 
> diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt
> index 2d9aecf630422aa6..c17b3e318169f9dc 100644
> --- a/tools/perf/Documentation/perf-lock.txt
> +++ b/tools/perf/Documentation/perf-lock.txt
> @@ -224,8 +224,12 @@ CONTENTION OPTIONS
>  	only with -b/--use-bpf.
>  
>  	The 'time' is specified in nsec but it can have a unit suffix.  Available
> -	units are "ms" and "us".  Note that it will busy-wait after it gets the
> -	lock.  Please use it at your own risk.
> +	units are "ms", "us" and "ns".  Currently it accepts up to 10ms of delays
> +	for safety reasons.
> +
> +	Note that it will busy-wait after it gets the lock. Delaying locks can
> +	have significant consequences including potential kernel crashes.  Please
> +	use it at your own risk.
>  
>  
>  SEE ALSO
> diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
> index 41f6f3d2b779b986..3b3ade7a39cad01f 100644
> --- a/tools/perf/builtin-lock.c
> +++ b/tools/perf/builtin-lock.c
> @@ -2537,6 +2537,11 @@ static bool add_lock_delay(char *spec)
>  		return false;
>  	}
>  
> +	if (duration > 10 * 1000 * 1000) {
> +		pr_err("lock delay is too long: %s (> 10ms)\n", spec);
> +		return false;
> +	}
> +

Please consider to replace those 1000 * 1000 her and in other places
with NSEC_PER_MSEC in a followup patch for the next merge window.

- Arnaldo

>  	tmp = realloc(delays, (nr_delays + 1) * sizeof(*delays));
>  	if (tmp == NULL) {
>  		pr_err("Memory allocation failure\n");
> -- 
> 2.49.0.1101.gccaa498523-goog

Re: [PATCH] perf lock contention: Reject more than 10ms delays for safety

Posted by Arnaldo Carvalho de Melo 6 months, 3 weeks ago

On Thu, May 15, 2025 at 11:10:42AM -0700, Namhyung Kim wrote:
> Delaying kernel operations can be dangerous and the kernel may kill
> (non-sleepable) BPF programs running for long in the future.
> 
> Limit the max delay to 10ms and update the document about it.
> 
>   $ sudo ./perf lock con -abl -J 100000us@cgroup_mutex true
>   lock delay is too long: 100000us (> 10ms)
> 
>    Usage: perf lock contention [<options>]
> 
>       -J, --inject-delay <TIME@FUNC>
>                             Inject delays to specific locks
> 
> Suggested-by: Alexei Starovoitov <ast@kernel.org>
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>



Thanks, applied to perf-tools-next,

- Arnaldo

> ---
>  tools/perf/Documentation/perf-lock.txt | 8 ++++++--
>  tools/perf/builtin-lock.c              | 5 +++++
>  2 files changed, 11 insertions(+), 2 deletions(-)
> 
> diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt
> index 2d9aecf630422aa6..c17b3e318169f9dc 100644
> --- a/tools/perf/Documentation/perf-lock.txt
> +++ b/tools/perf/Documentation/perf-lock.txt
> @@ -224,8 +224,12 @@ CONTENTION OPTIONS
>  	only with -b/--use-bpf.
>  
>  	The 'time' is specified in nsec but it can have a unit suffix.  Available
> -	units are "ms" and "us".  Note that it will busy-wait after it gets the
> -	lock.  Please use it at your own risk.
> +	units are "ms", "us" and "ns".  Currently it accepts up to 10ms of delays
> +	for safety reasons.
> +
> +	Note that it will busy-wait after it gets the lock. Delaying locks can
> +	have significant consequences including potential kernel crashes.  Please
> +	use it at your own risk.
>  
>  
>  SEE ALSO
> diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
> index 41f6f3d2b779b986..3b3ade7a39cad01f 100644
> --- a/tools/perf/builtin-lock.c
> +++ b/tools/perf/builtin-lock.c
> @@ -2537,6 +2537,11 @@ static bool add_lock_delay(char *spec)
>  		return false;
>  	}
>  
> +	if (duration > 10 * 1000 * 1000) {
> +		pr_err("lock delay is too long: %s (> 10ms)\n", spec);
> +		return false;
> +	}
> +
>  	tmp = realloc(delays, (nr_delays + 1) * sizeof(*delays));
>  	if (tmp == NULL) {
>  		pr_err("Memory allocation failure\n");
> -- 
> 2.49.0.1101.gccaa498523-goog

Re: [PATCH] perf lock contention: Reject more than 10ms delays for safety

Posted by Ian Rogers 7 months, 1 week ago

Nit, in the subject line for clarity perhaps rather than "perf lock
contention:" call it "perf lock contention/delay" or "perf lock
delay".

On Thu, May 15, 2025 at 11:10 AM Namhyung Kim <namhyung@kernel.org> wrote:
>
> Delaying kernel operations can be dangerous and the kernel may kill
> (non-sleepable) BPF programs running for long in the future.
>
> Limit the max delay to 10ms and update the document about it.
>
>   $ sudo ./perf lock con -abl -J 100000us@cgroup_mutex true
>   lock delay is too long: 100000us (> 10ms)
>
>    Usage: perf lock contention [<options>]
>
>       -J, --inject-delay <TIME@FUNC>
>                             Inject delays to specific locks
>
> Suggested-by: Alexei Starovoitov <ast@kernel.org>
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/Documentation/perf-lock.txt | 8 ++++++--
>  tools/perf/builtin-lock.c              | 5 +++++
>  2 files changed, 11 insertions(+), 2 deletions(-)
>
> diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt
> index 2d9aecf630422aa6..c17b3e318169f9dc 100644
> --- a/tools/perf/Documentation/perf-lock.txt
> +++ b/tools/perf/Documentation/perf-lock.txt
> @@ -224,8 +224,12 @@ CONTENTION OPTIONS
>         only with -b/--use-bpf.
>
>         The 'time' is specified in nsec but it can have a unit suffix.  Available
> -       units are "ms" and "us".  Note that it will busy-wait after it gets the
> -       lock.  Please use it at your own risk.
> +       units are "ms", "us" and "ns".  Currently it accepts up to 10ms of delays
> +       for safety reasons.
> +
> +       Note that it will busy-wait after it gets the lock. Delaying locks can
> +       have significant consequences including potential kernel crashes.  Please
> +       use it at your own risk.
>
>
>  SEE ALSO
> diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
> index 41f6f3d2b779b986..3b3ade7a39cad01f 100644
> --- a/tools/perf/builtin-lock.c
> +++ b/tools/perf/builtin-lock.c
> @@ -2537,6 +2537,11 @@ static bool add_lock_delay(char *spec)
>                 return false;
>         }
>
> +       if (duration > 10 * 1000 * 1000) {

nit: It's unfortunate the variable name isn't carrying the time unit.
For example, this could be:
```
if (duration_ns > 10 * NSEC_PER_SEC) {
```
which should hopefully make it clearer what the time units are and
that they aren't messed up.

Thanks,
Ian

> +               pr_err("lock delay is too long: %s (> 10ms)\n", spec);
> +               return false;
> +       }
> +
>         tmp = realloc(delays, (nr_delays + 1) * sizeof(*delays));
>         if (tmp == NULL) {
>                 pr_err("Memory allocation failure\n");
> --
> 2.49.0.1101.gccaa498523-goog
>