Documentation/admin-guide/sysctl/kernel.rst | 8 ++++++++ kernel/hung_task.c | 14 +++++++++++++- 2 files changed, 21 insertions(+), 1 deletion(-)
From: Li RongQing <lirongqing@baidu.com>
Currently, when hung_task_panic is enabled, kernel will panic immediately
upon detecting the first hung task. However, some hung tasks are transient
and the system can recover fully, while others are unrecoverable and
trigger consecutive hung task reports, and a panic is expected.
This commit adds a new sysctl parameter hung_task_count_to_panic to allows
specifying the number of consecutive hung tasks that must be detected
before triggering a kernel panic. This provides finer control for
environments where transient hangs maybe happen but persistent hangs should
still be fatal.
Acked-by: Lance Yang <lance.yang@linux.dev>
Signed-off-by: Li RongQing <lirongqing@baidu.com>
---
Diff with v1: change documentation as Lance suggested
Documentation/admin-guide/sysctl/kernel.rst | 8 ++++++++
kernel/hung_task.c | 14 +++++++++++++-
2 files changed, 21 insertions(+), 1 deletion(-)
diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
index 8b49eab..98b47a7 100644
--- a/Documentation/admin-guide/sysctl/kernel.rst
+++ b/Documentation/admin-guide/sysctl/kernel.rst
@@ -405,6 +405,14 @@ This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled.
1 Panic immediately.
= =================================================
+hung_task_count_to_panic
+=====================
+
+When set to a non-zero value, a kernel panic will be triggered if the
+number of detected hung tasks reaches this value.
+
+Note that setting hung_task_panic=1 will still cause an immediate panic
+on the first hung task.
hung_task_check_count
=====================
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 8708a12..87a6421 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -83,6 +83,8 @@ static unsigned int __read_mostly sysctl_hung_task_all_cpu_backtrace;
static unsigned int __read_mostly sysctl_hung_task_panic =
IS_ENABLED(CONFIG_BOOTPARAM_HUNG_TASK_PANIC);
+static unsigned int __read_mostly sysctl_hung_task_count_to_panic;
+
static int
hung_task_panic(struct notifier_block *this, unsigned long event, void *ptr)
{
@@ -219,7 +221,9 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout)
trace_sched_process_hang(t);
- if (sysctl_hung_task_panic) {
+ if (sysctl_hung_task_panic ||
+ (sysctl_hung_task_count_to_panic &&
+ (sysctl_hung_task_detect_count >= sysctl_hung_task_count_to_panic))) {
console_verbose();
hung_task_show_lock = true;
hung_task_call_panic = true;
@@ -388,6 +392,14 @@ static const struct ctl_table hung_task_sysctls[] = {
.extra2 = SYSCTL_ONE,
},
{
+ .procname = "hung_task_count_to_panic",
+ .data = &sysctl_hung_task_count_to_panic,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_minmax,
+ .extra1 = SYSCTL_ZERO,
+ },
+ {
.procname = "hung_task_check_count",
.data = &sysctl_hung_task_check_count,
.maxlen = sizeof(int),
--
2.9.4On Sun, 28 Sep 2025 13:31:37 +0800
lirongqing <lirongqing@baidu.com> wrote:
> From: Li RongQing <lirongqing@baidu.com>
>
> Currently, when hung_task_panic is enabled, kernel will panic immediately
> upon detecting the first hung task. However, some hung tasks are transient
> and the system can recover fully, while others are unrecoverable and
> trigger consecutive hung task reports, and a panic is expected.
>
> This commit adds a new sysctl parameter hung_task_count_to_panic to allows
> specifying the number of consecutive hung tasks that must be detected
> before triggering a kernel panic. This provides finer control for
> environments where transient hangs maybe happen but persistent hangs should
> still be fatal.
IIUC, perhaps there are multiple groups that require different timeouts
for hang checks, and you want to set the hung task timeout to match
the shorter one, but ignore the longer ones at that point.
If so, this is essentially a problem with a long process that is
performed under TASK_UNINTERRUPTIBLE. Ideally, the progress of such
process should be checked periodically and the hang check should be
reset unless it is real blocked.
But this is not currently implemented. (For example, depending on
the media, it may not be possible to check whether long IO is being
performed.)
The hung_tasks will even simulate these types of hangs as task
hang-ups. But if you set a long detection time accordingly, you
will also have to wait until that detection time for hangs that
occur in a short period of time.
The hung tasks on one major lock can spread in a domino effect.
So setting a reasonably short detection time, but not panicking
until there are enough of them, seems like a reasonable strategy.
But in this case, I think we also need a "hard timeout limit"
of hung tasks, which will detect longer ones. And also you should
use peak value not accumulation value.
If it is really transient (thus, it is not hung), accumulation of
such normal but just slow operation will still kick hung_tasks.
Thank you,
>
> Acked-by: Lance Yang <lance.yang@linux.dev>
> Signed-off-by: Li RongQing <lirongqing@baidu.com>
> ---
> Diff with v1: change documentation as Lance suggested
>
> Documentation/admin-guide/sysctl/kernel.rst | 8 ++++++++
> kernel/hung_task.c | 14 +++++++++++++-
> 2 files changed, 21 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
> index 8b49eab..98b47a7 100644
> --- a/Documentation/admin-guide/sysctl/kernel.rst
> +++ b/Documentation/admin-guide/sysctl/kernel.rst
> @@ -405,6 +405,14 @@ This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled.
> 1 Panic immediately.
> = =================================================
>
> +hung_task_count_to_panic
> +=====================
> +
> +When set to a non-zero value, a kernel panic will be triggered if the
> +number of detected hung tasks reaches this value.
> +
> +Note that setting hung_task_panic=1 will still cause an immediate panic
> +on the first hung task.
What happen if it is 0?
>
> hung_task_check_count
> =====================
> diff --git a/kernel/hung_task.c b/kernel/hung_task.c
> index 8708a12..87a6421 100644
> --- a/kernel/hung_task.c
> +++ b/kernel/hung_task.c
> @@ -83,6 +83,8 @@ static unsigned int __read_mostly sysctl_hung_task_all_cpu_backtrace;
> static unsigned int __read_mostly sysctl_hung_task_panic =
> IS_ENABLED(CONFIG_BOOTPARAM_HUNG_TASK_PANIC);
>
> +static unsigned int __read_mostly sysctl_hung_task_count_to_panic;
> +
> static int
> hung_task_panic(struct notifier_block *this, unsigned long event, void *ptr)
> {
> @@ -219,7 +221,9 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout)
>
> trace_sched_process_hang(t);
>
> - if (sysctl_hung_task_panic) {
> + if (sysctl_hung_task_panic ||
> + (sysctl_hung_task_count_to_panic &&
> + (sysctl_hung_task_detect_count >= sysctl_hung_task_count_to_panic))) {
> console_verbose();
> hung_task_show_lock = true;
> hung_task_call_panic = true;
> @@ -388,6 +392,14 @@ static const struct ctl_table hung_task_sysctls[] = {
> .extra2 = SYSCTL_ONE,
> },
> {
> + .procname = "hung_task_count_to_panic",
> + .data = &sysctl_hung_task_count_to_panic,
> + .maxlen = sizeof(int),
> + .mode = 0644,
> + .proc_handler = proc_dointvec_minmax,
> + .extra1 = SYSCTL_ZERO,
> + },
> + {
> .procname = "hung_task_check_count",
> .data = &sysctl_hung_task_check_count,
> .maxlen = sizeof(int),
> --
> 2.9.4
>
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
Hey Li,
On 2025/9/28 13:31, lirongqing wrote:
> From: Li RongQing <lirongqing@baidu.com>
>
> Currently, when hung_task_panic is enabled, kernel will panic immediately
> upon detecting the first hung task. However, some hung tasks are transient
> and the system can recover fully, while others are unrecoverable and
> trigger consecutive hung task reports, and a panic is expected.
>
> This commit adds a new sysctl parameter hung_task_count_to_panic to allows
> specifying the number of consecutive hung tasks that must be detected
> before triggering a kernel panic. This provides finer control for
> environments where transient hangs maybe happen but persistent hangs should
> still be fatal.
>
> Acked-by: Lance Yang <lance.yang@linux.dev>
> Signed-off-by: Li RongQing <lirongqing@baidu.com>
> ---
It's working as expect. So:
Tested-by: Lance Yang <lance.yang@linux.dev>
But on second thought: regarding this new sysctl parameter, I was wondering
if a name like max_hung_task_count_to_panic might be a bit more explicit,
just to follow the convention from max_rcu_stall_to_panic.
No strong opinion on this, though :)
Cheers,
Lance
> Diff with v1: change documentation as Lance suggested
>
> Documentation/admin-guide/sysctl/kernel.rst | 8 ++++++++
> kernel/hung_task.c | 14 +++++++++++++-
> 2 files changed, 21 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
> index 8b49eab..98b47a7 100644
> --- a/Documentation/admin-guide/sysctl/kernel.rst
> +++ b/Documentation/admin-guide/sysctl/kernel.rst
> @@ -405,6 +405,14 @@ This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled.
> 1 Panic immediately.
> = =================================================
>
> +hung_task_count_to_panic
> +=====================
> +
> +When set to a non-zero value, a kernel panic will be triggered if the
> +number of detected hung tasks reaches this value.
> +
> +Note that setting hung_task_panic=1 will still cause an immediate panic
> +on the first hung task.
>
> hung_task_check_count
> =====================
> diff --git a/kernel/hung_task.c b/kernel/hung_task.c
> index 8708a12..87a6421 100644
> --- a/kernel/hung_task.c
> +++ b/kernel/hung_task.c
> @@ -83,6 +83,8 @@ static unsigned int __read_mostly sysctl_hung_task_all_cpu_backtrace;
> static unsigned int __read_mostly sysctl_hung_task_panic =
> IS_ENABLED(CONFIG_BOOTPARAM_HUNG_TASK_PANIC);
>
> +static unsigned int __read_mostly sysctl_hung_task_count_to_panic;
> +
> static int
> hung_task_panic(struct notifier_block *this, unsigned long event, void *ptr)
> {
> @@ -219,7 +221,9 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout)
>
> trace_sched_process_hang(t);
>
> - if (sysctl_hung_task_panic) {
> + if (sysctl_hung_task_panic ||
> + (sysctl_hung_task_count_to_panic &&
> + (sysctl_hung_task_detect_count >= sysctl_hung_task_count_to_panic))) {
> console_verbose();
> hung_task_show_lock = true;
> hung_task_call_panic = true;
> @@ -388,6 +392,14 @@ static const struct ctl_table hung_task_sysctls[] = {
> .extra2 = SYSCTL_ONE,
> },
> {
> + .procname = "hung_task_count_to_panic",
> + .data = &sysctl_hung_task_count_to_panic,
> + .maxlen = sizeof(int),
> + .mode = 0644,
> + .proc_handler = proc_dointvec_minmax,
> + .extra1 = SYSCTL_ZERO,
> + },
> + {
> .procname = "hung_task_check_count",
> .data = &sysctl_hung_task_check_count,
> .maxlen = sizeof(int),
> On 2025/9/28 13:31, lirongqing wrote:
> > From: Li RongQing <lirongqing@baidu.com>
> >
> > Currently, when hung_task_panic is enabled, kernel will panic
> > immediately upon detecting the first hung task. However, some hung
> > tasks are transient and the system can recover fully, while others are
> > unrecoverable and trigger consecutive hung task reports, and a panic is
> expected.
> >
> > This commit adds a new sysctl parameter hung_task_count_to_panic to
> > allows specifying the number of consecutive hung tasks that must be
> > detected before triggering a kernel panic. This provides finer control
> > for environments where transient hangs maybe happen but persistent
> > hangs should still be fatal.
> >
> > Acked-by: Lance Yang <lance.yang@linux.dev>
> > Signed-off-by: Li RongQing <lirongqing@baidu.com>
> > ---
>
> It's working as expect. So:
> Tested-by: Lance Yang <lance.yang@linux.dev>
>
> But on second thought: regarding this new sysctl parameter, I was wondering if
> a name like max_hung_task_count_to_panic might be a bit more explicit, just to
> follow the convention from max_rcu_stall_to_panic.
>
I see that all the hung task sysctl parameters start with "hung_task"? Should we keep this convention? If so, we could name it "hung_task_max_to_panic". If not, we could call it "max_hang_task_to_panic"?
-Li
> No strong opinion on this, though :)
>
> Cheers,
> Lance
>
> > Diff with v1: change documentation as Lance suggested
> >
> > Documentation/admin-guide/sysctl/kernel.rst | 8 ++++++++
> > kernel/hung_task.c | 14 +++++++++++++-
> > 2 files changed, 21 insertions(+), 1 deletion(-)
> >
> > diff --git a/Documentation/admin-guide/sysctl/kernel.rst
> > b/Documentation/admin-guide/sysctl/kernel.rst
> > index 8b49eab..98b47a7 100644
> > --- a/Documentation/admin-guide/sysctl/kernel.rst
> > +++ b/Documentation/admin-guide/sysctl/kernel.rst
> > @@ -405,6 +405,14 @@ This file shows up if
> ``CONFIG_DETECT_HUNG_TASK`` is enabled.
> > 1 Panic immediately.
> > = =================================================
> >
> > +hung_task_count_to_panic
> > +=====================
> > +
> > +When set to a non-zero value, a kernel panic will be triggered if the
> > +number of detected hung tasks reaches this value.
> > +
> > +Note that setting hung_task_panic=1 will still cause an immediate
> > +panic on the first hung task.
> >
> > hung_task_check_count
> > =====================
> > diff --git a/kernel/hung_task.c b/kernel/hung_task.c index
> > 8708a12..87a6421 100644
> > --- a/kernel/hung_task.c
> > +++ b/kernel/hung_task.c
> > @@ -83,6 +83,8 @@ static unsigned int __read_mostly
> sysctl_hung_task_all_cpu_backtrace;
> > static unsigned int __read_mostly sysctl_hung_task_panic =
> > IS_ENABLED(CONFIG_BOOTPARAM_HUNG_TASK_PANIC);
> >
> > +static unsigned int __read_mostly sysctl_hung_task_count_to_panic;
> > +
> > static int
> > hung_task_panic(struct notifier_block *this, unsigned long event, void *ptr)
> > {
> > @@ -219,7 +221,9 @@ static void check_hung_task(struct task_struct *t,
> > unsigned long timeout)
> >
> > trace_sched_process_hang(t);
> >
> > - if (sysctl_hung_task_panic) {
> > + if (sysctl_hung_task_panic ||
> > + (sysctl_hung_task_count_to_panic &&
> > + (sysctl_hung_task_detect_count >=
> > +sysctl_hung_task_count_to_panic))) {
> > console_verbose();
> > hung_task_show_lock = true;
> > hung_task_call_panic = true;
> > @@ -388,6 +392,14 @@ static const struct ctl_table hung_task_sysctls[] = {
> > .extra2 = SYSCTL_ONE,
> > },
> > {
> > + .procname = "hung_task_count_to_panic",
> > + .data = &sysctl_hung_task_count_to_panic,
> > + .maxlen = sizeof(int),
> > + .mode = 0644,
> > + .proc_handler = proc_dointvec_minmax,
> > + .extra1 = SYSCTL_ZERO,
> > + },
> > + {
> > .procname = "hung_task_check_count",
> > .data = &sysctl_hung_task_check_count,
> > .maxlen = sizeof(int),
On 2025/9/28 15:03, Li,Rongqing wrote: >> On 2025/9/28 13:31, lirongqing wrote: >>> From: Li RongQing <lirongqing@baidu.com> >>> >>> Currently, when hung_task_panic is enabled, kernel will panic >>> immediately upon detecting the first hung task. However, some hung >>> tasks are transient and the system can recover fully, while others are >>> unrecoverable and trigger consecutive hung task reports, and a panic is >> expected. >>> >>> This commit adds a new sysctl parameter hung_task_count_to_panic to >>> allows specifying the number of consecutive hung tasks that must be >>> detected before triggering a kernel panic. This provides finer control >>> for environments where transient hangs maybe happen but persistent >>> hangs should still be fatal. >>> >>> Acked-by: Lance Yang <lance.yang@linux.dev> >>> Signed-off-by: Li RongQing <lirongqing@baidu.com> >>> --- >> >> It's working as expect. So: >> Tested-by: Lance Yang <lance.yang@linux.dev> >> >> But on second thought: regarding this new sysctl parameter, I was wondering if >> a name like max_hung_task_count_to_panic might be a bit more explicit, just to >> follow the convention from max_rcu_stall_to_panic. >> > > I see that all the hung task sysctl parameters start with "hung_task"? Should we keep this convention? If so, we could name it "hung_task_max_to_panic". If not, we could call it "max_hang_task_to_panic"? Well, let's see what other folks think ;)
© 2016 - 2025 Red Hat, Inc.