Documentation/admin-guide/kernel-parameters.txt | 15 ++++++++------- Documentation/admin-guide/sysctl/kernel.rst | 1 + kernel/hung_task.c | 5 +++-- lib/Kconfig.debug | 4 ++-- 4 files changed, 14 insertions(+), 11 deletions(-)
From: Li RongQing <lirongqing@baidu.com>
Currently the hung task detector can either panic immediately or continue
operation when hung tasks are detected. However, there are scenarios
where we want a more balanced approach:
- We don't want the system to panic immediately when a few hung tasks
are detected, as the system may be able to recover
- And we also don't want the system to stall indefinitely with multiple
hung tasks
This commit introduces a new mode (value 2) for the hung task panic behavior.
When set to 2, the system will panic only after the maximum number of hung
task warnings (hung_task_warnings) has been reached.
This provides a middle ground between immediate panic and potentially
infinite stall, allowing for automated vmcore generation after a reasonable
number of hung task incidents.
Signed-off-by: Li RongQing <lirongqing@baidu.com>
---
Documentation/admin-guide/kernel-parameters.txt | 15 ++++++++-------
Documentation/admin-guide/sysctl/kernel.rst | 1 +
kernel/hung_task.c | 5 +++--
lib/Kconfig.debug | 4 ++--
4 files changed, 14 insertions(+), 11 deletions(-)
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 5a7a83c..f2a9876 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -1993,13 +1993,14 @@
hung_task_panic=
[KNL] Should the hung task detector generate panics.
- Format: 0 | 1
-
- A value of 1 instructs the kernel to panic when a
- hung task is detected. The default value is controlled
- by the CONFIG_BOOTPARAM_HUNG_TASK_PANIC build-time
- option. The value selected by this boot parameter can
- be changed later by the kernel.hung_task_panic sysctl.
+ Format: 0 | 1 | 2
+
+ A value of 1 instructs the kernel to panic when a hung task is detected.
+ A value of 2 instructs the kernel to panic when hung_task_warnings is
+ decreased to 0. The default value is controlled by the
+ CONFIG_BOOTPARAM_HUNG_TASK_PANIC build-time option. The value selected
+ by this boot parameter can be changed later by the kernel.hung_task_panic
+ sysctl.
hvc_iucv= [S390] Number of z/VM IUCV hypervisor console (HVC)
terminal devices. Valid values: 0..8
diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
index 8b49eab..6f77241 100644
--- a/Documentation/admin-guide/sysctl/kernel.rst
+++ b/Documentation/admin-guide/sysctl/kernel.rst
@@ -403,6 +403,7 @@ This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled.
= =================================================
0 Continue operation. This is the default behavior.
1 Panic immediately.
+2 Panic when hung_task_warnings is decreased to 0.
= =================================================
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 8708a12..b052ec7 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -219,7 +219,8 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout)
trace_sched_process_hang(t);
- if (sysctl_hung_task_panic) {
+ if ((sysctl_hung_task_panic == 1) ||
+ (!sysctl_hung_task_warnings && sysctl_hung_task_panic == 2)) {
console_verbose();
hung_task_show_lock = true;
hung_task_call_panic = true;
@@ -385,7 +386,7 @@ static const struct ctl_table hung_task_sysctls[] = {
.mode = 0644,
.proc_handler = proc_dointvec_minmax,
.extra1 = SYSCTL_ZERO,
- .extra2 = SYSCTL_ONE,
+ .extra2 = SYSCTL_TWO,
},
{
.procname = "hung_task_check_count",
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index dc0e0c6..e7cf166 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1264,10 +1264,10 @@ config DEFAULT_HUNG_TASK_TIMEOUT
Keeping the default should be fine in most cases.
config BOOTPARAM_HUNG_TASK_PANIC
- bool "Panic (Reboot) On Hung Tasks"
+ int "Panic (Reboot) On Hung Tasks"
depends on DETECT_HUNG_TASK
help
- Say Y here to enable the kernel to panic on "hung tasks",
+ Say 1|2 here to enable the kernel to panic on "hung tasks",
which are bugs that cause the kernel to leave a task stuck
in uninterruptible "D" state.
--
2.9.4On 9/22/25 8:37 PM, lirongqing wrote: > diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug > index dc0e0c6..e7cf166 100644 > --- a/lib/Kconfig.debug > +++ b/lib/Kconfig.debug > @@ -1264,10 +1264,10 @@ config DEFAULT_HUNG_TASK_TIMEOUT > Keeping the default should be fine in most cases. > > config BOOTPARAM_HUNG_TASK_PANIC > - bool "Panic (Reboot) On Hung Tasks" > + int "Panic (Reboot) On Hung Tasks" > depends on DETECT_HUNG_TASK > help > - Say Y here to enable the kernel to panic on "hung tasks", > + Say 1|2 here to enable the kernel to panic on "hung tasks", Please make that "1|2" more user-friendly, e.g., "1 or 2". > which are bugs that cause the kernel to leave a task stuck > in uninterruptible "D" state. -- ~Randy
On Tue, 23 Sep 2025 11:37:40 +0800 lirongqing <lirongqing@baidu.com> wrote: > Currently the hung task detector can either panic immediately or continue > operation when hung tasks are detected. However, there are scenarios > where we want a more balanced approach: > > - We don't want the system to panic immediately when a few hung tasks > are detected, as the system may be able to recover > - And we also don't want the system to stall indefinitely with multiple > hung tasks > > This commit introduces a new mode (value 2) for the hung task panic behavior. > When set to 2, the system will panic only after the maximum number of hung > task warnings (hung_task_warnings) has been reached. > > This provides a middle ground between immediate panic and potentially > infinite stall, allowing for automated vmcore generation after a reasonable I assume the same argument applies to the NMI watchdog, to the softlockup detector and to the RCU stall detector? A general framework to handle all of these might be better. But why do it in kernel at all? What about a userspace detector which parses kernel logs (or new procfs counters) and makes such decisions?
> -----Original Message----- > From: Andrew Morton <akpm@linux-foundation.org> > Sent: 2025年9月23日 11:46 > To: Li,Rongqing <lirongqing@baidu.com> > Cc: corbet@lwn.net; lance.yang@linux.dev; mhiramat@kernel.org; > paulmck@kernel.org; pawan.kumar.gupta@linux.intel.com; mingo@kernel.org; > dave.hansen@linux.intel.com; rostedt@goodmis.org; kees@kernel.org; > arnd@arndb.de; feng.tang@linux.alibaba.com; pauld@redhat.com; > joel.granados@kernel.org; linux-doc@vger.kernel.org; > linux-kernel@vger.kernel.org > Subject: [????] Re: [PATCH][RFC] hung_task: Support to panic when the > maximum number of hung task warnings is reached > > On Tue, 23 Sep 2025 11:37:40 +0800 lirongqing <lirongqing@baidu.com> wrote: > > > Currently the hung task detector can either panic immediately or > > continue operation when hung tasks are detected. However, there are > > scenarios where we want a more balanced approach: > > > > - We don't want the system to panic immediately when a few hung tasks > > are detected, as the system may be able to recover > > - And we also don't want the system to stall indefinitely with multiple > > hung tasks > > > > This commit introduces a new mode (value 2) for the hung task panic behavior. > > When set to 2, the system will panic only after the maximum number of > > hung task warnings (hung_task_warnings) has been reached. > > > > This provides a middle ground between immediate panic and potentially > > infinite stall, allowing for automated vmcore generation after a > > reasonable > > I assume the same argument applies to the NMI watchdog, to the softlockup > detector and to the RCU stall detector? True, especial RCU stall detector > > A general framework to handle all of these might be better. But why do it in > kernel at all? What about a userspace detector which parses kernel logs (or > new procfs counters) and makes such decisions? By leveraging existing kernel mechanisms, implementation in kernel is very simple and reliable, I think Thanks -Li
On Tue, Sep 23, 2025 at 04:00:03AM +0000, Li,Rongqing wrote: > > > > -----Original Message----- > > From: Andrew Morton <akpm@linux-foundation.org> > > Sent: 2025年9月23日 11:46 > > To: Li,Rongqing <lirongqing@baidu.com> > > Cc: corbet@lwn.net; lance.yang@linux.dev; mhiramat@kernel.org; > > paulmck@kernel.org; pawan.kumar.gupta@linux.intel.com; mingo@kernel.org; > > dave.hansen@linux.intel.com; rostedt@goodmis.org; kees@kernel.org; > > arnd@arndb.de; feng.tang@linux.alibaba.com; pauld@redhat.com; > > joel.granados@kernel.org; linux-doc@vger.kernel.org; > > linux-kernel@vger.kernel.org > > Subject: [????] Re: [PATCH][RFC] hung_task: Support to panic when the > > maximum number of hung task warnings is reached > > > > On Tue, 23 Sep 2025 11:37:40 +0800 lirongqing <lirongqing@baidu.com> wrote: > > > > > Currently the hung task detector can either panic immediately or > > > continue operation when hung tasks are detected. However, there are > > > scenarios where we want a more balanced approach: > > > > > > - We don't want the system to panic immediately when a few hung tasks > > > are detected, as the system may be able to recover > > > - And we also don't want the system to stall indefinitely with multiple > > > hung tasks > > > > > > This commit introduces a new mode (value 2) for the hung task panic behavior. > > > When set to 2, the system will panic only after the maximum number of > > > hung task warnings (hung_task_warnings) has been reached. > > > > > > This provides a middle ground between immediate panic and potentially > > > infinite stall, allowing for automated vmcore generation after a > > > reasonable > > > > I assume the same argument applies to the NMI watchdog, to the softlockup > > detector and to the RCU stall detector? > > True, especial RCU stall detector There are the panic_on_rcu_stall and max_rcu_stall_to_panic sysctls, which together allow you to panic after (say) three RCU CPU stall warnings. Does those do what you need? Thanx, Paul > > A general framework to handle all of these might be better. But why do it in > > kernel at all? What about a userspace detector which parses kernel logs (or > > new procfs counters) and makes such decisions? > > > By leveraging existing kernel mechanisms, implementation in kernel is very simple and reliable, I think > > Thanks > > -Li >
> -----Original Message----- > From: Paul E. McKenney <paulmck@kernel.org> > Sent: 2025年9月23日 14:04 > To: Li,Rongqing <lirongqing@baidu.com> > Cc: Andrew Morton <akpm@linux-foundation.org>; corbet@lwn.net; > lance.yang@linux.dev; mhiramat@kernel.org; > pawan.kumar.gupta@linux.intel.com; mingo@kernel.org; > dave.hansen@linux.intel.com; rostedt@goodmis.org; kees@kernel.org; > arnd@arndb.de; feng.tang@linux.alibaba.com; pauld@redhat.com; > joel.granados@kernel.org; linux-doc@vger.kernel.org; > linux-kernel@vger.kernel.org > Subject: [外部邮件] Re: [????] Re: [PATCH][RFC] hung_task: Support to panic > when the maximum number of hung task warnings is reached > > On Tue, Sep 23, 2025 at 04:00:03AM +0000, Li,Rongqing wrote: > > > > > > > -----Original Message----- > > > From: Andrew Morton <akpm@linux-foundation.org> > > > Sent: 2025年9月23日 11:46 > > > To: Li,Rongqing <lirongqing@baidu.com> > > > Cc: corbet@lwn.net; lance.yang@linux.dev; mhiramat@kernel.org; > > > paulmck@kernel.org; pawan.kumar.gupta@linux.intel.com; > > > mingo@kernel.org; dave.hansen@linux.intel.com; rostedt@goodmis.org; > > > kees@kernel.org; arnd@arndb.de; feng.tang@linux.alibaba.com; > > > pauld@redhat.com; joel.granados@kernel.org; > > > linux-doc@vger.kernel.org; linux-kernel@vger.kernel.org > > > Subject: [????] Re: [PATCH][RFC] hung_task: Support to panic when > > > the maximum number of hung task warnings is reached > > > > > > On Tue, 23 Sep 2025 11:37:40 +0800 lirongqing <lirongqing@baidu.com> > wrote: > > > > > > > Currently the hung task detector can either panic immediately or > > > > continue operation when hung tasks are detected. However, there > > > > are scenarios where we want a more balanced approach: > > > > > > > > - We don't want the system to panic immediately when a few hung tasks > > > > are detected, as the system may be able to recover > > > > - And we also don't want the system to stall indefinitely with multiple > > > > hung tasks > > > > > > > > This commit introduces a new mode (value 2) for the hung task panic > behavior. > > > > When set to 2, the system will panic only after the maximum number > > > > of hung task warnings (hung_task_warnings) has been reached. > > > > > > > > This provides a middle ground between immediate panic and > > > > potentially infinite stall, allowing for automated vmcore > > > > generation after a reasonable > > > > > > I assume the same argument applies to the NMI watchdog, to the > > > softlockup detector and to the RCU stall detector? > > > > True, especial RCU stall detector > > There are the panic_on_rcu_stall and max_rcu_stall_to_panic sysctls, which > together allow you to panic after (say) three RCU CPU stall warnings. > Does those do what you need? Yes, this is what I need. RCU has been implemented. Thanks -Li > > Thanx, Paul > > > > A general framework to handle all of these might be better. But why > > > do it in kernel at all? What about a userspace detector which > > > parses kernel logs (or new procfs counters) and makes such decisions? > > > > > > By leveraging existing kernel mechanisms, implementation in kernel is > > very simple and reliable, I think > > > > Thanks > > > > -Li > >
> There are the panic_on_rcu_stall and max_rcu_stall_to_panic sysctls, which
> together allow you to panic after (say) three RCU CPU stall warnings.
> Does those do what you need?
>
> Thanx, Paul
>
inspired by dfe564045c653d "(rcu: Panic after fixed number of stalls)", add a new knod for hung task, how about?
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 8708a12..b93592d 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -83,6 +83,8 @@ static unsigned int __read_mostly sysctl_hung_task_all_cpu_backtrace;
static unsigned int __read_mostly sysctl_hung_task_panic =
IS_ENABLED(CONFIG_BOOTPARAM_HUNG_TASK_PANIC);
+static unsigned int __read_mostly sysctl_hung_task_panic_count;
+
static int
hung_task_panic(struct notifier_block *this, unsigned long event, void *ptr)
{
@@ -219,7 +221,9 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout)
trace_sched_process_hang(t);
- if (sysctl_hung_task_panic) {
+ if (sysctl_hung_task_panic ||
+ (sysctl_hung_task_panic_count &&
+ (sysctl_hung_task_detect_count > sysctl_hung_task_panic_count))) {
console_verbose();
hung_task_show_lock = true;
hung_task_call_panic = true;
@@ -388,6 +392,14 @@ static const struct ctl_table hung_task_sysctls[] = {
.extra2 = SYSCTL_ONE,
},
{
+ .procname = "hung_task_panic_count",
+ .data = &sysctl_hung_task_panic_count,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_minmax,
+ .extra1 = SYSCTL_ZERO,
+ },
+ {
.procname = "hung_task_check_count",
.data = &sysctl_hung_task_check_count,
.maxlen = sizeof(int),
On 2025/9/23 11:45, Andrew Morton wrote: > On Tue, 23 Sep 2025 11:37:40 +0800 lirongqing <lirongqing@baidu.com> wrote: > >> Currently the hung task detector can either panic immediately or continue >> operation when hung tasks are detected. However, there are scenarios >> where we want a more balanced approach: >> >> - We don't want the system to panic immediately when a few hung tasks >> are detected, as the system may be able to recover >> - And we also don't want the system to stall indefinitely with multiple >> hung tasks >> >> This commit introduces a new mode (value 2) for the hung task panic behavior. >> When set to 2, the system will panic only after the maximum number of hung >> task warnings (hung_task_warnings) has been reached. >> >> This provides a middle ground between immediate panic and potentially >> infinite stall, allowing for automated vmcore generation after a reasonable > > I assume the same argument applies to the NMI watchdog, to the > softlockup detector and to the RCU stall detector? > > A general framework to handle all of these might be better. But why do > it in kernel at all? What about a userspace detector which parses > kernel logs (or new procfs counters) and makes such decisions? +1. I agree that a userspace detector seems more appropriate for this. We already have the hung_task_detect_count counter, so a userspace detector could easily use that to implement custom policies ;)
> > I assume the same argument applies to the NMI watchdog, to the > > softlockup detector and to the RCU stall detector? > > > > A general framework to handle all of these might be better. But why > > do it in kernel at all? What about a userspace detector which parses > > kernel logs (or new procfs counters) and makes such decisions? > > +1. I agree that a userspace detector seems more appropriate for this. > I think the user-space maybe flexibility, but incurs relatively higher overhead and is less reliable. When the system hangs, this task may have already hung as well. > We already have the hung_task_detect_count counter, so a userspace detector > could easily use that to implement custom policies ;)
On 2025/9/23 13:22, Li,Rongqing wrote: >>> I assume the same argument applies to the NMI watchdog, to the >>> softlockup detector and to the RCU stall detector? >>> >>> A general framework to handle all of these might be better. But why >>> do it in kernel at all? What about a userspace detector which parses >>> kernel logs (or new procfs counters) and makes such decisions? >> >> +1. I agree that a userspace detector seems more appropriate for this. >> > > I think the user-space maybe flexibility, but incurs relatively higher overhead and is less reliable. When the system hangs, this task may have already hung as well. Emmm... if the system is so degraded that a userspace monitor cannot get scheduled, it's very likely that khungtaskd itself is also struggling to run, right? > > >> We already have the hung_task_detect_count counter, so a userspace detector >> could easily use that to implement custom policies ;)
> >> +1. I agree that a userspace detector seems more appropriate for this. > >> > > > > I think the user-space maybe flexibility, but incurs relatively higher overhead > and is less reliable. When the system hangs, this task may have already hung as > well. > > Emmm... if the system is so degraded that a userspace monitor cannot get > scheduled, it's very likely that khungtaskd itself is also struggling to run, right? > Yes -Li > > > > > >> We already have the hung_task_detect_count counter, so a userspace > >> detector could easily use that to implement custom policies ;)
© 2016 - 2025 Red Hat, Inc.