From nobody Fri Apr 17 23:07:33 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A710C1D130E; Thu, 12 Feb 2026 21:12:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770930733; cv=none; b=C65HSrNwkNTF2aN1AA5zfS2EwOsc7yYuYF2ch5YvJCWnTTYFlZCSOacA/wGHLACnN3fp8j/wgqcbUkm6nWK0lbFu87t06sWX+omWYto0UJoT4yNnpviOLY9bIEn3H0BM+HNy09FnVKGucyirlP2bELELuhwr++3h4rh7dAQF+Xg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770930733; c=relaxed/simple; bh=+s+4P6HGCAeRYpPcFFirWuyM7SRq/7NhLTjERkoTthM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=tnKk94fvSiy1kFSp/GuS0zrYg4yFlygCRROVjk3C8AkVh8N7R1s3MAlHD/lU0fHa1aS87IO6oMsTLTnT8azUF4UXiFKiQcC2HTqEiQsCRuomeiarvxSPWDzm3A4LMI0OSsXvULAyGELnSifBU5JbfLZRl4pPB/ENqlwAHUt0KoI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ocqfJnoa; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ocqfJnoa" Received: by smtp.kernel.org (Postfix) with ESMTPS id 6538DC16AAE; Thu, 12 Feb 2026 21:12:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1770930733; bh=+s+4P6HGCAeRYpPcFFirWuyM7SRq/7NhLTjERkoTthM=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=ocqfJnoa2CNR5K9okIe4dKxZOfbQ+N+Lv/phLhZqEIEuaN4Dz3GY0yCkQzeNliQD8 3Kkz1M2DKRmB39Bj7iKAmJjz9dVhn62dSccYMv03O9RXCQyYoaznO5AAu4wiNujsn+ hiem/an1AB7f1egMD8w+2wuP5rmlj03VspRXN+IG8fT5SvfLd6lMIxkx8ORdbfxwfd Fxc1orJOgWw/5V/TaIBBmPbLKdYMK21Jta2M/8spXBsoB2U+qIrzIskypznKPQyiCJ t+8Wzh8+xJbQ2DyGNJh9Nc2UX+Vwd4gfuykorl+r1gS7nMbWKTqb2MaYitUAGdGg2v 6r+9QsnautiQg== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 586A4EEA850; Thu, 12 Feb 2026 21:12:13 +0000 (UTC) From: Mayank Rungta via B4 Relay Date: Thu, 12 Feb 2026 14:12:10 -0700 Subject: [PATCH 1/4] watchdog/hardlockup: Always update saved interrupts during check Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260212-hardlockup-watchdog-fixes-v1-1-745f1dce04c3@google.com> References: <20260212-hardlockup-watchdog-fixes-v1-0-745f1dce04c3@google.com> In-Reply-To: <20260212-hardlockup-watchdog-fixes-v1-0-745f1dce04c3@google.com> To: Jonathan Corbet , Petr Mladek , Jinchao Wang , Yunhui Cui , Stephane Eranian , Ian Rogers , Li Huafei , Feng Tang , Max Kellermann , Jonathan Corbet , Douglas Anderson , Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Mayank Rungta X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1770930732; l=2116; i=mrungta@google.com; s=20260212; h=from:subject:message-id; bh=jeXyM9Zz7zH/TkvXATjr1ijroPIgJAbT1ZySlVxvX+0=; b=EmrlxcFLuYgP97uSm+OPQWshCcXqiYiX/U9ySJP+u6v+S8ri2MGxrjnlbzHiWziBLLVQq67/z PyUOFRmfLP/Bu/cARWpsnG6+ROx7Q8wk3Ov4Zum4cfI4UiAHhwZ5Hts X-Developer-Key: i=mrungta@google.com; a=ed25519; pk=2Bjwbv/ibL10QnyvK9G7DoKpffXy7z6+M4NawEYgYDI= X-Endpoint-Received: by B4 Relay for mrungta@google.com/20260212 with auth_id=634 X-Original-From: Mayank Rungta Reply-To: mrungta@google.com From: Mayank Rungta Currently, arch_touch_nmi_watchdog() causes an early return that skips updating hrtimer_interrupts_saved. This leads to stale comparisons and delayed lockup detection. Update the saved interrupt count before checking the touched flag to ensure detection timeliness. Signed-off-by: Mayank Rungta Reviewed-by: Douglas Anderson --- kernel/watchdog.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/kernel/watchdog.c b/kernel/watchdog.c index 7d675781bc917d709aa3fb499629eeac86934f55..b71aa814edcf9ad8f73644eb5bc= d1eeb3264e4ed 100644 --- a/kernel/watchdog.c +++ b/kernel/watchdog.c @@ -186,7 +186,21 @@ static void watchdog_hardlockup_kick(void) =20 void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs) { + bool is_hl; int hardlockup_all_cpu_backtrace; + /* + * Check for a hardlockup by making sure the CPU's timer + * interrupt is incrementing. The timer interrupt should have + * fired multiple times before we overflow'd. If it hasn't + * then this is a good indication the cpu is stuck + * + * Purposely check this _before_ checking watchdog_hardlockup_touched + * so we make sure we still update the saved value of the interrupts. + * Without that we'll take an extra round through this function before + * we can detect a lockup. + */ + + is_hl =3D is_hardlockup(cpu); =20 if (per_cpu(watchdog_hardlockup_touched, cpu)) { per_cpu(watchdog_hardlockup_touched, cpu) =3D false; @@ -195,13 +209,8 @@ void watchdog_hardlockup_check(unsigned int cpu, struc= t pt_regs *regs) =20 hardlockup_all_cpu_backtrace =3D (hardlockup_si_mask & SYS_INFO_ALL_BT) ? 1 : sysctl_hardlockup_all_cpu_backtrace; - /* - * Check for a hardlockup by making sure the CPU's timer - * interrupt is incrementing. The timer interrupt should have - * fired multiple times before we overflow'd. If it hasn't - * then this is a good indication the cpu is stuck - */ - if (is_hardlockup(cpu)) { + + if (is_hl) { unsigned int this_cpu =3D smp_processor_id(); unsigned long flags; =20 --=20 2.53.0.273.g2a3d683680-goog From nobody Fri Apr 17 23:07:33 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A71601DF27D; Thu, 12 Feb 2026 21:12:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770930733; cv=none; b=TCrKAxKEgAMhnk2Eq97HT/t7UonZqr0aD9qDm8qWDbrp/CI6ppIUxtrMZPITnMpQkocWATyGnKBQ6ZCZnw2gs5R5T+xmzyTj7M9IBYlhdj2/vuVTuvmLu6WSnJZx+JWEPClKPOR6nz2x1SH7eoDm7JlzuMHbn66wbwEFub6hgvc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770930733; c=relaxed/simple; bh=pBYkUxk5sriANA7AD+AFfUWPlwyQ7z+ZcPf2VHhn1KY=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=hZbb4PGmYX8LRyd2t6v+b10ralchrY4oLB92nSk36SuAN8jSvq6cqNII7NectDccAaHpN6JP6eus829txbMROvsW6cr5xBiy+Nsz/QMfFR06UidmiWhZ2q4r1sRfMDZA32xGX2TMvjUTz7gASPpmcbMhnjF6YeL2p1193sNqbIY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oplOu05P; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oplOu05P" Received: by smtp.kernel.org (Postfix) with ESMTPS id 84612C19421; Thu, 12 Feb 2026 21:12:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1770930733; bh=pBYkUxk5sriANA7AD+AFfUWPlwyQ7z+ZcPf2VHhn1KY=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=oplOu05P6bmfFlxXQSG/nQf3wyC+PGZzQ3QivleEsSP9YZVi+8P1UElWmGoLsrLiG abthGSTlM9+8eNCFl//u2dcklwfCX3v76ezhLgNNzxC0wRSl64mbhD9eCuzKVshGXO g7qmB44D2PS6mDBGX7LFSC3Avth8/Ds0ysjTad3OdMAi3tKoYRJCM2IBdXBkAzkueN 4S78cYdDjiqA0Ai2LHhFw0HpfJbHNUaZMtSVn229Aagylw+b0d6fxXlsl91AEJZ8ui AdoGXlm6a9PBbZIx0OWvgEJb6sTuzo2wbPGleEFLMoNXSeR/KlXba0RXb2mtsjunnE KEZ+bPJJgOL0g== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71317EEA852; Thu, 12 Feb 2026 21:12:13 +0000 (UTC) From: Mayank Rungta via B4 Relay Date: Thu, 12 Feb 2026 14:12:11 -0700 Subject: [PATCH 2/4] doc: watchdog: Clarify hardlockup detection timing Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260212-hardlockup-watchdog-fixes-v1-2-745f1dce04c3@google.com> References: <20260212-hardlockup-watchdog-fixes-v1-0-745f1dce04c3@google.com> In-Reply-To: <20260212-hardlockup-watchdog-fixes-v1-0-745f1dce04c3@google.com> To: Jonathan Corbet , Petr Mladek , Jinchao Wang , Yunhui Cui , Stephane Eranian , Ian Rogers , Li Huafei , Feng Tang , Max Kellermann , Jonathan Corbet , Douglas Anderson , Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Mayank Rungta X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1770930732; l=3659; i=mrungta@google.com; s=20260212; h=from:subject:message-id; bh=++s1kaAYhwZaNT4DoKQnleAcHjzb5H5RPunWGGQZXyE=; b=P6tB2IvFkwwyq1h/kVwBVF6voCYYnE72u6l9PJLok7d0W7hMtpFTRn4NoE8HwXu3pELeoM9dp d6IAMGMwF0hDzbv+6dWmc8+J539YyagHreV1SAv7S6/rgKL7e5zdbJF X-Developer-Key: i=mrungta@google.com; a=ed25519; pk=2Bjwbv/ibL10QnyvK9G7DoKpffXy7z6+M4NawEYgYDI= X-Endpoint-Received: by B4 Relay for mrungta@google.com/20260212 with auth_id=634 X-Original-From: Mayank Rungta Reply-To: mrungta@google.com From: Mayank Rungta The current documentation implies that a hardlockup is strictly defined as looping for "more than 10 seconds." However, the detection mechanism is periodic (based on `watchdog_thresh`), meaning detection time varies significantly depending on when the lockup occurs relative to the NMI perf event. Update the definition to remove the strict "more than 10 seconds" constraint in the introduction and defer details to the Implementation section. Additionally, add a "Detection Overhead" section illustrating the Best Case (~6s) and Worst Case (~20s) detection scenarios to provide administrators with a clearer understanding of the watchdog's latency. Signed-off-by: Mayank Rungta Reviewed-by: Douglas Anderson Reviewed-by: Petr Mladek --- Documentation/admin-guide/lockup-watchdogs.rst | 41 ++++++++++++++++++++++= +++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/Documentation/admin-guide/lockup-watchdogs.rst b/Documentation= /admin-guide/lockup-watchdogs.rst index 3e09284a8b9bef75c0ac1607a1809ac3b8a4c1ea..1b374053771f676d874716b3210= cade55ae89b28 100644 --- a/Documentation/admin-guide/lockup-watchdogs.rst +++ b/Documentation/admin-guide/lockup-watchdogs.rst @@ -16,7 +16,7 @@ details), and a compile option, "BOOTPARAM_SOFTLOCKUP_PAN= IC", are provided for this. =20 A 'hardlockup' is defined as a bug that causes the CPU to loop in -kernel mode for more than 10 seconds (see "Implementation" below for +kernel mode for several seconds (see "Implementation" below for details), without letting other interrupts have a chance to run. Similarly to the softlockup case, the current stack trace is displayed upon detection and the system will stay locked up unless the default @@ -64,6 +64,45 @@ administrators to configure the period of the hrtimer an= d the perf event. The right value for a particular environment is a trade-off between fast response to lockups and detection overhead. =20 +Detection Overhead +------------------ + +The hardlockup detector checks for lockups using a periodic NMI perf +event. This means the time to detect a lockup can vary depending on +when the lockup occurs relative to the NMI check window. + +**Best Case:** +In the best case scenario, the lockup occurs just before the first +heartbeat is due. The detector will notice the missing hrtimer +interrupt almost immediately during the next check. + +:: + + Time 100.0: cpu 1 heartbeat + Time 100.1: hardlockup_check, cpu1 stores its state + Time 103.9: Hard Lockup on cpu1 + Time 104.0: cpu 1 heartbeat never comes + Time 110.1: hardlockup_check, cpu1 checks the state again, should be the= same, declares lockup + + Time to detection: ~6 seconds + +**Worst Case:** +In the worst case scenario, the lockup occurs shortly after a valid +interrupt (heartbeat) which itself happened just after the NMI check. +The next NMI check sees that the interrupt count has changed (due to +that one heartbeat), assumes the CPU is healthy, and resets the +baseline. The lockup is only detected at the subsequent check. + +:: + + Time 100.0: hardlockup_check, cpu1 stores its state + Time 100.1: cpu 1 heartbeat + Time 100.2: Hard Lockup on cpu1 + Time 110.0: hardlockup_check, cpu1 stores its state (misses lockup as st= ate changed) + Time 120.0: hardlockup_check, cpu1 checks the state again, should be the= same, declares lockup + + Time to detection: ~20 seconds + By default, the watchdog runs on all online cores. However, on a kernel configured with NO_HZ_FULL, by default the watchdog runs only on the housekeeping cores, not the cores specified in the "nohz_full" --=20 2.53.0.273.g2a3d683680-goog From nobody Fri Apr 17 23:07:33 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B2DEB2C234A; Thu, 12 Feb 2026 21:12:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770930733; cv=none; b=EyaXODf0OrMI4w+n++e5ZzniPYg2smUhj8mW/ZPBiBeSMM/Ficzn7OAULgs/WvXUF2x6jnNRl3bWAyF3hIEFSaUT5K70NB8kxMGu7yLRclUJKfJkrkPqk+9LYi5Uhxwog2nM32+XlexvvI9VZr7C1lJR/qK8E6xa8KifXJdZ5gE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770930733; c=relaxed/simple; bh=aa62AWDGWIXB2c5ffhzHBqIyvtOS5YeFj6iLLNWaYio=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=j6b8veaXVSLLf3b1J40GUdA/imUYIwrUMhOL38dgyp+FLMCvC/cPxIycNn33n/cHW29iuepeVVIjXZpBAhi3tUU60bG/2H9mmg0jizp+ufIgaalJo6gQRGNQpkrvN+VQZkTrsJz0d3zuvzXiLTVy1Qjt4zbbVLwKcfXImVKDSEk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=rcPNyqMF; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="rcPNyqMF" Received: by smtp.kernel.org (Postfix) with ESMTPS id 962D8C2BC86; Thu, 12 Feb 2026 21:12:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1770930733; bh=aa62AWDGWIXB2c5ffhzHBqIyvtOS5YeFj6iLLNWaYio=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=rcPNyqMFvLFEH66nRjlvOUtS9E0c35ky1a/aVfLnq+ygjhobvt/GXTR6xS+hBL/v0 /oNyjwCuGrxNmia8bW6l7L8Txj3zm2qrtpKxkyGXa9II0oyEdE197qvDDNgA+NM2/a 5WdcQrIeU3KrzONyp3/fLOqjzDBZHhMry9TInC+TLkfSrnFjp/DMF0L89QnNW1v6NK gBu5yInQj+F50zRsarnB+JKjaqFUmekdOFGANRN9023XSL+o1uatH/xqkB0I5SALjj yjACr3Df0V2s/ob52I2Wm8HowHEZVwR1HEH1LWP9RnMo/L8kIIT8brcKS1DxnIkAW6 gwpit+rptzwvw== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CD64EEA851; Thu, 12 Feb 2026 21:12:13 +0000 (UTC) From: Mayank Rungta via B4 Relay Date: Thu, 12 Feb 2026 14:12:12 -0700 Subject: [PATCH 3/4] watchdog/hardlockup: improve buddy system detection timeliness Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260212-hardlockup-watchdog-fixes-v1-3-745f1dce04c3@google.com> References: <20260212-hardlockup-watchdog-fixes-v1-0-745f1dce04c3@google.com> In-Reply-To: <20260212-hardlockup-watchdog-fixes-v1-0-745f1dce04c3@google.com> To: Jonathan Corbet , Petr Mladek , Jinchao Wang , Yunhui Cui , Stephane Eranian , Ian Rogers , Li Huafei , Feng Tang , Max Kellermann , Jonathan Corbet , Douglas Anderson , Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Mayank Rungta X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1770930732; l=4597; i=mrungta@google.com; s=20260212; h=from:subject:message-id; bh=u/DAjpi2f2lGgrbnQjjFY9XiKDSfKSESiWB7qXT8xhI=; b=KpTEJDqImE50mOTCS0QDfLkUhHr5Bd19Yb12yj8rRfQsM0Swv2jtD6wS3WgsKfy+xBK/RaEeb PSgEkKo+5YjCh9Kh6giYxa+0RxYt8XDUuAQdC5vO/7sPyIS5SjZTKoO X-Developer-Key: i=mrungta@google.com; a=ed25519; pk=2Bjwbv/ibL10QnyvK9G7DoKpffXy7z6+M4NawEYgYDI= X-Endpoint-Received: by B4 Relay for mrungta@google.com/20260212 with auth_id=634 X-Original-From: Mayank Rungta Reply-To: mrungta@google.com From: Mayank Rungta Currently, the buddy system only performs checks every 3rd sample. With a 4-second interval. If a check window is missed, the next check occurs 12 seconds later, potentially delaying hard lockup detection for up to 24 seconds. Modify the buddy system to perform checks at every interval (4s). Introduce a missed-interrupt threshold to maintain the existing grace period while reducing the detection window to 8-12 seconds. Best and worst case detection scenarios: Before (12s check window): - Best case: Lockup occurs after first check but just before heartbeat interval. Detected in ~8s (8s till next check). - Worst case: Lockup occurs just after a check. Detected in ~24s (missed check + 12s till next check + 12s logic). After (4s check window with threshold of 3): - Best case: Lockup occurs just before a check. Detected in ~8s (0s till 1st check + 4s till 2nd + 4s till 3rd). - Worst case: Lockup occurs just after a check. Detected in ~12s (4s till 1st check + 4s till 2nd + 4s till 3rd). Signed-off-by: Mayank Rungta Reviewed-by: Douglas Anderson --- include/linux/nmi.h | 1 + kernel/watchdog.c | 18 ++++++++++++++++-- kernel/watchdog_buddy.c | 9 +-------- 3 files changed, 18 insertions(+), 10 deletions(-) diff --git a/include/linux/nmi.h b/include/linux/nmi.h index 207156f2143c5f43e89e81cbf0215331eae9bd49..bc1162895f3558bff178dd6c2c8= 39344162f8adc 100644 --- a/include/linux/nmi.h +++ b/include/linux/nmi.h @@ -21,6 +21,7 @@ void lockup_detector_soft_poweroff(void); extern int watchdog_user_enabled; extern int watchdog_thresh; extern unsigned long watchdog_enabled; +extern int watchdog_hardlockup_miss_thresh; =20 extern struct cpumask watchdog_cpumask; extern unsigned long *watchdog_cpumask_bits; diff --git a/kernel/watchdog.c b/kernel/watchdog.c index b71aa814edcf9ad8f73644eb5bcd1eeb3264e4ed..30199eaeb5d7e0fd229657a31ff= ff4463c97332c 100644 --- a/kernel/watchdog.c +++ b/kernel/watchdog.c @@ -60,6 +60,13 @@ unsigned long *watchdog_cpumask_bits =3D cpumask_bits(&w= atchdog_cpumask); int __read_mostly sysctl_hardlockup_all_cpu_backtrace; # endif /* CONFIG_SMP */ =20 +/* + * Number of consecutive missed interrupts before declaring a lockup. + * Default to 1 (immediate) for NMI/Perf. Buddy will overwrite this to 3. + */ +int __read_mostly watchdog_hardlockup_miss_thresh =3D 1; +EXPORT_SYMBOL_GPL(watchdog_hardlockup_miss_thresh); + /* * Should we panic when a soft-lockup or hard-lockup occurs: */ @@ -137,6 +144,7 @@ __setup("nmi_watchdog=3D", hardlockup_panic_setup); =20 static DEFINE_PER_CPU(atomic_t, hrtimer_interrupts); static DEFINE_PER_CPU(int, hrtimer_interrupts_saved); +static DEFINE_PER_CPU(int, hrtimer_interrupts_missed); static DEFINE_PER_CPU(bool, watchdog_hardlockup_warned); static DEFINE_PER_CPU(bool, watchdog_hardlockup_touched); static unsigned long hard_lockup_nmi_warn; @@ -163,8 +171,13 @@ static bool is_hardlockup(unsigned int cpu) { int hrint =3D atomic_read(&per_cpu(hrtimer_interrupts, cpu)); =20 - if (per_cpu(hrtimer_interrupts_saved, cpu) =3D=3D hrint) - return true; + if (per_cpu(hrtimer_interrupts_saved, cpu) =3D=3D hrint) { + per_cpu(hrtimer_interrupts_missed, cpu)++; + if (per_cpu(hrtimer_interrupts_missed, cpu) >=3D watchdog_hardlockup_mis= s_thresh) + return true; + + return false; + } =20 /* * NOTE: we don't need any fancy atomic_t or READ_ONCE/WRITE_ONCE @@ -172,6 +185,7 @@ static bool is_hardlockup(unsigned int cpu) * written/read by a single CPU. */ per_cpu(hrtimer_interrupts_saved, cpu) =3D hrint; + per_cpu(hrtimer_interrupts_missed, cpu) =3D 0; =20 return false; } diff --git a/kernel/watchdog_buddy.c b/kernel/watchdog_buddy.c index ee754d767c2131e3cd34bccf26d8e6cf0e0b5f75..3a1e57080c1c6a645c974b3b6ee= bec87df9e69e9 100644 --- a/kernel/watchdog_buddy.c +++ b/kernel/watchdog_buddy.c @@ -21,6 +21,7 @@ static unsigned int watchdog_next_cpu(unsigned int cpu) =20 int __init watchdog_hardlockup_probe(void) { + watchdog_hardlockup_miss_thresh =3D 3; return 0; } =20 @@ -86,14 +87,6 @@ void watchdog_buddy_check_hardlockup(int hrtimer_interru= pts) { unsigned int next_cpu; =20 - /* - * Test for hardlockups every 3 samples. The sample period is - * watchdog_thresh * 2 / 5, so 3 samples gets us back to slightly over - * watchdog_thresh (over by 20%). - */ - if (hrtimer_interrupts % 3 !=3D 0) - return; - /* check for a hardlockup on the next CPU */ next_cpu =3D watchdog_next_cpu(smp_processor_id()); if (next_cpu >=3D nr_cpu_ids) --=20 2.53.0.273.g2a3d683680-goog From nobody Fri Apr 17 23:07:33 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF4C631A044; Thu, 12 Feb 2026 21:12:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770930733; cv=none; b=FzGiWOzFmghPU/WlQpGgcab/vkZRrE9fen8M/ZLAD0/zeQsAwKXhb3KvYwFFVllzG2KQLS7OBAtoaIrrBYN1Pux5mHM8jktmQpIZWJUL9rSkPUkqrt9naEUQn67wcWqaWb1zel6zsKLXAPNlqciEvt2mtThBfI2nPityGW/FzTY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770930733; c=relaxed/simple; bh=OzjdI4PRkfQOu56QOGogikmXev7K2jhS7+htkQFK1rA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=R6MpCJdNAYEBdIk5vn6SPPDZ3jV9kpqj9D2e7Ri8YRXpaid7L9zFqSpzOjuN8pH+9svb2/dUxE6wsNG3X4G1QCIekqO8Kgh5IkMM3SEfI3T44ECSu0WWGNhQiNsScmceoqRPnmD/nkCV0MPal96u6z46P+n++ixV7CBuO7DA6rA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BlwG8gEb; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BlwG8gEb" Received: by smtp.kernel.org (Postfix) with ESMTPS id AF6D7C19424; Thu, 12 Feb 2026 21:12:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1770930733; bh=OzjdI4PRkfQOu56QOGogikmXev7K2jhS7+htkQFK1rA=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=BlwG8gEblEXOBj13tNy6TYSiOI0MjZO4JSs+d4Jo0PN4+G9ml3o0H9pl9/c/NHA0z i5cC6dZZ1994vnPdy5wGQwIrGalPYRZ5HcfbJ0Jk7I/ogwJA2gDRBPhw3MvRl4SN7U dBqZNIqgkEtxfO6pD70/e7QwWoINf8KzY7JfA6ktkmOiRRU/pKG/3UQ6U7CfZ8BCyh U4CSpKY2XVr7zNE7YJMFXmmmX2xlD2i2K9gc/Ktv1ex4Dqn91eQZ3NDGUb5NQZZWzC zTmOjxe05oUCvS3QYhi/agtUrQoF41LOp6wv9KHIwkg12dOTJvvvx6UuKNrAARIwJW ExDXwYEe4EL0g== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3C68EEA850; Thu, 12 Feb 2026 21:12:13 +0000 (UTC) From: Mayank Rungta via B4 Relay Date: Thu, 12 Feb 2026 14:12:13 -0700 Subject: [PATCH 4/4] doc: watchdog: Document buddy detector Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260212-hardlockup-watchdog-fixes-v1-4-745f1dce04c3@google.com> References: <20260212-hardlockup-watchdog-fixes-v1-0-745f1dce04c3@google.com> In-Reply-To: <20260212-hardlockup-watchdog-fixes-v1-0-745f1dce04c3@google.com> To: Jonathan Corbet , Petr Mladek , Jinchao Wang , Yunhui Cui , Stephane Eranian , Ian Rogers , Li Huafei , Feng Tang , Max Kellermann , Jonathan Corbet , Douglas Anderson , Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Mayank Rungta X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1770930732; l=9260; i=mrungta@google.com; s=20260212; h=from:subject:message-id; bh=zQJ5AME8IRJnyw3jfYBu6R/B2xO/DNdkDnl61rje4aw=; b=toFHZ4xPh9Rwtyl9nRS1I8kdpONsLEyLmZky1cWcVD5z9MmcoaspJHHjJPhCzdCKOs0J88MOO pkF7YBQHDqKDBfWl3iNyx8/WKdqAFz4MOpiHXhNbgdzT3dOFpzroXsK X-Developer-Key: i=mrungta@google.com; a=ed25519; pk=2Bjwbv/ibL10QnyvK9G7DoKpffXy7z6+M4NawEYgYDI= X-Endpoint-Received: by B4 Relay for mrungta@google.com/20260212 with auth_id=634 X-Original-From: Mayank Rungta Reply-To: mrungta@google.com From: Mayank Rungta The current documentation generalizes the hardlockup detector as primarily NMI-perf-based and lacks details on the SMP "Buddy" detector. Update the documentation to add a detailed description of the Buddy detector, and also restructure the "Implementation" section to explicitly separate "Softlockup Detector", "Hardlockup Detector (NMI/Perf)", and "Hardlockup Detector (Buddy)". Clarify that the softlockup hrtimer acts as the heartbeat generator for both hardlockup mechanisms and centralize the configuration details in a "Frequency and Heartbeats" section. Signed-off-by: Mayank Rungta Reviewed-by: Douglas Anderson --- Documentation/admin-guide/lockup-watchdogs.rst | 149 +++++++++++++++++----= ---- 1 file changed, 101 insertions(+), 48 deletions(-) diff --git a/Documentation/admin-guide/lockup-watchdogs.rst b/Documentation= /admin-guide/lockup-watchdogs.rst index 1b374053771f676d874716b3210cade55ae89b28..7ae7ce3abd2c838ff29c70f7a32= ffaf58531e150 100644 --- a/Documentation/admin-guide/lockup-watchdogs.rst +++ b/Documentation/admin-guide/lockup-watchdogs.rst @@ -30,22 +30,23 @@ timeout is set through the confusingly named "kernel.pa= nic" sysctl), to cause the system to reboot automatically after a specified amount of time. =20 +Configuration +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +A kernel knob is provided that allows administrators to configure +this period. The "watchdog_thresh" parameter (default 10 seconds) +controls the threshold. The right value for a particular environment +is a trade-off between fast response to lockups and detection overhead. + Implementation =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 -The soft and hard lockup detectors are built on top of the hrtimer and -perf subsystems, respectively. A direct consequence of this is that, -in principle, they should work in any architecture where these -subsystems are present. +The soft lockup detector is built on top of the hrtimer subsystem. +The hard lockup detector is built on top of the perf subsystem +(on architectures that support it) or uses an SMP "buddy" system. =20 -A periodic hrtimer runs to generate interrupts and kick the watchdog -job. An NMI perf event is generated every "watchdog_thresh" -(compile-time initialized to 10 and configurable through sysctl of the -same name) seconds to check for hardlockups. If any CPU in the system -does not receive any hrtimer interrupt during that time the -'hardlockup detector' (the handler for the NMI perf event) will -generate a kernel warning or call panic, depending on the -configuration. +Softlockup Detector +------------------- =20 The watchdog job runs in a stop scheduling thread that updates a timestamp every time it is scheduled. If that timestamp is not updated @@ -55,53 +56,105 @@ will dump useful debug information to the system log, = after which it will call panic if it was instructed to do so or resume execution of other kernel code. =20 -The period of the hrtimer is 2*watchdog_thresh/5, which means it has -two or three chances to generate an interrupt before the hardlockup -detector kicks in. +Frequency and Heartbeats +------------------------ + +The hrtimer used by the softlockup detector serves a dual purpose: +it detects softlockups, and it also generates the interrupts +(heartbeats) that the hardlockup detectors use to verify CPU liveness. + +The period of this hrtimer is 2*watchdog_thresh/5. This means the +hrtimer has two or three chances to generate an interrupt before the +NMI hardlockup detector kicks in. + +Hardlockup Detector (NMI/Perf) +------------------------------ + +On architectures that support NMI (Non-Maskable Interrupt) perf events, +a periodic NMI is generated every "watchdog_thresh" seconds. + +If any CPU in the system does not receive any hrtimer interrupt +(heartbeat) during the "watchdog_thresh" window, the 'hardlockup +detector' (the handler for the NMI perf event) will generate a kernel +warning or call panic. + +**Detection Overhead (NMI):** + +The time to detect a lockup can vary depending on when the lockup +occurs relative to the NMI check window. Examples below assume a watchdog_= thresh of 10. + +* **Best Case:** The lockup occurs just before the first heartbeat is + due. The detector will notice the missing hrtimer interrupt almost + immediately during the next check. + + :: + + Time 100.0: cpu 1 heartbeat + Time 100.1: hardlockup_check, cpu1 stores its state + Time 103.9: Hard Lockup on cpu1 + Time 104.0: cpu 1 heartbeat never comes + Time 110.1: hardlockup_check, cpu1 checks the state again, should be t= he same, declares lockup + + Time to detection: ~6 seconds + +* **Worst Case:** The lockup occurs shortly after a valid interrupt + (heartbeat) which itself happened just after the NMI check. The next + NMI check sees that the interrupt count has changed (due to that one + heartbeat), assumes the CPU is healthy, and resets the baseline. The + lockup is only detected at the subsequent check. + + :: + + Time 100.0: hardlockup_check, cpu1 stores its state + Time 100.1: cpu 1 heartbeat + Time 100.2: Hard Lockup on cpu1 + Time 110.0: hardlockup_check, cpu1 stores its state (misses lockup as = state changed) + Time 120.0: hardlockup_check, cpu1 checks the state again, should be t= he same, declares lockup =20 -As explained above, a kernel knob is provided that allows -administrators to configure the period of the hrtimer and the perf -event. The right value for a particular environment is a trade-off -between fast response to lockups and detection overhead. + Time to detection: ~20 seconds =20 -Detection Overhead ------------------- +Hardlockup Detector (Buddy) +--------------------------- =20 -The hardlockup detector checks for lockups using a periodic NMI perf -event. This means the time to detect a lockup can vary depending on -when the lockup occurs relative to the NMI check window. +On architectures or configurations where NMI perf events are not +available (or disabled), the kernel may use the "buddy" hardlockup +detector. This mechanism requires SMP (Symmetric Multi-Processing). =20 -**Best Case:** -In the best case scenario, the lockup occurs just before the first -heartbeat is due. The detector will notice the missing hrtimer -interrupt almost immediately during the next check. +In this mode, each CPU is assigned a "buddy" CPU to monitor. The +monitoring CPU runs its own hrtimer (the same one used for softlockup +detection) and checks if the buddy CPU's hrtimer interrupt count has +increased. =20 -:: +To ensure timeliness and avoid false positives, the buddy system performs +checks at every hrtimer interval (2*watchdog_thresh/5, which is 4 seconds +by default). It uses a missed-interrupt threshold of 3. If the buddy's +interrupt count has not changed for 3 consecutive checks, it is assumed +that the buddy CPU is hardlocked (interrupts disabled). The monitoring +CPU will then trigger the hardlockup response (warning or panic). =20 - Time 100.0: cpu 1 heartbeat - Time 100.1: hardlockup_check, cpu1 stores its state - Time 103.9: Hard Lockup on cpu1 - Time 104.0: cpu 1 heartbeat never comes - Time 110.1: hardlockup_check, cpu1 checks the state again, should be the= same, declares lockup +**Detection Overhead (Buddy):** =20 - Time to detection: ~6 seconds +With a default check interval of 4 seconds (watchdog_thresh =3D 10): =20 -**Worst Case:** -In the worst case scenario, the lockup occurs shortly after a valid -interrupt (heartbeat) which itself happened just after the NMI check. -The next NMI check sees that the interrupt count has changed (due to -that one heartbeat), assumes the CPU is healthy, and resets the -baseline. The lockup is only detected at the subsequent check. +* **Best case:** Lockup occurs just before a check. + Detected in ~8s (0s till 1st check + 4s till 2nd + 4s till 3rd). +* **Worst case:** Lockup occurs just after a check. + Detected in ~12s (4s till 1st check + 4s till 2nd + 4s till 3rd). =20 -:: +**Limitations of the Buddy Detector:** =20 - Time 100.0: hardlockup_check, cpu1 stores its state - Time 100.1: cpu 1 heartbeat - Time 100.2: Hard Lockup on cpu1 - Time 110.0: hardlockup_check, cpu1 stores its state (misses lockup as st= ate changed) - Time 120.0: hardlockup_check, cpu1 checks the state again, should be the= same, declares lockup +1. **All-CPU Lockup:** If all CPUs lock up simultaneously, the buddy + detector cannot detect the condition because the monitoring CPUs + are also frozen. +2. **Stack Traces:** Unlike the NMI detector, the buddy detector + cannot directly interrupt the locked CPU to grab a stack trace. + It relies on architecture-specific mechanisms (like NMI backtrace + support) to try and retrieve the status of the locked CPU. If + such support is missing, the log may only show that a lockup + occurred without providing the locked CPU's stack. =20 - Time to detection: ~20 seconds +Watchdog Core Exclusion +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 By default, the watchdog runs on all online cores. However, on a kernel configured with NO_HZ_FULL, by default the watchdog runs only --=20 2.53.0.273.g2a3d683680-goog