From nobody Fri Dec 19 14:04:46 2025 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 18E7C495CE for ; Wed, 20 Dec 2023 21:17:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="MgJD/Z3U" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-1d3dee5f534so9098485ad.1 for ; Wed, 20 Dec 2023 13:17:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1703107026; x=1703711826; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=t1CDklyqCRANoyYKv7Digk6bbgZ534wMCluTfdczIuw=; b=MgJD/Z3UnTeItdFfsGqEyxxSFlGTUtQ6xL2moU4MO/TcdgyUQRXTU+jRyvUDLTaMkP sOcjH3DXZ+mc3XTVAK567F5grC4UZufmQNXphmdjfI3OqkoYLys0AnYUC+HvJpQkEbdd 0wO1G+a1lsQIyhsQnDZCVCXQ7XA3zARvR71Fc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703107026; x=1703711826; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t1CDklyqCRANoyYKv7Digk6bbgZ534wMCluTfdczIuw=; b=UZ4lZxiR4d4OieR1I+jLOukEPSOaaWiYYXuWhvnUG6rayLBweyWbAXwS+oYpdlZv4p iXy7Ux0ndKNJj/zvN5kIzdXWP2f+xLnG4SW106CxETyJFKNlv3pUMwmslZavpH9IhnIT 540r+G9yNZKwvWm5fpEsCNAKWzrw1o8HeDwhG1CB+4abwDGjqqD7hHBzkGzQIlvpfO91 lkEikQxGPKupTIovF2Iow5sCIW3M9OQdmRsKr6tLsp7xT5Qlp9dYptpX5tkE2Ujqs2HE 2V+2yJCpnFn58gFLdEQIPn/DtDgPbGYvpmseNIhdV73b2Yfpctg4tLSJv89I1X4RQXGh w4sA== X-Gm-Message-State: AOJu0YypVMTZYn7HtEWMfECMoWqOwwedWsDNl+PygCwPGlWHxWK/FwiC 47y6V7WzmuqNnVhR76Aj1SiFeg== X-Google-Smtp-Source: AGHT+IG1+hNe6AZ8nrjh+nxE2uTo64+F0hmVK7+/1GxKBw9whzadq/uppDOX7bJc5sb6qxLBe9FLKA== X-Received: by 2002:a17:902:e543:b0:1d0:bf4c:166c with SMTP id n3-20020a170902e54300b001d0bf4c166cmr4634775plf.1.1703107026411; Wed, 20 Dec 2023 13:17:06 -0800 (PST) Received: from tictac2.mtv.corp.google.com ([2620:15c:9d:2:5a22:d46c:eec1:e5d4]) by smtp.gmail.com with ESMTPSA id u10-20020a170902b28a00b001d3dfebc05esm175023plr.21.2023.12.20.13.17.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Dec 2023 13:17:05 -0800 (PST) From: Douglas Anderson To: Andrew Morton Cc: Petr Mladek , Li Zhe , Pingfan Liu , John Ogness , Lecopzer Chen , Douglas Anderson , linux-kernel@vger.kernel.org Subject: [PATCH 1/4] watchdog/hardlockup: Adopt softlockup logic avoiding double-dumps Date: Wed, 20 Dec 2023 13:15:34 -0800 Message-ID: <20231220131534.1.I4f35a69fbb124b5f0c71f75c631e11fabbe188ff@changeid> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog In-Reply-To: <20231220211640.2023645-1-dianders@chromium.org> References: <20231220211640.2023645-1-dianders@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The hardlockup detector and softlockup detector both have the ability to dump the stack of all CPUs (`kernel.hardlockup_all_cpu_backtrace` and `kernel.softlockup_all_cpu_backtrace`). Both detectors also have some logic to attempt to avoid interleaving printouts if two CPUs were trying to do dumps of all CPUs at the same time. However: - The hardlockup detector's logic still allowed interleaving some information. Specifically another CPU could print modules and dump the stack of the locked CPU at the same time we were dumping all CPUs. - In the case where `kernel.hardlockup_panic` was set in addition to `kernel.hardlockup_all_cpu_backtrace`, when two CPUs both detected hardlockups at the same time the second CPU could call panic() while the first was still dumping stacks. This was especially bad if the locked up CPU wasn't responding to the request for a backtrace since the function nmi_trigger_cpumask_backtrace() can wait up to 10 seconds. Let's resolve this by adopting the softlockup logic in the hardlockup handler. NOTES: - As part of this, one might think that we should make a helper function that both the hard and softlockup detectors call. This turns out not to be super trivial since it would have to be parameterized quite a bit since there are separate global variables controlling each lockup detector and they print log messages that are just different enough that it would be a pain. We probably don't want to change the messages that are printed without good reason to avoid throwing log parsers for a loop. - One might also think that it would be a good idea to have the hardlockup and softlockup detector use the same global variable to prevent interleaving. This would make sure that softlockups and hardlockups can't interleave each other. That _almost_ works but has a dangerous flaw if `kernel.hardlockup_panic` is not the same as `kernel.softlockup_panic` because we might skip a call to panic() if one type of lockup was detected at the same time as another. Signed-off-by: Douglas Anderson --- kernel/watchdog.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/kernel/watchdog.c b/kernel/watchdog.c index bf30a6fac665..b4fd2f12137f 100644 --- a/kernel/watchdog.c +++ b/kernel/watchdog.c @@ -91,7 +91,7 @@ static DEFINE_PER_CPU(atomic_t, hrtimer_interrupts); static DEFINE_PER_CPU(int, hrtimer_interrupts_saved); static DEFINE_PER_CPU(bool, watchdog_hardlockup_warned); static DEFINE_PER_CPU(bool, watchdog_hardlockup_touched); -static unsigned long watchdog_hardlockup_all_cpu_dumped; +static unsigned long hard_lockup_nmi_warn; =20 notrace void arch_touch_nmi_watchdog(void) { @@ -156,6 +156,15 @@ void watchdog_hardlockup_check(unsigned int cpu, struc= t pt_regs *regs) if (per_cpu(watchdog_hardlockup_warned, cpu)) return; =20 + /* + * Prevent multiple hard-lockup reports if one cpu is already + * engaged in dumping all cpu back traces. + */ + if (sysctl_hardlockup_all_cpu_backtrace) { + if (test_and_set_bit_lock(0, &hard_lockup_nmi_warn)) + return; + } + pr_emerg("Watchdog detected hard LOCKUP on cpu %d\n", cpu); print_modules(); print_irqtrace_events(current); @@ -168,13 +177,10 @@ void watchdog_hardlockup_check(unsigned int cpu, stru= ct pt_regs *regs) trigger_single_cpu_backtrace(cpu); } =20 - /* - * Perform multi-CPU dump only once to avoid multiple - * hardlockups generating interleaving traces - */ - if (sysctl_hardlockup_all_cpu_backtrace && - !test_and_set_bit(0, &watchdog_hardlockup_all_cpu_dumped)) + if (sysctl_hardlockup_all_cpu_backtrace) { trigger_allbutcpu_cpu_backtrace(cpu); + clear_bit_unlock(0, &hard_lockup_nmi_warn); + } =20 if (hardlockup_panic) nmi_panic(regs, "Hard LOCKUP"); --=20 2.43.0.472.g3155946c3a-goog From nobody Fri Dec 19 14:04:46 2025 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A12C498AE for ; Wed, 20 Dec 2023 21:17:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="OyV7ZQac" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-1d3fc184b6dso1202545ad.2 for ; Wed, 20 Dec 2023 13:17:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1703107028; x=1703711828; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Pmb1hFIok+9eMKNNtq9Gk+/VDYxh0leTwTc9coy3jGk=; b=OyV7ZQacnPHW04yoE+2yXbC1IJY3TUmM2UOiubF3/YOK6NtOyys99/puF1N5+Lo6L8 FVgDqJSlH78X5mJS7H4taC0IFH35E68V3+fz6/C7SggokjrkGG7RCvyhsLLTHy4A6jYW rI5/fjae+yg7LRiPRSNzMWLyeLUNuvjgaozZQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703107028; x=1703711828; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Pmb1hFIok+9eMKNNtq9Gk+/VDYxh0leTwTc9coy3jGk=; b=p9b4DmEsshz35mLue9a3FE2dUPzMBKlLd6uAnoQ1WQOkVzMSgbtE2wgL2LOF9ajEdo xoSz8KDUdSBsxrBB0FDP7JV392cKs4Ii4mE+mNEVrbF4XUDcFqrHik6gjTd69iheYQIx VadGWkWAPmHW2OzGeeVfr4o4hYqFcQoDs+xa7n8L3z1s8DZlhvwCfhlqRy/F72OlV5al yHXgYfkr17WuIE8E0XSYt3XNVaEEmsNoXRn7ksJsJ7B4szF64ssxDmDxgirZ2WfWx+8L 4PNzXV5y5mgae+5j9h56m9PEQbi46JXp+wLeOyES5mj5zaeVDvjWz7BEWjOVLjULVTD5 wd5A== X-Gm-Message-State: AOJu0Ywr3TTgoWaMjLu0QZa9GXFCnvyyQ/Yvp9qbLSYV8cbOVMu22w8i qKiqGEOzUMFHQojO5DJuAUW6MQ== X-Google-Smtp-Source: AGHT+IGMzn78qfxYuCyS1ARY45cL6P2MuHjWyw4/lUIp6wjTkKc2tsB9vxOdISIru88jXT2X5qm6jg== X-Received: by 2002:a17:902:b28b:b0:1d3:f43a:a2d5 with SMTP id u11-20020a170902b28b00b001d3f43aa2d5mr1056238plr.120.1703107028565; Wed, 20 Dec 2023 13:17:08 -0800 (PST) Received: from tictac2.mtv.corp.google.com ([2620:15c:9d:2:5a22:d46c:eec1:e5d4]) by smtp.gmail.com with ESMTPSA id u10-20020a170902b28a00b001d3dfebc05esm175023plr.21.2023.12.20.13.17.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Dec 2023 13:17:07 -0800 (PST) From: Douglas Anderson To: Andrew Morton Cc: Petr Mladek , Li Zhe , Pingfan Liu , John Ogness , Lecopzer Chen , Douglas Anderson , linux-kernel@vger.kernel.org Subject: [PATCH 2/4] watchdog/softlockup: Use printk_cpu_sync_get_irqsave() to serialize reporting Date: Wed, 20 Dec 2023 13:15:35 -0800 Message-ID: <20231220131534.2.Ia5906525d440d8e8383cde31b7c61c2aadc8f907@changeid> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog In-Reply-To: <20231220211640.2023645-1-dianders@chromium.org> References: <20231220211640.2023645-1-dianders@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Instead of introducing a spinlock, use printk_cpu_sync_get_irqsave() and printk_cpu_sync_put_irqrestore() to serialize softlockup reporting. Alone this doesn't have any real advantage over the spinlock, but this will allow us to use the same function in a future change to also serialize hardlockup crawls. NOTE: for the most part this serialization is important because we often end up in the show_regs() path and that has no built-in serialization if there are multiple callers at once. However, even in the case where we end up in the dump_stack() path this still has some advantages because the stack will be guaranteed to be together in the logs with the lockup message with no interleaving. NOTE: the fact that printk_cpu_sync_get_irqsave() is allowed to be called multiple times on the same CPU is important here. Specifically we hold the "lock" while calling dump_stack() which also gets the same "lock". This is explicitly documented to be OK and means we don't need to introduce a variant of dump_stack() that doesn't grab the lock. Signed-off-by: Douglas Anderson Reviewed-by: John Ogness Reviewed-by: Li Zhe --- kernel/watchdog.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/watchdog.c b/kernel/watchdog.c index b4fd2f12137f..526041a1100a 100644 --- a/kernel/watchdog.c +++ b/kernel/watchdog.c @@ -454,7 +454,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hr= timer *hrtimer) struct pt_regs *regs =3D get_irq_regs(); int duration; int softlockup_all_cpu_backtrace =3D sysctl_softlockup_all_cpu_backtrace; - static DEFINE_SPINLOCK(watchdog_output_lock); + unsigned long flags; =20 if (!watchdog_enabled) return HRTIMER_NORESTART; @@ -521,7 +521,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hr= timer *hrtimer) /* Start period for the next softlockup warning. */ update_report_ts(); =20 - spin_lock(&watchdog_output_lock); + printk_cpu_sync_get_irqsave(flags); pr_emerg("BUG: soft lockup - CPU#%d stuck for %us! [%s:%d]\n", smp_processor_id(), duration, current->comm, task_pid_nr(current)); @@ -531,7 +531,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hr= timer *hrtimer) show_regs(regs); else dump_stack(); - spin_unlock(&watchdog_output_lock); + printk_cpu_sync_put_irqrestore(flags); =20 if (softlockup_all_cpu_backtrace) { trigger_allbutcpu_cpu_backtrace(smp_processor_id()); --=20 2.43.0.472.g3155946c3a-goog From nobody Fri Dec 19 14:04:46 2025 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CBDA34A9B3 for ; Wed, 20 Dec 2023 21:17:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Rm/li2ew" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1d3aa0321b5so1291455ad.2 for ; Wed, 20 Dec 2023 13:17:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1703107031; x=1703711831; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nqMAcmRjFD5lYv9eaz6Li/2mY5/nRWhBk3z0a51JdoE=; b=Rm/li2ewUQ2jK1rUJEjKu5kv4mMWZiKi+s0VsGzL71uSuDmezamhntuxbQUmlSoHCQ q6kPeX2kxU8+X4UF1/rg9Hi3nUpMPDGzMk1lfG5D8h1BIXxw17Uu166+p5aXFxkBQfmm ktpEbQicnMv5ns1or4ClMoLB/uaGq3beS9jcE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703107031; x=1703711831; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nqMAcmRjFD5lYv9eaz6Li/2mY5/nRWhBk3z0a51JdoE=; b=YUWmzYnxHLVFRV/EAoYhA5329UD9Ju5Pin4CDFnvAL7DgJheQGjwqWTUoYyCe4vCIq EeuKL6JR5K3BjtoHQr1MQbzv1XOtqgIWsQMiOOj9MvsC7b1w+6hl6rpec2mKuyLY7M+6 ev3CpE9CkLLaPkmVKS/4CfkXsHKpIH7sQh2EiEFs2UWmDW62c3UTtN+5RpnLurz80egc rSRbgnKoIi/k8CLWJ4leUpKDUBqGIGJF/OelPFtBA4Cdjuu+LZlmkqRxIwQOBm0GzCQ0 fZZ+zuA/T+vvnJ8o9Tj9sNGId8u2vQFBQGsCW46qTh/5WQNbq7Y8wdswQuEWWkVoiWjA Jzsw== X-Gm-Message-State: AOJu0YzXHFZeHHEcsYYIbX9JpylXhCm/l4y20u2kB8e3p1JB/HhzCsOO f3LKU53JDPlnYZnitsGQpTBs0VY2ulpxUweYi34= X-Google-Smtp-Source: AGHT+IHiT2pM/zIM7SEA7jKcjzR0uskgBKayIOmSJ+RyhBRQQ2xNi/wESLtEkm0sR2Q1jxnvPlu0Fw== X-Received: by 2002:a17:903:496:b0:1d4:445:ba7b with SMTP id jj22-20020a170903049600b001d40445ba7bmr24892plb.36.1703107031029; Wed, 20 Dec 2023 13:17:11 -0800 (PST) Received: from tictac2.mtv.corp.google.com ([2620:15c:9d:2:5a22:d46c:eec1:e5d4]) by smtp.gmail.com with ESMTPSA id u10-20020a170902b28a00b001d3dfebc05esm175023plr.21.2023.12.20.13.17.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Dec 2023 13:17:10 -0800 (PST) From: Douglas Anderson To: Andrew Morton Cc: Petr Mladek , Li Zhe , Pingfan Liu , John Ogness , Lecopzer Chen , Douglas Anderson , linux-kernel@vger.kernel.org Subject: [PATCH 3/4] watchdog/hardlockup: Use printk_cpu_sync_get_irqsave() to serialize reporting Date: Wed, 20 Dec 2023 13:15:36 -0800 Message-ID: <20231220131534.3.I6ff691b3b40f0379bc860f80c6e729a0485b5247@changeid> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog In-Reply-To: <20231220211640.2023645-1-dianders@chromium.org> References: <20231220211640.2023645-1-dianders@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If two CPUs end up reporting a hardlockup at the same time then their logs could get interleaved which is hard to read. The interleaving problem was especially bad with the "perf" hardlockup detector where the locked up CPU is always the same as the running CPU and we end up in show_regs(). show_regs() has no inherent serialization so we could mix together two crawls if two hardlockups happened at the same time (and if we didn't have `sysctl_hardlockup_all_cpu_backtrace` set). With this change we'll fully serialize hardlockups when using the "perf" hardlockup detector. The interleaving problem was less bad with the "buddy" hardlockup detector. With "buddy" we always end up calling `trigger_single_cpu_backtrace(cpu)` on some CPU other than the running one. trigger_single_cpu_backtrace() always at least serializes the individual stack crawls because it eventually uses printk_cpu_sync_get_irqsave(). Unfortunately the fact that trigger_single_cpu_backtrace() eventually calls printk_cpu_sync_get_irqsave() (on a different CPU) means that we have to drop the "lock" before calling it and we can't fully serialize all printouts associated with a given hardlockup. However, we still do get the advantage of serializing the output of print_modules() and print_irqtrace_events(). Aside from serializing hardlockups from each other, this change also has the advantage of serializing hardlockups and softlockups from each other if they happen to happen at the same time since they are both using the same "lock". Even though nobody is expected to hang while holding the lock associated with printk_cpu_sync_get_irqsave(), out of an abundance of caution, we don't call printk_cpu_sync_get_irqsave() until after we print out about the hardlockup. This makes extra sure that, even if printk_cpu_sync_get_irqsave() somehow never runs we at least print that we saw the hardlockup. This is different than the choice made for softlockup because hardlockup is really our last resort. Signed-off-by: Douglas Anderson Reviewed-by: John Ogness --- kernel/watchdog.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/kernel/watchdog.c b/kernel/watchdog.c index 526041a1100a..11f9577accca 100644 --- a/kernel/watchdog.c +++ b/kernel/watchdog.c @@ -151,6 +151,7 @@ void watchdog_hardlockup_check(unsigned int cpu, struct= pt_regs *regs) */ if (is_hardlockup(cpu)) { unsigned int this_cpu =3D smp_processor_id(); + unsigned long flags; =20 /* Only print hardlockups once. */ if (per_cpu(watchdog_hardlockup_warned, cpu)) @@ -165,7 +166,17 @@ void watchdog_hardlockup_check(unsigned int cpu, struc= t pt_regs *regs) return; } =20 + /* + * NOTE: we call printk_cpu_sync_get_irqsave() after printing + * the lockup message. While it would be nice to serialize + * that printout, we really want to make sure that if some + * other CPU somehow locked up while holding the lock associated + * with printk_cpu_sync_get_irqsave() that we can still at least + * get the message about the lockup out. + */ pr_emerg("Watchdog detected hard LOCKUP on cpu %d\n", cpu); + printk_cpu_sync_get_irqsave(flags); + print_modules(); print_irqtrace_events(current); if (cpu =3D=3D this_cpu) { @@ -173,7 +184,9 @@ void watchdog_hardlockup_check(unsigned int cpu, struct= pt_regs *regs) show_regs(regs); else dump_stack(); + printk_cpu_sync_put_irqrestore(flags); } else { + printk_cpu_sync_put_irqrestore(flags); trigger_single_cpu_backtrace(cpu); } =20 --=20 2.43.0.472.g3155946c3a-goog From nobody Fri Dec 19 14:04:46 2025 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA6634AF63 for ; Wed, 20 Dec 2023 21:17:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="WMsXp5la" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-1d3ddccfaffso1323145ad.1 for ; Wed, 20 Dec 2023 13:17:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1703107033; x=1703711833; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sGegIddggKFG/zutIaei4eHzxrpz6/dja4W8/BctOAU=; b=WMsXp5lad1ptNXHEjUl8VA87VrrCWANP2+oOSskHvW08BCjLlInH3WWbGorhzR5IKG Q3t95JdFtKdYbMRYuP6lCmOSzzjDOtGuQwzIr0F2QKrF6hsQv/uRi3IUn7p5uYwqrV4x u2e1M/EpEU/NXfkzSPDrGyiLFplp2ZLpYLOJA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703107033; x=1703711833; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sGegIddggKFG/zutIaei4eHzxrpz6/dja4W8/BctOAU=; b=NHZN/j2xGHI8SbJe/k5aOVmelUqpvEv7hjTXXjriSUThpciRRGFBSlrdjuDcwtVIUV 4aBmjcux0OBrqlhe9TofySHhSSM7uKwBY75MKScWcM5uMMJDf1MDfZaky35sm6GaP0Sh hUO1vOcziL/VoekAjJJo9eEGD3Arl2y4s/0o4KA4Jcr4oXEUM8tPaONIHP/lKflrw/Sv MpxR5n+mC5V6yZl8ohgNglLYvrmKt6FKAJB+gZNWyUT5D7VWngmuRkGTLgj19IUwFcpP diBRLtEty5VI7BNlvbQJbnIr6I3IYBsPBq+epuFtRcFdZ6qGZr2UGbTKPl9R42YKH4tZ 0Q+w== X-Gm-Message-State: AOJu0Yy/ax01Ab5qBh6KgRqkg6F7Z3XK0sGkvrvXGn4N0q/Vgk0fK5AV SjFmpyKKaN1i/gGW2jsYml9lyw== X-Google-Smtp-Source: AGHT+IEePv04N2Xkrvd8k65DZuDoY0n0riLwskKakblHowU4xsnnlhRcp1chgs2Vk6H2fJPV3kaKjw== X-Received: by 2002:a17:902:e847:b0:1d0:acfc:a653 with SMTP id t7-20020a170902e84700b001d0acfca653mr23942838plg.84.1703107033190; Wed, 20 Dec 2023 13:17:13 -0800 (PST) Received: from tictac2.mtv.corp.google.com ([2620:15c:9d:2:5a22:d46c:eec1:e5d4]) by smtp.gmail.com with ESMTPSA id u10-20020a170902b28a00b001d3dfebc05esm175023plr.21.2023.12.20.13.17.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Dec 2023 13:17:12 -0800 (PST) From: Douglas Anderson To: Andrew Morton Cc: Petr Mladek , Li Zhe , Pingfan Liu , John Ogness , Lecopzer Chen , Douglas Anderson , linux-kernel@vger.kernel.org Subject: [PATCH 4/4] watchdog: If panicking and we dumped everything, don't re-enable dumping Date: Wed, 20 Dec 2023 13:15:37 -0800 Message-ID: <20231220131534.4.Id3a9c7ec2d7d83e4080da6f8662ba2226b40543f@changeid> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog In-Reply-To: <20231220211640.2023645-1-dianders@chromium.org> References: <20231220211640.2023645-1-dianders@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If, as part of handling a hardlockup or softlockup, we've already dumped all CPUs and we're just about to panic, don't reenable dumping and give some other CPU a chance to hop in there and add some confusing logs right as the panic is happening. Signed-off-by: Douglas Anderson --- kernel/watchdog.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/watchdog.c b/kernel/watchdog.c index 11f9577accca..81a8862295d6 100644 --- a/kernel/watchdog.c +++ b/kernel/watchdog.c @@ -192,7 +192,8 @@ void watchdog_hardlockup_check(unsigned int cpu, struct= pt_regs *regs) =20 if (sysctl_hardlockup_all_cpu_backtrace) { trigger_allbutcpu_cpu_backtrace(cpu); - clear_bit_unlock(0, &hard_lockup_nmi_warn); + if (!hardlockup_panic) + clear_bit_unlock(0, &hard_lockup_nmi_warn); } =20 if (hardlockup_panic) @@ -548,7 +549,8 @@ static enum hrtimer_restart watchdog_timer_fn(struct hr= timer *hrtimer) =20 if (softlockup_all_cpu_backtrace) { trigger_allbutcpu_cpu_backtrace(smp_processor_id()); - clear_bit_unlock(0, &soft_lockup_nmi_warn); + if (!softlockup_panic) + clear_bit_unlock(0, &soft_lockup_nmi_warn); } =20 add_taint(TAINT_SOFTLOCKUP, LOCKDEP_STILL_OK); --=20 2.43.0.472.g3155946c3a-goog