[PATCH v4] sched/debug: Dump end of stack when stack corruption detected

Feng Tang posted 1 patch 1 week, 4 days ago
kernel/sched/core.c | 13 ++++++++++++-
1 file changed, 12 insertions(+), 1 deletion(-)
[PATCH v4] sched/debug: Dump end of stack when stack corruption detected
Posted by Feng Tang 1 week, 4 days ago
When debugging a kernel hang during suspend/resume [1], there were random
memory corruptions in different places like being reported by
slub_debug+KASAN, corrupted linked list or detected by scheduler with
error message:
  "Kernel panic - not syncing: corrupted stack end detected inside scheduler"

Dump the corrupted memory around the stack end to give more direct hints
about how the memory is corrupted, which could be cross-checked with the
memory dump from other victims for root-causing.

Something like:
 "
 Corrupted Stack: ff11000122770000: ff ff ff ff ff ff 14 91 82 3b 78 e8 08 00 45 00  .........;x...E.
 Corrupted Stack: ff11000122770010: 00 1d 2a ff 40 00 40 11 98 c8 0a ef 30 2c 0a ef  ..*.@.@.....0,..
 Corrupted Stack: ff11000122770020: 30 ff a2 00 22 3d 00 09 9a 95 2a 00 00 00 00 00  0..."=....*.....
 ...
 Kernel panic - not syncing: corrupted stack end detected inside scheduler
"

By comparing it with the dumped memory from other places like KASAN, the
same data pattern was found, which was later used to check the data buffer
of ethernet driver and got exact match. Then the culprit was identified to
be ethernet driver, that it frees RX related memory back to kernel in
its suspend hook, but its hardware is not really stopped, and still send
data through DMA to (corrupt) those old RX buffer which is now used by
different kernel component.

[1]. https://bugzilla.kernel.org/show_bug.cgi?id=217854

Signed-off-by: Feng Tang <feng.tang@intel.com>
Reviewed-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>
Signed-off-by: Feng Tang <feng.tang@linux.alibaba.com>
---
Changlog:
  since v3:
    * Modify commit log and rebase against 7.2-rc1
	* Update author info

  since v2:
    * Change code format (Adrian)
    * Add Reviewed tag from Adrian

  since v1:
    * Refine the commit log with more info, and rebase againt 6.8-rc3

 kernel/sched/core.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 96226707c2f6..0bd898927047 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6070,8 +6070,19 @@ static noinline void __schedule_bug(struct task_struct *prev)
 static inline void schedule_debug(struct task_struct *prev, bool preempt)
 {
 #ifdef CONFIG_SCHED_STACK_END_CHECK
-	if (task_stack_end_corrupted(prev))
+	if (task_stack_end_corrupted(prev)) {
+		unsigned long *ptr = end_of_stack(prev);
+
+		/* Dump 16 ulong words around the corruption point */
+#ifdef CONFIG_STACK_GROWSUP
+		ptr -= 15;
+#endif
+		print_hex_dump(KERN_ERR, "Corrupted Stack: ",
+			DUMP_PREFIX_ADDRESS, 16, 1, ptr,
+			16 * sizeof(unsigned long), true);
+
 		panic("corrupted stack end detected inside scheduler\n");
+	}
 
 	if (task_scs_end_corrupted(prev))
 		panic("corrupted shadow stack detected inside scheduler\n");
-- 
2.39.5 (Apple Git-154)