[v1] Dump instructions on panic for pKVM/nvhe

[PATCH 1/2] KVM: arm64: Dump instruction on hyp panic

Posted by Mostafa Saleh 6 months, 3 weeks ago

Similar to the kernel panic, where the instruction code is printed,
we can do the same for hypervisor panics.

This patch does that only in case of “CONFIG_NVHE_EL2_DEBUG” or nvhe.

The next patch adds support for pKVM.

Also, remove the hardcoded argument dump_kernel_instr().

Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
 arch/arm64/include/asm/traps.h |  1 +
 arch/arm64/kernel/traps.c      | 20 +++++++++++++-------
 arch/arm64/kvm/handle_exit.c   |  5 +++++
 3 files changed, 19 insertions(+), 7 deletions(-)

diff --git a/arch/arm64/include/asm/traps.h b/arch/arm64/include/asm/traps.h
index 82cf1f879c61..0d7e86a95d62 100644
--- a/arch/arm64/include/asm/traps.h
+++ b/arch/arm64/include/asm/traps.h
@@ -30,6 +30,7 @@ void arm64_force_sig_mceerr(int code, unsigned long far, short lsb, const char *
 void arm64_force_sig_ptrace_errno_trap(int errno, unsigned long far, const char *str);
 
 int early_brk64(unsigned long addr, unsigned long esr, struct pt_regs *regs);
+void dump_instr(unsigned long addr);
 
 /*
  * Move regs->pc to next instruction and do necessary setup before it
diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
index 9bfa5c944379..d692c05e3686 100644
--- a/arch/arm64/kernel/traps.c
+++ b/arch/arm64/kernel/traps.c
@@ -149,15 +149,11 @@ pstate_check_t * const aarch32_opcode_cond_checks[16] = {
 
 int show_unhandled_signals = 0;
 
-static void dump_kernel_instr(const char *lvl, struct pt_regs *regs)
+void dump_instr(unsigned long addr)
 {
-	unsigned long addr = instruction_pointer(regs);
 	char str[sizeof("00000000 ") * 5 + 2 + 1], *p = str;
 	int i;
 
-	if (user_mode(regs))
-		return;
-
 	for (i = -4; i < 1; i++) {
 		unsigned int val, bad;
 
@@ -169,7 +165,17 @@ static void dump_kernel_instr(const char *lvl, struct pt_regs *regs)
 			p += sprintf(p, i == 0 ? "(????????) " : "???????? ");
 	}
 
-	printk("%sCode: %s\n", lvl, str);
+	printk(KERN_EMERG "Code: %s\n", str);
+}
+
+static void dump_kernel_instr(struct pt_regs *regs)
+{
+	unsigned long addr = instruction_pointer(regs);
+
+	if (user_mode(regs))
+		return;
+
+	dump_instr(addr);
 }
 
 #define S_SMP " SMP"
@@ -190,7 +196,7 @@ static int __die(const char *str, long err, struct pt_regs *regs)
 	print_modules();
 	show_regs(regs);
 
-	dump_kernel_instr(KERN_EMERG, regs);
+	dump_kernel_instr(regs);
 
 	return ret;
 }
diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c
index 453266c96481..de12b4d4bccd 100644
--- a/arch/arm64/kvm/handle_exit.c
+++ b/arch/arm64/kvm/handle_exit.c
@@ -565,6 +565,11 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr,
 	/* Dump the nVHE hypervisor backtrace */
 	kvm_nvhe_dump_backtrace(hyp_offset);
 
+	/* Dump the faulting instruction */
+	if (!is_protected_kvm_enabled() ||
+	    IS_ENABLED(CONFIG_NVHE_EL2_DEBUG))
+		dump_instr(panic_addr + kaslr_offset());
+
 	/*
 	 * Hyp has panicked and we're going to handle that by panicking the
 	 * kernel. The kernel offset will be revealed in the panic so we're
-- 
2.50.0.727.gbf7dc18ff4-goog

Re: [PATCH 1/2] KVM: arm64: Dump instruction on hyp panic

Posted by Will Deacon 5 months ago

On Thu, Jul 17, 2025 at 11:47:43PM +0000, Mostafa Saleh wrote:
> Similar to the kernel panic, where the instruction code is printed,
> we can do the same for hypervisor panics.
> 
> This patch does that only in case of “CONFIG_NVHE_EL2_DEBUG” or nvhe.
> 
> The next patch adds support for pKVM.
> 
> Also, remove the hardcoded argument dump_kernel_instr().
> 
> Signed-off-by: Mostafa Saleh <smostafa@google.com>
> ---
>  arch/arm64/include/asm/traps.h |  1 +
>  arch/arm64/kernel/traps.c      | 20 +++++++++++++-------
>  arch/arm64/kvm/handle_exit.c   |  5 +++++
>  3 files changed, 19 insertions(+), 7 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/traps.h b/arch/arm64/include/asm/traps.h
> index 82cf1f879c61..0d7e86a95d62 100644
> --- a/arch/arm64/include/asm/traps.h
> +++ b/arch/arm64/include/asm/traps.h
> @@ -30,6 +30,7 @@ void arm64_force_sig_mceerr(int code, unsigned long far, short lsb, const char *
>  void arm64_force_sig_ptrace_errno_trap(int errno, unsigned long far, const char *str);
>  
>  int early_brk64(unsigned long addr, unsigned long esr, struct pt_regs *regs);
> +void dump_instr(unsigned long addr);
>  
>  /*
>   * Move regs->pc to next instruction and do necessary setup before it
> diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
> index 9bfa5c944379..d692c05e3686 100644
> --- a/arch/arm64/kernel/traps.c
> +++ b/arch/arm64/kernel/traps.c
> @@ -149,15 +149,11 @@ pstate_check_t * const aarch32_opcode_cond_checks[16] = {
>  
>  int show_unhandled_signals = 0;
>  
> -static void dump_kernel_instr(const char *lvl, struct pt_regs *regs)
> +void dump_instr(unsigned long addr)
>  {
> -	unsigned long addr = instruction_pointer(regs);
>  	char str[sizeof("00000000 ") * 5 + 2 + 1], *p = str;
>  	int i;
>  
> -	if (user_mode(regs))
> -		return;
> -
>  	for (i = -4; i < 1; i++) {
>  		unsigned int val, bad;

I'm a little worried that this function might be used to try and dump
instructions from userspace now that it just takes an address.

Maybe we could:

  - Keep the name unchanged, e.g. void dump_kernel_instr(unsigned long kaddr)
  - Inline the user_mode(regs) and instruction_pointer(regs) calls into
    __die()
  - Check is_ttbr1_addr(kaddr) in dump_kernel_instr()

WDYT?

Will

Re: [PATCH 1/2] KVM: arm64: Dump instruction on hyp panic

Posted by Mostafa Saleh 5 months ago

On Mon, Sep 08, 2025 at 01:16:45PM +0100, Will Deacon wrote:
> On Thu, Jul 17, 2025 at 11:47:43PM +0000, Mostafa Saleh wrote:
> > Similar to the kernel panic, where the instruction code is printed,
> > we can do the same for hypervisor panics.
> > 
> > This patch does that only in case of “CONFIG_NVHE_EL2_DEBUG” or nvhe.
> > 
> > The next patch adds support for pKVM.
> > 
> > Also, remove the hardcoded argument dump_kernel_instr().
> > 
> > Signed-off-by: Mostafa Saleh <smostafa@google.com>
> > ---
> >  arch/arm64/include/asm/traps.h |  1 +
> >  arch/arm64/kernel/traps.c      | 20 +++++++++++++-------
> >  arch/arm64/kvm/handle_exit.c   |  5 +++++
> >  3 files changed, 19 insertions(+), 7 deletions(-)
> > 
> > diff --git a/arch/arm64/include/asm/traps.h b/arch/arm64/include/asm/traps.h
> > index 82cf1f879c61..0d7e86a95d62 100644
> > --- a/arch/arm64/include/asm/traps.h
> > +++ b/arch/arm64/include/asm/traps.h
> > @@ -30,6 +30,7 @@ void arm64_force_sig_mceerr(int code, unsigned long far, short lsb, const char *
> >  void arm64_force_sig_ptrace_errno_trap(int errno, unsigned long far, const char *str);
> >  
> >  int early_brk64(unsigned long addr, unsigned long esr, struct pt_regs *regs);
> > +void dump_instr(unsigned long addr);
> >  
> >  /*
> >   * Move regs->pc to next instruction and do necessary setup before it
> > diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
> > index 9bfa5c944379..d692c05e3686 100644
> > --- a/arch/arm64/kernel/traps.c
> > +++ b/arch/arm64/kernel/traps.c
> > @@ -149,15 +149,11 @@ pstate_check_t * const aarch32_opcode_cond_checks[16] = {
> >  
> >  int show_unhandled_signals = 0;
> >  
> > -static void dump_kernel_instr(const char *lvl, struct pt_regs *regs)
> > +void dump_instr(unsigned long addr)
> >  {
> > -	unsigned long addr = instruction_pointer(regs);
> >  	char str[sizeof("00000000 ") * 5 + 2 + 1], *p = str;
> >  	int i;
> >  
> > -	if (user_mode(regs))
> > -		return;
> > -
> >  	for (i = -4; i < 1; i++) {
> >  		unsigned int val, bad;
> 
> I'm a little worried that this function might be used to try and dump
> instructions from userspace now that it just takes an address.
> 
> Maybe we could:
> 
>   - Keep the name unchanged, e.g. void dump_kernel_instr(unsigned long kaddr)
>   - Inline the user_mode(regs) and instruction_pointer(regs) calls into
>     __die()
>   - Check is_ttbr1_addr(kaddr) in dump_kernel_instr()
> 
> WDYT?

Makes sense, I will do that and respin v2.

Thanks,
Mostafa

> 
> Will

Re: [PATCH 1/2] KVM: arm64: Dump instruction on hyp panic

Posted by Kunwu Chan 6 months, 1 week ago

Hi Mostafa,
On 2025/7/18 07:47, Mostafa Saleh wrote:

... ....

> +	/* Dump the faulting instruction */
> +	if (!is_protected_kvm_enabled() ||
> +	    IS_ENABLED(CONFIG_NVHE_EL2_DEBUG))
> +		dump_instr(panic_addr + kaslr_offset());
> +
This part seem like unnecessary, cause it'll be remove in patch 2(Only 
the comment left).

>   	/*
>   	 * Hyp has panicked and we're going to handle that by panicking the
>   	 * kernel. The kernel offset will be revealed in the panic so we're
Another confusion is that no similar wording to what you mentioned in 
the cover—specifically "Cannot dump pKVM nVHE stacktrace: 
!CONFIG_PROTECTED_NVHE_STACKTRACE"—has been found in the code.


-- 
Thanks,
        Kunwu Chan.

Re: [PATCH 1/2] KVM: arm64: Dump instruction on hyp panic

Posted by Mostafa Saleh 6 months, 1 week ago

Hi Kunwu,

On Thu, Jul 31, 2025 at 1:59 PM Kunwu Chan <kunwu.chan@linux.dev> wrote:
>
> Hi Mostafa,
> On 2025/7/18 07:47, Mostafa Saleh wrote:
>
> ... ....
>
> > +     /* Dump the faulting instruction */
> > +     if (!is_protected_kvm_enabled() ||
> > +         IS_ENABLED(CONFIG_NVHE_EL2_DEBUG))
> > +             dump_instr(panic_addr + kaslr_offset());
> > +
> This part seem like unnecessary, cause it'll be remove in patch 2(Only
> the comment left).
>

Yes, the idea is that the first patch adds that only for CONFIG_NVHE_EL2_DEBUG
while the second does that for all configs, I split it this way as
doing that with stage-2
requires intrusive changes, so at least this patch can be picked
separately if needed.

> >       /*
> >        * Hyp has panicked and we're going to handle that by panicking the
> >        * kernel. The kernel offset will be revealed in the panic so we're
> Another confusion is that no similar wording to what you mentioned in
> the cover—specifically "Cannot dump pKVM nVHE stacktrace:
> !CONFIG_PROTECTED_NVHE_STACKTRACE"—has been found in the code.
>
I am not sure I follow, this has nothing to do with
"CONFIG_PROTECTED_NVHE_STACKTRACE"
This series added the print for for instructions as:
[   12.016044] Code: a8c17bfd d50323bf d65f03c0 d503245f (d4210000)

All other prints are from already existing code.

Thanks,
Mostafa


>
> --
> Thanks,
>         Kunwu Chan.
>

Re: [PATCH 1/2] KVM: arm64: Dump instruction on hyp panic

Posted by Kunwu Chan 6 months, 1 week ago

Hi Mostafa,

On 2025/7/31 21:05, Mostafa Saleh wrote:
> Hi Kunwu,
>
> On Thu, Jul 31, 2025 at 1:59 PM Kunwu Chan <kunwu.chan@linux.dev> wrote:
>> Hi Mostafa,
>> On 2025/7/18 07:47, Mostafa Saleh wrote:
>>
>> ... ....
>>
>>> +     /* Dump the faulting instruction */
>>> +     if (!is_protected_kvm_enabled() ||
>>> +         IS_ENABLED(CONFIG_NVHE_EL2_DEBUG))
>>> +             dump_instr(panic_addr + kaslr_offset());
>>> +
>> This part seem like unnecessary, cause it'll be remove in patch 2(Only
>> the comment left).
>>
> Yes, the idea is that the first patch adds that only for CONFIG_NVHE_EL2_DEBUG
> while the second does that for all configs, I split it this way as
> doing that with stage-2
> requires intrusive changes, so at least this patch can be picked
> separately if needed.
>
>>>        /*
>>>         * Hyp has panicked and we're going to handle that by panicking the
>>>         * kernel. The kernel offset will be revealed in the panic so we're
>> Another confusion is that no similar wording to what you mentioned in
>> the cover—specifically "Cannot dump pKVM nVHE stacktrace:
>> !CONFIG_PROTECTED_NVHE_STACKTRACE"—has been found in the code.
>>
> I am not sure I follow, this has nothing to do with
> "CONFIG_PROTECTED_NVHE_STACKTRACE"
> This series added the print for for instructions as:
> [   12.016044] Code: a8c17bfd d50323bf d65f03c0 d503245f (d4210000)
>
> All other prints are from already existing code.

Got it—I see what happened now. Turns out the confusion was caused

  by my CONFIG_PROTECTED_NVHE_STACKTRACE being enabled.

After turning that off and testing Patch 1 standalone, everything works 
exactly as you described.

The test results:

1: disable CONFIG_NVHE_EL2_DEBUG

--> "kvm [5375]: Hyp Offset: 0xfffec95693400000"

2: enable CONFIG_NVHE_EL2_DEBUG

--> "[ 684.715883][ T5525] Code: d51d991f d51d9901 d5159001 00000000 
(d4210000)

                   [ 684.715974][ T5525] kvm [5525]: Hyp Offset: 
0xfffe992b13400000"

3: without this patch :

--> "kvm [5497]: Hyp Offset: 0xfffedd4993400000"

Thanks for the clarification—really appreciate your help!

>
> Thanks,
> Mostafa


Feel free to add :

Tested-by: Kunwu Chan <chentao@kylinos.cn>
Reviewed-by: Kunwu Chan <chentao@kylinos.cn>


-- 
Thanks,
        Kunwu Chan.