accel/tcg/translator.c | 9 + block/blkreplay.c | 8 + block/io.c | 22 +++ block/qapi.c | 17 ++- block/qcow2-snapshot.c | 9 + block/qcow2.h | 2 blockdev.c | 10 ++ cpus.c | 19 ++- docs/interop/qcow2.txt | 4 + docs/replay.txt | 45 +++++++ exec.c | 6 + gdbstub.c | 50 +++++++- hmp-commands-info.hx | 14 ++ hmp-commands.hx | 30 +++++ hmp.h | 3 include/block/snapshot.h | 1 include/sysemu/replay.h | 18 +++ migration/savevm.c | 15 +- qapi/block-core.json | 5 + qapi/block.json | 3 qapi/misc.json | 68 +++++++++++ replay/Makefile.objs | 3 replay/replay-debugging.c | 287 +++++++++++++++++++++++++++++++++++++++++++++ replay/replay-events.c | 14 -- replay/replay-internal.h | 10 +- replay/replay-snapshot.c | 17 ++- replay/replay-time.c | 27 ++-- replay/replay.c | 22 +++ stubs/replay.c | 10 ++ util/qemu-timer.c | 11 -- vl.c | 18 ++- 31 files changed, 696 insertions(+), 81 deletions(-) create mode 100644 replay/replay-debugging.c
GDB remote protocol supports reverse debugging of the targets.
It includes 'reverse step' and 'reverse continue' operations.
The first one finds the previous step of the execution,
and the second one is intended to stop at the last breakpoint that
would happen when the program is executed normally.
Reverse debugging is possible in the replay mode, when at least
one snapshot was created at the record or replay phase.
QEMU can use these snapshots for travelling back in time with GDB.
Running the execution in replay mode allows using GDB reverse debugging
commands:
- reverse-stepi (or rsi): Steps one instruction to the past.
QEMU loads on of the prior snapshots and proceeds to the desired
instruction forward. When that step is reaches, execution stops.
- reverse-continue (or rc): Runs execution "backwards".
QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
and replaying the execution. Then QEMU loads snapshots again and
replays to the latest breakpoint. When there are no breakpoints in
the examined section of the execution, QEMU finds one more snapshot
and tries again. After the first snapshot is processed, execution
stops at this snapshot.
The set of patches include the following modifications:
- gdbstub update for reverse debugging support
- functions that automatically perform reverse step and reverse
continue operations
- hmp/qmp commands for manipulating the replay process
- improvement of the snapshotting for saving the execution step
in the snapshot parameters
- other record/replay fixes
The patches are available in the repository:
https://github.com/ispras/qemu/tree/rr-180524
v4 changes:
- changed 'since 2.13' to 'since 3.0' in json (as suggested by Eric Blake)
v3 changes:
- Fixed PS/2 bug with save/load vm, which caused failures of the replay.
The patch was sent separately.
- Rebased to the new code base.
- Minor fixes.
v2 changes:
- documented reverse debugging
- fixed start vmstate loading in record mode
- documented qcow2 changes (as suggested by Eric Blake)
- made icount SnapshotInfo field optional (as suggested by Eric Blake)
- renamed qmp commands (as suggested by Eric Blake)
- minor changes
---
Pavel Dovgalyuk (19):
block: implement bdrv_snapshot_goto for blkreplay
replay: disable default snapshot for record/replay
replay: update docs for record/replay with block devices
replay: don't drain/flush bdrv queue while RR is working
replay: finish record/replay before closing the disks
qcow2: introduce icount field for snapshots
migration: introduce icount field for snapshots
replay: introduce info hmp/qmp command
replay: introduce breakpoint at the specified step
replay: implement replay-seek command to proceed to the desired step
replay: flush events when exiting
timer: remove replay clock probe in deadline calculation
replay: refine replay-time module
translator: fix breakpoint processing
replay: flush rr queue before loading the vmstate
gdbstub: add reverse step support in replay mode
gdbstub: add reverse continue support in replay mode
replay: describe reverse debugging in docs/replay.txt
replay: allow loading any snapshots before recording
accel/tcg/translator.c | 9 +
block/blkreplay.c | 8 +
block/io.c | 22 +++
block/qapi.c | 17 ++-
block/qcow2-snapshot.c | 9 +
block/qcow2.h | 2
blockdev.c | 10 ++
cpus.c | 19 ++-
docs/interop/qcow2.txt | 4 +
docs/replay.txt | 45 +++++++
exec.c | 6 +
gdbstub.c | 50 +++++++-
hmp-commands-info.hx | 14 ++
hmp-commands.hx | 30 +++++
hmp.h | 3
include/block/snapshot.h | 1
include/sysemu/replay.h | 18 +++
migration/savevm.c | 15 +-
qapi/block-core.json | 5 +
qapi/block.json | 3
qapi/misc.json | 68 +++++++++++
replay/Makefile.objs | 3
replay/replay-debugging.c | 287 +++++++++++++++++++++++++++++++++++++++++++++
replay/replay-events.c | 14 --
replay/replay-internal.h | 10 +-
replay/replay-snapshot.c | 17 ++-
replay/replay-time.c | 27 ++--
replay/replay.c | 22 +++
stubs/replay.c | 10 ++
util/qemu-timer.c | 11 --
vl.c | 18 ++-
31 files changed, 696 insertions(+), 81 deletions(-)
create mode 100644 replay/replay-debugging.c
--
Pavel Dovgalyuk
Ping? Отправлено с помощью BlueMail На 28 Май 2018 г., 10:13, в 10:13, Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru> написал:п>GDB remote protocol supports reverse debugging of the targets. >It includes 'reverse step' and 'reverse continue' operations. >The first one finds the previous step of the execution, >and the second one is intended to stop at the last breakpoint that >would happen when the program is executed normally. > >Reverse debugging is possible in the replay mode, when at least >one snapshot was created at the record or replay phase. >QEMU can use these snapshots for travelling back in time with GDB. > >Running the execution in replay mode allows using GDB reverse debugging >commands: > - reverse-stepi (or rsi): Steps one instruction to the past. > QEMU loads on of the prior snapshots and proceeds to the desired > instruction forward. When that step is reaches, execution stops. > - reverse-continue (or rc): Runs execution "backwards". > QEMU tries to find breakpoint or watchpoint by loaded prior snapshot > and replaying the execution. Then QEMU loads snapshots again and > replays to the latest breakpoint. When there are no breakpoints in > the examined section of the execution, QEMU finds one more snapshot > and tries again. After the first snapshot is processed, execution > stops at this snapshot. > >The set of patches include the following modifications: > - gdbstub update for reverse debugging support > - functions that automatically perform reverse step and reverse > continue operations > - hmp/qmp commands for manipulating the replay process > - improvement of the snapshotting for saving the execution step > in the snapshot parameters > - other record/replay fixes > >The patches are available in the repository: >https://github.com/ispras/qemu/tree/rr-180524 > >v4 changes: >- changed 'since 2.13' to 'since 3.0' in json (as suggested by Eric >Blake) > >v3 changes: >- Fixed PS/2 bug with save/load vm, which caused failures of the >replay. > The patch was sent separately. > - Rebased to the new code base. > - Minor fixes. > >v2 changes: > - documented reverse debugging > - fixed start vmstate loading in record mode > - documented qcow2 changes (as suggested by Eric Blake) > - made icount SnapshotInfo field optional (as suggested by Eric Blake) > - renamed qmp commands (as suggested by Eric Blake) > - minor changes > >--- > >Pavel Dovgalyuk (19): > block: implement bdrv_snapshot_goto for blkreplay > replay: disable default snapshot for record/replay > replay: update docs for record/replay with block devices > replay: don't drain/flush bdrv queue while RR is working > replay: finish record/replay before closing the disks > qcow2: introduce icount field for snapshots > migration: introduce icount field for snapshots > replay: introduce info hmp/qmp command > replay: introduce breakpoint at the specified step > replay: implement replay-seek command to proceed to the desired step > replay: flush events when exiting > timer: remove replay clock probe in deadline calculation > replay: refine replay-time module > translator: fix breakpoint processing > replay: flush rr queue before loading the vmstate > gdbstub: add reverse step support in replay mode > gdbstub: add reverse continue support in replay mode > replay: describe reverse debugging in docs/replay.txt > replay: allow loading any snapshots before recording > > > accel/tcg/translator.c | 9 + > block/blkreplay.c | 8 + > block/io.c | 22 +++ > block/qapi.c | 17 ++- > block/qcow2-snapshot.c | 9 + > block/qcow2.h | 2 > blockdev.c | 10 ++ > cpus.c | 19 ++- > docs/interop/qcow2.txt | 4 + > docs/replay.txt | 45 +++++++ > exec.c | 6 + > gdbstub.c | 50 +++++++- > hmp-commands-info.hx | 14 ++ > hmp-commands.hx | 30 +++++ > hmp.h | 3 > include/block/snapshot.h | 1 > include/sysemu/replay.h | 18 +++ > migration/savevm.c | 15 +- > qapi/block-core.json | 5 + > qapi/block.json | 3 > qapi/misc.json | 68 +++++++++++ > replay/Makefile.objs | 3 >replay/replay-debugging.c | 287 >+++++++++++++++++++++++++++++++++++++++++++++ > replay/replay-events.c | 14 -- > replay/replay-internal.h | 10 +- > replay/replay-snapshot.c | 17 ++- > replay/replay-time.c | 27 ++-- > replay/replay.c | 22 +++ > stubs/replay.c | 10 ++ > util/qemu-timer.c | 11 -- > vl.c | 18 ++- > 31 files changed, 696 insertions(+), 81 deletions(-) > create mode 100644 replay/replay-debugging.c > >-- >Pavel Dovgalyuk
Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru> writes: > Ping? I started having a look but I ran into this straight away. First I recorded a boot of the kernel: ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=record,rrfile=replay.bin Then played back: ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=replay,rrfile=replay.bin -s -S And did the following on gdb: (gdb) i 0x0000000040000004 in ?? () => 0x40000004: mov x1, xzr 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr (gdb) 0x0000000040000008 in ?? () => 0x40000008: mov x2, xzr 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 (gdb) 0x000000004000000c in ?? () => 0x4000000c: mov x3, xzr 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 (gdb) 0x0000000040000010 in ?? () => 0x40000010: ldr x4, 0x40000020 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined (gdb) 0x0000000040000014 in ?? () => 0x40000014: br x4 0x40000018: .inst 0x44000000 ; undefined 0x4000001c: .inst 0x00000000 ; undefined (gdb) p/x $x4 $1 = 0x40080000 (gdb) reverse-stepi warning: Remote failure reply: E14 Surely this is the simple case and doesn't require any snapshots for block devices as there are none. Am I missing something? > > > > Отправлено с помощью BlueMail > > На 28 Май 2018 г., 10:13, в 10:13, Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru> написал:п>GDB remote protocol supports reverse debugging of the targets. >>It includes 'reverse step' and 'reverse continue' operations. >>The first one finds the previous step of the execution, >>and the second one is intended to stop at the last breakpoint that >>would happen when the program is executed normally. >> >>Reverse debugging is possible in the replay mode, when at least >>one snapshot was created at the record or replay phase. >>QEMU can use these snapshots for travelling back in time with GDB. >> >>Running the execution in replay mode allows using GDB reverse debugging >>commands: >> - reverse-stepi (or rsi): Steps one instruction to the past. >> QEMU loads on of the prior snapshots and proceeds to the desired >> instruction forward. When that step is reaches, execution stops. >> - reverse-continue (or rc): Runs execution "backwards". >> QEMU tries to find breakpoint or watchpoint by loaded prior snapshot >> and replaying the execution. Then QEMU loads snapshots again and >> replays to the latest breakpoint. When there are no breakpoints in >> the examined section of the execution, QEMU finds one more snapshot >> and tries again. After the first snapshot is processed, execution >> stops at this snapshot. >> >>The set of patches include the following modifications: >> - gdbstub update for reverse debugging support >> - functions that automatically perform reverse step and reverse >> continue operations >> - hmp/qmp commands for manipulating the replay process >> - improvement of the snapshotting for saving the execution step >> in the snapshot parameters >> - other record/replay fixes >> >>The patches are available in the repository: >>https://github.com/ispras/qemu/tree/rr-180524 >> >>v4 changes: >>- changed 'since 2.13' to 'since 3.0' in json (as suggested by Eric >>Blake) >> >>v3 changes: >>- Fixed PS/2 bug with save/load vm, which caused failures of the >>replay. >> The patch was sent separately. >> - Rebased to the new code base. >> - Minor fixes. >> >>v2 changes: >> - documented reverse debugging >> - fixed start vmstate loading in record mode >> - documented qcow2 changes (as suggested by Eric Blake) >> - made icount SnapshotInfo field optional (as suggested by Eric Blake) >> - renamed qmp commands (as suggested by Eric Blake) >> - minor changes >> >>--- >> >>Pavel Dovgalyuk (19): >> block: implement bdrv_snapshot_goto for blkreplay >> replay: disable default snapshot for record/replay >> replay: update docs for record/replay with block devices >> replay: don't drain/flush bdrv queue while RR is working >> replay: finish record/replay before closing the disks >> qcow2: introduce icount field for snapshots >> migration: introduce icount field for snapshots >> replay: introduce info hmp/qmp command >> replay: introduce breakpoint at the specified step >> replay: implement replay-seek command to proceed to the desired step >> replay: flush events when exiting >> timer: remove replay clock probe in deadline calculation >> replay: refine replay-time module >> translator: fix breakpoint processing >> replay: flush rr queue before loading the vmstate >> gdbstub: add reverse step support in replay mode >> gdbstub: add reverse continue support in replay mode >> replay: describe reverse debugging in docs/replay.txt >> replay: allow loading any snapshots before recording >> >> >> accel/tcg/translator.c | 9 + >> block/blkreplay.c | 8 + >> block/io.c | 22 +++ >> block/qapi.c | 17 ++- >> block/qcow2-snapshot.c | 9 + >> block/qcow2.h | 2 >> blockdev.c | 10 ++ >> cpus.c | 19 ++- >> docs/interop/qcow2.txt | 4 + >> docs/replay.txt | 45 +++++++ >> exec.c | 6 + >> gdbstub.c | 50 +++++++- >> hmp-commands-info.hx | 14 ++ >> hmp-commands.hx | 30 +++++ >> hmp.h | 3 >> include/block/snapshot.h | 1 >> include/sysemu/replay.h | 18 +++ >> migration/savevm.c | 15 +- >> qapi/block-core.json | 5 + >> qapi/block.json | 3 >> qapi/misc.json | 68 +++++++++++ >> replay/Makefile.objs | 3 >>replay/replay-debugging.c | 287 >>+++++++++++++++++++++++++++++++++++++++++++++ >> replay/replay-events.c | 14 -- >> replay/replay-internal.h | 10 +- >> replay/replay-snapshot.c | 17 ++- >> replay/replay-time.c | 27 ++-- >> replay/replay.c | 22 +++ >> stubs/replay.c | 10 ++ >> util/qemu-timer.c | 11 -- >> vl.c | 18 ++- >> 31 files changed, 696 insertions(+), 81 deletions(-) >> create mode 100644 replay/replay-debugging.c >> >>-- >>Pavel Dovgalyuk -- Alex Bennée
> From: Alex Bennée [mailto:alex.bennee@linaro.org] > Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru> writes: > > > Ping? > > I started having a look but I ran into this straight away. First I > recorded a boot of the kernel: > > ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic- > version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel > ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=record,rrfile=replay.bin > > Then played back: > > ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic- > version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel > ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=replay,rrfile=replay.bin - > s -S This looks ok, but... > And did the following on gdb: > > (gdb) i > 0x0000000040000004 in ?? () > => 0x40000004: mov x1, xzr > 0x40000008: mov x2, xzr > 0x4000000c: mov x3, xzr > (gdb) > 0x0000000040000008 in ?? () > => 0x40000008: mov x2, xzr > 0x4000000c: mov x3, xzr > 0x40000010: ldr x4, 0x40000020 > (gdb) > 0x000000004000000c in ?? () > => 0x4000000c: mov x3, xzr > 0x40000010: ldr x4, 0x40000020 > 0x40000014: br x4 > (gdb) > 0x0000000040000010 in ?? () > => 0x40000010: ldr x4, 0x40000020 > 0x40000014: br x4 > 0x40000018: .inst 0x44000000 ; undefined > (gdb) > 0x0000000040000014 in ?? () > => 0x40000014: br x4 > 0x40000018: .inst 0x44000000 ; undefined > 0x4000001c: .inst 0x00000000 ; undefined > (gdb) p/x $x4 > $1 = 0x40080000 > (gdb) reverse-stepi > warning: Remote failure reply: E14 > > Surely this is the simple case and doesn't require any snapshots for > block devices as there are none. Am I missing something? Reverse debugging requires the snapshotting. QEMU can't revert the VM state without the snapshots. You can try adding an empty qcow2 image to allow snapshotting there. Pavel Dovgalyuk
Pavel Dovgalyuk <dovgaluk@ispras.ru> writes: >> From: Alex Bennée [mailto:alex.bennee@linaro.org] >> Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru> writes: >> >> > Ping? >> >> I started having a look but I ran into this straight away. First I >> recorded a boot of the kernel: >> >> ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic- >> version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel >> ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=record,rrfile=replay.bin >> >> Then played back: >> >> ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic- >> version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel >> ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=replay,rrfile=replay.bin - >> s -S > > This looks ok, but... > >> And did the following on gdb: >> >> (gdb) i >> 0x0000000040000004 in ?? () >> => 0x40000004: mov x1, xzr >> 0x40000008: mov x2, xzr >> 0x4000000c: mov x3, xzr >> (gdb) >> 0x0000000040000008 in ?? () >> => 0x40000008: mov x2, xzr >> 0x4000000c: mov x3, xzr >> 0x40000010: ldr x4, 0x40000020 >> (gdb) >> 0x000000004000000c in ?? () >> => 0x4000000c: mov x3, xzr >> 0x40000010: ldr x4, 0x40000020 >> 0x40000014: br x4 >> (gdb) >> 0x0000000040000010 in ?? () >> => 0x40000010: ldr x4, 0x40000020 >> 0x40000014: br x4 >> 0x40000018: .inst 0x44000000 ; undefined >> (gdb) >> 0x0000000040000014 in ?? () >> => 0x40000014: br x4 >> 0x40000018: .inst 0x44000000 ; undefined >> 0x4000001c: .inst 0x00000000 ; undefined >> (gdb) p/x $x4 >> $1 = 0x40080000 >> (gdb) reverse-stepi >> warning: Remote failure reply: E14 >> >> Surely this is the simple case and doesn't require any snapshots for >> block devices as there are none. Am I missing something? > > Reverse debugging requires the snapshotting. QEMU can't revert the VM state without the snapshots. > You can try adding an empty qcow2 image to allow snapshotting there. I got confused with block device snapshots and rr snapshots. Let me try again ;-) -- Alex Bennée
Pavel Dovgalyuk <dovgaluk@ispras.ru> writes: >> From: Alex Bennée [mailto:alex.bennee@linaro.org] >> Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru> writes: >> >> > Ping? >> >> I started having a look but I ran into this straight away. First I >> recorded a boot of the kernel: >> >> ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic- >> version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel >> ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=record,rrfile=replay.bin >> >> Then played back: >> >> ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic- >> version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel >> ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=replay,rrfile=replay.bin - >> s -S > > This looks ok, but... > >> And did the following on gdb: >> >> (gdb) i >> 0x0000000040000004 in ?? () >> => 0x40000004: mov x1, xzr >> 0x40000008: mov x2, xzr >> 0x4000000c: mov x3, xzr >> (gdb) >> 0x0000000040000008 in ?? () >> => 0x40000008: mov x2, xzr >> 0x4000000c: mov x3, xzr >> 0x40000010: ldr x4, 0x40000020 >> (gdb) >> 0x000000004000000c in ?? () >> => 0x4000000c: mov x3, xzr >> 0x40000010: ldr x4, 0x40000020 >> 0x40000014: br x4 >> (gdb) >> 0x0000000040000010 in ?? () >> => 0x40000010: ldr x4, 0x40000020 >> 0x40000014: br x4 >> 0x40000018: .inst 0x44000000 ; undefined >> (gdb) >> 0x0000000040000014 in ?? () >> => 0x40000014: br x4 >> 0x40000018: .inst 0x44000000 ; undefined >> 0x4000001c: .inst 0x00000000 ; undefined >> (gdb) p/x $x4 >> $1 = 0x40080000 >> (gdb) reverse-stepi >> warning: Remote failure reply: E14 >> >> Surely this is the simple case and doesn't require any snapshots for >> block devices as there are none. Am I missing something? > > Reverse debugging requires the snapshotting. QEMU can't revert the VM state without the snapshots. > You can try adding an empty qcow2 image to allow snapshotting there. I suspect a recent patch has broken locking again: Starting program: /home/alex/lsrc/qemu/qemu.git/aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount shift=7,rr=replay,rrfile=replay.bin,rrsnapshot=debug -drive file=rr.qcow2,if=none,snapshot,id=rr -s -S [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7fffd8536700 (LWP 32452)] [New Thread 0x7fffd5bb3700 (LWP 32453)] [New Thread 0x7fffd4eab700 (LWP 32454)] ** ERROR:replay/replay-time.c:49:replay_read_clock: assertion failed: (replay_file && replay_mutex_locked()) Once I have the linux-user TCG tests merged I'm planning on focusing on the system emulation tests and we should be able to add some record/replay tests to defend the behaviour. -- Alex Bennée
> From: Alex Bennée [mailto:alex.bennee@linaro.org] > Pavel Dovgalyuk <dovgaluk@ispras.ru> writes: > > >> From: Alex Bennée [mailto:alex.bennee@linaro.org] > >> Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru> writes: > >> > >> > Ping? > >> > >> I started having a look but I ran into this straight away. First I > >> recorded a boot of the kernel: > >> > >> ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic- > >> version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel > >> ../images/aarch64-current-linux-initrd-guest.img -icount > shift=7,rr=record,rrfile=replay.bin > >> > >> Then played back: > >> > >> ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic- > >> version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel > >> ../images/aarch64-current-linux-initrd-guest.img -icount > shift=7,rr=replay,rrfile=replay.bin - > >> s -S > > > > This looks ok, but... > > > >> And did the following on gdb: > >> > >> (gdb) i > >> 0x0000000040000004 in ?? () > >> => 0x40000004: mov x1, xzr > >> 0x40000008: mov x2, xzr > >> 0x4000000c: mov x3, xzr > >> (gdb) > >> 0x0000000040000008 in ?? () > >> => 0x40000008: mov x2, xzr > >> 0x4000000c: mov x3, xzr > >> 0x40000010: ldr x4, 0x40000020 > >> (gdb) > >> 0x000000004000000c in ?? () > >> => 0x4000000c: mov x3, xzr > >> 0x40000010: ldr x4, 0x40000020 > >> 0x40000014: br x4 > >> (gdb) > >> 0x0000000040000010 in ?? () > >> => 0x40000010: ldr x4, 0x40000020 > >> 0x40000014: br x4 > >> 0x40000018: .inst 0x44000000 ; undefined > >> (gdb) > >> 0x0000000040000014 in ?? () > >> => 0x40000014: br x4 > >> 0x40000018: .inst 0x44000000 ; undefined > >> 0x4000001c: .inst 0x00000000 ; undefined > >> (gdb) p/x $x4 > >> $1 = 0x40080000 > >> (gdb) reverse-stepi > >> warning: Remote failure reply: E14 > >> > >> Surely this is the simple case and doesn't require any snapshots for > >> block devices as there are none. Am I missing something? > > > > Reverse debugging requires the snapshotting. QEMU can't revert the VM state without the > snapshots. > > You can try adding an empty qcow2 image to allow snapshotting there. > > I suspect a recent patch has broken locking again: > > Starting program: /home/alex/lsrc/qemu/qemu.git/aarch64-softmmu/qemu-system-aarch64 -machine > virt,graphics=on,gic-version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display > none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount > shift=7,rr=replay,rrfile=replay.bin,rrsnapshot=debug -drive > file=rr.qcow2,if=none,snapshot,id=rr -s -S > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > [New Thread 0x7fffd8536700 (LWP 32452)] > [New Thread 0x7fffd5bb3700 (LWP 32453)] > [New Thread 0x7fffd4eab700 (LWP 32454)] > ** > ERROR:replay/replay-time.c:49:replay_read_clock: assertion failed: (replay_file && > replay_mutex_locked()) Have you recorded it with the attached disk before replaying? Are you using the latest version? If the both answers are 'yes', then can you share the kernel? My i386 runs work normally. > Once I have the linux-user TCG tests merged I'm planning on focusing on > the system emulation tests and we should be able to add some > record/replay tests to defend the behaviour. That will be great. There are some Ciro's attempts on that: https://github.com/cirosantilli/qemu-test/blob/master/arm/rr Pavel Dovgalyuk
Pavel Dovgalyuk <dovgaluk@ispras.ru> writes: >> From: Alex Bennée [mailto:alex.bennee@linaro.org] >> Pavel Dovgalyuk <dovgaluk@ispras.ru> writes: >> >> >> From: Alex Bennée [mailto:alex.bennee@linaro.org] >> >> Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru> writes: >> >> >> >> > Ping? >> >> >> >> I started having a look but I ran into this straight away. First I >> >> recorded a boot of the kernel: >> >> >> >> ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic- >> >> version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel >> >> ../images/aarch64-current-linux-initrd-guest.img -icount >> shift=7,rr=record,rrfile=replay.bin >> >> >> >> Then played back: >> >> >> >> ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic- >> >> version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel >> >> ../images/aarch64-current-linux-initrd-guest.img -icount >> shift=7,rr=replay,rrfile=replay.bin - >> >> s -S >> > >> > This looks ok, but... >> > >> >> And did the following on gdb: >> >> >> >> (gdb) i >> >> 0x0000000040000004 in ?? () >> >> => 0x40000004: mov x1, xzr >> >> 0x40000008: mov x2, xzr >> >> 0x4000000c: mov x3, xzr >> >> (gdb) >> >> 0x0000000040000008 in ?? () >> >> => 0x40000008: mov x2, xzr >> >> 0x4000000c: mov x3, xzr >> >> 0x40000010: ldr x4, 0x40000020 >> >> (gdb) >> >> 0x000000004000000c in ?? () >> >> => 0x4000000c: mov x3, xzr >> >> 0x40000010: ldr x4, 0x40000020 >> >> 0x40000014: br x4 >> >> (gdb) >> >> 0x0000000040000010 in ?? () >> >> => 0x40000010: ldr x4, 0x40000020 >> >> 0x40000014: br x4 >> >> 0x40000018: .inst 0x44000000 ; undefined >> >> (gdb) >> >> 0x0000000040000014 in ?? () >> >> => 0x40000014: br x4 >> >> 0x40000018: .inst 0x44000000 ; undefined >> >> 0x4000001c: .inst 0x00000000 ; undefined >> >> (gdb) p/x $x4 >> >> $1 = 0x40080000 >> >> (gdb) reverse-stepi >> >> warning: Remote failure reply: E14 >> >> >> >> Surely this is the simple case and doesn't require any snapshots for >> >> block devices as there are none. Am I missing something? >> > >> > Reverse debugging requires the snapshotting. QEMU can't revert the VM state without the >> snapshots. >> > You can try adding an empty qcow2 image to allow snapshotting there. >> >> I suspect a recent patch has broken locking again: >> >> Starting program: /home/alex/lsrc/qemu/qemu.git/aarch64-softmmu/qemu-system-aarch64 -machine >> virt,graphics=on,gic-version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display >> none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount >> shift=7,rr=replay,rrfile=replay.bin,rrsnapshot=debug -drive >> file=rr.qcow2,if=none,snapshot,id=rr -s -S >> [Thread debugging using libthread_db enabled] >> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". >> [New Thread 0x7fffd8536700 (LWP 32452)] >> [New Thread 0x7fffd5bb3700 (LWP 32453)] >> [New Thread 0x7fffd4eab700 (LWP 32454)] >> ** >> ERROR:replay/replay-time.c:49:replay_read_clock: assertion failed: (replay_file && >> replay_mutex_locked()) > > Have you recorded it with the attached disk before replaying? Yes. I assume the -drive doesn't actually have to be visible to the guest, it's just the mechanism rr needs for saving snapshots? > Are you using the latest version? > If the both answers are 'yes', then can you share the kernel? My i386 > runs work normally. I'll have a go with x86 first as aarch64 hasn't been proven yet. > >> Once I have the linux-user TCG tests merged I'm planning on focusing on >> the system emulation tests and we should be able to add some >> record/replay tests to defend the behaviour. > > That will be great. > There are some Ciro's attempts on that: https://github.com/cirosantilli/qemu-test/blob/master/arm/rr > > Pavel Dovgalyuk -- Alex Bennée
> From: Alex Bennée [mailto:alex.bennee@linaro.org] > Pavel Dovgalyuk <dovgaluk@ispras.ru> writes: > > >> From: Alex Bennée [mailto:alex.bennee@linaro.org] > >> Pavel Dovgalyuk <dovgaluk@ispras.ru> writes: > >> > >> >> From: Alex Bennée [mailto:alex.bennee@linaro.org] > >> >> Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru> writes: > >> >> > >> >> > Ping? > >> >> > >> >> I started having a look but I ran into this straight away. First I > >> >> recorded a boot of the kernel: > >> >> > >> >> ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic- > >> >> version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel > >> >> ../images/aarch64-current-linux-initrd-guest.img -icount > >> shift=7,rr=record,rrfile=replay.bin > >> >> > >> >> Then played back: > >> >> > >> >> ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic- > >> >> version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel > >> >> ../images/aarch64-current-linux-initrd-guest.img -icount > >> shift=7,rr=replay,rrfile=replay.bin - > >> >> s -S > >> > > >> > This looks ok, but... > >> > > >> >> And did the following on gdb: > >> >> > >> >> (gdb) i > >> >> 0x0000000040000004 in ?? () > >> >> => 0x40000004: mov x1, xzr > >> >> 0x40000008: mov x2, xzr > >> >> 0x4000000c: mov x3, xzr > >> >> (gdb) > >> >> 0x0000000040000008 in ?? () > >> >> => 0x40000008: mov x2, xzr > >> >> 0x4000000c: mov x3, xzr > >> >> 0x40000010: ldr x4, 0x40000020 > >> >> (gdb) > >> >> 0x000000004000000c in ?? () > >> >> => 0x4000000c: mov x3, xzr > >> >> 0x40000010: ldr x4, 0x40000020 > >> >> 0x40000014: br x4 > >> >> (gdb) > >> >> 0x0000000040000010 in ?? () > >> >> => 0x40000010: ldr x4, 0x40000020 > >> >> 0x40000014: br x4 > >> >> 0x40000018: .inst 0x44000000 ; undefined > >> >> (gdb) > >> >> 0x0000000040000014 in ?? () > >> >> => 0x40000014: br x4 > >> >> 0x40000018: .inst 0x44000000 ; undefined > >> >> 0x4000001c: .inst 0x00000000 ; undefined > >> >> (gdb) p/x $x4 > >> >> $1 = 0x40080000 > >> >> (gdb) reverse-stepi > >> >> warning: Remote failure reply: E14 > >> >> > >> >> Surely this is the simple case and doesn't require any snapshots for > >> >> block devices as there are none. Am I missing something? > >> > > >> > Reverse debugging requires the snapshotting. QEMU can't revert the VM state without the > >> snapshots. > >> > You can try adding an empty qcow2 image to allow snapshotting there. > >> > >> I suspect a recent patch has broken locking again: > >> > >> Starting program: /home/alex/lsrc/qemu/qemu.git/aarch64-softmmu/qemu-system-aarch64 - > machine > >> virt,graphics=on,gic-version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio - > display > >> none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount > >> shift=7,rr=replay,rrfile=replay.bin,rrsnapshot=debug -drive > >> file=rr.qcow2,if=none,snapshot,id=rr -s -S Just noticed. If you are using VM snapshots, then you should disable "snapshot" option of the drive. Like that: -drive file=rr.qcow2,if=none BTW, similar command line for aarch64 worked for me. I just removed "-display=none" for convenience. Pavel Dovgalyuk
> From: Alex Bennée [mailto:alex.bennee@linaro.org] > Pavel Dovgalyuk <dovgaluk@ispras.ru> writes: > > >> From: Alex Bennée [mailto:alex.bennee@linaro.org] > >> Pavel Dovgalyuk <dovgaluk@ispras.ru> writes: > >> > >> >> From: Alex Bennée [mailto:alex.bennee@linaro.org] > >> >> Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru> writes: > >> >> > >> >> > Ping? > >> >> > >> >> I started having a look but I ran into this straight away. First I > >> >> recorded a boot of the kernel: > >> >> > >> >> ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic- > >> >> version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel > >> >> ../images/aarch64-current-linux-initrd-guest.img -icount > >> shift=7,rr=record,rrfile=replay.bin > >> >> > >> >> Then played back: > >> >> > >> >> ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic- > >> >> version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel > >> >> ../images/aarch64-current-linux-initrd-guest.img -icount > >> shift=7,rr=replay,rrfile=replay.bin - > >> >> s -S > >> > > >> > This looks ok, but... > >> > > >> >> And did the following on gdb: > >> >> > >> >> (gdb) i > >> >> 0x0000000040000004 in ?? () > >> >> => 0x40000004: mov x1, xzr > >> >> 0x40000008: mov x2, xzr > >> >> 0x4000000c: mov x3, xzr > >> >> (gdb) > >> >> 0x0000000040000008 in ?? () > >> >> => 0x40000008: mov x2, xzr > >> >> 0x4000000c: mov x3, xzr > >> >> 0x40000010: ldr x4, 0x40000020 > >> >> (gdb) > >> >> 0x000000004000000c in ?? () > >> >> => 0x4000000c: mov x3, xzr > >> >> 0x40000010: ldr x4, 0x40000020 > >> >> 0x40000014: br x4 > >> >> (gdb) > >> >> 0x0000000040000010 in ?? () > >> >> => 0x40000010: ldr x4, 0x40000020 > >> >> 0x40000014: br x4 > >> >> 0x40000018: .inst 0x44000000 ; undefined > >> >> (gdb) > >> >> 0x0000000040000014 in ?? () > >> >> => 0x40000014: br x4 > >> >> 0x40000018: .inst 0x44000000 ; undefined > >> >> 0x4000001c: .inst 0x00000000 ; undefined > >> >> (gdb) p/x $x4 > >> >> $1 = 0x40080000 > >> >> (gdb) reverse-stepi > >> >> warning: Remote failure reply: E14 > >> >> > >> >> Surely this is the simple case and doesn't require any snapshots for > >> >> block devices as there are none. Am I missing something? > >> > > >> > Reverse debugging requires the snapshotting. QEMU can't revert the VM state without the > >> snapshots. > >> > You can try adding an empty qcow2 image to allow snapshotting there. > >> > >> I suspect a recent patch has broken locking again: > >> > >> Starting program: /home/alex/lsrc/qemu/qemu.git/aarch64-softmmu/qemu-system-aarch64 - > machine > >> virt,graphics=on,gic-version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio - > display > >> none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount > >> shift=7,rr=replay,rrfile=replay.bin,rrsnapshot=debug -drive > >> file=rr.qcow2,if=none,snapshot,id=rr -s -S > >> [Thread debugging using libthread_db enabled] > >> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > >> [New Thread 0x7fffd8536700 (LWP 32452)] > >> [New Thread 0x7fffd5bb3700 (LWP 32453)] > >> [New Thread 0x7fffd4eab700 (LWP 32454)] > >> ** > >> ERROR:replay/replay-time.c:49:replay_read_clock: assertion failed: (replay_file && > >> replay_mutex_locked()) > > > > Have you recorded it with the attached disk before replaying? > > Yes. I assume the -drive doesn't actually have to be visible to the > guest, it's just the mechanism rr needs for saving snapshots? > > > Are you using the latest version? > > If the both answers are 'yes', then can you share the kernel? My i386 > > runs work normally. > > I'll have a go with x86 first as aarch64 hasn't been proven yet. Any news about that? Pavel Dovgalyuk
Pavel Dovgalyuk <dovgaluk@ispras.ru> writes: >> From: Alex Bennée [mailto:alex.bennee@linaro.org] >> Pavel Dovgalyuk <dovgaluk@ispras.ru> writes: >> >> >> From: Alex Bennée [mailto:alex.bennee@linaro.org] >> >> Pavel Dovgalyuk <dovgaluk@ispras.ru> writes: >> >> >> >> >> From: Alex Bennée [mailto:alex.bennee@linaro.org] >> >> >> Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru> writes: >> >> >> >> >> >> > Ping? >> >> >> >> >> >> I started having a look but I ran into this straight away. First I >> >> >> recorded a boot of the kernel: >> >> >> >> >> >> ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic- >> >> >> version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel >> >> >> ../images/aarch64-current-linux-initrd-guest.img -icount >> >> shift=7,rr=record,rrfile=replay.bin >> >> >> >> >> >> Then played back: >> >> >> >> >> >> ./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic- >> >> >> version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio -display none -kernel >> >> >> ../images/aarch64-current-linux-initrd-guest.img -icount >> >> shift=7,rr=replay,rrfile=replay.bin - >> >> >> s -S >> >> > >> >> > This looks ok, but... >> >> > >> >> >> And did the following on gdb: >> >> >> >> >> >> (gdb) i >> >> >> 0x0000000040000004 in ?? () >> >> >> => 0x40000004: mov x1, xzr >> >> >> 0x40000008: mov x2, xzr >> >> >> 0x4000000c: mov x3, xzr >> >> >> (gdb) >> >> >> 0x0000000040000008 in ?? () >> >> >> => 0x40000008: mov x2, xzr >> >> >> 0x4000000c: mov x3, xzr >> >> >> 0x40000010: ldr x4, 0x40000020 >> >> >> (gdb) >> >> >> 0x000000004000000c in ?? () >> >> >> => 0x4000000c: mov x3, xzr >> >> >> 0x40000010: ldr x4, 0x40000020 >> >> >> 0x40000014: br x4 >> >> >> (gdb) >> >> >> 0x0000000040000010 in ?? () >> >> >> => 0x40000010: ldr x4, 0x40000020 >> >> >> 0x40000014: br x4 >> >> >> 0x40000018: .inst 0x44000000 ; undefined >> >> >> (gdb) >> >> >> 0x0000000040000014 in ?? () >> >> >> => 0x40000014: br x4 >> >> >> 0x40000018: .inst 0x44000000 ; undefined >> >> >> 0x4000001c: .inst 0x00000000 ; undefined >> >> >> (gdb) p/x $x4 >> >> >> $1 = 0x40080000 >> >> >> (gdb) reverse-stepi >> >> >> warning: Remote failure reply: E14 >> >> >> >> >> >> Surely this is the simple case and doesn't require any snapshots for >> >> >> block devices as there are none. Am I missing something? >> >> > >> >> > Reverse debugging requires the snapshotting. QEMU can't revert the VM state without the >> >> snapshots. >> >> > You can try adding an empty qcow2 image to allow snapshotting there. >> >> >> >> I suspect a recent patch has broken locking again: >> >> >> >> Starting program: /home/alex/lsrc/qemu/qemu.git/aarch64-softmmu/qemu-system-aarch64 - >> machine >> >> virt,graphics=on,gic-version=3,virtualization=on -cpu cortex-a53 --serial mon:stdio - >> display >> >> none -kernel ../images/aarch64-current-linux-initrd-guest.img -icount >> >> shift=7,rr=replay,rrfile=replay.bin,rrsnapshot=debug -drive >> >> file=rr.qcow2,if=none,snapshot,id=rr -s -S >> >> [Thread debugging using libthread_db enabled] >> >> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". >> >> [New Thread 0x7fffd8536700 (LWP 32452)] >> >> [New Thread 0x7fffd5bb3700 (LWP 32453)] >> >> [New Thread 0x7fffd4eab700 (LWP 32454)] >> >> ** >> >> ERROR:replay/replay-time.c:49:replay_read_clock: assertion failed: (replay_file && >> >> replay_mutex_locked()) >> > >> > Have you recorded it with the attached disk before replaying? >> >> Yes. I assume the -drive doesn't actually have to be visible to the >> guest, it's just the mechanism rr needs for saving snapshots? >> >> > Are you using the latest version? >> > If the both answers are 'yes', then can you share the kernel? My i386 >> > runs work normally. >> >> I'll have a go with x86 first as aarch64 hasn't been proven yet. > > Any news about that? Sorry I got caught up with the pre-softfreeze rush. I'm hoping to get to it this week. > > > Pavel Dovgalyuk -- Alex Bennée
© 2016 - 2025 Red Hat, Inc.