GDB remote protocol supports reverse debugging of the targets.
It includes 'reverse step' and 'reverse continue' operations.
The first one finds the previous step of the execution,
and the second one is intended to stop at the last breakpoint that
would happen when the program is executed normally.

Reverse debugging is possible in the replay mode, when at least
one snapshot was created at the record or replay phase.
QEMU can use these snapshots for travelling back in time with GDB.

Running the execution in replay mode allows using GDB reverse debugging
commands:
 - reverse-stepi (or rsi): Steps one instruction to the past.
   QEMU loads on of the prior snapshots and proceeds to the desired
   instruction forward. When that step is reaches, execution stops.
 - reverse-continue (or rc): Runs execution "backwards".
   QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
   and replaying the execution. Then QEMU loads snapshots again and
   replays to the latest breakpoint. When there are no breakpoints in
   the examined section of the execution, QEMU finds one more snapshot
   and tries again. After the first snapshot is processed, execution
   stops at this snapshot.

The set of patches include the following modifications:
 - gdbstub update for reverse debugging support
 - functions that automatically perform reverse step and reverse
   continue operations
 - hmp/qmp commands for manipulating the replay process
 - improvement of the snapshotting for saving the execution step
   in the snapshot parameters

The patches are available in the repository:
https://github.com/ispras/qemu/tree/rr-191223

---

Pavel Dovgaluk (11):
      replay: provide an accessor for rr filename
      qcow2: introduce icount field for snapshots
      migration: introduce icount field for snapshots
      qapi: introduce replay.json for record/replay-related stuff
      replay: introduce info hmp/qmp command
      replay: introduce breakpoint at the specified step
      replay: implement replay-seek command
      replay: flush rr queue before loading the vmstate
      gdbstub: add reverse step support in replay mode
      gdbstub: add reverse continue support in replay mode
      replay: describe reverse debugging in docs/replay.txt


 MAINTAINERS               |    1 
 accel/tcg/translator.c    |    1 
 block/qapi.c              |   18 ++
 block/qcow2-snapshot.c    |    9 +
 block/qcow2.h             |    3 
 blockdev.c                |   10 +
 cpus.c                    |   19 ++-
 docs/interop/qcow2.txt    |    4 +
 docs/replay.txt           |   33 +++++
 exec.c                    |    8 +
 gdbstub.c                 |   64 ++++++++-
 hmp-commands-info.hx      |   14 ++
 hmp-commands.hx           |   53 +++++++
 include/block/snapshot.h  |    1 
 include/monitor/hmp.h     |    4 +
 include/sysemu/replay.h   |   24 +++
 migration/savevm.c        |   17 ++
 qapi/Makefile.objs        |    2 
 qapi/block-core.json      |    8 +
 qapi/block.json           |    3 
 qapi/misc.json            |   18 --
 qapi/qapi-schema.json     |    1 
 qapi/replay.json          |  121 +++++++++++++++++
 replay/Makefile.objs      |    1 
 replay/replay-debugging.c |  325 +++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h  |    6 +
 replay/replay.c           |   22 +++
 stubs/replay.c            |   10 +
 28 files changed, 761 insertions(+), 39 deletions(-)
 create mode 100644 qapi/replay.json
 create mode 100644 replay/replay-debugging.c

-- 
Pavel Dovgalyuk

Ping.



Pavel Dovgalyuk

> -----Original Message-----
> From: Pavel Dovgalyuk [mailto:pavel.dovgaluk@gmail.com]
> Sent: Monday, December 23, 2019 12:46 PM
> To: qemu-devel@nongnu.org
> Cc: kwolf@redhat.com; peter.maydell@linaro.org; crosthwaite.peter@gmail.com;
> boost.lists@gmail.com; artem.k.pisarenko@gmail.com; quintela@redhat.com;
> ciro.santilli@gmail.com; jasowang@redhat.com; mst@redhat.com; armbru@redhat.com;
> mreitz@redhat.com; maria.klimushenkova@ispras.ru; dovgaluk@ispras.ru; kraxel@redhat.com;
> pavel.dovgaluk@ispras.ru; thomas.dullien@googlemail.com; pbonzini@redhat.com;
> alex.bennee@linaro.org; dgilbert@redhat.com; rth@twiddle.net
> Subject: [for-5.0 PATCH 00/11] Support for reverse debugging with GDB
> 
> GDB remote protocol supports reverse debugging of the targets.
> It includes 'reverse step' and 'reverse continue' operations.
> The first one finds the previous step of the execution,
> and the second one is intended to stop at the last breakpoint that
> would happen when the program is executed normally.
> 
> Reverse debugging is possible in the replay mode, when at least
> one snapshot was created at the record or replay phase.
> QEMU can use these snapshots for travelling back in time with GDB.
> 
> Running the execution in replay mode allows using GDB reverse debugging
> commands:
>  - reverse-stepi (or rsi): Steps one instruction to the past.
>    QEMU loads on of the prior snapshots and proceeds to the desired
>    instruction forward. When that step is reaches, execution stops.
>  - reverse-continue (or rc): Runs execution "backwards".
>    QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
>    and replaying the execution. Then QEMU loads snapshots again and
>    replays to the latest breakpoint. When there are no breakpoints in
>    the examined section of the execution, QEMU finds one more snapshot
>    and tries again. After the first snapshot is processed, execution
>    stops at this snapshot.
> 
> The set of patches include the following modifications:
>  - gdbstub update for reverse debugging support
>  - functions that automatically perform reverse step and reverse
>    continue operations
>  - hmp/qmp commands for manipulating the replay process
>  - improvement of the snapshotting for saving the execution step
>    in the snapshot parameters
> 
> The patches are available in the repository:
> https://github.com/ispras/qemu/tree/rr-191223
> 
> ---
> 
> Pavel Dovgaluk (11):
>       replay: provide an accessor for rr filename
>       qcow2: introduce icount field for snapshots
>       migration: introduce icount field for snapshots
>       qapi: introduce replay.json for record/replay-related stuff
>       replay: introduce info hmp/qmp command
>       replay: introduce breakpoint at the specified step
>       replay: implement replay-seek command
>       replay: flush rr queue before loading the vmstate
>       gdbstub: add reverse step support in replay mode
>       gdbstub: add reverse continue support in replay mode
>       replay: describe reverse debugging in docs/replay.txt
> 
> 
>  MAINTAINERS               |    1
>  accel/tcg/translator.c    |    1
>  block/qapi.c              |   18 ++
>  block/qcow2-snapshot.c    |    9 +
>  block/qcow2.h             |    3
>  blockdev.c                |   10 +
>  cpus.c                    |   19 ++-
>  docs/interop/qcow2.txt    |    4 +
>  docs/replay.txt           |   33 +++++
>  exec.c                    |    8 +
>  gdbstub.c                 |   64 ++++++++-
>  hmp-commands-info.hx      |   14 ++
>  hmp-commands.hx           |   53 +++++++
>  include/block/snapshot.h  |    1
>  include/monitor/hmp.h     |    4 +
>  include/sysemu/replay.h   |   24 +++
>  migration/savevm.c        |   17 ++
>  qapi/Makefile.objs        |    2
>  qapi/block-core.json      |    8 +
>  qapi/block.json           |    3
>  qapi/misc.json            |   18 --
>  qapi/qapi-schema.json     |    1
>  qapi/replay.json          |  121 +++++++++++++++++
>  replay/Makefile.objs      |    1
>  replay/replay-debugging.c |  325 +++++++++++++++++++++++++++++++++++++++++++++
>  replay/replay-internal.h  |    6 +
>  replay/replay.c           |   22 +++
>  stubs/replay.c            |   10 +
>  28 files changed, 761 insertions(+), 39 deletions(-)
>  create mode 100644 qapi/replay.json
>  create mode 100644 replay/replay-debugging.c
> 
> --
> Pavel Dovgalyuk

Am 09.01.2020 um 07:13 hat Pavel Dovgalyuk geschrieben:
> Ping.

I think you have my Acked-by for the block-related patches in this
series now. If I missed something, please let me know.

Kevin

Kevin Wolf <kwolf@redhat.com> writes:

> Am 09.01.2020 um 07:13 hat Pavel Dovgalyuk geschrieben:
>> Ping.
>
> I think you have my Acked-by for the block-related patches in this
> series now. If I missed something, please let me know.

Pavel, whom are you nudging to do what?

> From: Markus Armbruster [mailto:armbru@redhat.com]
> Kevin Wolf <kwolf@redhat.com> writes:
> 
> > Am 09.01.2020 um 07:13 hat Pavel Dovgalyuk geschrieben:
> >> Ping.
> >
> > I think you have my Acked-by for the block-related patches in this
> > series now. If I missed something, please let me know.
> 
> Pavel, whom are you nudging to do what?

I'm not sure.
My prior patches for record/replay were pulled by Paolo.
But reverse debugging is more like a modification of things,
and not a completely new subsystem. 
Everything but gdbstub was already acked by the maintainers.

Pavel Dovgalyuk

Am 13.01.2020 um 10:35 hat Pavel Dovgalyuk geschrieben:
> > From: Markus Armbruster [mailto:armbru@redhat.com]
> > Kevin Wolf <kwolf@redhat.com> writes:
> > 
> > > Am 09.01.2020 um 07:13 hat Pavel Dovgalyuk geschrieben:
> > >> Ping.
> > >
> > > I think you have my Acked-by for the block-related patches in this
> > > series now. If I missed something, please let me know.
> > 
> > Pavel, whom are you nudging to do what?
> 
> I'm not sure.
> My prior patches for record/replay were pulled by Paolo.
> But reverse debugging is more like a modification of things,
> and not a completely new subsystem. 

In MAINTAINERS, you are listed yourself as the maintainer for
record/replay. I wonder whether you shouldn't just be sending pull
requests after getting Acked-by or Reviewed-by from the maintainers of
other subsystems you touch.

> Everything but gdbstub was already acked by the maintainers.

The GDB stub seems to be maintained by Alex Bennée, so he is probably
the one you need to nudge to give at least an Acked-by.

Kevin

On Mon, 13 Jan 2020 at 10:07, Kevin Wolf <kwolf@redhat.com> wrote:
> In MAINTAINERS, you are listed yourself as the maintainer for
> record/replay. I wonder whether you shouldn't just be sending pull
> requests after getting Acked-by or Reviewed-by from the maintainers of
> other subsystems you touch.

Ideally somebody else should be interested enough in record/replay
to review patches. "I'm a subsystem maintainer and send pull
requests" ideally shouldn't be something we give out just because
patches aren't getting code review, though I know that it
does sometimes degenerate into that...

thanks
-- PMM

Am 13.01.2020 um 11:14 hat Peter Maydell geschrieben:
> On Mon, 13 Jan 2020 at 10:07, Kevin Wolf <kwolf@redhat.com> wrote:
> > In MAINTAINERS, you are listed yourself as the maintainer for
> > record/replay. I wonder whether you shouldn't just be sending pull
> > requests after getting Acked-by or Reviewed-by from the maintainers of
> > other subsystems you touch.
> 
> Ideally somebody else should be interested enough in record/replay
> to review patches. "I'm a subsystem maintainer and send pull
> requests" ideally shouldn't be something we give out just because
> patches aren't getting code review, though I know that it
> does sometimes degenerate into that...

I had the impression that he said he had collected (almost) all of the
necessary reviews, but nobody really seems to be interested to take the
series through their tree because no matter who you ask, the majority of
changes will always be for other subsystems.

And as record/replay is already listed as a separate subsystem in
MAINTAINERS, it seems to make sense to me that it also gets its own pull
requests rather than trying to get patches merged though the trees of
various subsystem maintainers who all aren't really responsible for it.

Kevin

On Mon, 13 Jan 2020 at 10:27, Kevin Wolf <kwolf@redhat.com> wrote:
>
> Am 13.01.2020 um 11:14 hat Peter Maydell geschrieben:
> > On Mon, 13 Jan 2020 at 10:07, Kevin Wolf <kwolf@redhat.com> wrote:
> > > In MAINTAINERS, you are listed yourself as the maintainer for
> > > record/replay. I wonder whether you shouldn't just be sending pull
> > > requests after getting Acked-by or Reviewed-by from the maintainers of
> > > other subsystems you touch.
> >
> > Ideally somebody else should be interested enough in record/replay
> > to review patches. "I'm a subsystem maintainer and send pull
> > requests" ideally shouldn't be something we give out just because
> > patches aren't getting code review, though I know that it
> > does sometimes degenerate into that...
>
> I had the impression that he said he had collected (almost) all of the
> necessary reviews, but nobody really seems to be interested to take the
> series through their tree because no matter who you ask, the majority of
> changes will always be for other subsystems.

No, the series has only got acked-bys so far, except for the one patch
that touches qapi, which Markus reviewed. (The bulk of the changes
here are in replay/, and so far nobody's looked at those AFAIK.)

> And as record/replay is already listed as a separate subsystem in
> MAINTAINERS, it seems to make sense to me that it also gets its own pull
> requests rather than trying to get patches merged though the trees of
> various subsystem maintainers who all aren't really responsible for it.

Yeah, pull requests would be fine. Pull requests of whole
patchsets that are basically unreviewed are something I think
we should try to avoid if we can.

thanks
-- PMM

(looping the list back in)

Pavel Dovgalyuk <dovgaluk@ispras.ru> writes:

>> From: Alex Bennée [mailto:alex.bennee@linaro.org]
>> Pavel Dovgalyuk <dovgaluk@ispras.ru> writes:
>> 
>> >> From: Markus Armbruster [mailto:armbru@redhat.com]
>> >> Kevin Wolf <kwolf@redhat.com> writes:
>> >>
>> >> > Am 09.01.2020 um 07:13 hat Pavel Dovgalyuk geschrieben:
>> >> >> Ping.
>> >> >
>> >> > I think you have my Acked-by for the block-related patches in this
>> >> > series now. If I missed something, please let me know.
>> >>
>> >> Pavel, whom are you nudging to do what?
>> >
>> > I'm not sure.
>> > My prior patches for record/replay were pulled by Paolo.
>> > But reverse debugging is more like a modification of things,
>> > and not a completely new subsystem.
>> > Everything but gdbstub was already acked by the maintainers.
>> 
>> I'm having a look at the series now. What would help is if we can have
>> some sort of test case to ensure expected behaviour is protected. We
>> have a basic smoke test for record/replay and my gdbstub series:
>> 
>>   https://github.com/stsquad/qemu/tree/gdbstub/sve-registers-v5
>> 
>> introduces some infrastructure for testing thing via gdbstub. The main
>> thing needed now is some sort of gdb capability test to ensure the gdb
>> we are using supports whatever extensions it needs to work.
>
> The smoke test may be rather easy:
> 1. Execute qemu record with qcow2 image and rrsnapshot
> 2. Start qemu replay with -S
> 3. Connect gdb to qemu
> 4. stepi
> 5. break $pc
> 6. Save pc to X
> 7. stepi 10
> 8. Save pc to Y
> 9. stepi
> 10. reverse-stepi
> 11. pc should be Y
> 12. reverse-continue
> 13. pc should be X
>
> Is it possible with your infrastructure?

I think so - even if the reverse step instructions are not in the python
API the script can always do:

 gdb.parse_and_eval("command")

The main thing is ensuring probing the gdb for a version/feature first
so we can safely exit and skip the test if the host gdb isn't capable.
For the SVE testing I've had to use my own compiled version of GDB which
is a little ugly:

 ~/lsrc/qemu.git/tests/guest-debug/run-test.py \
   --qemu ./aarch64-linux-user/qemu-aarch64 \
   --qargs "-cpu max" \
   --bin ./tests/tcg/aarch64-linux-user/sve-ioctls \
   --test ~/lsrc/qemu.git/tests/tcg/aarch64/gdbstub/test-sve-ioctl.py \
   --gdb ~/src/tools/binutils-gdb.git/builds/all/install/bin/gdb

We should probably wrap that up into the tests/tcg configure.sh code so
the GDB used can persist with the build. Maybe via:

  ./configure --with-gdb=~/src/tools/binutils-gdb.git/builds/all/install/bin/gdb

-- 
Alex Bennée

> From: Kevin Wolf [mailto:kwolf@redhat.com]
> Am 09.01.2020 um 07:13 hat Pavel Dovgalyuk geschrieben:
> > Ping.
> 
> I think you have my Acked-by for the block-related patches in this
> series now. If I missed something, please let me know.

Thank you.

Pavel Dovgalyuk

Pavel Dovgalyuk <pavel.dovgaluk@gmail.com> writes:

> GDB remote protocol supports reverse debugging of the targets.
> It includes 'reverse step' and 'reverse continue' operations.
> The first one finds the previous step of the execution,
> and the second one is intended to stop at the last breakpoint that
> would happen when the program is executed normally.
>
> Reverse debugging is possible in the replay mode, when at least
> one snapshot was created at the record or replay phase.
> QEMU can use these snapshots for travelling back in time with GDB.
>
> Running the execution in replay mode allows using GDB reverse debugging
> commands:
>  - reverse-stepi (or rsi): Steps one instruction to the past.
>    QEMU loads on of the prior snapshots and proceeds to the desired
>    instruction forward. When that step is reaches, execution stops.
>  - reverse-continue (or rc): Runs execution "backwards".
>    QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
>    and replaying the execution. Then QEMU loads snapshots again and
>    replays to the latest breakpoint. When there are no breakpoints in
>    the examined section of the execution, QEMU finds one more snapshot
>    and tries again. After the first snapshot is processed, execution
>    stops at this snapshot.
>
> The set of patches include the following modifications:
>  - gdbstub update for reverse debugging support
>  - functions that automatically perform reverse step and reverse
>    continue operations
>  - hmp/qmp commands for manipulating the replay process
>  - improvement of the snapshotting for saving the execution step
>    in the snapshot parameters
>
> The patches are available in the repository:
> https://github.com/ispras/qemu/tree/rr-191223

So I tried with your additional patch. Launching QEMU as:

  ./aarch64-softmmu//qemu-system-aarch64 -monitor none \
     -display none -M virt -cpu max -display none \
     -semihosting-config enable=on \
     -kernel ./tests/tcg/aarch64-softmmu/memory \
     -icount shift=5,rr=replay,rrfile=record.bin \
     -s -S -d trace:gdbstub\*

And gdb:

  gdb tests/tcg/aarch64-softmmu/memory \
    -ex "target remote localhost:1234"

I get the following log:

  (gdb) x/3i $pc
  => 0x400037b0 <__start>:        adr     x0, 0x40003000 <vector_table>
     0x400037b4 <__start+4>:      msr     vbar_el1, x0
     0x400037b8 <__start+8>:      adrp    x0, 0x40200000
  (gdb) p/x $x0
  $1 = 0x0
  (gdb) si
  92              msr     vbar_el1, x0
  (gdb) p/x $x0
  $2 = 0x40003000
  (gdb) rsi
  warning: Remote failure reply: E14

  Program stopped.
  __start () at /home/alex.bennee/lsrc/qemu.git/tests/tcg/aarch64/system/boot.S:92
  92              msr     vbar_el1, x0
  (gdb) p/x $x0
  $3 = 0x40003000

So it doesn't seem to be working.

>
> ---
>
> Pavel Dovgaluk (11):
>       replay: provide an accessor for rr filename
>       qcow2: introduce icount field for snapshots
>       migration: introduce icount field for snapshots
>       qapi: introduce replay.json for record/replay-related stuff
>       replay: introduce info hmp/qmp command
>       replay: introduce breakpoint at the specified step
>       replay: implement replay-seek command
>       replay: flush rr queue before loading the vmstate
>       gdbstub: add reverse step support in replay mode
>       gdbstub: add reverse continue support in replay mode
>       replay: describe reverse debugging in docs/replay.txt
>
>
>  MAINTAINERS               |    1 
>  accel/tcg/translator.c    |    1 
>  block/qapi.c              |   18 ++
>  block/qcow2-snapshot.c    |    9 +
>  block/qcow2.h             |    3 
>  blockdev.c                |   10 +
>  cpus.c                    |   19 ++-
>  docs/interop/qcow2.txt    |    4 +
>  docs/replay.txt           |   33 +++++
>  exec.c                    |    8 +
>  gdbstub.c                 |   64 ++++++++-
>  hmp-commands-info.hx      |   14 ++
>  hmp-commands.hx           |   53 +++++++
>  include/block/snapshot.h  |    1 
>  include/monitor/hmp.h     |    4 +
>  include/sysemu/replay.h   |   24 +++
>  migration/savevm.c        |   17 ++
>  qapi/Makefile.objs        |    2 
>  qapi/block-core.json      |    8 +
>  qapi/block.json           |    3 
>  qapi/misc.json            |   18 --
>  qapi/qapi-schema.json     |    1 
>  qapi/replay.json          |  121 +++++++++++++++++
>  replay/Makefile.objs      |    1 
>  replay/replay-debugging.c |  325 +++++++++++++++++++++++++++++++++++++++++++++
>  replay/replay-internal.h  |    6 +
>  replay/replay.c           |   22 +++
>  stubs/replay.c            |   10 +
>  28 files changed, 761 insertions(+), 39 deletions(-)
>  create mode 100644 qapi/replay.json
>  create mode 100644 replay/replay-debugging.c


-- 
Alex Bennée

> From: Alex Bennée [mailto:alex.bennee@linaro.org]
> Pavel Dovgalyuk <pavel.dovgaluk@gmail.com> writes:
> 
> > GDB remote protocol supports reverse debugging of the targets.
> > It includes 'reverse step' and 'reverse continue' operations.
> > The first one finds the previous step of the execution,
> > and the second one is intended to stop at the last breakpoint that
> > would happen when the program is executed normally.
> >
> > Reverse debugging is possible in the replay mode, when at least
> > one snapshot was created at the record or replay phase.
> > QEMU can use these snapshots for travelling back in time with GDB.
> >
> > Running the execution in replay mode allows using GDB reverse debugging
> > commands:
> >  - reverse-stepi (or rsi): Steps one instruction to the past.
> >    QEMU loads on of the prior snapshots and proceeds to the desired
> >    instruction forward. When that step is reaches, execution stops.
> >  - reverse-continue (or rc): Runs execution "backwards".
> >    QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
> >    and replaying the execution. Then QEMU loads snapshots again and
> >    replays to the latest breakpoint. When there are no breakpoints in
> >    the examined section of the execution, QEMU finds one more snapshot
> >    and tries again. After the first snapshot is processed, execution
> >    stops at this snapshot.
> >
> > The set of patches include the following modifications:
> >  - gdbstub update for reverse debugging support
> >  - functions that automatically perform reverse step and reverse
> >    continue operations
> >  - hmp/qmp commands for manipulating the replay process
> >  - improvement of the snapshotting for saving the execution step
> >    in the snapshot parameters
> >
> > The patches are available in the repository:
> > https://github.com/ispras/qemu/tree/rr-191223
> 
> So I tried with your additional patch. Launching QEMU as:
> 
>   ./aarch64-softmmu//qemu-system-aarch64 -monitor none \
>      -display none -M virt -cpu max -display none \
>      -semihosting-config enable=on \
>      -kernel ./tests/tcg/aarch64-softmmu/memory \
>      -icount shift=5,rr=replay,rrfile=record.bin \
>      -s -S -d trace:gdbstub\*
> 
> And gdb:
> 
>   gdb tests/tcg/aarch64-softmmu/memory \
>     -ex "target remote localhost:1234"
> 
> I get the following log:
> 
>   (gdb) x/3i $pc
>   => 0x400037b0 <__start>:        adr     x0, 0x40003000 <vector_table>
>      0x400037b4 <__start+4>:      msr     vbar_el1, x0
>      0x400037b8 <__start+8>:      adrp    x0, 0x40200000
>   (gdb) p/x $x0
>   $1 = 0x0
>   (gdb) si
>   92              msr     vbar_el1, x0
>   (gdb) p/x $x0
>   $2 = 0x40003000
>   (gdb) rsi
>   warning: Remote failure reply: E14
> 
>   Program stopped.
>   __start () at /home/alex.bennee/lsrc/qemu.git/tests/tcg/aarch64/system/boot.S:92
>   92              msr     vbar_el1, x0
>   (gdb) p/x $x0
>   $3 = 0x40003000
> 
> So it doesn't seem to be working.

That's ok, you'll need at least one VM snapshot available to recover the initial VM state.
Try changing the command lines in the following way:

First, create empty.qcow2 which will be used for saving the snapshots.
Then record with initial snapshot and attached empty.qcow2:

   ./aarch64-softmmu//qemu-system-aarch64 -monitor none \
      -display none -M virt -cpu max \
      -kernel ./tests/tcg/aarch64-softmmu/memory \
      -icount shift=5,rr=record,rrfile=record.bin,rrsnapshot=init \
      -drive file=empty.qcow2

After that you can replay the execution with the support of reverse debugging:

   ./aarch64-softmmu//qemu-system-aarch64 -monitor none \
      -display none -M virt -cpu max \
      -kernel ./tests/tcg/aarch64-softmmu/memory \
      -icount shift=5,rr=replay,rrfile=record.bin,rrsnapshot=init \
      -drive file=empty.qcow2

I removed "-semihosting-config enable=on", because I'm not sure what is it for.
It may break the replay, because there is no implementation of
semihosting input record/replay.

Pavel Dovgalyuk

Pavel Dovgalyuk <dovgaluk@ispras.ru> writes:

>> From: Alex Bennée [mailto:alex.bennee@linaro.org]
>> Pavel Dovgalyuk <pavel.dovgaluk@gmail.com> writes:
>> 
>> > GDB remote protocol supports reverse debugging of the targets.
>> > It includes 'reverse step' and 'reverse continue' operations.
>> > The first one finds the previous step of the execution,
>> > and the second one is intended to stop at the last breakpoint that
>> > would happen when the program is executed normally.
>> >
>> > Reverse debugging is possible in the replay mode, when at least
>> > one snapshot was created at the record or replay phase.
>> > QEMU can use these snapshots for travelling back in time with GDB.
>> >
>> > Running the execution in replay mode allows using GDB reverse debugging
>> > commands:
>> >  - reverse-stepi (or rsi): Steps one instruction to the past.
>> >    QEMU loads on of the prior snapshots and proceeds to the desired
>> >    instruction forward. When that step is reaches, execution stops.
>> >  - reverse-continue (or rc): Runs execution "backwards".
>> >    QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
>> >    and replaying the execution. Then QEMU loads snapshots again and
>> >    replays to the latest breakpoint. When there are no breakpoints in
>> >    the examined section of the execution, QEMU finds one more snapshot
>> >    and tries again. After the first snapshot is processed, execution
>> >    stops at this snapshot.
>> >
>> > The set of patches include the following modifications:
>> >  - gdbstub update for reverse debugging support
>> >  - functions that automatically perform reverse step and reverse
>> >    continue operations
>> >  - hmp/qmp commands for manipulating the replay process
>> >  - improvement of the snapshotting for saving the execution step
>> >    in the snapshot parameters
>> >
>> > The patches are available in the repository:
>> > https://github.com/ispras/qemu/tree/rr-191223
>> 
>> So I tried with your additional patch. Launching QEMU as:
>> 
>>   ./aarch64-softmmu//qemu-system-aarch64 -monitor none \
>>      -display none -M virt -cpu max -display none \
>>      -semihosting-config enable=on \
>>      -kernel ./tests/tcg/aarch64-softmmu/memory \
>>      -icount shift=5,rr=replay,rrfile=record.bin \
>>      -s -S -d trace:gdbstub\*
>> 
>> And gdb:
>> 
>>   gdb tests/tcg/aarch64-softmmu/memory \
>>     -ex "target remote localhost:1234"
>> 
>> I get the following log:
>> 
>>   (gdb) x/3i $pc
>>   => 0x400037b0 <__start>:        adr     x0, 0x40003000 <vector_table>
>>      0x400037b4 <__start+4>:      msr     vbar_el1, x0
>>      0x400037b8 <__start+8>:      adrp    x0, 0x40200000
>>   (gdb) p/x $x0
>>   $1 = 0x0
>>   (gdb) si
>>   92              msr     vbar_el1, x0
>>   (gdb) p/x $x0
>>   $2 = 0x40003000
>>   (gdb) rsi
>>   warning: Remote failure reply: E14
>> 
>>   Program stopped.
>>   __start () at /home/alex.bennee/lsrc/qemu.git/tests/tcg/aarch64/system/boot.S:92
>>   92              msr     vbar_el1, x0
>>   (gdb) p/x $x0
>>   $3 = 0x40003000
>> 
>> So it doesn't seem to be working.
>
> That's ok, you'll need at least one VM snapshot available to recover the initial VM state.
> Try changing the command lines in the following way:
>
> First, create empty.qcow2 which will be used for saving the snapshots.
> Then record with initial snapshot and attached empty.qcow2:
>
>    ./aarch64-softmmu//qemu-system-aarch64 -monitor none \
>       -display none -M virt -cpu max \
>       -kernel ./tests/tcg/aarch64-softmmu/memory \
>       -icount shift=5,rr=record,rrfile=record.bin,rrsnapshot=init \
>       -drive file=empty.qcow2

./aarch64-softmmu//qemu-system-aarch64 -monitor none -display none -M virt -cpu max -display none -semihosting-config enable=on -kernel ./tests/tcg/aarch64-softmmu/memory -icount shift=5,rr=record,rrfile=record.bin,rrsnapshot=init -drive file=empty.qcow2
qemu-system-aarch64: invalid accelerator kvm
qemu-system-aarch64: falling back to tcg
qemu-system-aarch64: The qcow format used by node '#block163' does not support live migration
qemu-system-aarch64: Could not create snapshot for icount record

I don't know if this is down to the current state of the tree but I
haven't been able to get it working.

>
> After that you can replay the execution with the support of reverse debugging:
>
>    ./aarch64-softmmu//qemu-system-aarch64 -monitor none \
>       -display none -M virt -cpu max \
>       -kernel ./tests/tcg/aarch64-softmmu/memory \
>       -icount shift=5,rr=replay,rrfile=record.bin,rrsnapshot=init \
>       -drive file=empty.qcow2
>
> I removed "-semihosting-config enable=on", because I'm not sure what is it for.
> It may break the replay, because there is no implementation of
> semihosting input record/replay.

For this testcase semihosting in just a convenient output device (in
lieu of a serial device). We probably need to come up with a strategy on
how we handle all these devices otherwise we will end up with a random
selection of hardware combinations which work.

>
> Pavel Dovgalyuk


-- 
Alex Bennée

> From: Alex Bennée [mailto:alex.bennee@linaro.org]
> Pavel Dovgalyuk <dovgaluk@ispras.ru> writes:
> 
> >> From: Alex Bennée [mailto:alex.bennee@linaro.org]
> >> Pavel Dovgalyuk <pavel.dovgaluk@gmail.com> writes:
> >>
> >> > GDB remote protocol supports reverse debugging of the targets.
> >> > It includes 'reverse step' and 'reverse continue' operations.
> >> > The first one finds the previous step of the execution,
> >> > and the second one is intended to stop at the last breakpoint that
> >> > would happen when the program is executed normally.
> >> >
> >> > Reverse debugging is possible in the replay mode, when at least
> >> > one snapshot was created at the record or replay phase.
> >> > QEMU can use these snapshots for travelling back in time with GDB.
> >> >
> >> > Running the execution in replay mode allows using GDB reverse debugging
> >> > commands:
> >> >  - reverse-stepi (or rsi): Steps one instruction to the past.
> >> >    QEMU loads on of the prior snapshots and proceeds to the desired
> >> >    instruction forward. When that step is reaches, execution stops.
> >> >  - reverse-continue (or rc): Runs execution "backwards".
> >> >    QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
> >> >    and replaying the execution. Then QEMU loads snapshots again and
> >> >    replays to the latest breakpoint. When there are no breakpoints in
> >> >    the examined section of the execution, QEMU finds one more snapshot
> >> >    and tries again. After the first snapshot is processed, execution
> >> >    stops at this snapshot.
> >> >
> >> > The set of patches include the following modifications:
> >> >  - gdbstub update for reverse debugging support
> >> >  - functions that automatically perform reverse step and reverse
> >> >    continue operations
> >> >  - hmp/qmp commands for manipulating the replay process
> >> >  - improvement of the snapshotting for saving the execution step
> >> >    in the snapshot parameters
> >> >
> >> > The patches are available in the repository:
> >> > https://github.com/ispras/qemu/tree/rr-191223
> >>
> >> So I tried with your additional patch. Launching QEMU as:
> >>
> >>   ./aarch64-softmmu//qemu-system-aarch64 -monitor none \
> >>      -display none -M virt -cpu max -display none \
> >>      -semihosting-config enable=on \
> >>      -kernel ./tests/tcg/aarch64-softmmu/memory \
> >>      -icount shift=5,rr=replay,rrfile=record.bin \
> >>      -s -S -d trace:gdbstub\*
> >>
> >> And gdb:
> >>
> >>   gdb tests/tcg/aarch64-softmmu/memory \
> >>     -ex "target remote localhost:1234"
> >>
> >> I get the following log:
> >>
> >>   (gdb) x/3i $pc
> >>   => 0x400037b0 <__start>:        adr     x0, 0x40003000 <vector_table>
> >>      0x400037b4 <__start+4>:      msr     vbar_el1, x0
> >>      0x400037b8 <__start+8>:      adrp    x0, 0x40200000
> >>   (gdb) p/x $x0
> >>   $1 = 0x0
> >>   (gdb) si
> >>   92              msr     vbar_el1, x0
> >>   (gdb) p/x $x0
> >>   $2 = 0x40003000
> >>   (gdb) rsi
> >>   warning: Remote failure reply: E14
> >>
> >>   Program stopped.
> >>   __start () at /home/alex.bennee/lsrc/qemu.git/tests/tcg/aarch64/system/boot.S:92
> >>   92              msr     vbar_el1, x0
> >>   (gdb) p/x $x0
> >>   $3 = 0x40003000
> >>
> >> So it doesn't seem to be working.
> >
> > That's ok, you'll need at least one VM snapshot available to recover the initial VM state.
> > Try changing the command lines in the following way:
> >
> > First, create empty.qcow2 which will be used for saving the snapshots.
> > Then record with initial snapshot and attached empty.qcow2:
> >
> >    ./aarch64-softmmu//qemu-system-aarch64 -monitor none \
> >       -display none -M virt -cpu max \
> >       -kernel ./tests/tcg/aarch64-softmmu/memory \
> >       -icount shift=5,rr=record,rrfile=record.bin,rrsnapshot=init \
> >       -drive file=empty.qcow2
> 
> ./aarch64-softmmu//qemu-system-aarch64 -monitor none -display none -M virt -cpu max -display
> none -semihosting-config enable=on -kernel ./tests/tcg/aarch64-softmmu/memory -icount
> shift=5,rr=record,rrfile=record.bin,rrsnapshot=init -drive file=empty.qcow2


> qemu-system-aarch64: invalid accelerator kvm
> qemu-system-aarch64: falling back to tcg
> qemu-system-aarch64: The qcow format used by node '#block163' does not support live migration
> qemu-system-aarch64: Could not create snapshot for icount record

It seems that you have some problems with your disk image. Is it qcow2 or just qcow?

> For this testcase semihosting in just a convenient output device (in
> lieu of a serial device). 

I tried this test kernel with your options and everything was ok.

> We probably need to come up with a strategy on
> how we handle all these devices otherwise we will end up with a random
> selection of hardware combinations which work.

All correctly implemented virtual hardware should support record/replay.
But real semihosting (like file IO) should not, because it provides
untracked virtual machine inputs.

Pavel Dovgalyuk