[v1] Block-related record/replay fixes

[Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

Posted by Pavel Dovgalyuk 4 years, 7 months ago

The set of patches include the block-related updates
of the record/replay icount feature:
 - application of 'snapshot' option on the file layer instead of
   the top one: command line and documentation fix
 - implementation of bdrv_snapshot_goto for blkreplay driver
 - start/stop fix in replay mode with attached block devices
 - record/replay of bh oneshot events, used in the block layer

---

Pavel Dovgaluk (6):
      block: implement bdrv_snapshot_goto for blkreplay
      replay: disable default snapshot for record/replay
      replay: update docs for record/replay with block devices
      replay: don't drain/flush bdrv queue while RR is working
      replay: finish record/replay before closing the disks
      replay: add BH oneshot event for block layer


 block/blkreplay.c        |    8 ++++++++
 block/block-backend.c    |    9 ++++++---
 block/io.c               |   32 ++++++++++++++++++++++++++++++--
 block/iscsi.c            |    5 +++--
 block/nfs.c              |    6 ++++--
 block/null.c             |    4 +++-
 block/nvme.c             |    6 ++++--
 block/rbd.c              |    5 +++--
 block/vxhs.c             |    5 +++--
 cpus.c                   |    2 --
 docs/replay.txt          |   12 +++++++++---
 include/sysemu/replay.h  |    4 ++++
 replay/replay-events.c   |   16 ++++++++++++++++
 replay/replay-internal.h |    1 +
 replay/replay.c          |    2 ++
 stubs/Makefile.objs      |    1 +
 stubs/replay-user.c      |    9 +++++++++
 vl.c                     |   11 +++++++++--
 18 files changed, 115 insertions(+), 23 deletions(-)
 create mode 100644 stubs/replay-user.c

-- 
Pavel Dovgalyuk

Re: [Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

Posted by no-reply@patchew.org 4 years, 7 months ago

Patchew URL: https://patchew.org/QEMU/156872146565.1757.3033215873677512474.stgit@pasha-Precision-3630-Tower/



Hi,

This series failed the asan build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
=== TEST SCRIPT END ===

./tests/docker/docker.py --engine auto build qemu:fedora tests/docker/dockerfiles/fedora.docker   --add-current-user  
Image is up to date.
  LD      docker-test-debug@fedora.mo
cc: fatal error: no input files
compilation terminated.
make: *** [docker-test-debug@fedora.mo] Error 4



The full log is available at
http://patchew.org/logs/156872146565.1757.3033215873677512474.stgit@pasha-Precision-3630-Tower/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

Posted by Alex Bennée 4 years, 7 months ago

Pavel Dovgalyuk <pavel.dovgaluk@gmail.com> writes:

> The set of patches include the block-related updates
> of the record/replay icount feature:
>  - application of 'snapshot' option on the file layer instead of
>    the top one: command line and documentation fix
>  - implementation of bdrv_snapshot_goto for blkreplay driver
>  - start/stop fix in replay mode with attached block devices
>  - record/replay of bh oneshot events, used in the block layer

Can we come up with a reasonable smoke test to verify record/replay is
functional? AIUI we don't need a full OS but we do need a block device
to store the replay data. Could we use one of the simple system tests
like "memory" and run that through a record and replay cycle?

If we can defend the basic functionally in "make check" then breakage is
less likely to slip through and you'll have less bisecting to do.

>
> ---
>
> Pavel Dovgaluk (6):
>       block: implement bdrv_snapshot_goto for blkreplay
>       replay: disable default snapshot for record/replay
>       replay: update docs for record/replay with block devices
>       replay: don't drain/flush bdrv queue while RR is working
>       replay: finish record/replay before closing the disks
>       replay: add BH oneshot event for block layer
>
>
>  block/blkreplay.c        |    8 ++++++++
>  block/block-backend.c    |    9 ++++++---
>  block/io.c               |   32 ++++++++++++++++++++++++++++++--
>  block/iscsi.c            |    5 +++--
>  block/nfs.c              |    6 ++++--
>  block/null.c             |    4 +++-
>  block/nvme.c             |    6 ++++--
>  block/rbd.c              |    5 +++--
>  block/vxhs.c             |    5 +++--
>  cpus.c                   |    2 --
>  docs/replay.txt          |   12 +++++++++---
>  include/sysemu/replay.h  |    4 ++++
>  replay/replay-events.c   |   16 ++++++++++++++++
>  replay/replay-internal.h |    1 +
>  replay/replay.c          |    2 ++
>  stubs/Makefile.objs      |    1 +
>  stubs/replay-user.c      |    9 +++++++++
>  vl.c                     |   11 +++++++++--
>  18 files changed, 115 insertions(+), 23 deletions(-)
>  create mode 100644 stubs/replay-user.c


--
Alex Bennée

Re: [Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

Posted by Pavel Dovgalyuk 4 years, 7 months ago


> -----Original Message-----
> From: Alex Bennée [mailto:alex.bennee@linaro.org]
> Sent: Tuesday, September 17, 2019 10:02 PM
> To: Pavel Dovgalyuk
> Cc: qemu-devel@nongnu.org; kwolf@redhat.com; peter.maydell@linaro.org;
> crosthwaite.peter@gmail.com; boost.lists@gmail.com; artem.k.pisarenko@gmail.com;
> quintela@redhat.com; ciro.santilli@gmail.com; jasowang@redhat.com; mst@redhat.com;
> armbru@redhat.com; mreitz@redhat.com; maria.klimushenkova@ispras.ru; dovgaluk@ispras.ru;
> kraxel@redhat.com; pavel.dovgaluk@ispras.ru; thomas.dullien@googlemail.com;
> pbonzini@redhat.com; dgilbert@redhat.com; rth@twiddle.net
> Subject: Re: [for-4.2 PATCH 0/6] Block-related record/replay fixes
> 
> 
> Pavel Dovgalyuk <pavel.dovgaluk@gmail.com> writes:
> 
> > The set of patches include the block-related updates
> > of the record/replay icount feature:
> >  - application of 'snapshot' option on the file layer instead of
> >    the top one: command line and documentation fix
> >  - implementation of bdrv_snapshot_goto for blkreplay driver
> >  - start/stop fix in replay mode with attached block devices
> >  - record/replay of bh oneshot events, used in the block layer
> 
> Can we come up with a reasonable smoke test to verify record/replay is
> functional? AIUI we don't need a full OS but we do need a block device
> to store the replay data. Could we use one of the simple system tests
> like "memory" and run that through a record and replay cycle?

That's would be great.
I'm not familiar with testing framework and couldn't find "memory" test that you refer to.
Replay test must at least the following:
1. record some execution
2. replay that execution

And there could be several endings:
1. record stalled
2. record crashed
3. replay stalled
4. replay crashed
5. all executions finished successfully

Pavel Dovgalyuk

> 
> >
> > ---
> >
> > Pavel Dovgaluk (6):
> >       block: implement bdrv_snapshot_goto for blkreplay
> >       replay: disable default snapshot for record/replay
> >       replay: update docs for record/replay with block devices
> >       replay: don't drain/flush bdrv queue while RR is working
> >       replay: finish record/replay before closing the disks
> >       replay: add BH oneshot event for block layer
> >
> >
> >  block/blkreplay.c        |    8 ++++++++
> >  block/block-backend.c    |    9 ++++++---
> >  block/io.c               |   32 ++++++++++++++++++++++++++++++--
> >  block/iscsi.c            |    5 +++--
> >  block/nfs.c              |    6 ++++--
> >  block/null.c             |    4 +++-
> >  block/nvme.c             |    6 ++++--
> >  block/rbd.c              |    5 +++--
> >  block/vxhs.c             |    5 +++--
> >  cpus.c                   |    2 --
> >  docs/replay.txt          |   12 +++++++++---
> >  include/sysemu/replay.h  |    4 ++++
> >  replay/replay-events.c   |   16 ++++++++++++++++
> >  replay/replay-internal.h |    1 +
> >  replay/replay.c          |    2 ++
> >  stubs/Makefile.objs      |    1 +
> >  stubs/replay-user.c      |    9 +++++++++
> >  vl.c                     |   11 +++++++++--
> >  18 files changed, 115 insertions(+), 23 deletions(-)
> >  create mode 100644 stubs/replay-user.c
> 
> 
> --
> Alex Bennée

Re: [Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

Posted by Alex Bennée 4 years, 7 months ago

Pavel Dovgalyuk <dovgaluk@ispras.ru> writes:

>> -----Original Message-----
>> From: Alex Bennée [mailto:alex.bennee@linaro.org]
>> Sent: Tuesday, September 17, 2019 10:02 PM
>> To: Pavel Dovgalyuk
>> Cc: qemu-devel@nongnu.org; kwolf@redhat.com; peter.maydell@linaro.org;
>> crosthwaite.peter@gmail.com; boost.lists@gmail.com; artem.k.pisarenko@gmail.com;
>> quintela@redhat.com; ciro.santilli@gmail.com; jasowang@redhat.com; mst@redhat.com;
>> armbru@redhat.com; mreitz@redhat.com; maria.klimushenkova@ispras.ru; dovgaluk@ispras.ru;
>> kraxel@redhat.com; pavel.dovgaluk@ispras.ru; thomas.dullien@googlemail.com;
>> pbonzini@redhat.com; dgilbert@redhat.com; rth@twiddle.net
>> Subject: Re: [for-4.2 PATCH 0/6] Block-related record/replay fixes
>>
>>
>> Pavel Dovgalyuk <pavel.dovgaluk@gmail.com> writes:
>>
>> > The set of patches include the block-related updates
>> > of the record/replay icount feature:
>> >  - application of 'snapshot' option on the file layer instead of
>> >    the top one: command line and documentation fix
>> >  - implementation of bdrv_snapshot_goto for blkreplay driver
>> >  - start/stop fix in replay mode with attached block devices
>> >  - record/replay of bh oneshot events, used in the block layer
>>
>> Can we come up with a reasonable smoke test to verify record/replay is
>> functional? AIUI we don't need a full OS but we do need a block device
>> to store the replay data. Could we use one of the simple system tests
>> like "memory" and run that through a record and replay cycle?
>
> That's would be great.
> I'm not familiar with testing framework and couldn't find "memory"
> test that you refer to.

The memory test code is in tests/tcg/multiarch/system and gets combined
with the boot code for whichever target can build it. For example on
aarch64 it is run like:

  timeout 15  $BLD/aarch64-softmmu/qemu-system-aarch64 -monitor none -display none -chardev file,path=memory.out,id=output  -M virt -cpu max -display none -semihosting-config enable=on,target=native,chardev=output -kernel memory

The test binaries will be in $BLD/tests/tcg/aarch64-softmmu and are
built when you run "make check-tcg" and have either the appropriate
cross compilers installed on your system or docker enabled and setup
(see docs/devel/testing.rst).

> Replay test must at least the following:
> 1. record some execution
> 2. replay that execution
>
> And there could be several endings:
> 1. record stalled
> 2. record crashed
> 3. replay stalled
> 4. replay crashed
> 5. all executions finished successfully

If you can provide an appropriate set of invocations I can plumb them
into the Makefiles for you.

>
> Pavel Dovgalyuk
>
>>
>> >
>> > ---
>> >
>> > Pavel Dovgaluk (6):
>> >       block: implement bdrv_snapshot_goto for blkreplay
>> >       replay: disable default snapshot for record/replay
>> >       replay: update docs for record/replay with block devices
>> >       replay: don't drain/flush bdrv queue while RR is working
>> >       replay: finish record/replay before closing the disks
>> >       replay: add BH oneshot event for block layer
>> >
>> >
>> >  block/blkreplay.c        |    8 ++++++++
>> >  block/block-backend.c    |    9 ++++++---
>> >  block/io.c               |   32 ++++++++++++++++++++++++++++++--
>> >  block/iscsi.c            |    5 +++--
>> >  block/nfs.c              |    6 ++++--
>> >  block/null.c             |    4 +++-
>> >  block/nvme.c             |    6 ++++--
>> >  block/rbd.c              |    5 +++--
>> >  block/vxhs.c             |    5 +++--
>> >  cpus.c                   |    2 --
>> >  docs/replay.txt          |   12 +++++++++---
>> >  include/sysemu/replay.h  |    4 ++++
>> >  replay/replay-events.c   |   16 ++++++++++++++++
>> >  replay/replay-internal.h |    1 +
>> >  replay/replay.c          |    2 ++
>> >  stubs/Makefile.objs      |    1 +
>> >  stubs/replay-user.c      |    9 +++++++++
>> >  vl.c                     |   11 +++++++++--
>> >  18 files changed, 115 insertions(+), 23 deletions(-)
>> >  create mode 100644 stubs/replay-user.c
>>
>>
>> --
>> Alex Bennée


--
Alex Bennée

Re: [Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

Posted by Pavel Dovgalyuk 4 years, 7 months ago

> From: Alex Bennée [mailto:alex.bennee@linaro.org]
> Pavel Dovgalyuk <dovgaluk@ispras.ru> writes:
> 
> >> -----Original Message-----
> >> From: Alex Bennée [mailto:alex.bennee@linaro.org]
> >> Sent: Tuesday, September 17, 2019 10:02 PM
> >> To: Pavel Dovgalyuk
> >> Cc: qemu-devel@nongnu.org; kwolf@redhat.com; peter.maydell@linaro.org;
> >> crosthwaite.peter@gmail.com; boost.lists@gmail.com; artem.k.pisarenko@gmail.com;
> >> quintela@redhat.com; ciro.santilli@gmail.com; jasowang@redhat.com; mst@redhat.com;
> >> armbru@redhat.com; mreitz@redhat.com; maria.klimushenkova@ispras.ru; dovgaluk@ispras.ru;
> >> kraxel@redhat.com; pavel.dovgaluk@ispras.ru; thomas.dullien@googlemail.com;
> >> pbonzini@redhat.com; dgilbert@redhat.com; rth@twiddle.net
> >> Subject: Re: [for-4.2 PATCH 0/6] Block-related record/replay fixes
> >>
> >>
> >> Pavel Dovgalyuk <pavel.dovgaluk@gmail.com> writes:
> >>
> >> > The set of patches include the block-related updates
> >> > of the record/replay icount feature:
> >> >  - application of 'snapshot' option on the file layer instead of
> >> >    the top one: command line and documentation fix
> >> >  - implementation of bdrv_snapshot_goto for blkreplay driver
> >> >  - start/stop fix in replay mode with attached block devices
> >> >  - record/replay of bh oneshot events, used in the block layer
> >>
> >> Can we come up with a reasonable smoke test to verify record/replay is
> >> functional? AIUI we don't need a full OS but we do need a block device
> >> to store the replay data. Could we use one of the simple system tests
> >> like "memory" and run that through a record and replay cycle?
> >
> > That's would be great.
> > I'm not familiar with testing framework and couldn't find "memory"
> > test that you refer to.
> 
> The memory test code is in tests/tcg/multiarch/system and gets combined
> with the boot code for whichever target can build it. For example on
> aarch64 it is run like:
> 
>   timeout 15  $BLD/aarch64-softmmu/qemu-system-aarch64 -monitor none -display none -chardev
> file,path=memory.out,id=output  -M virt -cpu max -display none -semihosting-config
> enable=on,target=native,chardev=output -kernel memory

Yes, rr testing could be that simple, but in full system emulation mode.
Simplest tests may be run even without the block devices:

qemu-system-aarch64 -monitor none -display none -chardev file,path=memory.out,id=output
-M virt -cpu max -kernel memory -net none -icount shift=5,rr=record,rrfile=record.bin

Better testing must include some block device like Linux boot image or something similar.

> The test binaries will be in $BLD/tests/tcg/aarch64-softmmu and are
> built when you run "make check-tcg" and have either the appropriate
> cross compilers installed on your system or docker enabled and setup
> (see docs/devel/testing.rst).
> 
> > Replay test must at least the following:
> > 1. record some execution
> > 2. replay that execution
> >
> > And there could be several endings:
> > 1. record stalled
> > 2. record crashed
> > 3. replay stalled
> > 4. replay crashed
> > 5. all executions finished successfully
> 
> If you can provide an appropriate set of invocations I can plumb them
> into the Makefiles for you.

That will be great, thank you.

Pavel Dovgalyuk

Re: [for-4.2 PATCH 0/6] Block-related record/replay fixes

Posted by Kevin Wolf 4 years, 6 months ago

Am 17.09.2019 um 13:57 hat Pavel Dovgalyuk geschrieben:
> The set of patches include the block-related updates
> of the record/replay icount feature:
>  - application of 'snapshot' option on the file layer instead of
>    the top one: command line and documentation fix
>  - implementation of bdrv_snapshot_goto for blkreplay driver
>  - start/stop fix in replay mode with attached block devices
>  - record/replay of bh oneshot events, used in the block layer

Thanks, applied to the block branch.

Kevin

Re: [Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

Posted by no-reply@patchew.org 4 years, 7 months ago

Patchew URL: https://patchew.org/QEMU/156872146565.1757.3033215873677512474.stgit@pasha-Precision-3630-Tower/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#! /bin/bash
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===

./tests/docker/docker.py --engine auto build qemu:fedora tests/docker/dockerfiles/fedora.docker   --add-current-user  
Image is up to date.
  LD      docker-test-mingw@fedora.mo
cc: fatal error: no input files
compilation terminated.
make: *** [docker-test-mingw@fedora.mo] Error 4



The full log is available at
http://patchew.org/logs/156872146565.1757.3033215873677512474.stgit@pasha-Precision-3630-Tower/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com