This series adds a framework for coverage-guided fuzzing of
virtual-devices. Fuzzing targets are based on qtest and can make use of
the libqos abstractions.

V4:
 * add/transfer license headers to new files
 * restructure the added QTestClientTransportOps struct
 * restructure the FuzzTarget struct and fuzzer skeleton
 * fork-based fuzzer now directly mmaps shm over the coverage bitmaps
 * fixes to i440 and virtio-net fuzz targets
 * undo the changes to qtest_memwrite
 * possible to build /fuzz and /all in the same build-dir
 * misc fixes to address V3 comments

V3:
 * rebased onto v4.1.0+
 * add the fuzzer as a new build-target type in the build-system
 * add indirection to qtest client/server communication functions
 * remove ramfile and snapshot-based fuzzing support
 * add i440fx fuzz-target as a reference for developers.
 * add linker-script to assist with fork-based fuzzer

V2:
 * split off changes to qos virtio-net and qtest server to other patches
 * move vl:main initialization into new func: qemu_init
 * moved useful functions from qos-test.c to a separate object
 * use struct of function pointers for add_fuzz_target(), instead of
   arguments
 * move ramfile to migration/qemu-file
 * rewrite fork-based fuzzer pending patch to libfuzzer
 * pass check-patch

Alexander Oleinik (20):
  softmmu: split off vl.c:main() into main.c
  libqos: Rename i2c_send and i2c_recv
  fuzz: Add FUZZ_TARGET module type
  qtest: add qtest_server_send abstraction
  libqtest: Add a layer of abstraciton to send/recv
  module: check module wasn't already initialized
  qtest: add in-process incoming command handler
  tests: provide test variables to other targets
  libqos: split qos-test and libqos makefile vars
  libqos: move useful qos-test funcs to qos_external
  libqtest: make qtest_bufwrite send "atomic"
  libqtest: add in-process qtest.c tx/rx handlers
  fuzz: add configure flag --enable-fuzzing
  fuzz: Add target/fuzz makefile rules
  fuzz: add fuzzer skeleton
  fuzz: add support for fork-based fuzzing.
  fuzz: add support for qos-assisted fuzz targets
  fuzz: add i440fx fuzz targets
  fuzz: add virtio-net fuzz target
  fuzz: add documentation to docs/devel/

 Makefile                     |  16 ++-
 Makefile.objs                |   4 +
 Makefile.target              |  18 ++-
 configure                    |  39 ++++++
 docs/devel/fuzzing.txt       | 119 ++++++++++++++++++
 exec.c                       |  12 +-
 include/qemu/module.h        |   4 +-
 include/sysemu/qtest.h       |   4 +
 include/sysemu/sysemu.h      |   4 +
 main.c                       |  52 ++++++++
 qtest.c                      |  30 ++++-
 tests/Makefile.include       |  75 +++++------
 tests/fuzz/Makefile.include  |  11 ++
 tests/fuzz/fork_fuzz.c       |  51 ++++++++
 tests/fuzz/fork_fuzz.h       |  23 ++++
 tests/fuzz/fork_fuzz.ld      |  37 ++++++
 tests/fuzz/fuzz.c            | 177 ++++++++++++++++++++++++++
 tests/fuzz/fuzz.h            |  66 ++++++++++
 tests/fuzz/i440fx_fuzz.c     | 176 ++++++++++++++++++++++++++
 tests/fuzz/qos_fuzz.c        | 232 +++++++++++++++++++++++++++++++++++
 tests/fuzz/qos_fuzz.h        |  33 +++++
 tests/fuzz/virtio_net_fuzz.c | 123 +++++++++++++++++++
 tests/libqos/i2c-imx.c       |   8 +-
 tests/libqos/i2c-omap.c      |   8 +-
 tests/libqos/i2c.c           |  10 +-
 tests/libqos/i2c.h           |   4 +-
 tests/libqos/qos_external.c  | 168 +++++++++++++++++++++++++
 tests/libqos/qos_external.h  |  28 +++++
 tests/libqtest.c             | 109 ++++++++++++++--
 tests/libqtest.h             |   4 +
 tests/pca9552-test.c         |  10 +-
 tests/qos-test.c             | 140 +--------------------
 util/module.c                |   7 ++
 vl.c                         |  36 ++----
 34 files changed, 1601 insertions(+), 237 deletions(-)
 create mode 100644 docs/devel/fuzzing.txt
 create mode 100644 main.c
 create mode 100644 tests/fuzz/Makefile.include
 create mode 100644 tests/fuzz/fork_fuzz.c
 create mode 100644 tests/fuzz/fork_fuzz.h
 create mode 100644 tests/fuzz/fork_fuzz.ld
 create mode 100644 tests/fuzz/fuzz.c
 create mode 100644 tests/fuzz/fuzz.h
 create mode 100644 tests/fuzz/i440fx_fuzz.c
 create mode 100644 tests/fuzz/qos_fuzz.c
 create mode 100644 tests/fuzz/qos_fuzz.h
 create mode 100644 tests/fuzz/virtio_net_fuzz.c
 create mode 100644 tests/libqos/qos_external.c
 create mode 100644 tests/libqos/qos_external.h

-- 
2.23.0

Patchew URL: https://patchew.org/QEMU/20191030144926.11873-1-alxndr@bu.edu/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      tests/test-qapi-types-sub-sub-module.o
  CC      tests/test-qapi-visit.o
  CC      tests/include/test-qapi-visit-sub-module.o
/tmp/qemu-test/src/tests/test-char.c:31:13: error: static declaration of 'main_loop' follows non-static declaration
 static void main_loop(void)
             ^
In file included from /tmp/qemu-test/src/tests/test-char.c:10:0:
/tmp/qemu-test/src/include/sysemu/sysemu.h:117:6: note: previous declaration of 'main_loop' was here
 void main_loop(void);
      ^
make: *** [tests/test-char.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 662, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=24c0cf5ab5a24bf99c3e0023063c73e0', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-9qdq_pm7/src/docker-src.2019-10-30-11.19.26.32505:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=24c0cf5ab5a24bf99c3e0023063c73e0
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-9qdq_pm7/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    4m30.750s
user    0m8.740s


The full log is available at
http://patchew.org/logs/20191030144926.11873-1-alxndr@bu.edu/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

On Wed, Oct 30, 2019 at 08:23:57AM -0700, no-reply@patchew.org wrote:
> Patchew URL: https://patchew.org/QEMU/20191030144926.11873-1-alxndr@bu.edu/
> 
> 
> 
> Hi,
> 
> This series failed the docker-quick@centos7 build test. Please find the testing commands and
> their output below. If you have Docker installed, you can probably reproduce it
> locally.
> 
> === TEST SCRIPT BEGIN ===
> #!/bin/bash
> make docker-image-centos7 V=1 NETWORK=1
> time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
> === TEST SCRIPT END ===
> 
>   CC      tests/test-qapi-types-sub-sub-module.o
>   CC      tests/test-qapi-visit.o
>   CC      tests/include/test-qapi-visit-sub-module.o
> /tmp/qemu-test/src/tests/test-char.c:31:13: error: static declaration of 'main_loop' follows non-static declaration
>  static void main_loop(void)
>              ^
> In file included from /tmp/qemu-test/src/tests/test-char.c:10:0:
> /tmp/qemu-test/src/include/sysemu/sysemu.h:117:6: note: previous declaration of 'main_loop' was here
>  void main_loop(void);
>       ^
> make: *** [tests/test-char.o] Error 1
> make: *** Waiting for unfinished jobs....
> Traceback (most recent call last):
>   File "./tests/docker/docker.py", line 662, in <module>

Oops, the main_loop() change definitely broke tests/test-char.c.  Please
take a look.

Stefan

On Wed, Oct 30, 2019 at 02:49:47PM +0000, Oleinik, Alexander wrote:
> This series adds a framework for coverage-guided fuzzing of
> virtual-devices. Fuzzing targets are based on qtest and can make use of
> the libqos abstractions.
> 
> V4:
>  * add/transfer license headers to new files
>  * restructure the added QTestClientTransportOps struct
>  * restructure the FuzzTarget struct and fuzzer skeleton
>  * fork-based fuzzer now directly mmaps shm over the coverage bitmaps
>  * fixes to i440 and virtio-net fuzz targets
>  * undo the changes to qtest_memwrite
>  * possible to build /fuzz and /all in the same build-dir
>  * misc fixes to address V3 comments

I have finished reviewing this series.  Please fold in my Reviewed-by
tags when you send the next series.  That way I'll know which patches to
skip :).

Thanks,
Stefan

Hi Alexander,

I've been trying out these patches, and I'm seeing a high volume of
crashes - where for v3, there were none in a run of over 3 weeks -
so it was a bit of a surprise :)

The question is what may have changed that is causing that level of
crashes - are you seeing this for the virtio-net-fork-fuzz tests?

But also, I've been trying to debug some of these crashes - and the
expectation is that you pass the crash-XXXX file as an argument to
the qemu-fuzz-* binary - and when I do, I see the crash - but when I
try to debug it, it ends up running through and exiting.

My assumption is that because of the fork in the test, the crash is
in one of the children.

(ASIDE: I think it might be worth adding a debugging/analysing
section to the documentation you've added to help people debug such
crashes)

Setting follow-fork-mode to child does get me there, and each crash
seems, at least in the samples that I've taken, to be in iov_copy:

  #0  0x00007ffff4cff377 in raise () from /lib64/libc.so.6
  #1  0x00007ffff4d00a68 in abort () from /lib64/libc.so.6
  #2  0x00007ffff4cf8196 in __assert_fail_base () from
  /lib64/libc.so.6
  #3  0x00007ffff4cf8242 in __assert_fail () from /lib64/libc.so.6
  #4  0x00005555574d4026 in iov_copy ()
  #5  0x000055555640dbd8 in virtio_net_flush_tx ()
  #6  0x000055555640c8ef in virtio_net_tx_bh ()
  #7  0x00005555574a05bb in aio_bh_call ()
  #8  0x00005555574a0a34 in aio_bh_poll ()
  #9  0x00005555574b1687 in aio_dispatch ()
  #10 0x00005555574a35f9 in aio_ctx_dispatch ()
  #11 0x00007ffff5e5d099 in g_main_context_dispatch () from
  /lib64/libglib-2.0.so.0
  #12 0x00005555574ae9fd in glib_pollfds_poll ()
  #13 0x00005555574ad972 in os_host_main_loop_wait ()
  #14 0x00005555574ad62c in main_loop_wait ()
  #15 0x000055555736c653 in flush_events ()
  #16 0x00005555573710a4 in virtio_net_fork_fuzz ()
  #17 0x000055555736cb85 in LLVMFuzzerTestOneInput ()
  ...

Have you seen these kind of crashes, or is this just me?

Just wondering if I should dig into it as a real issue, or some
mis-merge I've done (not all the patches were cleanly applied for
me when I cloned from master).

Thanks,

Darren.

On Wed, Oct 30, 2019 at 02:49:47PM +0000, Oleinik, Alexander wrote:
>This series adds a framework for coverage-guided fuzzing of
>virtual-devices. Fuzzing targets are based on qtest and can make use of
>the libqos abstractions.
>
>V4:
> * add/transfer license headers to new files
> * restructure the added QTestClientTransportOps struct
> * restructure the FuzzTarget struct and fuzzer skeleton
> * fork-based fuzzer now directly mmaps shm over the coverage bitmaps
> * fixes to i440 and virtio-net fuzz targets
> * undo the changes to qtest_memwrite
> * possible to build /fuzz and /all in the same build-dir
> * misc fixes to address V3 comments
>
>V3:
> * rebased onto v4.1.0+
> * add the fuzzer as a new build-target type in the build-system
> * add indirection to qtest client/server communication functions
> * remove ramfile and snapshot-based fuzzing support
> * add i440fx fuzz-target as a reference for developers.
> * add linker-script to assist with fork-based fuzzer
>
>V2:
> * split off changes to qos virtio-net and qtest server to other patches
> * move vl:main initialization into new func: qemu_init
> * moved useful functions from qos-test.c to a separate object
> * use struct of function pointers for add_fuzz_target(), instead of
>   arguments
> * move ramfile to migration/qemu-file
> * rewrite fork-based fuzzer pending patch to libfuzzer
> * pass check-patch
>
>Alexander Oleinik (20):
>  softmmu: split off vl.c:main() into main.c
>  libqos: Rename i2c_send and i2c_recv
>  fuzz: Add FUZZ_TARGET module type
>  qtest: add qtest_server_send abstraction
>  libqtest: Add a layer of abstraciton to send/recv
>  module: check module wasn't already initialized
>  qtest: add in-process incoming command handler
>  tests: provide test variables to other targets
>  libqos: split qos-test and libqos makefile vars
>  libqos: move useful qos-test funcs to qos_external
>  libqtest: make qtest_bufwrite send "atomic"
>  libqtest: add in-process qtest.c tx/rx handlers
>  fuzz: add configure flag --enable-fuzzing
>  fuzz: Add target/fuzz makefile rules
>  fuzz: add fuzzer skeleton
>  fuzz: add support for fork-based fuzzing.
>  fuzz: add support for qos-assisted fuzz targets
>  fuzz: add i440fx fuzz targets
>  fuzz: add virtio-net fuzz target
>  fuzz: add documentation to docs/devel/
>
> Makefile                     |  16 ++-
> Makefile.objs                |   4 +
> Makefile.target              |  18 ++-
> configure                    |  39 ++++++
> docs/devel/fuzzing.txt       | 119 ++++++++++++++++++
> exec.c                       |  12 +-
> include/qemu/module.h        |   4 +-
> include/sysemu/qtest.h       |   4 +
> include/sysemu/sysemu.h      |   4 +
> main.c                       |  52 ++++++++
> qtest.c                      |  30 ++++-
> tests/Makefile.include       |  75 +++++------
> tests/fuzz/Makefile.include  |  11 ++
> tests/fuzz/fork_fuzz.c       |  51 ++++++++
> tests/fuzz/fork_fuzz.h       |  23 ++++
> tests/fuzz/fork_fuzz.ld      |  37 ++++++
> tests/fuzz/fuzz.c            | 177 ++++++++++++++++++++++++++
> tests/fuzz/fuzz.h            |  66 ++++++++++
> tests/fuzz/i440fx_fuzz.c     | 176 ++++++++++++++++++++++++++
> tests/fuzz/qos_fuzz.c        | 232 +++++++++++++++++++++++++++++++++++
> tests/fuzz/qos_fuzz.h        |  33 +++++
> tests/fuzz/virtio_net_fuzz.c | 123 +++++++++++++++++++
> tests/libqos/i2c-imx.c       |   8 +-
> tests/libqos/i2c-omap.c      |   8 +-
> tests/libqos/i2c.c           |  10 +-
> tests/libqos/i2c.h           |   4 +-
> tests/libqos/qos_external.c  | 168 +++++++++++++++++++++++++
> tests/libqos/qos_external.h  |  28 +++++
> tests/libqtest.c             | 109 ++++++++++++++--
> tests/libqtest.h             |   4 +
> tests/pca9552-test.c         |  10 +-
> tests/qos-test.c             | 140 +--------------------
> util/module.c                |   7 ++
> vl.c                         |  36 ++----
> 34 files changed, 1601 insertions(+), 237 deletions(-)
> create mode 100644 docs/devel/fuzzing.txt
> create mode 100644 main.c
> create mode 100644 tests/fuzz/Makefile.include
> create mode 100644 tests/fuzz/fork_fuzz.c
> create mode 100644 tests/fuzz/fork_fuzz.h
> create mode 100644 tests/fuzz/fork_fuzz.ld
> create mode 100644 tests/fuzz/fuzz.c
> create mode 100644 tests/fuzz/fuzz.h
> create mode 100644 tests/fuzz/i440fx_fuzz.c
> create mode 100644 tests/fuzz/qos_fuzz.c
> create mode 100644 tests/fuzz/qos_fuzz.h
> create mode 100644 tests/fuzz/virtio_net_fuzz.c
> create mode 100644 tests/libqos/qos_external.c
> create mode 100644 tests/libqos/qos_external.h
>
>-- 
>2.23.0
>
>

On 11/5/19 8:57 AM, Darren Kenny wrote:
> Hi Alexander,
> 
> I've been trying out these patches, and I'm seeing a high volume of
> crashes - where for v3, there were none in a run of over 3 weeks -
> so it was a bit of a surprise :)
> 
> The question is what may have changed that is causing that level of
> crashes - are you seeing this for the virtio-net-fork-fuzz tests?
Good question - my guess is that it may have to do with the change in 
how we run the main loop. I have not looked into it in much detail, but 
the crash below is likely triggered only after running the main_loop 
several times (the events handled in the first loop, schedule additional 
BHs). In v3, I believe the main_loop only ran once before the forked 
process exited. There are also changes to the linker-script used to 
facilitate communication between the forked process and the parent, but 
I think that would only impact the coverage information passed back to 
the parent.

> But also, I've been trying to debug some of these crashes - and the
> expectation is that you pass the crash-XXXX file as an argument to
> the qemu-fuzz-* binary - and when I do, I see the crash - but when I
> try to debug it, it ends up running through and exiting
> My assumption is that because of the fork in the test, the crash is
> in one of the children.
Right! Seems you are already using the follow-fork-mode option.

> (ASIDE: I think it might be worth adding a debugging/analysing
> section to the documentation you've added to help people debug such
> crashes)
Will do. Although it did not make it into v4, I am also working on an 
option to dump a trace of the qtest commands leading to a crashing 
input, which can then be replayed with a standard qtest program 
"replay.c". This seems like a good way to provide a reproducer to the 
device developers who may not be familiar with the fuzzer, or have time 
to build it.

> Setting follow-fork-mode to child does get me there, and each crash
> seems, at least in the samples that I've taken, to be in iov_copy:
Yes - this is what I have been using as well.

>   #0  0x00007ffff4cff377 in raise () from /lib64/libc.so.6
>   #1  0x00007ffff4d00a68 in abort () from /lib64/libc.so.6
>   #2  0x00007ffff4cf8196 in __assert_fail_base () from
>   /lib64/libc.so.6
>   #3  0x00007ffff4cf8242 in __assert_fail () from /lib64/libc.so.6
>   #4  0x00005555574d4026 in iov_copy ()
>   #5  0x000055555640dbd8 in virtio_net_flush_tx ()
>   #6  0x000055555640c8ef in virtio_net_tx_bh ()
>   #7  0x00005555574a05bb in aio_bh_call ()
>   #8  0x00005555574a0a34 in aio_bh_poll ()
>   #9  0x00005555574b1687 in aio_dispatch ()
>   #10 0x00005555574a35f9 in aio_ctx_dispatch ()
>   #11 0x00007ffff5e5d099 in g_main_context_dispatch () from
>   /lib64/libglib-2.0.so.0
>   #12 0x00005555574ae9fd in glib_pollfds_poll ()
>   #13 0x00005555574ad972 in os_host_main_loop_wait ()
>   #14 0x00005555574ad62c in main_loop_wait ()
>   #15 0x000055555736c653 in flush_events ()
>   #16 0x00005555573710a4 in virtio_net_fork_fuzz ()
>   #17 0x000055555736cb85 in LLVMFuzzerTestOneInput ()
>   ...
> 
> Have you seen these kind of crashes, or is this just me?
Not just you :) I posted a fix for this, but it may have not been 
complete. I think the fuzzer found it before we added forking, just by 
doing reboots in between runs:
https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg04882.html

> Just wondering if I should dig into it as a real issue, or some
> mis-merge I've done (not all the patches were cleanly applied for
> me when I cloned from master).

> Thanks,
> 
> Darren.
> 
> On Wed, Oct 30, 2019 at 02:49:47PM +0000, Oleinik, Alexander wrote:
>> This series adds a framework for coverage-guided fuzzing of
>> virtual-devices. Fuzzing targets are based on qtest and can make use of
>> the libqos abstractions.
>>
>> V4:
>> * add/transfer license headers to new files
>> * restructure the added QTestClientTransportOps struct
>> * restructure the FuzzTarget struct and fuzzer skeleton
>> * fork-based fuzzer now directly mmaps shm over the coverage bitmaps
>> * fixes to i440 and virtio-net fuzz targets
>> * undo the changes to qtest_memwrite
>> * possible to build /fuzz and /all in the same build-dir
>> * misc fixes to address V3 comments
>>
>> V3:
>> * rebased onto v4.1.0+
>> * add the fuzzer as a new build-target type in the build-system
>> * add indirection to qtest client/server communication functions
>> * remove ramfile and snapshot-based fuzzing support
>> * add i440fx fuzz-target as a reference for developers.
>> * add linker-script to assist with fork-based fuzzer
>>
>> V2:
>> * split off changes to qos virtio-net and qtest server to other patches
>> * move vl:main initialization into new func: qemu_init
>> * moved useful functions from qos-test.c to a separate object
>> * use struct of function pointers for add_fuzz_target(), instead of
>>   arguments
>> * move ramfile to migration/qemu-file
>> * rewrite fork-based fuzzer pending patch to libfuzzer
>> * pass check-patch
>>
>> Alexander Oleinik (20):
>>  softmmu: split off vl.c:main() into main.c
>>  libqos: Rename i2c_send and i2c_recv
>>  fuzz: Add FUZZ_TARGET module type
>>  qtest: add qtest_server_send abstraction
>>  libqtest: Add a layer of abstraciton to send/recv
>>  module: check module wasn't already initialized
>>  qtest: add in-process incoming command handler
>>  tests: provide test variables to other targets
>>  libqos: split qos-test and libqos makefile vars
>>  libqos: move useful qos-test funcs to qos_external
>>  libqtest: make qtest_bufwrite send "atomic"
>>  libqtest: add in-process qtest.c tx/rx handlers
>>  fuzz: add configure flag --enable-fuzzing
>>  fuzz: Add target/fuzz makefile rules
>>  fuzz: add fuzzer skeleton
>>  fuzz: add support for fork-based fuzzing.
>>  fuzz: add support for qos-assisted fuzz targets
>>  fuzz: add i440fx fuzz targets
>>  fuzz: add virtio-net fuzz target
>>  fuzz: add documentation to docs/devel/
>>
>> Makefile                     |  16 ++-
>> Makefile.objs                |   4 +
>> Makefile.target              |  18 ++-
>> configure                    |  39 ++++++
>> docs/devel/fuzzing.txt       | 119 ++++++++++++++++++
>> exec.c                       |  12 +-
>> include/qemu/module.h        |   4 +-
>> include/sysemu/qtest.h       |   4 +
>> include/sysemu/sysemu.h      |   4 +
>> main.c                       |  52 ++++++++
>> qtest.c                      |  30 ++++-
>> tests/Makefile.include       |  75 +++++------
>> tests/fuzz/Makefile.include  |  11 ++
>> tests/fuzz/fork_fuzz.c       |  51 ++++++++
>> tests/fuzz/fork_fuzz.h       |  23 ++++
>> tests/fuzz/fork_fuzz.ld      |  37 ++++++
>> tests/fuzz/fuzz.c            | 177 ++++++++++++++++++++++++++
>> tests/fuzz/fuzz.h            |  66 ++++++++++
>> tests/fuzz/i440fx_fuzz.c     | 176 ++++++++++++++++++++++++++
>> tests/fuzz/qos_fuzz.c        | 232 +++++++++++++++++++++++++++++++++++
>> tests/fuzz/qos_fuzz.h        |  33 +++++
>> tests/fuzz/virtio_net_fuzz.c | 123 +++++++++++++++++++
>> tests/libqos/i2c-imx.c       |   8 +-
>> tests/libqos/i2c-omap.c      |   8 +-
>> tests/libqos/i2c.c           |  10 +-
>> tests/libqos/i2c.h           |   4 +-
>> tests/libqos/qos_external.c  | 168 +++++++++++++++++++++++++
>> tests/libqos/qos_external.h  |  28 +++++
>> tests/libqtest.c             | 109 ++++++++++++++--
>> tests/libqtest.h             |   4 +
>> tests/pca9552-test.c         |  10 +-
>> tests/qos-test.c             | 140 +--------------------
>> util/module.c                |   7 ++
>> vl.c                         |  36 ++----
>> 34 files changed, 1601 insertions(+), 237 deletions(-)
>> create mode 100644 docs/devel/fuzzing.txt
>> create mode 100644 main.c
>> create mode 100644 tests/fuzz/Makefile.include
>> create mode 100644 tests/fuzz/fork_fuzz.c
>> create mode 100644 tests/fuzz/fork_fuzz.h
>> create mode 100644 tests/fuzz/fork_fuzz.ld
>> create mode 100644 tests/fuzz/fuzz.c
>> create mode 100644 tests/fuzz/fuzz.h
>> create mode 100644 tests/fuzz/i440fx_fuzz.c
>> create mode 100644 tests/fuzz/qos_fuzz.c
>> create mode 100644 tests/fuzz/qos_fuzz.h
>> create mode 100644 tests/fuzz/virtio_net_fuzz.c
>> create mode 100644 tests/libqos/qos_external.c
>> create mode 100644 tests/libqos/qos_external.h
>>
>> -- 
>> 2.23.0
>>
>>

On Tue, Nov 05, 2019 at 11:28:59AM -0500, Alexander Oleinik wrote:
>On 11/5/19 8:57 AM, Darren Kenny wrote:
>>Hi Alexander,
>>
>>I've been trying out these patches, and I'm seeing a high volume of
>>crashes - where for v3, there were none in a run of over 3 weeks -
>>so it was a bit of a surprise :)
>>
>>The question is what may have changed that is causing that level of
>>crashes - are you seeing this for the virtio-net-fork-fuzz tests?
>Good question - my guess is that it may have to do with the change in 
>how we run the main loop. I have not looked into it in much detail, 
>but the crash below is likely triggered only after running the 
>main_loop several times (the events handled in the first loop, 
>schedule additional BHs). In v3, I believe the main_loop only ran once 
>before the forked process exited. There are also changes to the 
>linker-script used to facilitate communication between the forked 
>process and the parent, but I think that would only impact the 
>coverage information passed back to the parent.

OK, sounds like this is a genuine issue then - great :)

>>But also, I've been trying to debug some of these crashes - and the
>>expectation is that you pass the crash-XXXX file as an argument to
>>the qemu-fuzz-* binary - and when I do, I see the crash - but when I
>>try to debug it, it ends up running through and exiting
>>My assumption is that because of the fork in the test, the crash is
>>in one of the children.
>Right! Seems you are already using the follow-fork-mode option.

Yup

>
>>(ASIDE: I think it might be worth adding a debugging/analysing
>>section to the documentation you've added to help people debug such
>>crashes)
>Will do. Although it did not make it into v4, I am also working on an 
>option to dump a trace of the qtest commands leading to a crashing 
>input, which can then be replayed with a standard qtest program 
>"replay.c". This seems like a good way to provide a reproducer to the 
>device developers who may not be familiar with the fuzzer, or have 
>time to build it.

That would be great!

For me the biggest problem was that I didn't have a version of clang
new enough (OL7/RHEL7) to have Libfuzzer, so if people don't have
it, then building with it's support is a bit more involved since you
also need to build clang, etc.

>>Setting follow-fork-mode to child does get me there, and each crash
>>seems, at least in the samples that I've taken, to be in iov_copy:
>Yes - this is what I have been using as well.
>
>>  #0  0x00007ffff4cff377 in raise () from /lib64/libc.so.6
>>  #1  0x00007ffff4d00a68 in abort () from /lib64/libc.so.6
>>  #2  0x00007ffff4cf8196 in __assert_fail_base () from
>>  /lib64/libc.so.6
>>  #3  0x00007ffff4cf8242 in __assert_fail () from /lib64/libc.so.6
>>  #4  0x00005555574d4026 in iov_copy ()
>>  #5  0x000055555640dbd8 in virtio_net_flush_tx ()
>>  #6  0x000055555640c8ef in virtio_net_tx_bh ()
>>  #7  0x00005555574a05bb in aio_bh_call ()
>>  #8  0x00005555574a0a34 in aio_bh_poll ()
>>  #9  0x00005555574b1687 in aio_dispatch ()
>>  #10 0x00005555574a35f9 in aio_ctx_dispatch ()
>>  #11 0x00007ffff5e5d099 in g_main_context_dispatch () from
>>  /lib64/libglib-2.0.so.0
>>  #12 0x00005555574ae9fd in glib_pollfds_poll ()
>>  #13 0x00005555574ad972 in os_host_main_loop_wait ()
>>  #14 0x00005555574ad62c in main_loop_wait ()
>>  #15 0x000055555736c653 in flush_events ()
>>  #16 0x00005555573710a4 in virtio_net_fork_fuzz ()
>>  #17 0x000055555736cb85 in LLVMFuzzerTestOneInput ()
>>  ...
>>
>>Have you seen these kind of crashes, or is this just me?
>Not just you :) I posted a fix for this, but it may have not been 
>complete. I think the fuzzer found it before we added forking, just by 
>doing reboots in between runs:
>https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg04882.html

Ah! So I guess this hasn't been pulled into master yet (or at least
wasn't when I pulled from there yesterday.

Thanks,

Darren.

>
>>Just wondering if I should dig into it as a real issue, or some
>>mis-merge I've done (not all the patches were cleanly applied for
>>me when I cloned from master).
>
>>Thanks,
>>
>>Darren.
>>
>>On Wed, Oct 30, 2019 at 02:49:47PM +0000, Oleinik, Alexander wrote:
>>>This series adds a framework for coverage-guided fuzzing of
>>>virtual-devices. Fuzzing targets are based on qtest and can make use of
>>>the libqos abstractions.
>>>
>>>V4:
>>>* add/transfer license headers to new files
>>>* restructure the added QTestClientTransportOps struct
>>>* restructure the FuzzTarget struct and fuzzer skeleton
>>>* fork-based fuzzer now directly mmaps shm over the coverage bitmaps
>>>* fixes to i440 and virtio-net fuzz targets
>>>* undo the changes to qtest_memwrite
>>>* possible to build /fuzz and /all in the same build-dir
>>>* misc fixes to address V3 comments
>>>
>>>V3:
>>>* rebased onto v4.1.0+
>>>* add the fuzzer as a new build-target type in the build-system
>>>* add indirection to qtest client/server communication functions
>>>* remove ramfile and snapshot-based fuzzing support
>>>* add i440fx fuzz-target as a reference for developers.
>>>* add linker-script to assist with fork-based fuzzer
>>>
>>>V2:
>>>* split off changes to qos virtio-net and qtest server to other patches
>>>* move vl:main initialization into new func: qemu_init
>>>* moved useful functions from qos-test.c to a separate object
>>>* use struct of function pointers for add_fuzz_target(), instead of
>>>  arguments
>>>* move ramfile to migration/qemu-file
>>>* rewrite fork-based fuzzer pending patch to libfuzzer
>>>* pass check-patch
>>>
>>>Alexander Oleinik (20):
>>> softmmu: split off vl.c:main() into main.c
>>> libqos: Rename i2c_send and i2c_recv
>>> fuzz: Add FUZZ_TARGET module type
>>> qtest: add qtest_server_send abstraction
>>> libqtest: Add a layer of abstraciton to send/recv
>>> module: check module wasn't already initialized
>>> qtest: add in-process incoming command handler
>>> tests: provide test variables to other targets
>>> libqos: split qos-test and libqos makefile vars
>>> libqos: move useful qos-test funcs to qos_external
>>> libqtest: make qtest_bufwrite send "atomic"
>>> libqtest: add in-process qtest.c tx/rx handlers
>>> fuzz: add configure flag --enable-fuzzing
>>> fuzz: Add target/fuzz makefile rules
>>> fuzz: add fuzzer skeleton
>>> fuzz: add support for fork-based fuzzing.
>>> fuzz: add support for qos-assisted fuzz targets
>>> fuzz: add i440fx fuzz targets
>>> fuzz: add virtio-net fuzz target
>>> fuzz: add documentation to docs/devel/
>>>
>>>Makefile                     |  16 ++-
>>>Makefile.objs                |   4 +
>>>Makefile.target              |  18 ++-
>>>configure                    |  39 ++++++
>>>docs/devel/fuzzing.txt       | 119 ++++++++++++++++++
>>>exec.c                       |  12 +-
>>>include/qemu/module.h        |   4 +-
>>>include/sysemu/qtest.h       |   4 +
>>>include/sysemu/sysemu.h      |   4 +
>>>main.c                       |  52 ++++++++
>>>qtest.c                      |  30 ++++-
>>>tests/Makefile.include       |  75 +++++------
>>>tests/fuzz/Makefile.include  |  11 ++
>>>tests/fuzz/fork_fuzz.c       |  51 ++++++++
>>>tests/fuzz/fork_fuzz.h       |  23 ++++
>>>tests/fuzz/fork_fuzz.ld      |  37 ++++++
>>>tests/fuzz/fuzz.c            | 177 ++++++++++++++++++++++++++
>>>tests/fuzz/fuzz.h            |  66 ++++++++++
>>>tests/fuzz/i440fx_fuzz.c     | 176 ++++++++++++++++++++++++++
>>>tests/fuzz/qos_fuzz.c        | 232 +++++++++++++++++++++++++++++++++++
>>>tests/fuzz/qos_fuzz.h        |  33 +++++
>>>tests/fuzz/virtio_net_fuzz.c | 123 +++++++++++++++++++
>>>tests/libqos/i2c-imx.c       |   8 +-
>>>tests/libqos/i2c-omap.c      |   8 +-
>>>tests/libqos/i2c.c           |  10 +-
>>>tests/libqos/i2c.h           |   4 +-
>>>tests/libqos/qos_external.c  | 168 +++++++++++++++++++++++++
>>>tests/libqos/qos_external.h  |  28 +++++
>>>tests/libqtest.c             | 109 ++++++++++++++--
>>>tests/libqtest.h             |   4 +
>>>tests/pca9552-test.c         |  10 +-
>>>tests/qos-test.c             | 140 +--------------------
>>>util/module.c                |   7 ++
>>>vl.c                         |  36 ++----
>>>34 files changed, 1601 insertions(+), 237 deletions(-)
>>>create mode 100644 docs/devel/fuzzing.txt
>>>create mode 100644 main.c
>>>create mode 100644 tests/fuzz/Makefile.include
>>>create mode 100644 tests/fuzz/fork_fuzz.c
>>>create mode 100644 tests/fuzz/fork_fuzz.h
>>>create mode 100644 tests/fuzz/fork_fuzz.ld
>>>create mode 100644 tests/fuzz/fuzz.c
>>>create mode 100644 tests/fuzz/fuzz.h
>>>create mode 100644 tests/fuzz/i440fx_fuzz.c
>>>create mode 100644 tests/fuzz/qos_fuzz.c
>>>create mode 100644 tests/fuzz/qos_fuzz.h
>>>create mode 100644 tests/fuzz/virtio_net_fuzz.c
>>>create mode 100644 tests/libqos/qos_external.c
>>>create mode 100644 tests/libqos/qos_external.h
>>>
>>>-- 
>>>2.23.0
>>>
>>>
>