This attempts to bring together my understanding of the requirements
for icount behaviour into one reference document for our developer
notes. It currently make one piece of conjecture which I think is true
that we don't need gen_io_start/end statements for non-MMIO related
I/O operations.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Pavel Dovgalyuk <dovgaluk@ispras.ru>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Peter Maydell <peter.maydell@linaro.org>
Message-Id: <20200619135844.23307-1-alex.bennee@linaro.org>

---
v2
  - fix copyright date
  - it's -> its
  - drop mentioned of gen_io_end()
  - remove and correct original conjecture
---
 docs/devel/tcg-icount.rst | 89 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 89 insertions(+)
 create mode 100644 docs/devel/tcg-icount.rst

diff --git a/docs/devel/tcg-icount.rst b/docs/devel/tcg-icount.rst
new file mode 100644
index 00000000000..cb51cb34dde
--- /dev/null
+++ b/docs/devel/tcg-icount.rst
@@ -0,0 +1,89 @@
+..
+   Copyright (c) 2020, Linaro Limited
+   Written by Alex Bennée
+
+
+========================
+TCG Instruction Counting
+========================
+
+TCG has long supported a feature known as icount which allows for
+instruction counting during execution. This should be confused with
+cycle accurate emulation - QEMU does not attempt to emulate how long
+an instruction would take on real hardware. That is a job for other
+more detailed (and slower) tools that simulate the rest of a
+micro-architecture.
+
+This feature is only available for system emulation and is
+incompatible with multi-threaded TCG. It can be used to better align
+execution time with wall-clock time so a "slow" device doesn't run too
+fast on modern hardware. It can also provides for a degree of
+deterministic execution and is an essential part of the record/replay
+support in QEMU.
+
+Core Concepts
+=============
+
+At its heart icount is simply a count of executed instructions which
+is stored in the TimersState of QEMU's timer sub-system. The number of
+executed instructions can then be used to calculate QEMU_CLOCK_VIRTUAL
+which represents the amount of elapsed time in the system since
+execution started. Depending on the icount mode this may either be a
+fixed number of ns per instructions or adjusted as execution continues
+to keep wall clock time and virtual time in sync.
+
+To be able to calculate the number of executed instructions the
+translator starts by allocating a budget of instructions to be
+executed. The budget of instructions is limited by how long it will be
+until the next timer will expire. We store this budget as part of a
+vCPU icount_decr field which shared with the machinery for handling
+cpu_exit(). The whole field is checked at the start of every
+translated block and will cause a return to the outer loop to deal
+with whatever caused the exit.
+
+In the case of icount before the flag is checked we subtract the
+number of instructions the translation block would execute. If this
+would cause the instruction budget to got negative we exit the main
+loop and regenerate a new translation block with exactly the right
+number of instructions to take the budget to 0 meaning whatever timer
+was due to expire will expire exactly when we exit the main run loop.
+
+Dealing with MMIO
+-----------------
+
+While we can adjust the instruction budget for known events like timer
+expiry we can not do the same for MMIO. Every load/store we execute
+might potentially trigger an I/O event at which point we will need an
+up to date and accurate reading of the icount number.
+
+To deal with this case when an I/O access is made we:
+
+  - restore un-executed instructions to the icount budget
+  - re-compile a single [1]_ instruction block for the current PC
+  - exit the cpu loop and execute the re-compiled block
+
+The new block is created with the CF_LAST_IO compile flag which
+ensures the final instruction translation starts with a call to
+gen_io_start() so we don't enter a perpetual loop constantly
+recompiling a single instruction block. For translators using the
+common translator_loop this is done automatically.
+  
+.. [1] sometimes two instructions if dealing with delay slots  
+
+Other I/O operations
+--------------------
+
+MMIO isn't the only type of operation for which we might need a
+correct and accurate clock. IO port instructions and accesses to
+system registers are the common examples here. These instructions have
+to be handled by the individual translators which have the knowledge
+of which operations are I/O operations.
+
+.. warning:: Any instruction that eventually causes an access to
+             QEMU_CLOCK_VIRTUAL needs to be preceded by a
+             gen_io_start() and must also be the last instruction
+             translated in the block.
+   
+
+
+
-- 
2.20.1

Patchew URL: https://patchew.org/QEMU/20200619170930.11704-1-alex.bennee@linaro.org/



Hi,

This series failed the asan build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      stubs/blk-commit-all.o
  CC      stubs/cpu-get-clock.o

Warning, treated as error:
/tmp/qemu-test/src/docs/devel/tcg-icount.rst:document isn't included in any toctree
  CC      stubs/cpu-get-icount.o
  CC      stubs/dump.o
---
  CC      stubs/replay.o
  CC      stubs/runstate-check.o
  CC      stubs/semihost.o
make: *** [Makefile:1088: docs/devel/index.html] Error 2
make: *** Waiting for unfinished jobs....
  CC      stubs/set-fd-handler.o
  CC      stubs/vmgenid.o
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=d091cf91d9d64d0da268b1af62b2e324', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=x86_64-softmmu', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-1grco2tx/src/docker-src.2020-06-19-13.29.03.27165:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-debug']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=d091cf91d9d64d0da268b1af62b2e324
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-1grco2tx/src'
make: *** [docker-run-test-debug@fedora] Error 2

real    4m16.201s
user    0m7.834s


The full log is available at
http://patchew.org/logs/20200619170930.11704-1-alex.bennee@linaro.org/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

On 6/19/20 10:09 AM, Alex Bennée wrote:
> +TCG has long supported a feature known as icount which allows for
> +instruction counting during execution. This should be confused with

should not be

> +cycle accurate emulation - QEMU does not attempt to emulate how long

--- for em-dash, iirc, and (technically) no surrounding spaces.

Otherwise,
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>


r~

Alex Bennée <alex.bennee@linaro.org> writes:

> This attempts to bring together my understanding of the requirements
> for icount behaviour into one reference document for our developer
> notes. It currently make one piece of conjecture which I think is true
> that we don't need gen_io_start/end statements for non-MMIO related
> I/O operations.

Well I should have totally removed that line from the commit message...

-- 
Alex Bennée

Patchew URL: https://patchew.org/QEMU/20200619170930.11704-1-alex.bennee@linaro.org/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      crypto/tlssession.o
  CC      crypto/secret_common.o

Warning, treated as error:
/tmp/qemu-test/src/docs/devel/tcg-icount.rst:document isn't included in any toctree
  CC      crypto/secret.o
  CC      crypto/pbkdf.o
---
  CC      crypto/ivgen.o
  CC      crypto/ivgen-essiv.o
  CC      crypto/ivgen-plain.o
make: *** [Makefile:1088: docs/devel/index.html] Error 2
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 669, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=4b2654e9124546568f87572000aac245', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-1xzpr3pw/src/docker-src.2020-06-19-13.25.26.20175:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=4b2654e9124546568f87572000aac245
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-1xzpr3pw/src'
make: *** [docker-run-test-mingw@fedora] Error 2

real    2m44.263s
user    0m9.068s


The full log is available at
http://patchew.org/logs/20200619170930.11704-1-alex.bennee@linaro.org/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Patchew URL: https://patchew.org/QEMU/20200619170930.11704-1-alex.bennee@linaro.org/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [PATCH v2] docs/devel: add some notes on tcg-icount for developers
Type: series
Message-id: 20200619170930.11704-1-alex.bennee@linaro.org

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]         patchew/20200619171809.30984-1-dgilbert@redhat.com -> patchew/20200619171809.30984-1-dgilbert@redhat.com
Switched to a new branch 'test'
db0d6ec docs/devel: add some notes on tcg-icount for developers

=== OUTPUT BEGIN ===
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#23: 
new file mode 100644

ERROR: trailing whitespace
#97: FILE: docs/devel/tcg-icount.rst:70:
+  $

ERROR: trailing whitespace
#98: FILE: docs/devel/tcg-icount.rst:71:
+.. [1] sometimes two instructions if dealing with delay slots  $

ERROR: trailing whitespace
#113: FILE: docs/devel/tcg-icount.rst:86:
+   $

total: 3 errors, 1 warnings, 89 lines checked

Commit db0d6ec8c02b (docs/devel: add some notes on tcg-icount for developers) has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20200619170930.11704-1-alex.bennee@linaro.org/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com