MemoryRegionCache was reverted to "normal" address_space_* operations
for 2.9, due to lack of support for IOMMUs.  This series reinstates
optimizations, caching only the IOMMU translation but not the IOMMU
lookup and target AddressSpace translation.

Patches 1 to 3 prepare by introducing a new function
address_space_translate_iommu (split out of address_space_translate)
and new header files for declarations shared by regular and "cached"
address_space_* functions.  Patch 4 uses them to introduce both
the slow path for IOMMU and MMIO cached regions, and the fast path
for RAM cached regions.

Paolo

Paolo Bonzini (4):
  exec: move memory access declarations to a common header, inline
    *_phys functions
  exec: small changes to flatview_do_translate
  exec: extract address_space_translate_iommu, fix page_mask corner case
  exec: reintroduce MemoryRegion caching

 exec.c                                | 245 ++++++++++++++++++++------
 include/exec/cpu-all.h                |  79 ++++-----
 include/exec/memory-internal.h        |   3 +
 include/exec/memory.h                 | 209 ++++++++++------------
 include/exec/memory_ldst.inc.h        |  71 ++++++++
 include/exec/memory_ldst_cached.inc.h | 108 ++++++++++++
 include/exec/memory_ldst_phys.inc.h   | 147 ++++++++++++++++
 memory.c                              |   4 +-
 memory_ldst.inc.c                     | 126 -------------
 9 files changed, 651 insertions(+), 341 deletions(-)
 create mode 100644 include/exec/memory_ldst.inc.h
 create mode 100644 include/exec/memory_ldst_cached.inc.h
 create mode 100644 include/exec/memory_ldst_phys.inc.h

-- 
2.17.0

Hi Paolo,

On 04/17/2018 04:07 PM, Paolo Bonzini wrote:
> MemoryRegionCache was reverted to "normal" address_space_* operations
> for 2.9, due to lack of support for IOMMUs.  This series reinstates
> optimizations, caching only the IOMMU translation but not the IOMMU
> lookup and target AddressSpace translation.
> 
> Patches 1 to 3 prepare by introducing a new function
> address_space_translate_iommu (split out of address_space_translate)
> and new header files for declarations shared by regular and "cached"
> address_space_* functions.  Patch 4 uses them to introduce both
> the slow path for IOMMU and MMIO cached regions, and the fast path
> for RAM cached regions.

This patch seems to cause a regression with ARM vsmmu + virtio-blk-pci. Reverting it looks to fix the issue. Otherwise I get:

Program received signal SIGSEGV, Segmentation fault.
address_space_lduw_internal_cached_slow (endian=DEVICE_LITTLE_ENDIAN, result=0x0, attrs=..., addr=2, cache=0xffffa81110a0) at /home/augere/UPSTREAM/qemu/memory_ldst.inc.c:242
242	            val = lduw_le_p(ptr);
(gdb) where
#0  address_space_lduw_internal_cached_slow (endian=DEVICE_LITTLE_ENDIAN, result=0x0, attrs=..., addr=2, cache=0xffffa81110a0) at /home/augere/UPSTREAM/qemu/memory_ldst.inc.c:242
#1  address_space_lduw_le_cached_slow (cache=0xffffa81110a0, addr=addr@entry=2, attrs=..., attrs@entry=..., result=0x0) at /home/augere/UPSTREAM/qemu/memory_ldst.inc.c:273
#2  0x0000000000511c74 in address_space_lduw_le_cached (result=0x0, attrs=..., addr=2, cache=<optimized out>) at /home/augere/UPSTREAM/qemu/include/exec/memory_ldst_cached.inc.h:56
#3  lduw_le_phys_cached (addr=2, cache=<optimized out>) at /home/augere/UPSTREAM/qemu/include/exec/memory_ldst_phys.inc.h:91
#4  virtio_lduw_phys_cached (pa=2, cache=<optimized out>, vdev=<optimized out>) at /home/augere/UPSTREAM/qemu/include/hw/virtio/virtio-access.h:166
#5  vring_avail_idx (vq=0x1cfe2d0) at /home/augere/UPSTREAM/qemu/hw/virtio/virtio.c:227
#6  virtio_queue_set_notification (vq=0x1cfe2d0, enable=0) at /home/augere/UPSTREAM/qemu/hw/virtio/virtio.c:324
#7  0x0000000000511d2c in virtio_queue_set_notification (vq=<optimized out>, enable=<optimized out>) at /home/augere/UPSTREAM/qemu/hw/virtio/virtio.c:318
#8  0x00000000004aa158 in virtio_blk_handle_vq (s=0x1cf32b0, vq=0x1cfe2d0) at /home/augere/UPSTREAM/qemu/hw/block/virtio-blk.c:605
#9  0x00000000005113f0 in virtio_queue_notify_aio_vq (vq=0x1cfe2d0) at /home/augere/UPSTREAM/qemu/hw/virtio/virtio.c:1515
#10 0x000000000087cbf0 in aio_dispatch_handlers (ctx=ctx@entry=0x15300e0) at util/aio-posix.c:406
#11 0x000000000087d3f8 in aio_dispatch (ctx=0x15300e0) at util/aio-posix.c:437
#12 0x0000000000879f30 in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>) at util/async.c:261
#13 0x0000ffffbeee97a0 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
#14 0x000000000087c678 in glib_pollfds_poll () at util/main-loop.c:215
#15 os_host_main_loop_wait (timeout=<optimized out>) at util/main-loop.c:263
#16 main_loop_wait (nonblocking=nonblocking@entry=0) at util/main-loop.c:522
#17 0x0000000000423dd8 in main_loop () at vl.c:1943
#18 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4675

Investigating further ...

Thanks

Eric
> 
> Paolo
> 
> Paolo Bonzini (4):
>   exec: move memory access declarations to a common header, inline
>     *_phys functions
>   exec: small changes to flatview_do_translate
>   exec: extract address_space_translate_iommu, fix page_mask corner case
>   exec: reintroduce MemoryRegion caching
> 
>  exec.c                                | 245 ++++++++++++++++++++------
>  include/exec/cpu-all.h                |  79 ++++-----
>  include/exec/memory-internal.h        |   3 +
>  include/exec/memory.h                 | 209 ++++++++++------------
>  include/exec/memory_ldst.inc.h        |  71 ++++++++
>  include/exec/memory_ldst_cached.inc.h | 108 ++++++++++++
>  include/exec/memory_ldst_phys.inc.h   | 147 ++++++++++++++++
>  memory.c                              |   4 +-
>  memory_ldst.inc.c                     | 126 -------------
>  9 files changed, 651 insertions(+), 341 deletions(-)
>  create mode 100644 include/exec/memory_ldst.inc.h
>  create mode 100644 include/exec/memory_ldst_cached.inc.h
>  create mode 100644 include/exec/memory_ldst_phys.inc.h
> 

On 16/05/2018 12:41, Auger Eric wrote:
> This patch seems to cause a regression with ARM vsmmu +
> virtio-blk-pci. Reverting it looks to fix the issue. Otherwise I
> get:

What's the command line for a reproducer?

Thanks,

Paolo

Hi Paolo

On 05/16/2018 12:42 PM, Paolo Bonzini wrote:
> On 16/05/2018 12:41, Auger Eric wrote:
>> This patch seems to cause a regression with ARM vsmmu +
>> virtio-blk-pci. Reverting it looks to fix the issue. Otherwise I
>> get:
> 
> What's the command line for a reproducer?
> 
> Thanks,
> 
> Paolo
> 

here is my command line:

TCG:

sudo /home/augere/UPSTREAM/qemu/aarch64-softmmu/qemu-system-aarch64 -M virt-2.12,gic-version=3,iommu=smmuv3 \
-cpu cortex-a57 -smp 8 -m 4096 -display none -machine accel=tcg \
-serial tcp:localhost:4444,server -trace events=/home/augere/TEST/QEMU/hw-arm-smmu \
-qmp unix:/home/augere/TEST/QEMU/qmp-sock,server,nowait \
-device virtio-blk-pci,bus=pcie.0,scsi=off,drive=drv0,id=drv0,bootindex=1,iommu_platform,disable-modern=off,disable-legacy=on,werror=stop,rerror=stop \
-drive file=/home/augere/VM/IMAGES/vm0.qcow2,format=qcow2,if=none,id=drv0 \
-device virtio-net-pci,bus=pcie.0,netdev=nic0,mac=6a:f5:10:b1:3d:d2,iommu_platform,disable-modern=off,disable-legacy=on \
-netdev tap,id=nic0,script=/home/augere/TEST/SCRIPTS/qemu-ifup,downscript=/home/augere/TEST/SCRIPTS/qemu-ifdown \
-net none

or with acceleration:

sudo /home/augere/UPSTREAM/qemu/aarch64-softmmu/qemu-system-aarch64 -M virt-2.12,gic-version=3,iommu=smmuv3 -cpu host -smp 8 -m 4096 -display none --enable-kvm ../..

I reproduce with both, 100% of the cases.

Trying to reproduce with intel iommu as well.

Thanks

Eric

Hi Paolo,

On 05/16/2018 03:38 PM, Auger Eric wrote:
> Hi Paolo
> 
> On 05/16/2018 12:42 PM, Paolo Bonzini wrote:
>> On 16/05/2018 12:41, Auger Eric wrote:
>>> This patch seems to cause a regression with ARM vsmmu +
>>> virtio-blk-pci. Reverting it looks to fix the issue. Otherwise I
>>> get:
>>
>> What's the command line for a reproducer?
>>
>> Thanks,
>>
>> Paolo
>>
> 
> here is my command line:
> 
> TCG:
> 
> sudo /home/augere/UPSTREAM/qemu/aarch64-softmmu/qemu-system-aarch64 -M virt-2.12,gic-version=3,iommu=smmuv3 \
> -cpu cortex-a57 -smp 8 -m 4096 -display none -machine accel=tcg \
> -serial tcp:localhost:4444,server -trace events=/home/augere/TEST/QEMU/hw-arm-smmu \
> -qmp unix:/home/augere/TEST/QEMU/qmp-sock,server,nowait \
> -device virtio-blk-pci,bus=pcie.0,scsi=off,drive=drv0,id=drv0,bootindex=1,iommu_platform,disable-modern=off,disable-legacy=on,werror=stop,rerror=stop \
> -drive file=/home/augere/VM/IMAGES/vm0.qcow2,format=qcow2,if=none,id=drv0 \
> -device virtio-net-pci,bus=pcie.0,netdev=nic0,mac=6a:f5:10:b1:3d:d2,iommu_platform,disable-modern=off,disable-legacy=on \
> -netdev tap,id=nic0,script=/home/augere/TEST/SCRIPTS/qemu-ifup,downscript=/home/augere/TEST/SCRIPTS/qemu-ifdown \
> -net none
> 
> or with acceleration:
> 
> sudo /home/augere/UPSTREAM/qemu/aarch64-softmmu/qemu-system-aarch64 -M virt-2.12,gic-version=3,iommu=smmuv3 -cpu host -smp 8 -m 4096 -display none --enable-kvm ../..
> 
> I reproduce with both, 100% of the cases.
> 
> Trying to reproduce with intel iommu as well.

Yep I just reproduced on x86 as well. Here is the used cmd line.

/home/augere/UPSTREAM/qemu/x86_64-softmmu/qemu-system-x86_64 \
-M q35,accel=kvm,usb=off,dump-guest-core=off -cpu Haswell,-hle,-rtm -smp 4,sockets=4,cores=1,threads=1 -m 8192 \
-display none --enable-kvm -serial tcp:localhost:4444,server -qmp unix:/home/augere/TEST/QEMU/qmp-sock,server,nowait -rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay -realtime mlock=off -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 \
-boot strict=on -machine kernel_irqchip=split -device ioh3420,chassis=1,addr=4,id=pcie.1 -device ioh3420,chassis=2,addr=5,id=pcie.2 \
-device intel-iommu,intremap=on \
-device virtio-blk-pci,scsi=off,drive=drv0,id=virtio-disk0,bootindex=1,iommu_platform,disable-modern=off,disable-legacy=on \
-drive file=/home/augere/VM/IMAGES/vm0.qcow2,format=qcow2,if=none,id=drv0 \
-device virtio-net-pci,netdev=nic0,mac=6a:f5:10:b1:3d:d2,iommu_platform,disable-modern=off,disable-legacy=on \
-netdev tap,id=nic0,script=/home/augere/TEST/SCRIPTS/qemu-ifup,downscript=/home/augere/TEST/SCRIPTS/qemu-ifdown \
-net none

Program received signal SIGSEGV, Segmentation fault.
address_space_lduw_internal_cached_slow (endian=DEVICE_LITTLE_ENDIAN, result=<optimized out>, attrs=...,
    addr=2, cache=<optimized out>) at /home/augere/UPSTREAM/qemu/memory_ldst.inc.c:242
242	            val = lduw_le_p(ptr);
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 celt051-0.5.1.3-8.el7.x86_64 cyrus-sasl-lib-2.1.26-23.el7.x86_64 elfutils-libelf-0.170-4.el7.x86_64
elfutils-libs-0.170-4.el7.x86_64 glibc-2.17-222.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-18.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-52.el7.x86_64
libcacard-2.5.2-2.el7.x86_64 libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-11.el7.x86_64 libffi-3.0.13-18.el7.x86_64 libgcc-4.8.5-28.el7.x86_64 libjpeg-turbo-1.2.90-5.el7.x86_64
libmount-2.23.2-52.el7.x86_64 libselinux-2.5-12.el7.x86_64 libstdc++-4.8.5-28.el7.x86_64 libusbx-1.0.21-1.el7.x86_64 libuuid-2.23.2-52.el7.x86_64 lz4-1.7.5-2.el7.x86_64
ncurses-libs-5.9-14.20130511.el7_4.x86_64 nspr-4.17.0-1.el7.x86_64 nss-3.34.0-4.el7.x86_64 nss-softokn-freebl-3.34.0-2.el7.x86_64 nss-util-3.34.0-2.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64
openssl-libs-1.0.2k-12.el7.x86_64 opus-1.0.2-6.el7.x86_64 pcre-8.32-17.el7.x86_64 pixman-0.34.0-1.el7.x86_64 spice-server-0.14.0-2.el7.x86_64 systemd-libs-219-57.el7.x86_64 usbredir-0.7.1-3.el7.x86_64
xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) where
#0  0x000055555580e773 in address_space_lduw_le_cached_slow (endian=
    DEVICE_LITTLE_ENDIAN, result=<optimized out>, attrs=..., addr=2, cache=<optimized out>)
    at /home/augere/UPSTREAM/qemu/memory_ldst.inc.c:242
#1  0x000055555580e773 in address_space_lduw_le_cached_slow (cache=<optimized out>, addr=addr@entry=2, attrs=..., attrs@entry=..., result=result@entry=0x0) at
/home/augere/UPSTREAM/qemu/memory_ldst.inc.c:273
#2  0x00005555558b0a9b in virtio_queue_set_notification (result=0x0, attrs=..., addr=2, cache=<optimized out>) at /home/augere/UPSTREAM/qemu/include/exec/memory_ldst_cached.inc.h:56
#3  0x00005555558b0a9b in virtio_queue_set_notification (addr=2, cache=<optimized out>)
    at /home/augere/UPSTREAM/qemu/include/exec/memory_ldst_phys.inc.h:91
#4  0x00005555558b0a9b in virtio_queue_set_notification (vdev=<optimized out>, pa=2, cache=<optimized out>) at /home/augere/UPSTREAM/qemu/include/hw/virtio/virtio-access.h:166
#5  0x00005555558b0a9b in virtio_queue_set_notification (vq=0x7fffec260010)
    at /home/augere/UPSTREAM/qemu/hw/virtio/virtio.c:227
#6  0x00005555558b0a9b in virtio_queue_set_notification (vq=vq@entry=0x7fffec260010, enable=enable@entry=0) at /home/augere/UPSTREAM/qemu/hw/virtio/virtio.c:324
#7  0x00005555558b0b12 in virtio_queue_set_notification (vq=vq@entry=0x7fffec260010, enable=enable@entry=0) at /home/augere/UPSTREAM/qemu/hw/virtio/virtio.c:318
#8  0x0000555555880fea in virtio_blk_handle_vq (s=0x555557dc9660, vq=0x7fffec260010)
    at /home/augere/UPSTREAM/qemu/hw/block/virtio-blk.c:605
#9  0x00005555558b0680 in virtio_queue_notify_aio_vq (vq=0x7fffec260010)
    at /home/augere/UPSTREAM/qemu/hw/virtio/virtio.c:1515
#10 0x0000555555bef7c8 in aio_dispatch_handlers (ctx=ctx@entry=0x555556a35ce0) at util/aio-posix.c:406
#11 0x0000555555bf0068 in aio_dispatch (ctx=0x555556a35ce0) at util/aio-posix.c:437
#12 0x0000555555becbce in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>) at util/async.c:261
#13 0x00007ffff79058f9 in g_main_context_dispatch (context=0x555556a360b0) at gmain.c:3146
#14 0x00007ffff79058f9 in g_main_context_dispatch (context=context@entry=0x555556a360b0) at gmain.c:3811
#15 0x0000555555bef326 in main_loop_wait () at util/main-loop.c:215
#16 0x0000555555bef326 in main_loop_wait (timeout=<optimized out>) at util/main-loop.c:263
#17 0x0000555555bef326 in main_loop_wait (nonblocking=nonblocking@entry=0) at util/main-loop.c:522
#18 0x00005555557fdb1f in main () at vl.c:1943
#19 0x00005555557fdb1f in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
    at vl.c:4675
(gdb) quit


Thanks

Eric

> 
> Thanks
> 
> Eric
>