When the VM exits with an data abort, we check the ISV field in the ESR and when
ISV=1, that means the processor has filled the remaining fields with information
needed to determine the access that caused the abort: address, access width, and
the register operand. However, only a limited set of instructions which can
cause a data abort is nice enough for the processor to decode this way. Many
instructions such as LDP/STP and SIMD can cause an data abort with ISV=0 and for
that the hypervisor needs to manually decode the instruction, find the operands,
and emulate the access.

QEMU already ships with the ability to do this: TCG. However, TCG currently
operates as a stand-alone accelerator. This patch set enables HVF to call into
TCG when needed in order to perform a memory access that caused the abort.

One thing this enables is the ability to use virtio-vga with Windows for ARM64.
Currently, graphics support for Windows is flakey because you must first boot
with ramfb to get to the desktop where you can then install the virtio-gpu
drivers and then start up with virtio-gpu. Even then, there is a known issue
where Windows mistakingly thinks there are two monitors connected because the
boot display does not share a framebuffer with the GPU. This results in
sometimes a black screen when updating Windows.

Another issue is that the TPM driver uses LDP/STP to access the command buffer
and so the QEMU device which maps registers as MMIO will not work.

There are a couple major issues with the patch as it currently stands. First of
all, it is very slow. Because we do not track writes to code pages, to be safe
we flush TLBs and TBs every time we switch to emulation mode. We also need to
synchronize the register states between HVF and QEMU each time we enter and
exit emulation mode. Since we enter/exit emulation mode for every instruction
that causes the data abort, in the case of the VGA buffer being cleared in a
loop, this means we need to enter-exit emulation mode to execute a single
instruction for every pixel. Second, we don't support plugins at all. Lastly,
some of the CPU state used by TCG is not properly synchronized with HVF and so
subtle issues can occur. We may want to constrain the emulator to only run with
a known allowlist of instructions we wish to handle in a data abort.

I think these issues can be worked around but I want to know if people think
this approach is worth doing or if instead we should pursue alternatives such
as a more basic instruction decoder which only supports a subset of instructions
which are interesting for memory accesses.

Joelle van Dyne (4):
  cpu-exec: support single-step without debug
  cpu-target: support emulation from non-TCG accels
  hvf: arm: emulate instruction when ISV=0
  hw/arm/virt: enable VGA

 include/exec/cpu-common.h |   1 +
 include/hw/core/cpu.h     |  11 +++++
 include/system/hvf_int.h  |   2 +-
 target/arm/hvf_arm.h      |   5 ++
 target/arm/internals.h    |   3 +-
 accel/hvf/hvf-accel-ops.c |   2 +-
 accel/tcg/cpu-exec.c      |  35 +++++++++----
 accel/tcg/plugin-gen.c    |   4 ++
 accel/tcg/tb-maint.c      |   2 +-
 accel/tcg/tcg-accel-ops.c |   3 +-
 cpu-target.c              |  20 +++++++-
 plugins/core.c            |  12 +++++
 system/physmem.c          |   7 +--
 target/arm/hvf/hvf.c      | 100 ++++++++++++++++++++++++++++++++++++--
 target/i386/hvf/hvf.c     |   2 +-
 hw/arm/Kconfig            |   1 +
 16 files changed, 186 insertions(+), 24 deletions(-)

-- 
2.41.0

On Sun, 9 Feb 2025 at 03:33, Joelle van Dyne <j@getutm.app> wrote:
>
> When the VM exits with an data abort, we check the ISV field in the ESR and when
> ISV=1, that means the processor has filled the remaining fields with information
> needed to determine the access that caused the abort: address, access width, and
> the register operand. However, only a limited set of instructions which can
> cause a data abort is nice enough for the processor to decode this way. Many
> instructions such as LDP/STP and SIMD can cause an data abort with ISV=0 and for
> that the hypervisor needs to manually decode the instruction, find the operands,
> and emulate the access.
>
> QEMU already ships with the ability to do this: TCG. However, TCG currently
> operates as a stand-alone accelerator. This patch set enables HVF to call into
> TCG when needed in order to perform a memory access that caused the abort.

So one problem with this is that it immediately puts all of TCG onto
the security boundary with the VM. We don't claim any kind of security
or can't-escape guarantees for TCG, and it's a lot of code, some of
which is old and some of which wasn't written with security as
a top-of-mind concern.

Our approach to these instructions with KVM on Arm is to say "don't
do those in the guest to MMIO regions". Most sensible guest code
doesn't do weird instruction forms for device accesses, and the
performance is going to be bad anyway if you need to fully emulate them.
(This includes in the past that Windows got fixed to not do this
kind of insn to a device in at least one case.)

> One thing this enables is the ability to use virtio-vga with Windows for ARM64.
> Currently, graphics support for Windows is flakey because you must first boot
> with ramfb to get to the desktop where you can then install the virtio-gpu
> drivers and then start up with virtio-gpu. Even then, there is a known issue
> where Windows mistakingly thinks there are two monitors connected because the
> boot display does not share a framebuffer with the GPU. This results in
> sometimes a black screen when updating Windows.

It's not really a good idea to use virtio-vga in an Arm VM,
because it requires FEAT_S2FWB in the host CPU to make it
work properly, and not every CPU has that, at least in the
KVM world. So you need to use virtio-gpu anyhow.

thanks
-- PMM