SDEI is for ARM "Software Delegated Exception Interface". AS ARM64 doesn't have
native non-maskable interrupt (NMI), we rely on higher privileged (larger
exception level) software to change the execution flow of lower privileged
(smaller exception level) software when certain events occur, to emulate NMI
mechanism, and SDEI is the standard interfaces between the two levels of
privileged software. It is based on SMC/HVC calls.

The higher privileged software implements an SDEI dispatcher to handle SDEI
related SMC/HVC calls and trigger SDEI events; the lower privileged software
implements an SDEI client to request SDEI services and handle SDEI events.

Core interfaces provided by SDEI include:

1. interrupt bind: client can request to bind an interrupt to an SDEI event, so
the interrupt will be a non-maskable event and the event number will be returned
to the caller. Only PPI and SPI can be bound to SDEI events.

2. register: client can request to register a handler to an SDEI event, so
dispatcher will change PC of lower privileged software to this handler when
certain event occurs.

3. complete: client notifies dispatcher that it has completed the event
handling, so dispatcher will restore the context of guest when it is
interrupted. 

In virtualization situation, guest OS is the lower privileged software and
hypervisor is the higher one.

KVM is supposed to pass SMC/HVC calls to qemu, and qemu will emulate an SDEI
dispatcher to serve the SDEI requests and trigger the events. If an interrupt is
requested to be bound to an event, qemu should not inject the interrupt to guest
any more; instead, it should save the context of VCPU and change the PC to event
handler which is registered by guest, and then return to guest.

To make the conversion of interrupt to SDEI event transparent to other modules
in qemu, we used qemu_irq and qemu_irq_intercept_in() to override the default
irq handler with SDEI event trigger. I saw qemu_irq_intercept_in() should be
only used in qemu MST, but it seemed fit to override interrupt injection with
event trigger after guest requests to bind interrupt to SDEI event. 

This patchset is trying to implement the whole SDEI framework in qemu with KVM
enabled, including all SDEI v1.0 interfaces, as well as event trigger conduit
from other qemu devices after interrupt binding. 

Key points:
- We propose to only support kvm enabled arm64 virtual machines, for
  non-kvm VMs can emulate EL3 and have Trusted Firmware run on it,
  which has a builtin SDEI dispatcher.
- New kvm capability KVM_CAP_FORWARD_HYPERCALL is added to probe if
  kvm supports forwarding hypercalls, and the capability should be
  enabled explicitly.
- We make the dispatcher as a logical device, to save the states
  during migration or save/restore operation; only one instance is
  allowed in one VM.
- We use qemu_irq as the bridge for other qemu modules to switch from
  irq injection to SDEI event trigger after VM binds the interrupt to
  SDEI event. We use qemu_irq_intercept_in() to override qemu_irq
  handler with SDEI event trigger, and a new interface
  qemu_irq_remove_intercept() is added to restore the handler to
  default one (i.e. ARM GIC).

More details are in the commit message of each patch.

Basic tests are done by emulating a watchdog timer and triggering SDEI
event in every 10s.

Please focus on the interfaces and framework first. We can refine the code for
several rounds after the big things have been determined.

Any comment or suggestion is welcome.

Thanks,

HG

Cc: Peter Maydell <peter.maydell@linaro.org>
Cc: Dave Martin <Dave.Martin@arm.com>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Shannon Zhao <shannon.zhaosl@gmail.com>
Cc: Igor Mammedov <imammedo@redhat.com>

v2:
- Import import linux/arm_sdei.h to standard-headers
- Drop SDEI table definition and add comments
- Some bugfix and code refinement

Heyi Guo (14):
  update-linux-headers.sh: import linux/arm_sdei.h to standard-headers
  standard-headers: import arm_sdei.h
  arm/sdei: add virtual device framework
  arm: add CONFIG_SDEI build flag
  arm/sdei: add support to handle SDEI requests from guest
  arm/sdei: add system reset callback
  arm/sdei: add support to trigger event by GIC interrupt ID
  core/irq: add qemu_irq_remove_intercept interface
  arm/sdei: override qemu_irq handler when binding interrupt
  arm/sdei: add support to register interrupt bind notifier
  linux-headers/kvm.h: add capability to forward hypercall
  arm/sdei: add stub to fix build failure when SDEI is not enabled
  arm/kvm: handle guest exit of hypercall
  virt/acpi: add SDEI table if SDEI is enabled

 default-configs/arm-softmmu.mak           |    1 +
 hw/arm/Kconfig                            |    4 +
 hw/arm/virt-acpi-build.c                  |   26 +
 hw/core/irq.c                             |   11 +
 include/hw/irq.h                          |    8 +-
 include/standard-headers/linux/arm_sdei.h |   73 +
 linux-headers/linux/kvm.h                 |    1 +
 scripts/update-linux-headers.sh           |    1 +
 target/arm/Makefile.objs                  |    4 +
 target/arm/kvm.c                          |   17 +
 target/arm/sdei-stub.c                    |   49 +
 target/arm/sdei.c                         | 1576 +++++++++++++++++++++
 target/arm/sdei.h                         |   60 +
 target/arm/sdei_int.h                     |  121 ++
 14 files changed, 1950 insertions(+), 2 deletions(-)
 create mode 100644 include/standard-headers/linux/arm_sdei.h
 create mode 100644 target/arm/sdei-stub.c
 create mode 100644 target/arm/sdei.c
 create mode 100644 target/arm/sdei.h
 create mode 100644 target/arm/sdei_int.h

-- 
2.19.1

On 2019/11/5 17:10, Heyi Guo wrote:
> SDEI is for ARM "Software Delegated Exception Interface". AS ARM64 doesn't have
> native non-maskable interrupt (NMI), we rely on higher privileged (larger
> exception level) software to change the execution flow of lower privileged
> (smaller exception level) software when certain events occur, to emulate NMI
> mechanism, and SDEI is the standard interfaces between the two levels of
> privileged software. It is based on SMC/HVC calls.
Sorry I forgot to attach the link of SDEI specification v1.0:

http://infocenter.arm.com/help/topic/com.arm.doc.den0054a/ARM_DEN0054A_Software_Delegated_Exception_Interface.pdf

>
> The higher privileged software implements an SDEI dispatcher to handle SDEI
> related SMC/HVC calls and trigger SDEI events; the lower privileged software
> implements an SDEI client to request SDEI services and handle SDEI events.
>
> Core interfaces provided by SDEI include:
>
> 1. interrupt bind: client can request to bind an interrupt to an SDEI event, so
> the interrupt will be a non-maskable event and the event number will be returned
> to the caller. Only PPI and SPI can be bound to SDEI events.
>
> 2. register: client can request to register a handler to an SDEI event, so
> dispatcher will change PC of lower privileged software to this handler when
> certain event occurs.
>
> 3. complete: client notifies dispatcher that it has completed the event
> handling, so dispatcher will restore the context of guest when it is
> interrupted.
>
> In virtualization situation, guest OS is the lower privileged software and
> hypervisor is the higher one.
>
> KVM is supposed to pass SMC/HVC calls to qemu, and qemu will emulate an SDEI
> dispatcher to serve the SDEI requests and trigger the events. If an interrupt is
> requested to be bound to an event, qemu should not inject the interrupt to guest
> any more; instead, it should save the context of VCPU and change the PC to event
> handler which is registered by guest, and then return to guest.
>
> To make the conversion of interrupt to SDEI event transparent to other modules
> in qemu, we used qemu_irq and qemu_irq_intercept_in() to override the default
> irq handler with SDEI event trigger. I saw qemu_irq_intercept_in() should be
> only used in qemu MST, but it seemed fit to override interrupt injection with
> event trigger after guest requests to bind interrupt to SDEI event.
>
> This patchset is trying to implement the whole SDEI framework in qemu with KVM
> enabled, including all SDEI v1.0 interfaces, as well as event trigger conduit
> from other qemu devices after interrupt binding.
>
> Key points:
> - We propose to only support kvm enabled arm64 virtual machines, for
>    non-kvm VMs can emulate EL3 and have Trusted Firmware run on it,
>    which has a builtin SDEI dispatcher.
> - New kvm capability KVM_CAP_FORWARD_HYPERCALL is added to probe if
>    kvm supports forwarding hypercalls, and the capability should be
>    enabled explicitly.
> - We make the dispatcher as a logical device, to save the states
>    during migration or save/restore operation; only one instance is
>    allowed in one VM.
> - We use qemu_irq as the bridge for other qemu modules to switch from
>    irq injection to SDEI event trigger after VM binds the interrupt to
>    SDEI event. We use qemu_irq_intercept_in() to override qemu_irq
>    handler with SDEI event trigger, and a new interface
>    qemu_irq_remove_intercept() is added to restore the handler to
>    default one (i.e. ARM GIC).
>
> More details are in the commit message of each patch.
>
> Basic tests are done by emulating a watchdog timer and triggering SDEI
> event in every 10s.
>
> Please focus on the interfaces and framework first. We can refine the code for
> several rounds after the big things have been determined.
>
> Any comment or suggestion is welcome.
>
> Thanks,
>
> HG
>
> Cc: Peter Maydell <peter.maydell@linaro.org>
> Cc: Dave Martin <Dave.Martin@arm.com>
> Cc: Marc Zyngier <marc.zyngier@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: James Morse <james.morse@arm.com>
> Cc: "Michael S. Tsirkin" <mst@redhat.com>
> Cc: Cornelia Huck <cohuck@redhat.com>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Shannon Zhao <shannon.zhaosl@gmail.com>
> Cc: Igor Mammedov <imammedo@redhat.com>
>
> v2:
> - Import import linux/arm_sdei.h to standard-headers
> - Drop SDEI table definition and add comments
> - Some bugfix and code refinement
>
> Heyi Guo (14):
>    update-linux-headers.sh: import linux/arm_sdei.h to standard-headers
>    standard-headers: import arm_sdei.h
>    arm/sdei: add virtual device framework
>    arm: add CONFIG_SDEI build flag
>    arm/sdei: add support to handle SDEI requests from guest
>    arm/sdei: add system reset callback
>    arm/sdei: add support to trigger event by GIC interrupt ID
>    core/irq: add qemu_irq_remove_intercept interface
>    arm/sdei: override qemu_irq handler when binding interrupt
>    arm/sdei: add support to register interrupt bind notifier
>    linux-headers/kvm.h: add capability to forward hypercall
>    arm/sdei: add stub to fix build failure when SDEI is not enabled
>    arm/kvm: handle guest exit of hypercall
>    virt/acpi: add SDEI table if SDEI is enabled
>
>   default-configs/arm-softmmu.mak           |    1 +
>   hw/arm/Kconfig                            |    4 +
>   hw/arm/virt-acpi-build.c                  |   26 +
>   hw/core/irq.c                             |   11 +
>   include/hw/irq.h                          |    8 +-
>   include/standard-headers/linux/arm_sdei.h |   73 +
>   linux-headers/linux/kvm.h                 |    1 +
>   scripts/update-linux-headers.sh           |    1 +
>   target/arm/Makefile.objs                  |    4 +
>   target/arm/kvm.c                          |   17 +
>   target/arm/sdei-stub.c                    |   49 +
>   target/arm/sdei.c                         | 1576 +++++++++++++++++++++
>   target/arm/sdei.h                         |   60 +
>   target/arm/sdei_int.h                     |  121 ++
>   14 files changed, 1950 insertions(+), 2 deletions(-)
>   create mode 100644 include/standard-headers/linux/arm_sdei.h
>   create mode 100644 target/arm/sdei-stub.c
>   create mode 100644 target/arm/sdei.c
>   create mode 100644 target/arm/sdei.h
>   create mode 100644 target/arm/sdei_int.h
>

Patchew URL: https://patchew.org/QEMU/20191105091056.9541-1-guoheyi@huawei.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=eed29e87c1cd46b29d4327f6effafd5b', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-nkpfmo47/src/docker-src.2019-11-05-04.33.58.25415:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=eed29e87c1cd46b29d4327f6effafd5b
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-nkpfmo47/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    2m10.919s
user    0m8.019s


The full log is available at
http://patchew.org/logs/20191105091056.9541-1-guoheyi@huawei.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Patchew URL: https://patchew.org/QEMU/20191105091056.9541-1-guoheyi@huawei.com/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===

    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=3708b2e272fa453aaa2f6dd31f5c4bff', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-aclbx6j2/src/docker-src.2019-11-05-04.36.51.1877:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=3708b2e272fa453aaa2f6dd31f5c4bff
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-aclbx6j2/src'
make: *** [docker-run-test-mingw@fedora] Error 2

real    1m57.553s
user    0m7.728s


The full log is available at
http://patchew.org/logs/20191105091056.9541-1-guoheyi@huawei.com/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Hi Peter,

Could you spare some time to review the framework and provide comments 
and advice?

Thanks,

HG


On 2019/11/5 17:10, Heyi Guo wrote:
> SDEI is for ARM "Software Delegated Exception Interface". AS ARM64 doesn't have
> native non-maskable interrupt (NMI), we rely on higher privileged (larger
> exception level) software to change the execution flow of lower privileged
> (smaller exception level) software when certain events occur, to emulate NMI
> mechanism, and SDEI is the standard interfaces between the two levels of
> privileged software. It is based on SMC/HVC calls.
>
> The higher privileged software implements an SDEI dispatcher to handle SDEI
> related SMC/HVC calls and trigger SDEI events; the lower privileged software
> implements an SDEI client to request SDEI services and handle SDEI events.
>
> Core interfaces provided by SDEI include:
>
> 1. interrupt bind: client can request to bind an interrupt to an SDEI event, so
> the interrupt will be a non-maskable event and the event number will be returned
> to the caller. Only PPI and SPI can be bound to SDEI events.
>
> 2. register: client can request to register a handler to an SDEI event, so
> dispatcher will change PC of lower privileged software to this handler when
> certain event occurs.
>
> 3. complete: client notifies dispatcher that it has completed the event
> handling, so dispatcher will restore the context of guest when it is
> interrupted.
>
> In virtualization situation, guest OS is the lower privileged software and
> hypervisor is the higher one.
>
> KVM is supposed to pass SMC/HVC calls to qemu, and qemu will emulate an SDEI
> dispatcher to serve the SDEI requests and trigger the events. If an interrupt is
> requested to be bound to an event, qemu should not inject the interrupt to guest
> any more; instead, it should save the context of VCPU and change the PC to event
> handler which is registered by guest, and then return to guest.
>
> To make the conversion of interrupt to SDEI event transparent to other modules
> in qemu, we used qemu_irq and qemu_irq_intercept_in() to override the default
> irq handler with SDEI event trigger. I saw qemu_irq_intercept_in() should be
> only used in qemu MST, but it seemed fit to override interrupt injection with
> event trigger after guest requests to bind interrupt to SDEI event.
>
> This patchset is trying to implement the whole SDEI framework in qemu with KVM
> enabled, including all SDEI v1.0 interfaces, as well as event trigger conduit
> from other qemu devices after interrupt binding.
>
> Key points:
> - We propose to only support kvm enabled arm64 virtual machines, for
>    non-kvm VMs can emulate EL3 and have Trusted Firmware run on it,
>    which has a builtin SDEI dispatcher.
> - New kvm capability KVM_CAP_FORWARD_HYPERCALL is added to probe if
>    kvm supports forwarding hypercalls, and the capability should be
>    enabled explicitly.
> - We make the dispatcher as a logical device, to save the states
>    during migration or save/restore operation; only one instance is
>    allowed in one VM.
> - We use qemu_irq as the bridge for other qemu modules to switch from
>    irq injection to SDEI event trigger after VM binds the interrupt to
>    SDEI event. We use qemu_irq_intercept_in() to override qemu_irq
>    handler with SDEI event trigger, and a new interface
>    qemu_irq_remove_intercept() is added to restore the handler to
>    default one (i.e. ARM GIC).
>
> More details are in the commit message of each patch.
>
> Basic tests are done by emulating a watchdog timer and triggering SDEI
> event in every 10s.
>
> Please focus on the interfaces and framework first. We can refine the code for
> several rounds after the big things have been determined.
>
> Any comment or suggestion is welcome.
>
> Thanks,
>
> HG
>
> Cc: Peter Maydell <peter.maydell@linaro.org>
> Cc: Dave Martin <Dave.Martin@arm.com>
> Cc: Marc Zyngier <marc.zyngier@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: James Morse <james.morse@arm.com>
> Cc: "Michael S. Tsirkin" <mst@redhat.com>
> Cc: Cornelia Huck <cohuck@redhat.com>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Shannon Zhao <shannon.zhaosl@gmail.com>
> Cc: Igor Mammedov <imammedo@redhat.com>
>
> v2:
> - Import import linux/arm_sdei.h to standard-headers
> - Drop SDEI table definition and add comments
> - Some bugfix and code refinement
>
> Heyi Guo (14):
>    update-linux-headers.sh: import linux/arm_sdei.h to standard-headers
>    standard-headers: import arm_sdei.h
>    arm/sdei: add virtual device framework
>    arm: add CONFIG_SDEI build flag
>    arm/sdei: add support to handle SDEI requests from guest
>    arm/sdei: add system reset callback
>    arm/sdei: add support to trigger event by GIC interrupt ID
>    core/irq: add qemu_irq_remove_intercept interface
>    arm/sdei: override qemu_irq handler when binding interrupt
>    arm/sdei: add support to register interrupt bind notifier
>    linux-headers/kvm.h: add capability to forward hypercall
>    arm/sdei: add stub to fix build failure when SDEI is not enabled
>    arm/kvm: handle guest exit of hypercall
>    virt/acpi: add SDEI table if SDEI is enabled
>
>   default-configs/arm-softmmu.mak           |    1 +
>   hw/arm/Kconfig                            |    4 +
>   hw/arm/virt-acpi-build.c                  |   26 +
>   hw/core/irq.c                             |   11 +
>   include/hw/irq.h                          |    8 +-
>   include/standard-headers/linux/arm_sdei.h |   73 +
>   linux-headers/linux/kvm.h                 |    1 +
>   scripts/update-linux-headers.sh           |    1 +
>   target/arm/Makefile.objs                  |    4 +
>   target/arm/kvm.c                          |   17 +
>   target/arm/sdei-stub.c                    |   49 +
>   target/arm/sdei.c                         | 1576 +++++++++++++++++++++
>   target/arm/sdei.h                         |   60 +
>   target/arm/sdei_int.h                     |  121 ++
>   14 files changed, 1950 insertions(+), 2 deletions(-)
>   create mode 100644 include/standard-headers/linux/arm_sdei.h
>   create mode 100644 target/arm/sdei-stub.c
>   create mode 100644 target/arm/sdei.c
>   create mode 100644 target/arm/sdei.h
>   create mode 100644 target/arm/sdei_int.h
>

On Mon, 18 Nov 2019 at 06:55, Guoheyi <guoheyi@huawei.com> wrote:
>
> Hi Peter,
>
> Could you spare some time to review the framework and provide comments
> and advice?

This patchset is on my to-review list but there are also
a lot of others on that list...

thanks
-- PMM

Thanks a lot. Please take your time.

HG


On 2019/11/18 21:35, Peter Maydell wrote:
> On Mon, 18 Nov 2019 at 06:55, Guoheyi <guoheyi@huawei.com> wrote:
>> Hi Peter,
>>
>> Could you spare some time to review the framework and provide comments
>> and advice?
> This patchset is on my to-review list but there are also
> a lot of others on that list...
>
> thanks
> -- PMM
>
> .
>

On Tue, 5 Nov 2019 at 09:12, Heyi Guo <guoheyi@huawei.com> wrote:
> SDEI is for ARM "Software Delegated Exception Interface". AS ARM64 doesn't have
> native non-maskable interrupt (NMI), we rely on higher privileged (larger
> exception level) software to change the execution flow of lower privileged
> (smaller exception level) software when certain events occur, to emulate NMI
> mechanism, and SDEI is the standard interfaces between the two levels of
> privileged software. It is based on SMC/HVC calls.
>
> The higher privileged software implements an SDEI dispatcher to handle SDEI
> related SMC/HVC calls and trigger SDEI events; the lower privileged software
> implements an SDEI client to request SDEI services and handle SDEI events.

Hi; I read through these patches last week, but I didn't reply
then because although there are some aspects to the design that
I don't like, I don't have a clear idea of what a better approach
to the problems it's trying to solve would be. However I didn't
want to go home for the end of the year without providing at
least some response. So I'm going to lay out the parts I have
issues with and perhaps somebody else will have a good idea.

The first part that I dislike here is that this is implementing
an entire ABI which in real hardware is provided by firmware. I
think that QEMU's design works best when QEMU provides emulation of
hardware or hardware-like facilities, which guest code (either
in the kernel, or firmware/bios running in the guest) can then
make use of. Once we start getting into implementing firmware
interfaces directly in QEMU this rapidly becomes a large amount
of work and code, and it's unclear where it should stop. Should
we implement also the equivalent of firmware for omap boards?
For imx* boards? For the raspberry pi? For xilinx boards?
Are we going to end up reimplementing more of ARM Trusted Firmware
functionality inside QEMU? The code to implement firmware-equivalent
ABIs in all these boards would I think quickly become a large part
of the codebase.

My second concern is that to do the things it wants to do,
the implementation here does some pretty invasive things:
 * intercepting interrupt lines which ought to just be
   emulated hardware signals between devices and the GIC
 * capturing register values of other CPUs, and arbitrarily
   stopping those other CPUs and making them run other code
   at a later point in time
I'm really uncomfortable with what's just an 'emulated firmware'
interface for one specific board model doing this kind of thing.

Finally, the stated rationale for the patchset ("we'd like an
emulated NMI equivalent") doesn't really feel to me like it's
strong enough to counterbalance the amount of code here and
the degree to which it's moving us into a swamp I'd prefer
it if we could stay out of.

I'd be much happier with a design where QEMU provides simple
facilities to the guest and the guest firmware and kernel
deal with making use of them. I appreciate that it's not
clear how that would work though, given that in real hardware
this works by the firmware running at EL3 and KVM not
providing a mechanism that allows guest code that runs at
a higher (effective or emulated) privilege level than the
guest kernel...

thanks
-- PMM

Hi Peter,

Really appreciate your comments.

For other platforms/boards emulated in qemu, like omap, imx*, etc, are 
they just TCG platforms? Can we just enable security and EL3 emulation 
for these platforms instead of implementing copies of firmware 
interfaces in qemu? Also I think it is possible to optimize the code to 
support all KVM enabled virtual boards with one single copy of SDEI 
code, so at least the duplication of code inside qemu might be avoided.

I can understand your concerns; the exsiting SDEI code in ARM Trusted 
Firmware also tempted me when I started to writing the code in qemu. I 
agree the ideal way is to use the existing firmware directly, but how 
can we achieve that? Either I don't think it is good to modify the 
firmware code too much, for firmware should be kept simple and reliable.

Does James or Marc have any idea?

Thanks,

Heyi

在 2019/12/20 21:44, Peter Maydell 写道:
> On Tue, 5 Nov 2019 at 09:12, Heyi Guo <guoheyi@huawei.com> wrote:
>> SDEI is for ARM "Software Delegated Exception Interface". AS ARM64 doesn't have
>> native non-maskable interrupt (NMI), we rely on higher privileged (larger
>> exception level) software to change the execution flow of lower privileged
>> (smaller exception level) software when certain events occur, to emulate NMI
>> mechanism, and SDEI is the standard interfaces between the two levels of
>> privileged software. It is based on SMC/HVC calls.
>>
>> The higher privileged software implements an SDEI dispatcher to handle SDEI
>> related SMC/HVC calls and trigger SDEI events; the lower privileged software
>> implements an SDEI client to request SDEI services and handle SDEI events.
> Hi; I read through these patches last week, but I didn't reply
> then because although there are some aspects to the design that
> I don't like, I don't have a clear idea of what a better approach
> to the problems it's trying to solve would be. However I didn't
> want to go home for the end of the year without providing at
> least some response. So I'm going to lay out the parts I have
> issues with and perhaps somebody else will have a good idea.
>
> The first part that I dislike here is that this is implementing
> an entire ABI which in real hardware is provided by firmware. I
> think that QEMU's design works best when QEMU provides emulation of
> hardware or hardware-like facilities, which guest code (either
> in the kernel, or firmware/bios running in the guest) can then
> make use of. Once we start getting into implementing firmware
> interfaces directly in QEMU this rapidly becomes a large amount
> of work and code, and it's unclear where it should stop. Should
> we implement also the equivalent of firmware for omap boards?
> For imx* boards? For the raspberry pi? For xilinx boards?
> Are we going to end up reimplementing more of ARM Trusted Firmware
> functionality inside QEMU? The code to implement firmware-equivalent
> ABIs in all these boards would I think quickly become a large part
> of the codebase.
>
> My second concern is that to do the things it wants to do,
> the implementation here does some pretty invasive things:
>   * intercepting interrupt lines which ought to just be
>     emulated hardware signals between devices and the GIC
>   * capturing register values of other CPUs, and arbitrarily
>     stopping those other CPUs and making them run other code
>     at a later point in time
> I'm really uncomfortable with what's just an 'emulated firmware'
> interface for one specific board model doing this kind of thing.
>
> Finally, the stated rationale for the patchset ("we'd like an
> emulated NMI equivalent") doesn't really feel to me like it's
> strong enough to counterbalance the amount of code here and
> the degree to which it's moving us into a swamp I'd prefer
> it if we could stay out of.
>
> I'd be much happier with a design where QEMU provides simple
> facilities to the guest and the guest firmware and kernel
> deal with making use of them. I appreciate that it's not
> clear how that would work though, given that in real hardware
> this works by the firmware running at EL3 and KVM not
> providing a mechanism that allows guest code that runs at
> a higher (effective or emulated) privilege level than the
> guest kernel...
>
> thanks
> -- PMM
>
> .

Update Marc's email address.

+cc Gavin as he is posting a RFC for ARM NMI.

Hi Marc,

Really sorry for missing to update your email address, for the initial 
topic was raised long time ago and I forgot to update the Cc list in the 
commit message of the patches.

Thanks Gavin for forwarding current discussion on ARM NMI to me.

For you said SDEI is "horrible", does it mean we'd better never 
implement SDEI in virtual world? Or do you have any advice on how to 
implement it?

Thanks,

Heyi

On 2019/12/23 16:20, Guoheyi wrote:
> Hi Peter,
>
> Really appreciate your comments.
>
> For other platforms/boards emulated in qemu, like omap, imx*, etc, are 
> they just TCG platforms? Can we just enable security and EL3 emulation 
> for these platforms instead of implementing copies of firmware 
> interfaces in qemu? Also I think it is possible to optimize the code 
> to support all KVM enabled virtual boards with one single copy of SDEI 
> code, so at least the duplication of code inside qemu might be avoided.
>
> I can understand your concerns; the exsiting SDEI code in ARM Trusted 
> Firmware also tempted me when I started to writing the code in qemu. I 
> agree the ideal way is to use the existing firmware directly, but how 
> can we achieve that? Either I don't think it is good to modify the 
> firmware code too much, for firmware should be kept simple and reliable.
>
> Does James or Marc have any idea?
>
> Thanks,
>
> Heyi
>
> 在 2019/12/20 21:44, Peter Maydell 写道:
>> On Tue, 5 Nov 2019 at 09:12, Heyi Guo <guoheyi@huawei.com> wrote:
>>> SDEI is for ARM "Software Delegated Exception Interface". AS ARM64 
>>> doesn't have
>>> native non-maskable interrupt (NMI), we rely on higher privileged 
>>> (larger
>>> exception level) software to change the execution flow of lower 
>>> privileged
>>> (smaller exception level) software when certain events occur, to 
>>> emulate NMI
>>> mechanism, and SDEI is the standard interfaces between the two 
>>> levels of
>>> privileged software. It is based on SMC/HVC calls.
>>>
>>> The higher privileged software implements an SDEI dispatcher to 
>>> handle SDEI
>>> related SMC/HVC calls and trigger SDEI events; the lower privileged 
>>> software
>>> implements an SDEI client to request SDEI services and handle SDEI 
>>> events.
>> Hi; I read through these patches last week, but I didn't reply
>> then because although there are some aspects to the design that
>> I don't like, I don't have a clear idea of what a better approach
>> to the problems it's trying to solve would be. However I didn't
>> want to go home for the end of the year without providing at
>> least some response. So I'm going to lay out the parts I have
>> issues with and perhaps somebody else will have a good idea.
>>
>> The first part that I dislike here is that this is implementing
>> an entire ABI which in real hardware is provided by firmware. I
>> think that QEMU's design works best when QEMU provides emulation of
>> hardware or hardware-like facilities, which guest code (either
>> in the kernel, or firmware/bios running in the guest) can then
>> make use of. Once we start getting into implementing firmware
>> interfaces directly in QEMU this rapidly becomes a large amount
>> of work and code, and it's unclear where it should stop. Should
>> we implement also the equivalent of firmware for omap boards?
>> For imx* boards? For the raspberry pi? For xilinx boards?
>> Are we going to end up reimplementing more of ARM Trusted Firmware
>> functionality inside QEMU? The code to implement firmware-equivalent
>> ABIs in all these boards would I think quickly become a large part
>> of the codebase.
>>
>> My second concern is that to do the things it wants to do,
>> the implementation here does some pretty invasive things:
>>   * intercepting interrupt lines which ought to just be
>>     emulated hardware signals between devices and the GIC
>>   * capturing register values of other CPUs, and arbitrarily
>>     stopping those other CPUs and making them run other code
>>     at a later point in time
>> I'm really uncomfortable with what's just an 'emulated firmware'
>> interface for one specific board model doing this kind of thing.
>>
>> Finally, the stated rationale for the patchset ("we'd like an
>> emulated NMI equivalent") doesn't really feel to me like it's
>> strong enough to counterbalance the amount of code here and
>> the degree to which it's moving us into a swamp I'd prefer
>> it if we could stay out of.
>>
>> I'd be much happier with a design where QEMU provides simple
>> facilities to the guest and the guest firmware and kernel
>> deal with making use of them. I appreciate that it's not
>> clear how that would work though, given that in real hardware
>> this works by the firmware running at EL3 and KVM not
>> providing a mechanism that allows guest code that runs at
>> a higher (effective or emulated) privilege level than the
>> guest kernel...
>>
>> thanks
>> -- PMM
>>
>> .

Hi Heyi,

On 2020-02-04 08:26, Heyi Guo wrote:
> Update Marc's email address.
> 
> +cc Gavin as he is posting a RFC for ARM NMI.
> 
> Hi Marc,
> 
> Really sorry for missing to update your email address, for the initial
> topic was raised long time ago and I forgot to update the Cc list in
> the commit message of the patches.
> 
> Thanks Gavin for forwarding current discussion on ARM NMI to me.
> 
> For you said SDEI is "horrible", does it mean we'd better never
> implement SDEI in virtual world? Or do you have any advice on how to
> implement it?

My concern is that SDEI implies having EL3. EL3 not being virtualizable
with KVM, you end-up baking SDEI in *hardware*. Of course, this hardware
is actually software (it is QEMU), but this isn't the way it was 
intended.

It's not the first time we've done that (PSCI is another example), but 
the
logic behind SDEI looks much more invasive.

         M.
-- 
Jazz is not dead. It just smells funny...

Hi Marc,

On 2020/2/5 21:15, Marc Zyngier wrote:
> Hi Heyi,
>
> On 2020-02-04 08:26, Heyi Guo wrote:
>> Update Marc's email address.
>>
>> +cc Gavin as he is posting a RFC for ARM NMI.
>>
>> Hi Marc,
>>
>> Really sorry for missing to update your email address, for the initial
>> topic was raised long time ago and I forgot to update the Cc list in
>> the commit message of the patches.
>>
>> Thanks Gavin for forwarding current discussion on ARM NMI to me.
>>
>> For you said SDEI is "horrible", does it mean we'd better never
>> implement SDEI in virtual world? Or do you have any advice on how to
>> implement it?
>
> My concern is that SDEI implies having EL3. EL3 not being virtualizable
> with KVM, you end-up baking SDEI in *hardware*. Of course, this hardware
> is actually software (it is QEMU), but this isn't the way it was 
> intended.

>
> It's not the first time we've done that (PSCI is another example), but 
> the
> logic behind SDEI looks much more invasive.

Thanks for your comments.

Thinking about them for quite a while, below is my understanding, please 
correct me if I'm wrong:

So should the KVM based virtual machine be treated as one with CPUs only 
having NS-EL1 and NS-EL0, ideally? And SDEI messes up this model, isn't it?

PSCI only contains some one-shot operations, so it is much less invasive 
than SDEI.


I've another question. The origin of "virtual" SDEI requirement comes 
from the lack of hard lockup detector in VM. We can have some kind of 
watchdog, but how can the watchdog trigger the VM OS to panic and run 
kdump, even in irq-off state?

Thanks,

Heyi

>
>         M.

On 2020-02-06 01:20, Heyi Guo wrote:
> Hi Marc,
> 
> On 2020/2/5 21:15, Marc Zyngier wrote:
>> Hi Heyi,
>> 
>> On 2020-02-04 08:26, Heyi Guo wrote:
>>> Update Marc's email address.
>>> 
>>> +cc Gavin as he is posting a RFC for ARM NMI.
>>> 
>>> Hi Marc,
>>> 
>>> Really sorry for missing to update your email address, for the 
>>> initial
>>> topic was raised long time ago and I forgot to update the Cc list in
>>> the commit message of the patches.
>>> 
>>> Thanks Gavin for forwarding current discussion on ARM NMI to me.
>>> 
>>> For you said SDEI is "horrible", does it mean we'd better never
>>> implement SDEI in virtual world? Or do you have any advice on how to
>>> implement it?
>> 
>> My concern is that SDEI implies having EL3. EL3 not being 
>> virtualizable
>> with KVM, you end-up baking SDEI in *hardware*. Of course, this 
>> hardware
>> is actually software (it is QEMU), but this isn't the way it was 
>> intended.
> 
>> 
>> It's not the first time we've done that (PSCI is another example), but 
>> the
>> logic behind SDEI looks much more invasive.
> 
> Thanks for your comments.
> 
> Thinking about them for quite a while, below is my understanding,
> please correct me if I'm wrong:
> 
> So should the KVM based virtual machine be treated as one with CPUs
> only having NS-EL1 and NS-EL0, ideally? And SDEI messes up this model,
> isn't it?

Well, that's exactly what it is (until we have nested virt, in which 
case
you will be able to add NS-EL2 to the mix).

> PSCI only contains some one-shot operations, so it is much less
> invasive than SDEI.
> 
> 
> I've another question. The origin of "virtual" SDEI requirement comes
> from the lack of hard lockup detector in VM.

Sure. But nothing guarantees that the guest is going to register a SDEI
entry point anyway.

> We can have some kind of
> watchdog, but how can the watchdog trigger the VM OS to panic and run
> kdump, even in irq-off state?

Nothing. All the events, including SDEI, are maskable, one way or 
another.

Gavin's approach to inject a SError is probably OK for Linux, given that
it tends to run with PSTATE.A==0. But that's not a guarantee either (if
you take a recursive exception, SError won't be delivered).

The long and the short of it is that there is no way to do what you want
with absolute guarantees on the ARM architecture. It just doesn't exist.

         M.
-- 
Jazz is not dead. It just smells funny...

Hi guys,

On 06/02/2020 17:30, Marc Zyngier wrote:
> On 2020-02-06 01:20, Heyi Guo wrote:
>> On 2020/2/5 21:15, Marc Zyngier wrote:
>>> My concern is that SDEI implies having EL3. EL3 not being virtualizable
>>> with KVM, you end-up baking SDEI in *hardware*. Of course, this hardware
>>> is actually software (it is QEMU), but this isn't the way it was intended.
>>
>>>
>>> It's not the first time we've done that (PSCI is another example), but the
>>> logic behind SDEI looks much more invasive.
>>
>> Thanks for your comments.
>>
>> Thinking about them for quite a while, below is my understanding,
>> please correct me if I'm wrong:
>>
>> So should the KVM based virtual machine be treated as one with CPUs
>> only having NS-EL1 and NS-EL0, ideally? And SDEI messes up this model,
>> isn't it?
> 
> Well, that's exactly what it is (until we have nested virt, in which case
> you will be able to add NS-EL2 to the mix).
> 
>> PSCI only contains some one-shot operations, so it is much less
>> invasive than SDEI.

Is there an established pattern for how Qemu 'gets' things that are done in secure-world?
For PSCI the kernel does it, but this obviously doesn't scale to something like OP-TEE.

Ideally we'd get the reference implementation (from ATF) in some form that is easy to use...


>> I've another question. The origin of "virtual" SDEI requirement comes
>> from the lack of hard lockup detector in VM.

(this is your use case. Its origin was just symmetry with EL3<->EL2)


> Sure. But nothing guarantees that the guest is going to register a SDEI
> entry point anyway.

>> We can have some kind of
>> watchdog, but how can the watchdog trigger the VM OS to panic and run
>> kdump, even in irq-off state?
> 
> Nothing. All the events, including SDEI, are maskable, one way or another.
> 
> Gavin's approach to inject a SError is probably OK for Linux, given that
> it tends to run with PSTATE.A==0. But that's not a guarantee either (if
> you take a recursive exception, SError won't be delivered).

Or get stuck in debug-state (for which we mask SError), power-management, the vectors or
somewhere weird, like KVM's world-switch.


If you just want to kill the OS if its sort-of-alive, there is another trick:

Synchronous exceptions can't be masked because they are caused by the instruction pointed
to by the ELR. You can't inject an emulated data-abort unless the ELR points to an
instruction that accesses memory, but...

synchronous external abort for instruction fetch is something that  could happen at any
time. If you have v8.2 you can make the severity uncontainable for extra points.

On real hardware, this would be as if this instruction missed in the i-cache, then got an
abort from the PoU-cache. The PoU-cache must have suffered some metadata corruption to
report an uncontained error. On real hardware its very likely the next instruction would
suffer the same fate, but linux should put up a good show of trying to panic().


> The long and the short of it is that there is no way to do what you want
> with absolute guarantees on the ARM architecture. It just doesn't exist.

Yes. By sort-of-alive it needs to be making some kind of progress. If the CPU is spinning
through the vectors (because some joker unmapped them), all bets are off.


Thanks,

James

On 2020/2/7 18:52, James Morse wrote:
> Hi guys,
>
> On 06/02/2020 17:30, Marc Zyngier wrote:
>> On 2020-02-06 01:20, Heyi Guo wrote:
>>> On 2020/2/5 21:15, Marc Zyngier wrote:
>>>> My concern is that SDEI implies having EL3. EL3 not being virtualizable
>>>> with KVM, you end-up baking SDEI in *hardware*. Of course, this hardware
>>>> is actually software (it is QEMU), but this isn't the way it was intended.
>>>> It's not the first time we've done that (PSCI is another example), but the
>>>> logic behind SDEI looks much more invasive.
>>> Thanks for your comments.
>>>
>>> Thinking about them for quite a while, below is my understanding,
>>> please correct me if I'm wrong:
>>>
>>> So should the KVM based virtual machine be treated as one with CPUs
>>> only having NS-EL1 and NS-EL0, ideally? And SDEI messes up this model,
>>> isn't it?
>> Well, that's exactly what it is (until we have nested virt, in which case
>> you will be able to add NS-EL2 to the mix).
>>
>>> PSCI only contains some one-shot operations, so it is much less
>>> invasive than SDEI.
> Is there an established pattern for how Qemu 'gets' things that are done in secure-world?
> For PSCI the kernel does it, but this obviously doesn't scale to something like OP-TEE.
>
> Ideally we'd get the reference implementation (from ATF) in some form that is easy to use...
>
>
>>> I've another question. The origin of "virtual" SDEI requirement comes
>>> from the lack of hard lockup detector in VM.
> (this is your use case. Its origin was just symmetry with EL3<->EL2)
>
>
>> Sure. But nothing guarantees that the guest is going to register a SDEI
>> entry point anyway.
>>> We can have some kind of
>>> watchdog, but how can the watchdog trigger the VM OS to panic and run
>>> kdump, even in irq-off state?
>> Nothing. All the events, including SDEI, are maskable, one way or another.
>>
>> Gavin's approach to inject a SError is probably OK for Linux, given that
>> it tends to run with PSTATE.A==0. But that's not a guarantee either (if
>> you take a recursive exception, SError won't be delivered).
> Or get stuck in debug-state (for which we mask SError), power-management, the vectors or
> somewhere weird, like KVM's world-switch.
>
>
> If you just want to kill the OS if its sort-of-alive, there is another trick:
>
> Synchronous exceptions can't be masked because they are caused by the instruction pointed
> to by the ELR. You can't inject an emulated data-abort unless the ELR points to an
> instruction that accesses memory, but...
>
> synchronous external abort for instruction fetch is something that  could happen at any
> time. If you have v8.2 you can make the severity uncontainable for extra points.
>
> On real hardware, this would be as if this instruction missed in the i-cache, then got an
> abort from the PoU-cache. The PoU-cache must have suffered some metadata corruption to
> report an uncontained error. On real hardware its very likely the next instruction would
> suffer the same fate, but linux should put up a good show of trying to panic().

Good idea. It seems to be able to cover all the possible the cases, 
while we only need to highlight this exception in the user document :)

Thanks,

Heyi

>
>
>> The long and the short of it is that there is no way to do what you want
>> with absolute guarantees on the ARM architecture. It just doesn't exist.
> Yes. By sort-of-alive it needs to be making some kind of progress. If the CPU is spinning
> through the vectors (because some joker unmapped them), all bets are off.
>
>
> Thanks,
>
> James
>
> .

On Fri, 7 Feb 2020 at 10:52, James Morse <james.morse@arm.com> wrote:
> Is there an established pattern for how Qemu 'gets' things that are done in secure-world?
> For PSCI the kernel does it, but this obviously doesn't scale to something like OP-TEE.

The answer broadly is "it doesn't get them", the same way
that a real hardware implementation that provides only EL1
and EL0 has no way to get them. Ideally there would be an
architecturally provided way to run a virtualized EL3 in the
guest so it could run (a QEMU-aware flavour of) firmware...

thanks
-- PMM

On 2020/2/7 1:30, Marc Zyngier wrote:
> On 2020-02-06 01:20, Heyi Guo wrote:
>> Hi Marc,
>>
>> On 2020/2/5 21:15, Marc Zyngier wrote:
>>> Hi Heyi,
>>>
>>> On 2020-02-04 08:26, Heyi Guo wrote:
>>>> Update Marc's email address.
>>>>
>>>> +cc Gavin as he is posting a RFC for ARM NMI.
>>>>
>>>> Hi Marc,
>>>>
>>>> Really sorry for missing to update your email address, for the initial
>>>> topic was raised long time ago and I forgot to update the Cc list in
>>>> the commit message of the patches.
>>>>
>>>> Thanks Gavin for forwarding current discussion on ARM NMI to me.
>>>>
>>>> For you said SDEI is "horrible", does it mean we'd better never
>>>> implement SDEI in virtual world? Or do you have any advice on how to
>>>> implement it?
>>>
>>> My concern is that SDEI implies having EL3. EL3 not being virtualizable
>>> with KVM, you end-up baking SDEI in *hardware*. Of course, this 
>>> hardware
>>> is actually software (it is QEMU), but this isn't the way it was 
>>> intended.
>>
>>>
>>> It's not the first time we've done that (PSCI is another example), 
>>> but the
>>> logic behind SDEI looks much more invasive.
>>
>> Thanks for your comments.
>>
>> Thinking about them for quite a while, below is my understanding,
>> please correct me if I'm wrong:
>>
>> So should the KVM based virtual machine be treated as one with CPUs
>> only having NS-EL1 and NS-EL0, ideally? And SDEI messes up this model,
>> isn't it?
>
> Well, that's exactly what it is (until we have nested virt, in which case
> you will be able to add NS-EL2 to the mix).
>
>> PSCI only contains some one-shot operations, so it is much less
>> invasive than SDEI.
>>
>>
>> I've another question. The origin of "virtual" SDEI requirement comes
>> from the lack of hard lockup detector in VM.
>
> Sure. But nothing guarantees that the guest is going to register a SDEI
> entry point anyway.
>
>> We can have some kind of
>> watchdog, but how can the watchdog trigger the VM OS to panic and run
>> kdump, even in irq-off state?
>
> Nothing. All the events, including SDEI, are maskable, one way or 
> another.

Indeed...

>
> Gavin's approach to inject a SError is probably OK for Linux, given that
> it tends to run with PSTATE.A==0. But that's not a guarantee either (if
> you take a recursive exception, SError won't be delivered).
>
> The long and the short of it is that there is no way to do what you want
> with absolute guarantees on the ARM architecture. It just doesn't exist.

Much appreciate your explanation and advice.

Thanks,

Heyi

>
>         M.