[Qemu-devel] [PATCH v2 0/6] s390x/vfio: VFIO-AP interrupt control interception

Pierre Morel posted 6 patches 5 years, 4 months ago
Test asan passed
Test checkpatch passed
Test docker-quick@centos7 passed
Test docker-mingw@fedora passed
Test docker-clang@ubuntu passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/1542904555-1136-1-git-send-email-pmorel@linux.ibm.com
hw/s390x/ap-bridge.c            |  12 ++++
hw/s390x/ap-device.c            |  22 +++++++
hw/vfio/ap.c                    | 131 +++++++++++++++++++++++++++++++++++++---
include/hw/s390x/ap-bridge.h    |   1 +
include/hw/s390x/ap-device.h    |  65 ++++++++++++++++++++
include/hw/s390x/css.h          |   1 +
linux-headers/linux/vfio.h      |  25 ++++++++
target/s390x/cpu_features.c     |   1 +
target/s390x/cpu_features_def.h |   1 +
target/s390x/cpu_models.c       |   1 +
target/s390x/gen-features.c     |   1 +
target/s390x/kvm.c              |  31 ++++++++++
12 files changed, 282 insertions(+), 10 deletions(-)
[Qemu-devel] [PATCH v2 0/6] s390x/vfio: VFIO-AP interrupt control interception
Posted by Pierre Morel 5 years, 4 months ago
This series has 3 different type of patches:

The first two:
  s390x/vfio: ap: Finding the AP bridge
  s390x/vfio: ap: Use the APdevice as a child of the APBus

Are dealing with the QEMU Object Model and how we retrieve the
AP devices from instruction interception.
A lifting of the AP bridge, bus and device was necessary to
ease the process and allow future extensions.

The third one is a place holder to ease reviewing:
  s390x/vfio: ap: Linux uapi VFIO place holder

The last three are really dealing with PQAP/AQIC instruction
interception and associate IOCTL calls to the VFIO AP device
driver.
  s390x/cpumodel: Set up CPU model for AQIC interception
  s390x/vfio: ap: Definition for AP Adapter type
  s390x/vfio: ap: Implementing AP Queue Interrupt Control

The S390 APQP/AQIC instruction is intercepted by the host
to configure the AP queues interruption redirection.
It retrieves the ISC used by the host and the one used
by the guest and setup the indicator address.

This patch series
- define the AQIC feature in the cpumodel,
- extend the APDevice type for per card and queue interrupt handling,
- intercept the APQP/AQIC instruction, uses the S390 adapter interface
  to setup the adapter 
- and use a VFIO ioctl to let the VFIO-AP driver handle the host
  instruction associated with the intercepted guest instruction.

This patch serie can be tested with the Linux/KVM patch series
for the VFIO-AP driver: "s390: vfio: ap: Using GISA for AP Interrupt"


Pierre Morel (6):
  s390x/vfio: ap: Finding the AP bridge
  s390x/vfio: ap: Use the APdevice as a child of the APBus
  s390x/vfio: ap: Linux uapi VFIO place holder
  s390x/cpumodel: Set up CPU model for AQIC interception
  s390x/vfio: ap: Definition for AP Adapter type
  s390x/vfio: ap: Implementing AP Queue Interrupt Control

 hw/s390x/ap-bridge.c            |  12 ++++
 hw/s390x/ap-device.c            |  22 +++++++
 hw/vfio/ap.c                    | 131 +++++++++++++++++++++++++++++++++++++---
 include/hw/s390x/ap-bridge.h    |   1 +
 include/hw/s390x/ap-device.h    |  65 ++++++++++++++++++++
 include/hw/s390x/css.h          |   1 +
 linux-headers/linux/vfio.h      |  25 ++++++++
 target/s390x/cpu_features.c     |   1 +
 target/s390x/cpu_features_def.h |   1 +
 target/s390x/cpu_models.c       |   1 +
 target/s390x/gen-features.c     |   1 +
 target/s390x/kvm.c              |  31 ++++++++++
 12 files changed, 282 insertions(+), 10 deletions(-)

-- 
2.7.4

Changelog from v1:
- legitimate "Huh" from Conny on global lead me to reconsider
  the QOM for AP Device, BUS and Bridge and remove globals
- squash patches for kvm.c and hw/vfio/ap.c
  (Conny) so related changes are easier to see
- modified the UAPI (Also comes from Linux)
- suppressed complicated structures to integer conversions
  and use masks to avoid endianess considerations,
  I did not change anything for endianess since we get all
  from the registers, AFAIU we do not need to.
  Tell me if I am wrong (I know you will :) )
- simplified the implementation of the ioctl

Tested with according Linux patch.
Take care that the UAPI changed betwen V1 and V2!
and do NOT use with the older version.



Re: [Qemu-devel] [PATCH v2 0/6] s390x/vfio: VFIO-AP interrupt control interception
Posted by Halil Pasic 5 years, 3 months ago
On Thu, 22 Nov 2018 17:35:49 +0100
Pierre Morel <pmorel@linux.ibm.com> wrote:

> This series has 3 different type of patches:
> 
> The first two:
>   s390x/vfio: ap: Finding the AP bridge
>   s390x/vfio: ap: Use the APdevice as a child of the APBus
> 
> Are dealing with the QEMU Object Model and how we retrieve the
> AP devices from instruction interception.
> A lifting of the AP bridge, bus and device was necessary to
> ease the process and allow future extensions.
> 
> The third one is a place holder to ease reviewing:
>   s390x/vfio: ap: Linux uapi VFIO place holder
> 
> The last three are really dealing with PQAP/AQIC instruction
> interception and associate IOCTL calls to the VFIO AP device
> driver.
>   s390x/cpumodel: Set up CPU model for AQIC interception
>   s390x/vfio: ap: Definition for AP Adapter type
>   s390x/vfio: ap: Implementing AP Queue Interrupt Control
> 
> The S390 APQP/AQIC instruction is intercepted by the host
> to configure the AP queues interruption redirection.
> It retrieves the ISC used by the host and the one used
> by the guest and setup the indicator address.
> 
> This patch series
> - define the AQIC feature in the cpumodel,
> - extend the APDevice type for per card and queue interrupt handling,
> - intercept the APQP/AQIC instruction, uses the S390 adapter interface
>   to setup the adapter 
> - and use a VFIO ioctl to let the VFIO-AP driver handle the host
>   instruction associated with the intercepted guest instruction.
> 
> This patch serie can be tested with the Linux/KVM patch series
> for the VFIO-AP driver: "s390: vfio: ap: Using GISA for AP Interrupt"
> 

Sorry for raising concern this late, I hope it's better late than
never.

I have strong doubts that handling PQAP/AQCI via userspace is worth
the effort. IMHO we could do what we have to do on AQCI in kernel
iff the ap is done SIE interpreted, the appropriate feature is presented
to the guest, and the queue in question belongs to the given guest. Or
am I wrong?

I do understand that doing it like this *may* end up being beneficial
*if* we decide to do some sort of ap virtualization in QEMU. But I don't
see it coming in the foreseeable future, and for ap virtualization we
would need a solution for making the host do an NQAP and an DQAP on
behalf of the guest/emulator, and not only to do the same for PQAP/QCI.

In my understanding, with this, we would end up with an infrastructure
that only makes sense in a perspective of some 'future features' which
may never come to existence.

What I ask for is, please, let us examine the other option.

Regards,
Halil


> 
> Pierre Morel (6):
>   s390x/vfio: ap: Finding the AP bridge
>   s390x/vfio: ap: Use the APdevice as a child of the APBus
>   s390x/vfio: ap: Linux uapi VFIO place holder
>   s390x/cpumodel: Set up CPU model for AQIC interception
>   s390x/vfio: ap: Definition for AP Adapter type
>   s390x/vfio: ap: Implementing AP Queue Interrupt Control
> 
>  hw/s390x/ap-bridge.c            |  12 ++++
>  hw/s390x/ap-device.c            |  22 +++++++
>  hw/vfio/ap.c                    | 131 +++++++++++++++++++++++++++++++++++++---
>  include/hw/s390x/ap-bridge.h    |   1 +
>  include/hw/s390x/ap-device.h    |  65 ++++++++++++++++++++
>  include/hw/s390x/css.h          |   1 +
>  linux-headers/linux/vfio.h      |  25 ++++++++
>  target/s390x/cpu_features.c     |   1 +
>  target/s390x/cpu_features_def.h |   1 +
>  target/s390x/cpu_models.c       |   1 +
>  target/s390x/gen-features.c     |   1 +
>  target/s390x/kvm.c              |  31 ++++++++++
>  12 files changed, 282 insertions(+), 10 deletions(-)
> 


Re: [Qemu-devel] [PATCH v2 0/6] s390x/vfio: VFIO-AP interrupt control interception
Posted by Pierre Morel 5 years, 3 months ago
On 29/11/2018 16:55, Halil Pasic wrote:
> On Thu, 22 Nov 2018 17:35:49 +0100
> Pierre Morel <pmorel@linux.ibm.com> wrote:
> 
>> This series has 3 different type of patches:
>>
>> The first two:
>>    s390x/vfio: ap: Finding the AP bridge
>>    s390x/vfio: ap: Use the APdevice as a child of the APBus
>>
>> Are dealing with the QEMU Object Model and how we retrieve the
>> AP devices from instruction interception.
>> A lifting of the AP bridge, bus and device was necessary to
>> ease the process and allow future extensions.
>>
>> The third one is a place holder to ease reviewing:
>>    s390x/vfio: ap: Linux uapi VFIO place holder
>>
>> The last three are really dealing with PQAP/AQIC instruction
>> interception and associate IOCTL calls to the VFIO AP device
>> driver.
>>    s390x/cpumodel: Set up CPU model for AQIC interception
>>    s390x/vfio: ap: Definition for AP Adapter type
>>    s390x/vfio: ap: Implementing AP Queue Interrupt Control
>>
>> The S390 APQP/AQIC instruction is intercepted by the host
>> to configure the AP queues interruption redirection.
>> It retrieves the ISC used by the host and the one used
>> by the guest and setup the indicator address.
>>
>> This patch series
>> - define the AQIC feature in the cpumodel,
>> - extend the APDevice type for per card and queue interrupt handling,
>> - intercept the APQP/AQIC instruction, uses the S390 adapter interface
>>    to setup the adapter
>> - and use a VFIO ioctl to let the VFIO-AP driver handle the host
>>    instruction associated with the intercepted guest instruction.
>>
>> This patch serie can be tested with the Linux/KVM patch series
>> for the VFIO-AP driver: "s390: vfio: ap: Using GISA for AP Interrupt"
>>
> 
> Sorry for raising concern this late, I hope it's better late than
> never.
> 
> I have strong doubts that handling PQAP/AQCI via userspace is worth
> the effort. IMHO we could do what we have to do on AQCI in kernel
> iff the ap is done SIE interpreted, the appropriate feature is presented
> to the guest, and the queue in question belongs to the given guest. Or
> am I wrong?
> 
> I do understand that doing it like this *may* end up being beneficial
> *if* we decide to do some sort of ap virtualization in QEMU. But I don't
> see it coming in the foreseeable future, and for ap virtualization we
> would need a solution for making the host do an NQAP and an DQAP on
> behalf of the guest/emulator, and not only to do the same for PQAP/QCI.
> 
> In my understanding, with this, we would end up with an infrastructure
> that only makes sense in a perspective of some 'future features' which
> may never come to existence.
> 
> What I ask for is, please, let us examine the other option.
> 
> Regards,
> Halil
> 
> 

As we discussed offline, I already began to implement prototypes for 
both options.
This is a clear design choice.

If we examine the pro and contra, I can list:

1- Pro kernel implementation of the PQAP/AQIC
-> rapidity of the reaction

Question: is this important?
Answer: NO,
Why: The PQAP/AQIC is rarely used by the AP driver of the guest. 
exclusively during RESET of the AP queue.
I do not think we need a rapid reaction there.


2- Pro userland implementation of PQAP/AQIC
-> standard implementation, already used by PCI, CCW

Question: is it important?
Answer: YES
Why:  like following the standard
It is easily extend-able to other virtualization implementation like 
interception based VFIO and emulation


There is no implementation which would be really more complicated as the 
other, for both we will need to introduce new pro APQN (queue) 
structures to hold the interrupt information (ISC, NIB), for both we 
will need to ping the NIB in memory.

So as long as there are no other opinion against the design I presented 
here I will continue this way while considering the comments I got on 
this series.

Regards,
Pierre


-- 
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany


Re: [Qemu-devel] [PATCH v2 0/6] s390x/vfio: VFIO-AP interrupt control interception
Posted by Halil Pasic 5 years, 3 months ago
On Fri, 30 Nov 2018 14:01:42 +0100
Pierre Morel <pmorel@linux.ibm.com> wrote:

> On 29/11/2018 16:55, Halil Pasic wrote:
> > On Thu, 22 Nov 2018 17:35:49 +0100
> > Pierre Morel <pmorel@linux.ibm.com> wrote:
> > 
> >> This series has 3 different type of patches:
> >>
> >> The first two:
> >>    s390x/vfio: ap: Finding the AP bridge
> >>    s390x/vfio: ap: Use the APdevice as a child of the APBus
> >>
> >> Are dealing with the QEMU Object Model and how we retrieve the
> >> AP devices from instruction interception.
> >> A lifting of the AP bridge, bus and device was necessary to
> >> ease the process and allow future extensions.
> >>
> >> The third one is a place holder to ease reviewing:
> >>    s390x/vfio: ap: Linux uapi VFIO place holder
> >>
> >> The last three are really dealing with PQAP/AQIC instruction
> >> interception and associate IOCTL calls to the VFIO AP device
> >> driver.
> >>    s390x/cpumodel: Set up CPU model for AQIC interception
> >>    s390x/vfio: ap: Definition for AP Adapter type
> >>    s390x/vfio: ap: Implementing AP Queue Interrupt Control
> >>
> >> The S390 APQP/AQIC instruction is intercepted by the host
> >> to configure the AP queues interruption redirection.
> >> It retrieves the ISC used by the host and the one used
> >> by the guest and setup the indicator address.
> >>
> >> This patch series
> >> - define the AQIC feature in the cpumodel,
> >> - extend the APDevice type for per card and queue interrupt handling,
> >> - intercept the APQP/AQIC instruction, uses the S390 adapter interface
> >>    to setup the adapter
> >> - and use a VFIO ioctl to let the VFIO-AP driver handle the host
> >>    instruction associated with the intercepted guest instruction.
> >>
> >> This patch serie can be tested with the Linux/KVM patch series
> >> for the VFIO-AP driver: "s390: vfio: ap: Using GISA for AP Interrupt"
> >>
> > 
> > Sorry for raising concern this late, I hope it's better late than
> > never.
> > 
> > I have strong doubts that handling PQAP/AQCI via userspace is worth
> > the effort. IMHO we could do what we have to do on AQCI in kernel
> > iff the ap is done SIE interpreted, the appropriate feature is presented
> > to the guest, and the queue in question belongs to the given guest. Or
> > am I wrong?
> > 
> > I do understand that doing it like this *may* end up being beneficial
> > *if* we decide to do some sort of ap virtualization in QEMU. But I don't
> > see it coming in the foreseeable future, and for ap virtualization we
> > would need a solution for making the host do an NQAP and an DQAP on
> > behalf of the guest/emulator, and not only to do the same for PQAP/QCI.
> > 
> > In my understanding, with this, we would end up with an infrastructure
> > that only makes sense in a perspective of some 'future features' which
> > may never come to existence.
> > 
> > What I ask for is, please, let us examine the other option.
> > 
> > Regards,
> > Halil
> > 
> > 
> 
> As we discussed offline, I already began to implement prototypes for 
> both options.
> This is a clear design choice.
> 
> If we examine the pro and contra, I can list:
> 
> 1- Pro kernel implementation of the PQAP/AQIC
> -> rapidity of the reaction
> 
> Question: is this important?
> Answer: NO,
> Why: The PQAP/AQIC is rarely used by the AP driver of the guest. 
> exclusively during RESET of the AP queue.
> I do not think we need a rapid reaction there.
> 
> 
> 2- Pro userland implementation of PQAP/AQIC
> -> standard implementation, already used by PCI, CCW
> 
> Question: is it important?
> Answer: YES
> Why:  like following the standard
> It is easily extend-able to other virtualization implementation like 
> interception based VFIO and emulation
> 
> 
> There is no implementation which would be really more complicated as the 
> other, for both we will need to introduce new pro APQN (queue) 
> structures to hold the interrupt information (ISC, NIB), for both we 
> will need to ping the NIB in memory.
> 
> So as long as there are no other opinion against the design I presented 
> here I will continue this way while considering the comments I got on 
> this series.
> 

I'm a bit confused. Your first sentence reads like you in a process of
providing a kernel-heavy version. Your last sentence however reads like,
based on the discussion following the first sentence, you decided to not
explore, if the kernel-heavy variant (we still need a cpu model feature
in QEMU) is simpler.

Frankly, my feeling was that a kernel heavy implementation can be done
with less lines of code (considering QEMU and Linux) and without
introducing new ioctl interfaces. I may be wrong. You certainly seem to
dispute that a kernel-heavy implementation is likely to be simpler. If
you don't want to explore that option, I would like to ask you for your
permission to do it myself if my time allows.

Regards,
Halil