docs/system/s390x/cpu-topology.rst | 87 ++++++++++ docs/system/target-s390x.rst | 1 + include/hw/s390x/cpu-topology.h | 52 ++++++ include/hw/s390x/s390-virtio-ccw.h | 6 + target/s390x/cpu.h | 78 +++++++++ target/s390x/kvm/kvm_s390x.h | 1 + hw/s390x/cpu-topology.c | 261 +++++++++++++++++++++++++++++ hw/s390x/s390-virtio-ccw.c | 7 + target/s390x/cpu-sysemu.c | 21 +++ target/s390x/cpu_models.c | 1 + target/s390x/kvm/cpu_topology.c | 186 ++++++++++++++++++++ target/s390x/kvm/kvm.c | 46 ++++- hw/s390x/meson.build | 1 + target/s390x/kvm/meson.build | 3 +- 14 files changed, 749 insertions(+), 2 deletions(-) create mode 100644 docs/system/s390x/cpu-topology.rst create mode 100644 include/hw/s390x/cpu-topology.h create mode 100644 hw/s390x/cpu-topology.c create mode 100644 target/s390x/kvm/cpu_topology.c
Hi, Implementation discussions ========================== CPU models ---------- Since the S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model for old QEMU we could not activate it as usual from KVM but needed a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. Checking and enabling this capability enables S390_FEAT_CONFIGURATION_TOPOLOGY. Migration --------- Once the S390_FEAT_CONFIGURATION_TOPOLOGY is enabled in the source host the STFL(11) is provided to the guest. Since the feature is already in the CPU model of older QEMU, a migration from a new QEMU enabling the topology to an old QEMU will keep STFL(11) enabled making the guest get an exception for illegal operation as soon as it uses the PTF instruction. A VMState keeping track of the S390_FEAT_CONFIGURATION_TOPOLOGY allows to forbid the migration in such a case. Note that the VMState will be used to hold information on the topology once we implement topology change for a running guest. Topology -------- Until we introduce bookss and drawers, polarization and dedication the topology is kept very simple and is specified uniquely by the core_id of the vCPU which is also the vCPU address. Testing ======= To use the QEMU patches, you will need Linux V6-rc1 or newer, or use the following Linux mainline patches: f5ecfee94493 2022-07-20 KVM: s390: resetting the Topology-Change-Report 24fe0195bc19 2022-07-20 KVM: s390: guest support for topology function 0130337ec45b 2022-07-20 KVM: s390: Cleanup ipte lock access and SIIF fac.. Currently this code is for KVM only, I have no idea if it is interesting to provide a TCG patch. If ever it will be done in another series. Documentation ============= To have a better understanding of the S390x CPU Topology and its implementation in QEMU you can have a look at the documentation in the last patch of this series. The admin will want to match the host and the guest topology, taking into account that the guest does not recognize multithreading. Consequently, two vCPU assigned to threads of the same real CPU should preferably be assigned to the same socket of the guest machine. Future developments =================== Two series are actively prepared: - Adding drawers, book, polarization and dedication to the vCPU. - changing the topology with a running guest Regards, Pierre Pierre Morel (7): s390x/cpu topology: Creating CPU topology device s390x/cpu topology: reporting the CPU topology to the guest s390x/cpu_topology: resetting the Topology-Change-Report s390x/cpu_topology: CPU topology migration s390x/cpu_topology: interception of PTF instruction s390x/cpu_topology: activating CPU topology docs/s390x: document s390x cpu topology docs/system/s390x/cpu-topology.rst | 87 ++++++++++ docs/system/target-s390x.rst | 1 + include/hw/s390x/cpu-topology.h | 52 ++++++ include/hw/s390x/s390-virtio-ccw.h | 6 + target/s390x/cpu.h | 78 +++++++++ target/s390x/kvm/kvm_s390x.h | 1 + hw/s390x/cpu-topology.c | 261 +++++++++++++++++++++++++++++ hw/s390x/s390-virtio-ccw.c | 7 + target/s390x/cpu-sysemu.c | 21 +++ target/s390x/cpu_models.c | 1 + target/s390x/kvm/cpu_topology.c | 186 ++++++++++++++++++++ target/s390x/kvm/kvm.c | 46 ++++- hw/s390x/meson.build | 1 + target/s390x/kvm/meson.build | 3 +- 14 files changed, 749 insertions(+), 2 deletions(-) create mode 100644 docs/system/s390x/cpu-topology.rst create mode 100644 include/hw/s390x/cpu-topology.h create mode 100644 hw/s390x/cpu-topology.c create mode 100644 target/s390x/kvm/cpu_topology.c -- 2.31.1 - since v12 - suppress new CPU flag "disable-topology" just use ctop - no use of special fields in CCW machine or in CPU - modifications in documentation - insert documentation in tree (Cedric) - moved cpu-topology.c from target/s390 to target/s390/kvm to compile smoothly (without topology) for TCG (Cedric) - since v11 - new CPU flag "disable-topology" I would have take "topology" if I was able to have it false on default. (Christian, Thomas) - Build the topology during the interception of the STSI instruction. (Cedric) - return CC3 in case the calculated SYSIB length is greater than 4096. (Janis) - minor corections on documentation - since v10 - change machine attribute "topology-disable" to "topology" (Cedric) - Add preliminary patch for machine properties (Cedric) - Use next machine as 7.2 (Cedric / Connie) - Remove unecessary mutex (Thomas) - use ENOTSUP return value for kvm_s390_topology_set_mtcr() (Cedric) - Add explanation on container and cpu TLEs (Thomas) - use again cpu and socket count in topology structure (Cedric) - Suppress the S390TopoTLE structure and integrate the TLE masks to the socket structure. (-) - the STSI instruction now finds the topology from the machine (Cedric) - since v9 - remove books and drawers - remove thread denying and replace with a merge of cores * threads to specify the CPUs available to the guest - add a class option to avoid topology on older machines (Cedric) - Allocate a SYSIB buffer of the maximal length to avoid overflow. (Nico, Janis) - suppress redundancy of smp parameters in topology and use directly the machine smp structure - Early check for topology support (Cedric) - since v8 - Linux patches are now mainline - simplification of the implementation (Janis) - Migration, new machine definition (Thomas) - Documentation - since v7 - Coherence with the Linux patch series changes for MTCR get (Pierre) - check return values during new CPU creation (Thomas) - Improving codding style and argument usages (Thomas) - since v6 - Changes on smp args in qemu-options (Daniel) - changed comments in machine.jason (Daniel) - Added reset (Janosch) - since v5 - rebasing on newer QEMU version - reworked most lines above 80 characters. - since v4 - Added drawer and books to topology - Added numa topology - Added documentation - since v3 - Added migration (Thomas) - Separated STSI instruction from KVM to prepare TCG (Thomas) - Take care of endianess to prepare TCG (Thomas) - Added comments on STSI CPU container and PFT instruction (Thomas) - Moved enabling the instructions as the last patch (Thomas)
On 08/12/2022 10.44, Pierre Morel wrote: > Hi, > > Implementation discussions > ========================== > > CPU models > ---------- > > Since the S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model > for old QEMU we could not activate it as usual from KVM but needed > a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. > Checking and enabling this capability enables > S390_FEAT_CONFIGURATION_TOPOLOGY. > > Migration > --------- > > Once the S390_FEAT_CONFIGURATION_TOPOLOGY is enabled in the source > host the STFL(11) is provided to the guest. > Since the feature is already in the CPU model of older QEMU, > a migration from a new QEMU enabling the topology to an old QEMU > will keep STFL(11) enabled making the guest get an exception for > illegal operation as soon as it uses the PTF instruction. I now thought that it is not possible to enable "ctop" on older QEMUs since the don't enable the KVM capability? ... or is it still somehow possible? What did I miss? Thomas > A VMState keeping track of the S390_FEAT_CONFIGURATION_TOPOLOGY > allows to forbid the migration in such a case. > > Note that the VMState will be used to hold information on the > topology once we implement topology change for a running guest.
On 12/9/22 14:32, Thomas Huth wrote: > On 08/12/2022 10.44, Pierre Morel wrote: >> Hi, >> >> Implementation discussions >> ========================== >> >> CPU models >> ---------- >> >> Since the S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model >> for old QEMU we could not activate it as usual from KVM but needed >> a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. >> Checking and enabling this capability enables >> S390_FEAT_CONFIGURATION_TOPOLOGY. >> >> Migration >> --------- >> >> Once the S390_FEAT_CONFIGURATION_TOPOLOGY is enabled in the source >> host the STFL(11) is provided to the guest. >> Since the feature is already in the CPU model of older QEMU, >> a migration from a new QEMU enabling the topology to an old QEMU >> will keep STFL(11) enabled making the guest get an exception for >> illegal operation as soon as it uses the PTF instruction. > > I now thought that it is not possible to enable "ctop" on older QEMUs > since the don't enable the KVM capability? ... or is it still somehow > possible? What did I miss? > > Thomas Enabling ctop with ctop=on on old QEMU is not possible, this is right. But, if STFL(11) is enable in the source KVM by a new QEMU, I can see that even with -ctop=off the STFL(11) is migrated to the destination. It is highly possible that I missed something in the cpu model. A solution proposed by Cedric was to add a new machine but we did not want this because we decided that we do not want to wait for a new machine. Another solution could be to have a we can have a new CPU feature overruling ctop like S390_FEAT_CPU_TOPOLOGY in the last series version 12. I am not sure it must be linked with the creation of a new machine. The solution here in this series is to add a VMState which will block the migration with older QEMU if the topology is activated with ctop on a new QEMU. Regards, Pierre > > >> A VMState keeping track of the S390_FEAT_CONFIGURATION_TOPOLOGY >> allows to forbid the migration in such a case. >> >> Note that the VMState will be used to hold information on the >> topology once we implement topology change for a running guest. > -- Pierre Morel IBM Lab Boeblingen
On 12/12/2022 09.51, Pierre Morel wrote: > > > On 12/9/22 14:32, Thomas Huth wrote: >> On 08/12/2022 10.44, Pierre Morel wrote: >>> Hi, >>> >>> Implementation discussions >>> ========================== >>> >>> CPU models >>> ---------- >>> >>> Since the S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model >>> for old QEMU we could not activate it as usual from KVM but needed >>> a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. >>> Checking and enabling this capability enables >>> S390_FEAT_CONFIGURATION_TOPOLOGY. >>> >>> Migration >>> --------- >>> >>> Once the S390_FEAT_CONFIGURATION_TOPOLOGY is enabled in the source >>> host the STFL(11) is provided to the guest. >>> Since the feature is already in the CPU model of older QEMU, >>> a migration from a new QEMU enabling the topology to an old QEMU >>> will keep STFL(11) enabled making the guest get an exception for >>> illegal operation as soon as it uses the PTF instruction. >> >> I now thought that it is not possible to enable "ctop" on older QEMUs >> since the don't enable the KVM capability? ... or is it still somehow >> possible? What did I miss? >> >> Thomas > > Enabling ctop with ctop=on on old QEMU is not possible, this is right. > But, if STFL(11) is enable in the source KVM by a new QEMU, I can see that > even with -ctop=off the STFL(11) is migrated to the destination. Is this with the "host" CPU model or another one? And did you explicitly specify "ctop=off" at the command line, or are you just using the default setting by not specifying it? Thomas
On 12/12/22 10:07, Thomas Huth wrote:
> On 12/12/2022 09.51, Pierre Morel wrote:
>>
>>
>> On 12/9/22 14:32, Thomas Huth wrote:
>>> On 08/12/2022 10.44, Pierre Morel wrote:
>>>> Hi,
>>>>
>>>> Implementation discussions
>>>> ==========================
>>>>
>>>> CPU models
>>>> ----------
>>>>
>>>> Since the S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model
>>>> for old QEMU we could not activate it as usual from KVM but needed
>>>> a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY.
>>>> Checking and enabling this capability enables
>>>> S390_FEAT_CONFIGURATION_TOPOLOGY.
>>>>
>>>> Migration
>>>> ---------
>>>>
>>>> Once the S390_FEAT_CONFIGURATION_TOPOLOGY is enabled in the source
>>>> host the STFL(11) is provided to the guest.
>>>> Since the feature is already in the CPU model of older QEMU,
>>>> a migration from a new QEMU enabling the topology to an old QEMU
>>>> will keep STFL(11) enabled making the guest get an exception for
>>>> illegal operation as soon as it uses the PTF instruction.
>>>
>>> I now thought that it is not possible to enable "ctop" on older QEMUs
>>> since the don't enable the KVM capability? ... or is it still somehow
>>> possible? What did I miss?
>>>
>>> Thomas
>>
>> Enabling ctop with ctop=on on old QEMU is not possible, this is right.
>> But, if STFL(11) is enable in the source KVM by a new QEMU, I can see
>> that even with -ctop=off the STFL(11) is migrated to the destination.
>
> Is this with the "host" CPU model or another one? And did you explicitly
> specify "ctop=off" at the command line, or are you just using the
> default setting by not specifying it?
With explicit cpumodel and using ctop=off like in
sudo /usr/local/bin/qemu-system-s390x_master \
-m 512M \
-enable-kvm -smp 4,sockets=4,cores=2,maxcpus=8 \
-cpu z14,ctop=off \
-machine s390-ccw-virtio-7.2,accel=kvm \
...
regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
On 12/12/2022 11.10, Pierre Morel wrote: > > > On 12/12/22 10:07, Thomas Huth wrote: >> On 12/12/2022 09.51, Pierre Morel wrote: >>> >>> >>> On 12/9/22 14:32, Thomas Huth wrote: >>>> On 08/12/2022 10.44, Pierre Morel wrote: >>>>> Hi, >>>>> >>>>> Implementation discussions >>>>> ========================== >>>>> >>>>> CPU models >>>>> ---------- >>>>> >>>>> Since the S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model >>>>> for old QEMU we could not activate it as usual from KVM but needed >>>>> a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. >>>>> Checking and enabling this capability enables >>>>> S390_FEAT_CONFIGURATION_TOPOLOGY. >>>>> >>>>> Migration >>>>> --------- >>>>> >>>>> Once the S390_FEAT_CONFIGURATION_TOPOLOGY is enabled in the source >>>>> host the STFL(11) is provided to the guest. >>>>> Since the feature is already in the CPU model of older QEMU, >>>>> a migration from a new QEMU enabling the topology to an old QEMU >>>>> will keep STFL(11) enabled making the guest get an exception for >>>>> illegal operation as soon as it uses the PTF instruction. >>>> >>>> I now thought that it is not possible to enable "ctop" on older QEMUs >>>> since the don't enable the KVM capability? ... or is it still somehow >>>> possible? What did I miss? >>>> >>>> Thomas >>> >>> Enabling ctop with ctop=on on old QEMU is not possible, this is right. >>> But, if STFL(11) is enable in the source KVM by a new QEMU, I can see >>> that even with -ctop=off the STFL(11) is migrated to the destination. >> >> Is this with the "host" CPU model or another one? And did you explicitly >> specify "ctop=off" at the command line, or are you just using the default >> setting by not specifying it? > > With explicit cpumodel and using ctop=off like in > > sudo /usr/local/bin/qemu-system-s390x_master \ > -m 512M \ > -enable-kvm -smp 4,sockets=4,cores=2,maxcpus=8 \ > -cpu z14,ctop=off \ > -machine s390-ccw-virtio-7.2,accel=kvm \ > ... Ok ... that sounds like a bug somewhere in your patches or in the kernel code ... the guest should never see STFL bit 11 if ctop=off, should it? Thomas
Am 12.12.22 um 11:17 schrieb Thomas Huth: > On 12/12/2022 11.10, Pierre Morel wrote: >> >> >> On 12/12/22 10:07, Thomas Huth wrote: >>> On 12/12/2022 09.51, Pierre Morel wrote: >>>> >>>> >>>> On 12/9/22 14:32, Thomas Huth wrote: >>>>> On 08/12/2022 10.44, Pierre Morel wrote: >>>>>> Hi, >>>>>> >>>>>> Implementation discussions >>>>>> ========================== >>>>>> >>>>>> CPU models >>>>>> ---------- >>>>>> >>>>>> Since the S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model >>>>>> for old QEMU we could not activate it as usual from KVM but needed >>>>>> a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. >>>>>> Checking and enabling this capability enables >>>>>> S390_FEAT_CONFIGURATION_TOPOLOGY. >>>>>> >>>>>> Migration >>>>>> --------- >>>>>> >>>>>> Once the S390_FEAT_CONFIGURATION_TOPOLOGY is enabled in the source >>>>>> host the STFL(11) is provided to the guest. >>>>>> Since the feature is already in the CPU model of older QEMU, >>>>>> a migration from a new QEMU enabling the topology to an old QEMU >>>>>> will keep STFL(11) enabled making the guest get an exception for >>>>>> illegal operation as soon as it uses the PTF instruction. >>>>> >>>>> I now thought that it is not possible to enable "ctop" on older QEMUs since the don't enable the KVM capability? ... or is it still somehow possible? What did I miss? >>>>> >>>>> Thomas >>>> >>>> Enabling ctop with ctop=on on old QEMU is not possible, this is right. >>>> But, if STFL(11) is enable in the source KVM by a new QEMU, I can see that even with -ctop=off the STFL(11) is migrated to the destination. This does not make sense. the cpu model and stfle values are not migrated. This is re-created during startup depending on the command line parameters of -cpu. Thats why source and host have the same command lines for -cpu. And STFLE.11 must not be set on the SOURCE of ctop is off. >>> >>> Is this with the "host" CPU model or another one? And did you explicitly specify "ctop=off" at the command line, or are you just using the default setting by not specifying it? >> >> With explicit cpumodel and using ctop=off like in >> >> sudo /usr/local/bin/qemu-system-s390x_master \ >> -m 512M \ >> -enable-kvm -smp 4,sockets=4,cores=2,maxcpus=8 \ >> -cpu z14,ctop=off \ >> -machine s390-ccw-virtio-7.2,accel=kvm \ >> ... > > Ok ... that sounds like a bug somewhere in your patches or in the kernel code ... the guest should never see STFL bit 11 if ctop=off, should it? Correct. If ctop=off then QEMU should disable STFLE.11 for the CPU model.
On 12/13/22 14:41, Christian Borntraeger wrote: > > > Am 12.12.22 um 11:17 schrieb Thomas Huth: >> On 12/12/2022 11.10, Pierre Morel wrote: >>> >>> >>> On 12/12/22 10:07, Thomas Huth wrote: >>>> On 12/12/2022 09.51, Pierre Morel wrote: >>>>> >>>>> >>>>> On 12/9/22 14:32, Thomas Huth wrote: >>>>>> On 08/12/2022 10.44, Pierre Morel wrote: >>>>>>> Hi, >>>>>>> >>>>>>> Implementation discussions >>>>>>> ========================== >>>>>>> >>>>>>> CPU models >>>>>>> ---------- >>>>>>> >>>>>>> Since the S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU >>>>>>> model >>>>>>> for old QEMU we could not activate it as usual from KVM but needed >>>>>>> a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. >>>>>>> Checking and enabling this capability enables >>>>>>> S390_FEAT_CONFIGURATION_TOPOLOGY. >>>>>>> >>>>>>> Migration >>>>>>> --------- >>>>>>> >>>>>>> Once the S390_FEAT_CONFIGURATION_TOPOLOGY is enabled in the source >>>>>>> host the STFL(11) is provided to the guest. >>>>>>> Since the feature is already in the CPU model of older QEMU, >>>>>>> a migration from a new QEMU enabling the topology to an old QEMU >>>>>>> will keep STFL(11) enabled making the guest get an exception for >>>>>>> illegal operation as soon as it uses the PTF instruction. >>>>>> >>>>>> I now thought that it is not possible to enable "ctop" on older >>>>>> QEMUs since the don't enable the KVM capability? ... or is it >>>>>> still somehow possible? What did I miss? >>>>>> >>>>>> Thomas >>>>> >>>>> Enabling ctop with ctop=on on old QEMU is not possible, this is right. >>>>> But, if STFL(11) is enable in the source KVM by a new QEMU, I can >>>>> see that even with -ctop=off the STFL(11) is migrated to the >>>>> destination. > > This does not make sense. the cpu model and stfle values are not > migrated. This is re-created during startup depending on the command > line parameters of -cpu. > Thats why source and host have the same command lines for -cpu. And > STFLE.11 must not be set on the SOURCE of ctop is off. OK, so it is a feature > > >>>> >>>> Is this with the "host" CPU model or another one? And did you >>>> explicitly specify "ctop=off" at the command line, or are you just >>>> using the default setting by not specifying it? >>> >>> With explicit cpumodel and using ctop=off like in >>> >>> sudo /usr/local/bin/qemu-system-s390x_master \ >>> -m 512M \ >>> -enable-kvm -smp 4,sockets=4,cores=2,maxcpus=8 \ >>> -cpu z14,ctop=off \ >>> -machine s390-ccw-virtio-7.2,accel=kvm \ >>> ... >> >> Ok ... that sounds like a bug somewhere in your patches or in the >> kernel code ... the guest should never see STFL bit 11 if ctop=off, >> should it? > > Correct. If ctop=off then QEMU should disable STFLE.11 for the CPU model. Sorry but not completely correct in the case of migration. After a migration if the source host specifies ctop=on and target host specifies ctop=off it does see the STFL bit 11. The admin should not, but can, specify ctop=off on target if the source set ctop=on. Then the target will start and run in a degraded mode. Admin should specify the same flags on both ends, as you said above the STFL bits are not migrated and target QEMU can not verify what the original flags were. However, isn't it a bug? Is there a reason to not prevent QEMU to start with a wrong cpu model like specifying different flags on both ends or even different cpu? Regards, Pierre -- Pierre Morel IBM Lab Boeblingen
On 13/12/2022 18.24, Pierre Morel wrote: > > > On 12/13/22 14:41, Christian Borntraeger wrote: >> >> >> Am 12.12.22 um 11:17 schrieb Thomas Huth: >>> On 12/12/2022 11.10, Pierre Morel wrote: >>>> >>>> >>>> On 12/12/22 10:07, Thomas Huth wrote: >>>>> On 12/12/2022 09.51, Pierre Morel wrote: >>>>>> >>>>>> >>>>>> On 12/9/22 14:32, Thomas Huth wrote: >>>>>>> On 08/12/2022 10.44, Pierre Morel wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> Implementation discussions >>>>>>>> ========================== >>>>>>>> >>>>>>>> CPU models >>>>>>>> ---------- >>>>>>>> >>>>>>>> Since the S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model >>>>>>>> for old QEMU we could not activate it as usual from KVM but needed >>>>>>>> a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. >>>>>>>> Checking and enabling this capability enables >>>>>>>> S390_FEAT_CONFIGURATION_TOPOLOGY. >>>>>>>> >>>>>>>> Migration >>>>>>>> --------- >>>>>>>> >>>>>>>> Once the S390_FEAT_CONFIGURATION_TOPOLOGY is enabled in the source >>>>>>>> host the STFL(11) is provided to the guest. >>>>>>>> Since the feature is already in the CPU model of older QEMU, >>>>>>>> a migration from a new QEMU enabling the topology to an old QEMU >>>>>>>> will keep STFL(11) enabled making the guest get an exception for >>>>>>>> illegal operation as soon as it uses the PTF instruction. >>>>>>> >>>>>>> I now thought that it is not possible to enable "ctop" on older QEMUs >>>>>>> since the don't enable the KVM capability? ... or is it still somehow >>>>>>> possible? What did I miss? >>>>>>> >>>>>>> Thomas >>>>>> >>>>>> Enabling ctop with ctop=on on old QEMU is not possible, this is right. >>>>>> But, if STFL(11) is enable in the source KVM by a new QEMU, I can see >>>>>> that even with -ctop=off the STFL(11) is migrated to the destination. >> >> This does not make sense. the cpu model and stfle values are not migrated. >> This is re-created during startup depending on the command line parameters >> of -cpu. >> Thats why source and host have the same command lines for -cpu. And >> STFLE.11 must not be set on the SOURCE of ctop is off. > > OK, so it is a feature > >> >> >>>>> >>>>> Is this with the "host" CPU model or another one? And did you >>>>> explicitly specify "ctop=off" at the command line, or are you just >>>>> using the default setting by not specifying it? >>>> >>>> With explicit cpumodel and using ctop=off like in >>>> >>>> sudo /usr/local/bin/qemu-system-s390x_master \ >>>> -m 512M \ >>>> -enable-kvm -smp 4,sockets=4,cores=2,maxcpus=8 \ >>>> -cpu z14,ctop=off \ >>>> -machine s390-ccw-virtio-7.2,accel=kvm \ >>>> ... >>> >>> Ok ... that sounds like a bug somewhere in your patches or in the kernel >>> code ... the guest should never see STFL bit 11 if ctop=off, should it? >> >> Correct. If ctop=off then QEMU should disable STFLE.11 for the CPU model. > > Sorry but not completely correct in the case of migration. > > After a migration if the source host specifies ctop=on and target host > specifies ctop=off it does see the STFL bit 11. > > The admin should not, but can, specify ctop=off on target if the source set > ctop=on. Then the target will start and run in a degraded mode. > > Admin should specify the same flags on both ends, as you said above the STFL > bits are not migrated and target QEMU can not verify what the original flags > were. > > However, isn't it a bug? > Is there a reason to not prevent QEMU to start with a wrong cpu model like > specifying different flags on both ends or even different cpu? It's clearly an user error if the two QEMUs are started with different flags on the source and destination ends. But it would be great if there was a generic way to check for this error condition and bail out early instead of doing the migration and let the user run into weird problems later... Does anybody have an idea whether there is a good and easy way to implement such a check? Thomas
On Tue, 2022-12-13 at 14:41 +0100, Christian Borntraeger wrote: > > Am 12.12.22 um 11:17 schrieb Thomas Huth: > > On 12/12/2022 11.10, Pierre Morel wrote: > > > > > > > > > On 12/12/22 10:07, Thomas Huth wrote: > > > > On 12/12/2022 09.51, Pierre Morel wrote: > > > > > > > > > > > > > > > On 12/9/22 14:32, Thomas Huth wrote: > > > > > > On 08/12/2022 10.44, Pierre Morel wrote: > > > > > > > Hi, > > > > > > > > > > > > > > Implementation discussions > > > > > > > ========================== > > > > > > > > > > > > > > CPU models > > > > > > > ---------- > > > > > > > > > > > > > > Since the S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model > > > > > > > for old QEMU we could not activate it as usual from KVM but needed > > > > > > > a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. > > > > > > > Checking and enabling this capability enables > > > > > > > S390_FEAT_CONFIGURATION_TOPOLOGY. > > > > > > > > > > > > > > Migration > > > > > > > --------- > > > > > > > > > > > > > > Once the S390_FEAT_CONFIGURATION_TOPOLOGY is enabled in the source > > > > > > > host the STFL(11) is provided to the guest. > > > > > > > Since the feature is already in the CPU model of older QEMU, > > > > > > > a migration from a new QEMU enabling the topology to an old QEMU > > > > > > > will keep STFL(11) enabled making the guest get an exception for > > > > > > > illegal operation as soon as it uses the PTF instruction. > > > > > > > > > > > > I now thought that it is not possible to enable "ctop" on older QEMUs since the don't enable the KVM capability? ... or is it still somehow possible? What did I miss? > > > > > > > > > > > > Thomas > > > > > > > > > > Enabling ctop with ctop=on on old QEMU is not possible, this is right. > > > > > But, if STFL(11) is enable in the source KVM by a new QEMU, I can see that even with -ctop=off the STFL(11) is migrated to the destination. > > This does not make sense. the cpu model and stfle values are not migrated. This is re-created during startup depending on the command line parameters of -cpu. > Thats why source and host have the same command lines for -cpu. And STFLE.11 must not be set on the SOURCE of ctop is off. > What about linux? I didn't look to thoroughly at it, but it caches the stfle bits, doesn't it? So if the migration succeeds, even tho it should not, it will look to the guest like the facility is enabled. > > > > > > > > > Is this with the "host" CPU model or another one? And did you explicitly specify "ctop=off" at the command line, or are you just using the default setting by not specifying it? > > > > > > With explicit cpumodel and using ctop=off like in > > > > > > sudo /usr/local/bin/qemu-system-s390x_master \ > > > -m 512M \ > > > -enable-kvm -smp 4,sockets=4,cores=2,maxcpus=8 \ > > > -cpu z14,ctop=off \ > > > -machine s390-ccw-virtio-7.2,accel=kvm \ > > > ... > > > > Ok ... that sounds like a bug somewhere in your patches or in the kernel code ... the guest should never see STFL bit 11 if ctop=off, should it? > > Correct. If ctop=off then QEMU should disable STFLE.11 for the CPU model.
Am 13.12.22 um 14:57 schrieb Janis Schoetterl-Glausch: > On Tue, 2022-12-13 at 14:41 +0100, Christian Borntraeger wrote: >> >> Am 12.12.22 um 11:17 schrieb Thomas Huth: >>> On 12/12/2022 11.10, Pierre Morel wrote: >>>> >>>> >>>> On 12/12/22 10:07, Thomas Huth wrote: >>>>> On 12/12/2022 09.51, Pierre Morel wrote: >>>>>> >>>>>> >>>>>> On 12/9/22 14:32, Thomas Huth wrote: >>>>>>> On 08/12/2022 10.44, Pierre Morel wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> Implementation discussions >>>>>>>> ========================== >>>>>>>> >>>>>>>> CPU models >>>>>>>> ---------- >>>>>>>> >>>>>>>> Since the S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model >>>>>>>> for old QEMU we could not activate it as usual from KVM but needed >>>>>>>> a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. >>>>>>>> Checking and enabling this capability enables >>>>>>>> S390_FEAT_CONFIGURATION_TOPOLOGY. >>>>>>>> >>>>>>>> Migration >>>>>>>> --------- >>>>>>>> >>>>>>>> Once the S390_FEAT_CONFIGURATION_TOPOLOGY is enabled in the source >>>>>>>> host the STFL(11) is provided to the guest. >>>>>>>> Since the feature is already in the CPU model of older QEMU, >>>>>>>> a migration from a new QEMU enabling the topology to an old QEMU >>>>>>>> will keep STFL(11) enabled making the guest get an exception for >>>>>>>> illegal operation as soon as it uses the PTF instruction. >>>>>>> >>>>>>> I now thought that it is not possible to enable "ctop" on older QEMUs since the don't enable the KVM capability? ... or is it still somehow possible? What did I miss? >>>>>>> >>>>>>> Thomas >>>>>> >>>>>> Enabling ctop with ctop=on on old QEMU is not possible, this is right. >>>>>> But, if STFL(11) is enable in the source KVM by a new QEMU, I can see that even with -ctop=off the STFL(11) is migrated to the destination. >> >> This does not make sense. the cpu model and stfle values are not migrated. This is re-created during startup depending on the command line parameters of -cpu. >> Thats why source and host have the same command lines for -cpu. And STFLE.11 must not be set on the SOURCE of ctop is off. >> > What about linux? I didn't look to thoroughly at it, but it caches the stfle bits, doesn't it? > So if the migration succeeds, even tho it should not, it will look to the guest like the facility is enabled. That is true, but the migration should not succeed in that case unless you use an unsafe way of migrating. And the cpu model was exactly added to block those unsafe way. One thing: -cpu host is unsafe (host-passthrough in libvirt speak). Either use host-model or a fully specified model like z14.2,ctop=on....
On 12/8/22 10:44, Pierre Morel wrote: > Hi, > > Implementation discussions > ========================== > > CPU models > ---------- > > Since the S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model > for old QEMU we could not activate it as usual from KVM but needed > a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. > Checking and enabling this capability enables > S390_FEAT_CONFIGURATION_TOPOLOGY. > > Migration > --------- > > Once the S390_FEAT_CONFIGURATION_TOPOLOGY is enabled in the source > host the STFL(11) is provided to the guest. > Since the feature is already in the CPU model of older QEMU, > a migration from a new QEMU enabling the topology to an old QEMU > will keep STFL(11) enabled making the guest get an exception for > illegal operation as soon as it uses the PTF instruction. > > A VMState keeping track of the S390_FEAT_CONFIGURATION_TOPOLOGY > allows to forbid the migration in such a case. > > Note that the VMState will be used to hold information on the > topology once we implement topology change for a running guest. > > Topology > -------- > > Until we introduce bookss and drawers, polarization and dedication > the topology is kept very simple and is specified uniquely by > the core_id of the vCPU which is also the vCPU address. > > Testing > ======= > > To use the QEMU patches, you will need Linux V6-rc1 or newer, > or use the following Linux mainline patches: > > f5ecfee94493 2022-07-20 KVM: s390: resetting the Topology-Change-Report > 24fe0195bc19 2022-07-20 KVM: s390: guest support for topology function > 0130337ec45b 2022-07-20 KVM: s390: Cleanup ipte lock access and SIIF fac.. > > Currently this code is for KVM only, I have no idea if it is interesting > to provide a TCG patch. If ever it will be done in another series. > > Documentation > ============= > > To have a better understanding of the S390x CPU Topology and its > implementation in QEMU you can have a look at the documentation in the > last patch of this series. > > The admin will want to match the host and the guest topology, taking > into account that the guest does not recognize multithreading. > Consequently, two vCPU assigned to threads of the same real CPU should > preferably be assigned to the same socket of the guest machine. > > Future developments > =================== > > Two series are actively prepared: > - Adding drawers, book, polarization and dedication to the vCPU. > - changing the topology with a running guest Since we have time before the next QEMU 8.0 release, could you please send the whole patchset ? Having the full picture would help in taking decisions for downstream also. I am still uncertain about the usefulness of S390Topology object because, as for now, the state can be computed on the fly, the reset can be handled at the machine level directly under s390_machine_reset() and so could migration if the machine had a vmstate (not the case today but quite easy to add). So before merging anything that could be difficult to maintain and/or backport, I would prefer to see it all ! Thanks, C. > > Regards, > Pierre > > Pierre Morel (7): > s390x/cpu topology: Creating CPU topology device > s390x/cpu topology: reporting the CPU topology to the guest > s390x/cpu_topology: resetting the Topology-Change-Report > s390x/cpu_topology: CPU topology migration > s390x/cpu_topology: interception of PTF instruction > s390x/cpu_topology: activating CPU topology > docs/s390x: document s390x cpu topology > > docs/system/s390x/cpu-topology.rst | 87 ++++++++++ > docs/system/target-s390x.rst | 1 + > include/hw/s390x/cpu-topology.h | 52 ++++++ > include/hw/s390x/s390-virtio-ccw.h | 6 + > target/s390x/cpu.h | 78 +++++++++ > target/s390x/kvm/kvm_s390x.h | 1 + > hw/s390x/cpu-topology.c | 261 +++++++++++++++++++++++++++++ > hw/s390x/s390-virtio-ccw.c | 7 + > target/s390x/cpu-sysemu.c | 21 +++ > target/s390x/cpu_models.c | 1 + > target/s390x/kvm/cpu_topology.c | 186 ++++++++++++++++++++ > target/s390x/kvm/kvm.c | 46 ++++- > hw/s390x/meson.build | 1 + > target/s390x/kvm/meson.build | 3 +- > 14 files changed, 749 insertions(+), 2 deletions(-) > create mode 100644 docs/system/s390x/cpu-topology.rst > create mode 100644 include/hw/s390x/cpu-topology.h > create mode 100644 hw/s390x/cpu-topology.c > create mode 100644 target/s390x/kvm/cpu_topology.c >
On 12/9/22 15:45, Cédric Le Goater wrote: > On 12/8/22 10:44, Pierre Morel wrote: >> Hi, >> >> Implementation discussions >> ========================== >> >> CPU models >> ---------- >> >> Since the S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model >> for old QEMU we could not activate it as usual from KVM but needed >> a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. >> Checking and enabling this capability enables >> S390_FEAT_CONFIGURATION_TOPOLOGY. >> >> Migration >> --------- >> >> Once the S390_FEAT_CONFIGURATION_TOPOLOGY is enabled in the source >> host the STFL(11) is provided to the guest. >> Since the feature is already in the CPU model of older QEMU, >> a migration from a new QEMU enabling the topology to an old QEMU >> will keep STFL(11) enabled making the guest get an exception for >> illegal operation as soon as it uses the PTF instruction. >> >> A VMState keeping track of the S390_FEAT_CONFIGURATION_TOPOLOGY >> allows to forbid the migration in such a case. >> >> Note that the VMState will be used to hold information on the >> topology once we implement topology change for a running guest. >> >> Topology >> -------- >> >> Until we introduce bookss and drawers, polarization and dedication >> the topology is kept very simple and is specified uniquely by >> the core_id of the vCPU which is also the vCPU address. >> >> Testing >> ======= >> >> To use the QEMU patches, you will need Linux V6-rc1 or newer, >> or use the following Linux mainline patches: >> >> f5ecfee94493 2022-07-20 KVM: s390: resetting the Topology-Change-Report >> 24fe0195bc19 2022-07-20 KVM: s390: guest support for topology function >> 0130337ec45b 2022-07-20 KVM: s390: Cleanup ipte lock access and SIIF >> fac.. >> >> Currently this code is for KVM only, I have no idea if it is interesting >> to provide a TCG patch. If ever it will be done in another series. >> >> Documentation >> ============= >> >> To have a better understanding of the S390x CPU Topology and its >> implementation in QEMU you can have a look at the documentation in the >> last patch of this series. >> >> The admin will want to match the host and the guest topology, taking >> into account that the guest does not recognize multithreading. >> Consequently, two vCPU assigned to threads of the same real CPU should >> preferably be assigned to the same socket of the guest machine. >> >> Future developments >> =================== >> >> Two series are actively prepared: >> - Adding drawers, book, polarization and dedication to the vCPU. >> - changing the topology with a running guest > > Since we have time before the next QEMU 8.0 release, could you please > send the whole patchset ? Having the full picture would help in taking > decisions for downstream also. > > I am still uncertain about the usefulness of S390Topology object because, > as for now, the state can be computed on the fly, the reset can be > handled at the machine level directly under s390_machine_reset() and so > could migration if the machine had a vmstate (not the case today but > quite easy to add). So before merging anything that could be difficult > to maintain and/or backport, I would prefer to see it all ! > The goal of dedicating an object for topology was to ease the maintenance and portability by using the QEMU object framework. If on contrary it is a problem for maintenance or backport we surely better not use it. Any other opinion? Regards, Pierre -- Pierre Morel IBM Lab Boeblingen
Am 12.12.22 um 11:01 schrieb Pierre Morel: > > > On 12/9/22 15:45, Cédric Le Goater wrote: >> On 12/8/22 10:44, Pierre Morel wrote: >>> Hi, >>> >>> Implementation discussions >>> ========================== >>> >>> CPU models >>> ---------- >>> >>> Since the S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model >>> for old QEMU we could not activate it as usual from KVM but needed >>> a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. >>> Checking and enabling this capability enables >>> S390_FEAT_CONFIGURATION_TOPOLOGY. >>> >>> Migration >>> --------- >>> >>> Once the S390_FEAT_CONFIGURATION_TOPOLOGY is enabled in the source >>> host the STFL(11) is provided to the guest. >>> Since the feature is already in the CPU model of older QEMU, >>> a migration from a new QEMU enabling the topology to an old QEMU >>> will keep STFL(11) enabled making the guest get an exception for >>> illegal operation as soon as it uses the PTF instruction. >>> >>> A VMState keeping track of the S390_FEAT_CONFIGURATION_TOPOLOGY >>> allows to forbid the migration in such a case. >>> >>> Note that the VMState will be used to hold information on the >>> topology once we implement topology change for a running guest. >>> >>> Topology >>> -------- >>> >>> Until we introduce bookss and drawers, polarization and dedication >>> the topology is kept very simple and is specified uniquely by >>> the core_id of the vCPU which is also the vCPU address. >>> >>> Testing >>> ======= >>> >>> To use the QEMU patches, you will need Linux V6-rc1 or newer, >>> or use the following Linux mainline patches: >>> >>> f5ecfee94493 2022-07-20 KVM: s390: resetting the Topology-Change-Report >>> 24fe0195bc19 2022-07-20 KVM: s390: guest support for topology function >>> 0130337ec45b 2022-07-20 KVM: s390: Cleanup ipte lock access and SIIF fac.. >>> >>> Currently this code is for KVM only, I have no idea if it is interesting >>> to provide a TCG patch. If ever it will be done in another series. >>> >>> Documentation >>> ============= >>> >>> To have a better understanding of the S390x CPU Topology and its >>> implementation in QEMU you can have a look at the documentation in the >>> last patch of this series. >>> >>> The admin will want to match the host and the guest topology, taking >>> into account that the guest does not recognize multithreading. >>> Consequently, two vCPU assigned to threads of the same real CPU should >>> preferably be assigned to the same socket of the guest machine. >>> >>> Future developments >>> =================== >>> >>> Two series are actively prepared: >>> - Adding drawers, book, polarization and dedication to the vCPU. >>> - changing the topology with a running guest >> >> Since we have time before the next QEMU 8.0 release, could you please >> send the whole patchset ? Having the full picture would help in taking >> decisions for downstream also. >> >> I am still uncertain about the usefulness of S390Topology object because, >> as for now, the state can be computed on the fly, the reset can be >> handled at the machine level directly under s390_machine_reset() and so >> could migration if the machine had a vmstate (not the case today but >> quite easy to add). So before merging anything that could be difficult >> to maintain and/or backport, I would prefer to see it all ! >> > > The goal of dedicating an object for topology was to ease the maintenance and portability by using the QEMU object framework. > > If on contrary it is a problem for maintenance or backport we surely better not use it. > > Any other opinion? I agree with Cedric. There is no point in a topology object. The state is calculated on the fly depending on the command line. This would change if we ever implement the PTF horizontal/vertical state. But this can then be another CPU state. So I think we could go forward with this patch as a simple patch set that allows to sets a static topology. This makes sense on its own, e.g. if you plan to pin your vCPUs to given host CPUs. Dynamic changes do come with CPU hotplug, not sure what libvirt does with new CPUs in that case during migration. I assume those are re-generated on the target with whatever topology was created on the source. So I guess this will just work, but it would be good if we could test that. A more fine-grained topology (drawer, book) could be added later or upfront. It does require common code and libvirt enhancements, though.
Am 13.12.22 um 14:50 schrieb Christian Borntraeger: > > > Am 12.12.22 um 11:01 schrieb Pierre Morel: >> >> >> On 12/9/22 15:45, Cédric Le Goater wrote: >>> On 12/8/22 10:44, Pierre Morel wrote: >>>> Hi, >>>> >>>> Implementation discussions >>>> ========================== >>>> >>>> CPU models >>>> ---------- >>>> >>>> Since the S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model >>>> for old QEMU we could not activate it as usual from KVM but needed >>>> a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. >>>> Checking and enabling this capability enables >>>> S390_FEAT_CONFIGURATION_TOPOLOGY. >>>> >>>> Migration >>>> --------- >>>> >>>> Once the S390_FEAT_CONFIGURATION_TOPOLOGY is enabled in the source >>>> host the STFL(11) is provided to the guest. >>>> Since the feature is already in the CPU model of older QEMU, >>>> a migration from a new QEMU enabling the topology to an old QEMU >>>> will keep STFL(11) enabled making the guest get an exception for >>>> illegal operation as soon as it uses the PTF instruction. >>>> >>>> A VMState keeping track of the S390_FEAT_CONFIGURATION_TOPOLOGY >>>> allows to forbid the migration in such a case. >>>> >>>> Note that the VMState will be used to hold information on the >>>> topology once we implement topology change for a running guest. >>>> >>>> Topology >>>> -------- >>>> >>>> Until we introduce bookss and drawers, polarization and dedication >>>> the topology is kept very simple and is specified uniquely by >>>> the core_id of the vCPU which is also the vCPU address. >>>> >>>> Testing >>>> ======= >>>> >>>> To use the QEMU patches, you will need Linux V6-rc1 or newer, >>>> or use the following Linux mainline patches: >>>> >>>> f5ecfee94493 2022-07-20 KVM: s390: resetting the Topology-Change-Report >>>> 24fe0195bc19 2022-07-20 KVM: s390: guest support for topology function >>>> 0130337ec45b 2022-07-20 KVM: s390: Cleanup ipte lock access and SIIF fac.. >>>> >>>> Currently this code is for KVM only, I have no idea if it is interesting >>>> to provide a TCG patch. If ever it will be done in another series. >>>> >>>> Documentation >>>> ============= >>>> >>>> To have a better understanding of the S390x CPU Topology and its >>>> implementation in QEMU you can have a look at the documentation in the >>>> last patch of this series. >>>> >>>> The admin will want to match the host and the guest topology, taking >>>> into account that the guest does not recognize multithreading. >>>> Consequently, two vCPU assigned to threads of the same real CPU should >>>> preferably be assigned to the same socket of the guest machine. >>>> >>>> Future developments >>>> =================== >>>> >>>> Two series are actively prepared: >>>> - Adding drawers, book, polarization and dedication to the vCPU. >>>> - changing the topology with a running guest >>> >>> Since we have time before the next QEMU 8.0 release, could you please >>> send the whole patchset ? Having the full picture would help in taking >>> decisions for downstream also. >>> >>> I am still uncertain about the usefulness of S390Topology object because, >>> as for now, the state can be computed on the fly, the reset can be >>> handled at the machine level directly under s390_machine_reset() and so >>> could migration if the machine had a vmstate (not the case today but >>> quite easy to add). So before merging anything that could be difficult >>> to maintain and/or backport, I would prefer to see it all ! >>> >> >> The goal of dedicating an object for topology was to ease the maintenance and portability by using the QEMU object framework. >> >> If on contrary it is a problem for maintenance or backport we surely better not use it. >> >> Any other opinion? > > I agree with Cedric. There is no point in a topology object. > The state is calculated on the fly depending on the command line. This > would change if we ever implement the PTF horizontal/vertical state. But this > can then be another CPU state. > > So I think we could go forward with this patch as a simple patch set that allows to > sets a static topology. This makes sense on its own, e.g. if you plan to pin your > vCPUs to given host CPUs. > > Dynamic changes do come with CPU hotplug, not sure what libvirt does with new CPUs > in that case during migration. I assume those are re-generated on the target with > whatever topology was created on the source. So I guess this will just work, but > it would be good if we could test that. > > A more fine-grained topology (drawer, book) could be added later or upfront. It > does require common code and libvirt enhancements, though. Now I have discussed with Pierre and there IS a state that we want to migrate. The topology change state is a guest wide bit that might be still set when topology is changed->bit is set guest is not yet started migration 2 options: 1. a vmstate with that bit and migrate the state 2. always set the topology change bit after migration
On Tue, 2022-12-13 at 16:12 +0100, Christian Borntraeger wrote: > > Am 13.12.22 um 14:50 schrieb Christian Borntraeger: > > > > > > Am 12.12.22 um 11:01 schrieb Pierre Morel: > > > > > > > > > On 12/9/22 15:45, Cédric Le Goater wrote: > > > > On 12/8/22 10:44, Pierre Morel wrote: > > > > > Hi, > > > > > > > > > > Implementation discussions > > > > > ========================== > > > > > > > > > > CPU models > > > > > ---------- > > > > > > > > > > Since the S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model > > > > > for old QEMU we could not activate it as usual from KVM but needed > > > > > a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. > > > > > Checking and enabling this capability enables > > > > > S390_FEAT_CONFIGURATION_TOPOLOGY. > > > > > > > > > > Migration > > > > > --------- > > > > > > > > > > Once the S390_FEAT_CONFIGURATION_TOPOLOGY is enabled in the source > > > > > host the STFL(11) is provided to the guest. > > > > > Since the feature is already in the CPU model of older QEMU, > > > > > a migration from a new QEMU enabling the topology to an old QEMU > > > > > will keep STFL(11) enabled making the guest get an exception for > > > > > illegal operation as soon as it uses the PTF instruction. > > > > > > > > > > A VMState keeping track of the S390_FEAT_CONFIGURATION_TOPOLOGY > > > > > allows to forbid the migration in such a case. > > > > > > > > > > Note that the VMState will be used to hold information on the > > > > > topology once we implement topology change for a running guest. > > > > > > > > > > Topology > > > > > -------- > > > > > > > > > > Until we introduce bookss and drawers, polarization and dedication > > > > > the topology is kept very simple and is specified uniquely by > > > > > the core_id of the vCPU which is also the vCPU address. > > > > > > > > > > Testing > > > > > ======= > > > > > > > > > > To use the QEMU patches, you will need Linux V6-rc1 or newer, > > > > > or use the following Linux mainline patches: > > > > > > > > > > f5ecfee94493 2022-07-20 KVM: s390: resetting the Topology-Change-Report > > > > > 24fe0195bc19 2022-07-20 KVM: s390: guest support for topology function > > > > > 0130337ec45b 2022-07-20 KVM: s390: Cleanup ipte lock access and SIIF fac.. > > > > > > > > > > Currently this code is for KVM only, I have no idea if it is interesting > > > > > to provide a TCG patch. If ever it will be done in another series. > > > > > > > > > > Documentation > > > > > ============= > > > > > > > > > > To have a better understanding of the S390x CPU Topology and its > > > > > implementation in QEMU you can have a look at the documentation in the > > > > > last patch of this series. > > > > > > > > > > The admin will want to match the host and the guest topology, taking > > > > > into account that the guest does not recognize multithreading. > > > > > Consequently, two vCPU assigned to threads of the same real CPU should > > > > > preferably be assigned to the same socket of the guest machine. > > > > > > > > > > Future developments > > > > > =================== > > > > > > > > > > Two series are actively prepared: > > > > > - Adding drawers, book, polarization and dedication to the vCPU. > > > > > - changing the topology with a running guest > > > > > > > > Since we have time before the next QEMU 8.0 release, could you please > > > > send the whole patchset ? Having the full picture would help in taking > > > > decisions for downstream also. > > > > > > > > I am still uncertain about the usefulness of S390Topology object because, > > > > as for now, the state can be computed on the fly, the reset can be > > > > handled at the machine level directly under s390_machine_reset() and so > > > > could migration if the machine had a vmstate (not the case today but > > > > quite easy to add). So before merging anything that could be difficult > > > > to maintain and/or backport, I would prefer to see it all ! > > > > > > > > > > The goal of dedicating an object for topology was to ease the maintenance and portability by using the QEMU object framework. > > > > > > If on contrary it is a problem for maintenance or backport we surely better not use it. > > > > > > Any other opinion? > > > > I agree with Cedric. There is no point in a topology object. > > The state is calculated on the fly depending on the command line. This > > would change if we ever implement the PTF horizontal/vertical state. But this > > can then be another CPU state. > > > > So I think we could go forward with this patch as a simple patch set that allows to > > sets a static topology. This makes sense on its own, e.g. if you plan to pin your > > vCPUs to given host CPUs. > > > > Dynamic changes do come with CPU hotplug, not sure what libvirt does with new CPUs > > in that case during migration. I assume those are re-generated on the target with > > whatever topology was created on the source. So I guess this will just work, but > > it would be good if we could test that. > > > > A more fine-grained topology (drawer, book) could be added later or upfront. It > > does require common code and libvirt enhancements, though. > > Now I have discussed with Pierre and there IS a state that we want to migrate. > The topology change state is a guest wide bit that might be still set when > topology is changed->bit is set > guest is not yet started > migration > > 2 options: > 1. a vmstate with that bit and migrate the state > 2. always set the topology change bit after migration 2. is the default behavior if you do nothing. VCPU creation on the target sets the change bit to 1. So 1. is only to prevent spurious topology change indication.
On 12/13/22 16:31, Janis Schoetterl-Glausch wrote: > On Tue, 2022-12-13 at 16:12 +0100, Christian Borntraeger wrote: >> >> Am 13.12.22 um 14:50 schrieb Christian Borntraeger: >>> >>> >>> Am 12.12.22 um 11:01 schrieb Pierre Morel: >>>> >>>> >>>> On 12/9/22 15:45, Cédric Le Goater wrote: >>>>> On 12/8/22 10:44, Pierre Morel wrote: >>>>>> Hi, >>>>>> >>>>>> Implementation discussions >>>>>> ========================== >>>>>> >>>>>> CPU models >>>>>> ---------- >>>>>> >>>>>> Since the S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model >>>>>> for old QEMU we could not activate it as usual from KVM but needed >>>>>> a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. >>>>>> Checking and enabling this capability enables >>>>>> S390_FEAT_CONFIGURATION_TOPOLOGY. >>>>>> >>>>>> Migration >>>>>> --------- >>>>>> >>>>>> Once the S390_FEAT_CONFIGURATION_TOPOLOGY is enabled in the source >>>>>> host the STFL(11) is provided to the guest. >>>>>> Since the feature is already in the CPU model of older QEMU, >>>>>> a migration from a new QEMU enabling the topology to an old QEMU >>>>>> will keep STFL(11) enabled making the guest get an exception for >>>>>> illegal operation as soon as it uses the PTF instruction. >>>>>> >>>>>> A VMState keeping track of the S390_FEAT_CONFIGURATION_TOPOLOGY >>>>>> allows to forbid the migration in such a case. >>>>>> >>>>>> Note that the VMState will be used to hold information on the >>>>>> topology once we implement topology change for a running guest. >>>>>> >>>>>> Topology >>>>>> -------- >>>>>> >>>>>> Until we introduce bookss and drawers, polarization and dedication >>>>>> the topology is kept very simple and is specified uniquely by >>>>>> the core_id of the vCPU which is also the vCPU address. >>>>>> >>>>>> Testing >>>>>> ======= >>>>>> >>>>>> To use the QEMU patches, you will need Linux V6-rc1 or newer, >>>>>> or use the following Linux mainline patches: >>>>>> >>>>>> f5ecfee94493 2022-07-20 KVM: s390: resetting the Topology-Change-Report >>>>>> 24fe0195bc19 2022-07-20 KVM: s390: guest support for topology function >>>>>> 0130337ec45b 2022-07-20 KVM: s390: Cleanup ipte lock access and SIIF fac.. >>>>>> >>>>>> Currently this code is for KVM only, I have no idea if it is interesting >>>>>> to provide a TCG patch. If ever it will be done in another series. >>>>>> >>>>>> Documentation >>>>>> ============= >>>>>> >>>>>> To have a better understanding of the S390x CPU Topology and its >>>>>> implementation in QEMU you can have a look at the documentation in the >>>>>> last patch of this series. >>>>>> >>>>>> The admin will want to match the host and the guest topology, taking >>>>>> into account that the guest does not recognize multithreading. >>>>>> Consequently, two vCPU assigned to threads of the same real CPU should >>>>>> preferably be assigned to the same socket of the guest machine. >>>>>> >>>>>> Future developments >>>>>> =================== >>>>>> >>>>>> Two series are actively prepared: >>>>>> - Adding drawers, book, polarization and dedication to the vCPU. >>>>>> - changing the topology with a running guest >>>>> >>>>> Since we have time before the next QEMU 8.0 release, could you please >>>>> send the whole patchset ? Having the full picture would help in taking >>>>> decisions for downstream also. >>>>> >>>>> I am still uncertain about the usefulness of S390Topology object because, >>>>> as for now, the state can be computed on the fly, the reset can be >>>>> handled at the machine level directly under s390_machine_reset() and so >>>>> could migration if the machine had a vmstate (not the case today but >>>>> quite easy to add). So before merging anything that could be difficult >>>>> to maintain and/or backport, I would prefer to see it all ! >>>>> >>>> >>>> The goal of dedicating an object for topology was to ease the maintenance and portability by using the QEMU object framework. >>>> >>>> If on contrary it is a problem for maintenance or backport we surely better not use it. >>>> >>>> Any other opinion? >>> >>> I agree with Cedric. There is no point in a topology object. >>> The state is calculated on the fly depending on the command line. This >>> would change if we ever implement the PTF horizontal/vertical state. But this >>> can then be another CPU state. >>> >>> So I think we could go forward with this patch as a simple patch set that allows to >>> sets a static topology. This makes sense on its own, e.g. if you plan to pin your >>> vCPUs to given host CPUs. >>> >>> Dynamic changes do come with CPU hotplug, not sure what libvirt does with new CPUs >>> in that case during migration. I assume those are re-generated on the target with >>> whatever topology was created on the source. So I guess this will just work, but >>> it would be good if we could test that. >>> >>> A more fine-grained topology (drawer, book) could be added later or upfront. It >>> does require common code and libvirt enhancements, though. >> >> Now I have discussed with Pierre and there IS a state that we want to migrate. >> The topology change state is a guest wide bit that might be still set when >> topology is changed->bit is set >> guest is not yet started >> migration >> >> 2 options: >> 1. a vmstate with that bit and migrate the state >> 2. always set the topology change bit after migration > > 2. is the default behavior if you do nothing. VCPU creation on the target sets > the change bit to 1. > So 1. is only to prevent spurious topology change indication. > OK, then I go the easy way and suppress the object. Regards, Pierre -- Pierre Morel IBM Lab Boeblingen
© 2016 - 2024 Red Hat, Inc.