include/hw/s390x/s390-virtio-ccw.h | 10 ++++ include/sysemu/kvm_int.h | 1 + accel/kvm/kvm-all.c | 79 ++++++++++++++++++------------ hw/s390x/s390-virtio-ccw.c | 30 ++---------- target/s390x/kvm.c | 1 + 5 files changed, 63 insertions(+), 58 deletions(-)
Changelog:
since v2:
- break migration from old QEMU (since 2.12-4.1) for guest with >8TB RAM
and drop migratable aliases patch as was agreed during v2 review
- drop 4.2 machines patch as it's not prerequisite anymore
since v1:
- include 4.2 machines patch for adding compat RAM layout on top
- 2/4 add missing in v1 patch for splitting too big MemorySection on
several memslots
- 3/4 amend code path on alias destruction to ensure that RAMBlock is
cleaned properly
- 4/4 add compat machine code to keep old layout (migration-wise) for
4.1 and older machines
While looking into unifying guest RAM allocation to use hostmem backends
for initial RAM (especially when -mempath is used) and retiring
memory_region_allocate_system_memory() API, leaving only single hostmem backend,
I was inspecting how currently it is used by boards and it turns out several
boards abuse it by calling the function several times (despite documented contract
forbiding it).
s390 is one of such boards where KVM limitation on memslot size got propagated
to board design and memory_region_allocate_system_memory() was abused to satisfy
KVM requirement for max RAM chunk where memory region alias would suffice.
Unfortunately, memory_region_allocate_system_memory() usage created migration
dependency where guest RAM is transferred in migration stream as several RAMBlocks
if it's more than KVM_SLOT_MAX_BYTES. During v2 review it was agreed to ignore
migration breakage (documenting it in release notes) and leaving only KVM fix.
In order to replace these several RAM chunks with a single memdev and keep it
working with KVM memslot size limit, following was done:
* [1/2] split too big RAM chunk inside of KVM code on several memory slots
if necessary
* [2/2] drop manual ram splitting in s390 code
CC: pbonzini@redhat.com
CC: qemu-s390x@nongnu.org
CC: borntraeger@de.ibm.com
CC: thuth@redhat.com
CC: david@redhat.com
CC: cohuck@redhat.com
Igor Mammedov (2):
kvm: s390: split too big memory section on several memslots
s390: do not call memory_region_allocate_system_memory() multiple
times
include/hw/s390x/s390-virtio-ccw.h | 10 ++++
include/sysemu/kvm_int.h | 1 +
accel/kvm/kvm-all.c | 79 ++++++++++++++++++------------
hw/s390x/s390-virtio-ccw.c | 30 ++----------
target/s390x/kvm.c | 1 +
5 files changed, 63 insertions(+), 58 deletions(-)
--
2.18.1
Patchew URL: https://patchew.org/QEMU/20190802133241.29298-1-imammedo@redhat.com/
Hi,
This series failed build test on s390x host. Please find the details below.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
# Testing script will be invoked under the git checkout with
# HEAD pointing to a commit that has the patches applied on top of "base"
# branch
set -e
echo
echo "=== ENV ==="
env
echo
echo "=== PACKAGES ==="
rpm -qa
echo
echo "=== UNAME ==="
uname -a
CC=$HOME/bin/cc
INSTALL=$PWD/install
BUILD=$PWD/build
mkdir -p $BUILD $INSTALL
SRC=$PWD
cd $BUILD
$SRC/configure --cc=$CC --prefix=$INSTALL
make -j4
# XXX: we need reliable clean up
# make check -j4 V=1
make install
=== TEST SCRIPT END ===
CC s390x-softmmu/target/s390x/sigp.o
CC s390x-softmmu/target/s390x/kvm.o
/var/tmp/patchew-tester-tmp-bcfxbkeg/src/target/s390x/kvm.c: In function ‘kvm_arch_init’:
/var/tmp/patchew-tester-tmp-bcfxbkeg/src/target/s390x/kvm.c:350:5: error: implicit declaration of function ‘kvm_set_max_memslot_size’; did you mean ‘kvm_get_max_memslots’? [-Werror=implicit-function-declaration]
350 | kvm_set_max_memslot_size(KVM_SLOT_MAX_BYTES);
| ^~~~~~~~~~~~~~~~~~~~~~~~
| kvm_get_max_memslots
/var/tmp/patchew-tester-tmp-bcfxbkeg/src/target/s390x/kvm.c:350:5: error: nested extern declaration of ‘kvm_set_max_memslot_size’ [-Werror=nested-externs]
cc1: all warnings being treated as errors
make[1]: *** [/var/tmp/patchew-tester-tmp-bcfxbkeg/src/rules.mak:69: target/s390x/kvm.o] Error 1
make: *** [Makefile:472: s390x-softmmu/all] Error 2
The full log is available at
http://patchew.org/logs/20190802133241.29298-1-imammedo@redhat.com/testing.s390x/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
On 02.08.19 15:32, Igor Mammedov wrote: > Changelog: > since v2: > - break migration from old QEMU (since 2.12-4.1) for guest with >8TB RAM > and drop migratable aliases patch as was agreed during v2 review > - drop 4.2 machines patch as it's not prerequisite anymore > since v1: > - include 4.2 machines patch for adding compat RAM layout on top > - 2/4 add missing in v1 patch for splitting too big MemorySection on > several memslots > - 3/4 amend code path on alias destruction to ensure that RAMBlock is > cleaned properly > - 4/4 add compat machine code to keep old layout (migration-wise) for > 4.1 and older machines > > > While looking into unifying guest RAM allocation to use hostmem backends > for initial RAM (especially when -mempath is used) and retiring > memory_region_allocate_system_memory() API, leaving only single hostmem backend, > I was inspecting how currently it is used by boards and it turns out several > boards abuse it by calling the function several times (despite documented contract > forbiding it). > > s390 is one of such boards where KVM limitation on memslot size got propagated > to board design and memory_region_allocate_system_memory() was abused to satisfy > KVM requirement for max RAM chunk where memory region alias would suffice. > > Unfortunately, memory_region_allocate_system_memory() usage created migration > dependency where guest RAM is transferred in migration stream as several RAMBlocks > if it's more than KVM_SLOT_MAX_BYTES. During v2 review it was agreed to ignore > migration breakage (documenting it in release notes) and leaving only KVM fix. > > In order to replace these several RAM chunks with a single memdev and keep it > working with KVM memslot size limit, following was done: > * [1/2] split too big RAM chunk inside of KVM code on several memory slots > if necessary > * [2/2] drop manual ram splitting in s390 code > > > CC: pbonzini@redhat.com > CC: qemu-s390x@nongnu.org > CC: borntraeger@de.ibm.com > CC: thuth@redhat.com > CC: david@redhat.com > CC: cohuck@redhat.com With the fixup this patch set seems to work on s390. I can start 9TB guests and I can migrate smaller guests between 4.1+patch and 4.0 and 3.1. I currently can not test migration for the 9TB guest due to lack of a 2nd system. > > Igor Mammedov (2): > kvm: s390: split too big memory section on several memslots > s390: do not call memory_region_allocate_system_memory() multiple > times > > include/hw/s390x/s390-virtio-ccw.h | 10 ++++ > include/sysemu/kvm_int.h | 1 + > accel/kvm/kvm-all.c | 79 ++++++++++++++++++------------ > hw/s390x/s390-virtio-ccw.c | 30 ++---------- > target/s390x/kvm.c | 1 + > 5 files changed, 63 insertions(+), 58 deletions(-) >
On 02.08.19 16:42, Christian Borntraeger wrote: > On 02.08.19 15:32, Igor Mammedov wrote: >> Changelog: >> since v2: >> - break migration from old QEMU (since 2.12-4.1) for guest with >8TB RAM >> and drop migratable aliases patch as was agreed during v2 review >> - drop 4.2 machines patch as it's not prerequisite anymore >> since v1: >> - include 4.2 machines patch for adding compat RAM layout on top >> - 2/4 add missing in v1 patch for splitting too big MemorySection on >> several memslots >> - 3/4 amend code path on alias destruction to ensure that RAMBlock is >> cleaned properly >> - 4/4 add compat machine code to keep old layout (migration-wise) for >> 4.1 and older machines >> >> >> While looking into unifying guest RAM allocation to use hostmem backends >> for initial RAM (especially when -mempath is used) and retiring >> memory_region_allocate_system_memory() API, leaving only single hostmem backend, >> I was inspecting how currently it is used by boards and it turns out several >> boards abuse it by calling the function several times (despite documented contract >> forbiding it). >> >> s390 is one of such boards where KVM limitation on memslot size got propagated >> to board design and memory_region_allocate_system_memory() was abused to satisfy >> KVM requirement for max RAM chunk where memory region alias would suffice. >> >> Unfortunately, memory_region_allocate_system_memory() usage created migration >> dependency where guest RAM is transferred in migration stream as several RAMBlocks >> if it's more than KVM_SLOT_MAX_BYTES. During v2 review it was agreed to ignore >> migration breakage (documenting it in release notes) and leaving only KVM fix. >> >> In order to replace these several RAM chunks with a single memdev and keep it >> working with KVM memslot size limit, following was done: >> * [1/2] split too big RAM chunk inside of KVM code on several memory slots >> if necessary >> * [2/2] drop manual ram splitting in s390 code >> >> >> CC: pbonzini@redhat.com >> CC: qemu-s390x@nongnu.org >> CC: borntraeger@de.ibm.com >> CC: thuth@redhat.com >> CC: david@redhat.com >> CC: cohuck@redhat.com > > With the fixup this patch set seems to work on s390. I can start 9TB guests and > I can migrate smaller guests between 4.1+patch and 4.0 and 3.1. I currently can > not test migration for the 9TB guest due to lack of a 2nd system. I have to correct myself. The 9TB guest started up but it does not seem to do anything useful (it hangs). I will try to investigate. >> >> Igor Mammedov (2): >> kvm: s390: split too big memory section on several memslots >> s390: do not call memory_region_allocate_system_memory() multiple >> times >> >> include/hw/s390x/s390-virtio-ccw.h | 10 ++++ >> include/sysemu/kvm_int.h | 1 + >> accel/kvm/kvm-all.c | 79 ++++++++++++++++++------------ >> hw/s390x/s390-virtio-ccw.c | 30 ++---------- >> target/s390x/kvm.c | 1 + >> 5 files changed, 63 insertions(+), 58 deletions(-) >>
On 02.08.19 16:59, Christian Borntraeger wrote:
>
>
> On 02.08.19 16:42, Christian Borntraeger wrote:
>> On 02.08.19 15:32, Igor Mammedov wrote:
>>> Changelog:
>>> since v2:
>>> - break migration from old QEMU (since 2.12-4.1) for guest with >8TB RAM
>>> and drop migratable aliases patch as was agreed during v2 review
>>> - drop 4.2 machines patch as it's not prerequisite anymore
>>> since v1:
>>> - include 4.2 machines patch for adding compat RAM layout on top
>>> - 2/4 add missing in v1 patch for splitting too big MemorySection on
>>> several memslots
>>> - 3/4 amend code path on alias destruction to ensure that RAMBlock is
>>> cleaned properly
>>> - 4/4 add compat machine code to keep old layout (migration-wise) for
>>> 4.1 and older machines
>>>
>>>
>>> While looking into unifying guest RAM allocation to use hostmem backends
>>> for initial RAM (especially when -mempath is used) and retiring
>>> memory_region_allocate_system_memory() API, leaving only single hostmem backend,
>>> I was inspecting how currently it is used by boards and it turns out several
>>> boards abuse it by calling the function several times (despite documented contract
>>> forbiding it).
>>>
>>> s390 is one of such boards where KVM limitation on memslot size got propagated
>>> to board design and memory_region_allocate_system_memory() was abused to satisfy
>>> KVM requirement for max RAM chunk where memory region alias would suffice.
>>>
>>> Unfortunately, memory_region_allocate_system_memory() usage created migration
>>> dependency where guest RAM is transferred in migration stream as several RAMBlocks
>>> if it's more than KVM_SLOT_MAX_BYTES. During v2 review it was agreed to ignore
>>> migration breakage (documenting it in release notes) and leaving only KVM fix.
>>>
>>> In order to replace these several RAM chunks with a single memdev and keep it
>>> working with KVM memslot size limit, following was done:
>>> * [1/2] split too big RAM chunk inside of KVM code on several memory slots
>>> if necessary
>>> * [2/2] drop manual ram splitting in s390 code
>>>
>>>
>>> CC: pbonzini@redhat.com
>>> CC: qemu-s390x@nongnu.org
>>> CC: borntraeger@de.ibm.com
>>> CC: thuth@redhat.com
>>> CC: david@redhat.com
>>> CC: cohuck@redhat.com
>>
>> With the fixup this patch set seems to work on s390. I can start 9TB guests and
>> I can migrate smaller guests between 4.1+patch and 4.0 and 3.1. I currently can
>> not test migration for the 9TB guest due to lack of a 2nd system.
>
> I have to correct myself. The 9TB guest started up but it does not seem to do
> anything useful (it hangs).
Seems that the userspace addr is wrong (its the same).
[pid 258234] ioctl(10, KVM_SET_USER_MEMORY_REGION, {slot=0, flags=0, guest_phys_addr=0, memory_size=8796091973632, userspace_addr=0x3fff7d00000}) = 0
[pid 258234] ioctl(10, KVM_SET_USER_MEMORY_REGION, {slot=1, flags=0, guest_phys_addr=0x7fffff00000, memory_size=1099512676352, userspace_addr=0x3fff7d00000}) = 0
On Fri, 2 Aug 2019 17:04:21 +0200
Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> On 02.08.19 16:59, Christian Borntraeger wrote:
> >
> >
> > On 02.08.19 16:42, Christian Borntraeger wrote:
> >> On 02.08.19 15:32, Igor Mammedov wrote:
> >>> Changelog:
> >>> since v2:
> >>> - break migration from old QEMU (since 2.12-4.1) for guest with >8TB RAM
> >>> and drop migratable aliases patch as was agreed during v2 review
> >>> - drop 4.2 machines patch as it's not prerequisite anymore
> >>> since v1:
> >>> - include 4.2 machines patch for adding compat RAM layout on top
> >>> - 2/4 add missing in v1 patch for splitting too big MemorySection on
> >>> several memslots
> >>> - 3/4 amend code path on alias destruction to ensure that RAMBlock is
> >>> cleaned properly
> >>> - 4/4 add compat machine code to keep old layout (migration-wise) for
> >>> 4.1 and older machines
> >>>
> >>>
> >>> While looking into unifying guest RAM allocation to use hostmem backends
> >>> for initial RAM (especially when -mempath is used) and retiring
> >>> memory_region_allocate_system_memory() API, leaving only single hostmem backend,
> >>> I was inspecting how currently it is used by boards and it turns out several
> >>> boards abuse it by calling the function several times (despite documented contract
> >>> forbiding it).
> >>>
> >>> s390 is one of such boards where KVM limitation on memslot size got propagated
> >>> to board design and memory_region_allocate_system_memory() was abused to satisfy
> >>> KVM requirement for max RAM chunk where memory region alias would suffice.
> >>>
> >>> Unfortunately, memory_region_allocate_system_memory() usage created migration
> >>> dependency where guest RAM is transferred in migration stream as several RAMBlocks
> >>> if it's more than KVM_SLOT_MAX_BYTES. During v2 review it was agreed to ignore
> >>> migration breakage (documenting it in release notes) and leaving only KVM fix.
> >>>
> >>> In order to replace these several RAM chunks with a single memdev and keep it
> >>> working with KVM memslot size limit, following was done:
> >>> * [1/2] split too big RAM chunk inside of KVM code on several memory slots
> >>> if necessary
> >>> * [2/2] drop manual ram splitting in s390 code
> >>>
> >>>
> >>> CC: pbonzini@redhat.com
> >>> CC: qemu-s390x@nongnu.org
> >>> CC: borntraeger@de.ibm.com
> >>> CC: thuth@redhat.com
> >>> CC: david@redhat.com
> >>> CC: cohuck@redhat.com
> >>
> >> With the fixup this patch set seems to work on s390. I can start 9TB guests and
> >> I can migrate smaller guests between 4.1+patch and 4.0 and 3.1. I currently can
> >> not test migration for the 9TB guest due to lack of a 2nd system.
> >
> > I have to correct myself. The 9TB guest started up but it does not seem to do
> > anything useful (it hangs).
>
> Seems that the userspace addr is wrong (its the same).
> [pid 258234] ioctl(10, KVM_SET_USER_MEMORY_REGION, {slot=0, flags=0, guest_phys_addr=0, memory_size=8796091973632, userspace_addr=0x3fff7d00000}) = 0
> [pid 258234] ioctl(10, KVM_SET_USER_MEMORY_REGION, {slot=1, flags=0, guest_phys_addr=0x7fffff00000, memory_size=1099512676352, userspace_addr=0x3fff7d00000}) = 0
It's a bug in 1/2, I forgot to advance mem->ram along with mem->start_addr.
Let me fix it and simulate it on small s390 host (/me sorry for messy patches)
it won't test migration properly but should be sufficient for testing KVM code patch.
On Mon, 5 Aug 2019 10:54:40 +0200
Igor Mammedov <imammedo@redhat.com> wrote:
> On Fri, 2 Aug 2019 17:04:21 +0200
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
>
> > On 02.08.19 16:59, Christian Borntraeger wrote:
> > >
> > >
> > > On 02.08.19 16:42, Christian Borntraeger wrote:
> > >> On 02.08.19 15:32, Igor Mammedov wrote:
> > >>> Changelog:
> > >>> since v2:
> > >>> - break migration from old QEMU (since 2.12-4.1) for guest with >8TB RAM
> > >>> and drop migratable aliases patch as was agreed during v2 review
FWIW, that seems reasonable to me as well.
> > >>> - drop 4.2 machines patch as it's not prerequisite anymore
> > >>> since v1:
> > >>> - include 4.2 machines patch for adding compat RAM layout on top
> > >>> - 2/4 add missing in v1 patch for splitting too big MemorySection on
> > >>> several memslots
> > >>> - 3/4 amend code path on alias destruction to ensure that RAMBlock is
> > >>> cleaned properly
> > >>> - 4/4 add compat machine code to keep old layout (migration-wise) for
> > >>> 4.1 and older machines
> > >>>
> > >>>
> > >>> While looking into unifying guest RAM allocation to use hostmem backends
> > >>> for initial RAM (especially when -mempath is used) and retiring
> > >>> memory_region_allocate_system_memory() API, leaving only single hostmem backend,
> > >>> I was inspecting how currently it is used by boards and it turns out several
> > >>> boards abuse it by calling the function several times (despite documented contract
> > >>> forbiding it).
> > >>>
> > >>> s390 is one of such boards where KVM limitation on memslot size got propagated
> > >>> to board design and memory_region_allocate_system_memory() was abused to satisfy
> > >>> KVM requirement for max RAM chunk where memory region alias would suffice.
> > >>>
> > >>> Unfortunately, memory_region_allocate_system_memory() usage created migration
> > >>> dependency where guest RAM is transferred in migration stream as several RAMBlocks
> > >>> if it's more than KVM_SLOT_MAX_BYTES. During v2 review it was agreed to ignore
> > >>> migration breakage (documenting it in release notes) and leaving only KVM fix.
> > >>>
> > >>> In order to replace these several RAM chunks with a single memdev and keep it
> > >>> working with KVM memslot size limit, following was done:
> > >>> * [1/2] split too big RAM chunk inside of KVM code on several memory slots
> > >>> if necessary
> > >>> * [2/2] drop manual ram splitting in s390 code
> > >>>
> > >>>
> > >>> CC: pbonzini@redhat.com
> > >>> CC: qemu-s390x@nongnu.org
> > >>> CC: borntraeger@de.ibm.com
> > >>> CC: thuth@redhat.com
> > >>> CC: david@redhat.com
> > >>> CC: cohuck@redhat.com
> > >>
> > >> With the fixup this patch set seems to work on s390. I can start 9TB guests and
> > >> I can migrate smaller guests between 4.1+patch and 4.0 and 3.1. I currently can
> > >> not test migration for the 9TB guest due to lack of a 2nd system.
> > >
> > > I have to correct myself. The 9TB guest started up but it does not seem to do
> > > anything useful (it hangs).
> >
> > Seems that the userspace addr is wrong (its the same).
> > [pid 258234] ioctl(10, KVM_SET_USER_MEMORY_REGION, {slot=0, flags=0, guest_phys_addr=0, memory_size=8796091973632, userspace_addr=0x3fff7d00000}) = 0
> > [pid 258234] ioctl(10, KVM_SET_USER_MEMORY_REGION, {slot=1, flags=0, guest_phys_addr=0x7fffff00000, memory_size=1099512676352, userspace_addr=0x3fff7d00000}) = 0
>
> It's a bug in 1/2, I forgot to advance mem->ram along with mem->start_addr.
> Let me fix it and simulate it on small s390 host (/me sorry for messy patches)
> it won't test migration properly but should be sufficient for testing KVM code patch.
>
Ok, I'll wait for a v4 before I spend any time on this :)
© 2016 - 2024 Red Hat, Inc.