This series has been successfully tested in s390x. Booted up a VM including:
/home/qemu/build/qemu-system-s390x \
...
-m 4G,maxmem=20G \
-object memory-backend-ram,id=mem0,size=16G,reserve=off \
-device virtio-mem-ccw,id=vmem0,memdev=mem0,dynamic-memslots=on \
...
Check the memory devices
(qemu) info memory-devices
Memory device [virtio-mem]: "vmem0"
memaddr: 0x100000000
node: 0
requested-size: 0
size: 0
max-size: 17179869184
block-size: 1048576
memdev: /objects/mem0
Hotplug memory to the maximum
(qemu) qom-set vmem0 requested-size 16G
(qemu) info memory-devices
Memory device [virtio-mem]: "vmem0"
memaddr: 0x100000000
node: 0
requested-size: 17179869184
size: 17179869184
max-size: 17179869184
block-size: 1048576
memdev: /objects/mem0
Try to unplug the device, check the error message is correct
(qemu) device_del vmem0
Error: virtio-mem device cannot get unplugged while some of its memory
is still plugged
Reduce the memory till 0 and device is successfully unplugged
(qemu) qom-set vmem0 requested-size 0
(qemu) device_del vmem0
(qemu) object_del mem0
Hotplug again the device
(qemu) object_add memory-backend-ram,id=mem0,size=8G,reserve=off
(qemu) device_add
virtio-mem-ccw,id=vmem0,memdev=mem0,dynamic-memslots=on,requested-size=1G
(qemu) info memory-devices
Memory device [virtio-mem]: "vmem0"
memaddr: 0x100000000
node: 0
requested-size: 1073741824
size: 1073741824
max-size: 8589934592
block-size: 1048576
memdev: /objects/mem0
Finally, add some extra memory and check the memslots
(qemu) qom-set vmem0 requested-size 4G
(qemu) info mtree
...
0000000100000000-00000002ffffffff (prio 0, i/o): virtio-mem
0000000100000000-000000013fffffff (prio 0, ram): alias memslot-0
@mem0 0000000000000000-000000003fffffff
0000000140000000-000000017fffffff (prio 0, ram): alias memslot-1
@mem0 0000000040000000-000000007fffffff
0000000180000000-00000001bfffffff (prio 0, ram): alias memslot-2
@mem0 0000000080000000-00000000bfffffff
00000001c0000000-00000001ffffffff (prio 0, ram): alias memslot-3
@mem0 00000000c0000000-00000000ffffffff
Tested-by: Mario Casquero <mcasquer@redhat.com>
On Tue, Oct 8, 2024 at 12:57 PM David Hildenbrand <david@redhat.com> wrote:
>
> Based on current master.
>
> There is really not much left to do on s390x, because virtio-mem already
> implements most things we need today (e.g., early-migration,
> unplugged-inaccessible). The biggest part of this series is just doing what
> we do with virtio-pci, wiring it up in the machine hotplug handler and ...
> well, messing with the physical memory layout where we can now exceed
> initial RAM size and have sparsity (memory holes).
>
> I tested a lot of things, including:
> * Memory hotplug/unplug
> * Device hotplug/unplug
> * System resets / reboots
> * Migrate to/from file (including storage attributes under KVM)
> * Basic live migration
> * Basic postcopy live migration
>
> More details on how to use it on s390x -- which is pretty much how
> we use it on other architectures, except
> s/virtio-mem-pci/virtio-mem-ccw/ --- is in the last patch.
>
> This series introduces a new diag(500) "STORAGE LIMIT" subcode that will
> be documented in the kernel and at [2] once this+kernel part go upstream.
>
> There are not many s390x-specific virtio-mem future work items, except:
> * Storage attribute migration might be improved
> * We might want to reset storage attributes of unplugged memory
> (might or might not be required for upcoming page table reclaim in
> Linux; TBD)
>
> The Linux driver is available at [3].
>
> [1] https://lkml.kernel.org/r/20240906101658.514470-1-pbonzini@redhat.com
> [2] https://gitlab.com/davidhildenbrand/s390x-os-virt-spec
> [3] https://lkml.kernel.org/r/20240910191541.2179655-6-david@redhat.com
>
> v1 -> v2:
> * "s390x/s390-virtio-hcall: prepare for more diag500 hypercalls"
> - Turn handle_diag_500() into a void function
> - Inject PGM_SPECIFICATION from handle_diag_500() directly
> * "s390x/s390-virtio-ccw: move setting the maximum guest size from sclp to
> machine code"
> - Move code to a new function to make further changes easier
> - Adjust s390_pv_vm_try_disable_async() to stay in sync with the
> maxram->ram change
> * "s390x: introduce s390_get_memory_limit()"
> - Store limit in machine
> - Move s390_set_memory_limit() from target code into machine code
> * "s390x/s390-hypercall: introduce DIAG500 STORAGE_LIMIT"
> - Move handling into a separate function now that we lookup the machine
> * "s390x/s390-stattrib-kvm: prepare for memory devices and sparse memory
> layouts"
> - Adjust to s390_get_memory_limit() changes
> * "s390x/s390-skeys: prepare for memory devices"
> - Adjust to s390_get_memory_limit() changes
> * "s390x/pv: prepare for memory devices"
> - Use s390_get_memory_limit()
> * "s390x: remember the maximum page size"
> - Store it in the machine
> - Move s390_set_max_pagesize() from target code into machine code
> - No need for s390_get_max_pagesize()
> * "s390x/virtio-ccw: add support for virtio based memory devices"
> - Move machine wire-up code from virtio-mem patch into this patch
> - Add stubs to make compilation without virtio-mem work
> * "s390x: virtio-mem support"
> - Move machine write-up code to previous patch
>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Thomas Huth <thuth@redhat.com>
> Cc: Halil Pasic <pasic@linux.ibm.com>
> Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
> Cc: Eric Farman <farman@linux.ibm.com>
> Cc: Richard Henderson <richard.henderson@linaro.org>
> Cc: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
> Cc: Heiko Carstens <hca@linux.ibm.com>
> Cc: Ilya Leoshkevich <iii@linux.ibm.com>
> Cc: Janosch Frank <frankja@linux.ibm.com>
> Cc: "Michael S. Tsirkin" <mst@redhat.com>
> Cc: Cornelia Huck <cohuck@redhat.com>
>
> David Hildenbrand (14):
> s390x/s390-virtio-ccw: don't crash on weird RAM sizes
> s390x/s390-virtio-hcall: remove hypercall registration mechanism
> s390x/s390-virtio-hcall: prepare for more diag500 hypercalls
> s390x: rename s390-virtio-hcall* to s390-hypercall*
> s390x/s390-virtio-ccw: move setting the maximum guest size from sclp
> to machine code
> s390x: introduce s390_get_memory_limit()
> s390x/s390-hypercall: introduce DIAG500 STORAGE_LIMIT
> s390x/s390-stattrib-kvm: prepare for memory devices and sparse memory
> layouts
> s390x/s390-skeys: prepare for memory devices
> s390x/s390-virtio-ccw: prepare for memory devices
> s390x/pv: prepare for memory devices
> s390x: remember the maximum page size
> s390x/virtio-ccw: add support for virtio based memory devices
> s390x: virtio-mem support
>
> MAINTAINERS | 5 +
> hw/s390x/Kconfig | 1 +
> hw/s390x/meson.build | 6 +-
> hw/s390x/s390-hypercall.c | 85 +++++++++++
> hw/s390x/s390-hypercall.h | 25 ++++
> hw/s390x/s390-skeys.c | 6 +-
> hw/s390x/s390-stattrib-kvm.c | 67 ++++++---
> hw/s390x/s390-virtio-ccw.c | 165 ++++++++++++++-------
> hw/s390x/s390-virtio-hcall.c | 41 ------
> hw/s390x/s390-virtio-hcall.h | 25 ----
> hw/s390x/sclp.c | 17 +--
> hw/s390x/virtio-ccw-md-stubs.c | 24 +++
> hw/s390x/virtio-ccw-md.c | 153 +++++++++++++++++++
> hw/s390x/virtio-ccw-md.h | 44 ++++++
> hw/s390x/virtio-ccw-mem.c | 226 +++++++++++++++++++++++++++++
> hw/s390x/virtio-ccw-mem.h | 34 +++++
> hw/virtio/Kconfig | 1 +
> hw/virtio/virtio-mem.c | 4 +-
> include/hw/s390x/s390-virtio-ccw.h | 4 +
> target/s390x/cpu-sysemu.c | 15 --
> target/s390x/cpu.h | 2 -
> target/s390x/kvm/kvm.c | 18 +--
> target/s390x/kvm/pv.c | 2 +-
> target/s390x/tcg/misc_helper.c | 7 +-
> 24 files changed, 782 insertions(+), 195 deletions(-)
> create mode 100644 hw/s390x/s390-hypercall.c
> create mode 100644 hw/s390x/s390-hypercall.h
> delete mode 100644 hw/s390x/s390-virtio-hcall.c
> delete mode 100644 hw/s390x/s390-virtio-hcall.h
> create mode 100644 hw/s390x/virtio-ccw-md-stubs.c
> create mode 100644 hw/s390x/virtio-ccw-md.c
> create mode 100644 hw/s390x/virtio-ccw-md.h
> create mode 100644 hw/s390x/virtio-ccw-mem.c
> create mode 100644 hw/s390x/virtio-ccw-mem.h
>
> --
> 2.46.1
>
>