[PATCH v2 00/14] s390x: virtio-mem support

David Hildenbrand posted 14 patches 1 month, 2 weeks ago
MAINTAINERS                        |   5 +
hw/s390x/Kconfig                   |   1 +
hw/s390x/meson.build               |   6 +-
hw/s390x/s390-hypercall.c          |  85 +++++++++++
hw/s390x/s390-hypercall.h          |  25 ++++
hw/s390x/s390-skeys.c              |   6 +-
hw/s390x/s390-stattrib-kvm.c       |  67 ++++++---
hw/s390x/s390-virtio-ccw.c         | 165 ++++++++++++++-------
hw/s390x/s390-virtio-hcall.c       |  41 ------
hw/s390x/s390-virtio-hcall.h       |  25 ----
hw/s390x/sclp.c                    |  17 +--
hw/s390x/virtio-ccw-md-stubs.c     |  24 +++
hw/s390x/virtio-ccw-md.c           | 153 +++++++++++++++++++
hw/s390x/virtio-ccw-md.h           |  44 ++++++
hw/s390x/virtio-ccw-mem.c          | 226 +++++++++++++++++++++++++++++
hw/s390x/virtio-ccw-mem.h          |  34 +++++
hw/virtio/Kconfig                  |   1 +
hw/virtio/virtio-mem.c             |   4 +-
include/hw/s390x/s390-virtio-ccw.h |   4 +
target/s390x/cpu-sysemu.c          |  15 --
target/s390x/cpu.h                 |   2 -
target/s390x/kvm/kvm.c             |  18 +--
target/s390x/kvm/pv.c              |   2 +-
target/s390x/tcg/misc_helper.c     |   7 +-
24 files changed, 782 insertions(+), 195 deletions(-)
create mode 100644 hw/s390x/s390-hypercall.c
create mode 100644 hw/s390x/s390-hypercall.h
delete mode 100644 hw/s390x/s390-virtio-hcall.c
delete mode 100644 hw/s390x/s390-virtio-hcall.h
create mode 100644 hw/s390x/virtio-ccw-md-stubs.c
create mode 100644 hw/s390x/virtio-ccw-md.c
create mode 100644 hw/s390x/virtio-ccw-md.h
create mode 100644 hw/s390x/virtio-ccw-mem.c
create mode 100644 hw/s390x/virtio-ccw-mem.h
[PATCH v2 00/14] s390x: virtio-mem support
Posted by David Hildenbrand 1 month, 2 weeks ago
Based on current master.

There is really not much left to do on s390x, because virtio-mem already
implements most things we need today (e.g., early-migration,
unplugged-inaccessible). The biggest part of this series is just doing what
we do with virtio-pci, wiring it up in the machine hotplug handler and ...
well, messing with the physical memory layout where we can now exceed
initial RAM size and have sparsity (memory holes).

I tested a lot of things, including:
 * Memory hotplug/unplug
 * Device hotplug/unplug
 * System resets / reboots
 * Migrate to/from file (including storage attributes under KVM)
 * Basic live migration
 * Basic postcopy live migration

More details on how to use it on s390x -- which is pretty much how
we use it on other architectures, except
s/virtio-mem-pci/virtio-mem-ccw/ --- is in the last patch.

This series introduces a new diag(500) "STORAGE LIMIT" subcode that will
be documented in the kernel and at [2] once this+kernel part go upstream.

There are not many s390x-specific virtio-mem future work items, except:
* Storage attribute migration might be improved
* We might want to reset storage attributes of unplugged memory
  (might or might not be required for upcoming page table reclaim in
   Linux; TBD)

The Linux driver is available at [3].

[1] https://lkml.kernel.org/r/20240906101658.514470-1-pbonzini@redhat.com
[2] https://gitlab.com/davidhildenbrand/s390x-os-virt-spec
[3] https://lkml.kernel.org/r/20240910191541.2179655-6-david@redhat.com

v1 -> v2:
* "s390x/s390-virtio-hcall: prepare for more diag500 hypercalls"
 - Turn handle_diag_500() into a void function
 - Inject PGM_SPECIFICATION from handle_diag_500() directly
* "s390x/s390-virtio-ccw: move setting the maximum guest size from sclp to
   machine code"
 - Move code to a new function to make further changes easier
 - Adjust s390_pv_vm_try_disable_async() to stay in sync with the
   maxram->ram change
* "s390x: introduce s390_get_memory_limit()"
 - Store limit in machine
 - Move s390_set_memory_limit() from target code into machine code
* "s390x/s390-hypercall: introduce DIAG500 STORAGE_LIMIT"
 - Move handling into a separate function now that we lookup the machine
* "s390x/s390-stattrib-kvm: prepare for memory devices and sparse memory
   layouts"
 - Adjust to s390_get_memory_limit() changes
* "s390x/s390-skeys: prepare for memory devices"
 - Adjust to s390_get_memory_limit() changes
* "s390x/pv: prepare for memory devices"
 - Use s390_get_memory_limit()
* "s390x: remember the maximum page size"
 - Store it in the machine
 - Move s390_set_max_pagesize() from target code into machine code
 - No need for s390_get_max_pagesize()
* "s390x/virtio-ccw: add support for virtio based memory devices"
 - Move machine wire-up code from virtio-mem patch into this patch
 - Add stubs to make compilation without virtio-mem work
* "s390x: virtio-mem support"
 - Move machine write-up code to previous patch

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Thomas Huth <thuth@redhat.com>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Eric Farman <farman@linux.ibm.com>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Ilya Leoshkevich <iii@linux.ibm.com>
Cc: Janosch Frank <frankja@linux.ibm.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Cornelia Huck <cohuck@redhat.com>

David Hildenbrand (14):
  s390x/s390-virtio-ccw: don't crash on weird RAM sizes
  s390x/s390-virtio-hcall: remove hypercall registration mechanism
  s390x/s390-virtio-hcall: prepare for more diag500 hypercalls
  s390x: rename s390-virtio-hcall* to s390-hypercall*
  s390x/s390-virtio-ccw: move setting the maximum guest size from sclp
    to machine code
  s390x: introduce s390_get_memory_limit()
  s390x/s390-hypercall: introduce DIAG500 STORAGE_LIMIT
  s390x/s390-stattrib-kvm: prepare for memory devices and sparse memory
    layouts
  s390x/s390-skeys: prepare for memory devices
  s390x/s390-virtio-ccw: prepare for memory devices
  s390x/pv: prepare for memory devices
  s390x: remember the maximum page size
  s390x/virtio-ccw: add support for virtio based memory devices
  s390x: virtio-mem support

 MAINTAINERS                        |   5 +
 hw/s390x/Kconfig                   |   1 +
 hw/s390x/meson.build               |   6 +-
 hw/s390x/s390-hypercall.c          |  85 +++++++++++
 hw/s390x/s390-hypercall.h          |  25 ++++
 hw/s390x/s390-skeys.c              |   6 +-
 hw/s390x/s390-stattrib-kvm.c       |  67 ++++++---
 hw/s390x/s390-virtio-ccw.c         | 165 ++++++++++++++-------
 hw/s390x/s390-virtio-hcall.c       |  41 ------
 hw/s390x/s390-virtio-hcall.h       |  25 ----
 hw/s390x/sclp.c                    |  17 +--
 hw/s390x/virtio-ccw-md-stubs.c     |  24 +++
 hw/s390x/virtio-ccw-md.c           | 153 +++++++++++++++++++
 hw/s390x/virtio-ccw-md.h           |  44 ++++++
 hw/s390x/virtio-ccw-mem.c          | 226 +++++++++++++++++++++++++++++
 hw/s390x/virtio-ccw-mem.h          |  34 +++++
 hw/virtio/Kconfig                  |   1 +
 hw/virtio/virtio-mem.c             |   4 +-
 include/hw/s390x/s390-virtio-ccw.h |   4 +
 target/s390x/cpu-sysemu.c          |  15 --
 target/s390x/cpu.h                 |   2 -
 target/s390x/kvm/kvm.c             |  18 +--
 target/s390x/kvm/pv.c              |   2 +-
 target/s390x/tcg/misc_helper.c     |   7 +-
 24 files changed, 782 insertions(+), 195 deletions(-)
 create mode 100644 hw/s390x/s390-hypercall.c
 create mode 100644 hw/s390x/s390-hypercall.h
 delete mode 100644 hw/s390x/s390-virtio-hcall.c
 delete mode 100644 hw/s390x/s390-virtio-hcall.h
 create mode 100644 hw/s390x/virtio-ccw-md-stubs.c
 create mode 100644 hw/s390x/virtio-ccw-md.c
 create mode 100644 hw/s390x/virtio-ccw-md.h
 create mode 100644 hw/s390x/virtio-ccw-mem.c
 create mode 100644 hw/s390x/virtio-ccw-mem.h

-- 
2.46.1
Re: [PATCH v2 00/14] s390x: virtio-mem support
Posted by Mario Casquero 1 month, 2 weeks ago
This series has been successfully tested in s390x. Booted up a VM including:

/home/qemu/build/qemu-system-s390x \
...
-m 4G,maxmem=20G \
-object memory-backend-ram,id=mem0,size=16G,reserve=off \
-device virtio-mem-ccw,id=vmem0,memdev=mem0,dynamic-memslots=on \
...

Check the memory devices
(qemu) info memory-devices
Memory device [virtio-mem]: "vmem0"
memaddr: 0x100000000
node: 0
requested-size: 0
size: 0
max-size: 17179869184
block-size: 1048576
memdev: /objects/mem0

Hotplug memory to the maximum
(qemu) qom-set vmem0 requested-size 16G
(qemu) info memory-devices
Memory device [virtio-mem]: "vmem0"
  memaddr: 0x100000000
  node: 0
  requested-size: 17179869184
  size: 17179869184
  max-size: 17179869184
  block-size: 1048576
  memdev: /objects/mem0

Try to unplug the device, check the error message is correct
(qemu) device_del vmem0
Error: virtio-mem device cannot get unplugged while some of its memory
is still plugged

Reduce the memory till 0 and device is successfully unplugged
(qemu) qom-set vmem0 requested-size 0
(qemu) device_del vmem0
(qemu) object_del mem0

Hotplug again the device
(qemu) object_add memory-backend-ram,id=mem0,size=8G,reserve=off
(qemu) device_add
virtio-mem-ccw,id=vmem0,memdev=mem0,dynamic-memslots=on,requested-size=1G
(qemu) info memory-devices
Memory device [virtio-mem]: "vmem0"
memaddr: 0x100000000
node: 0
requested-size: 1073741824
size: 1073741824
max-size: 8589934592
block-size: 1048576
memdev: /objects/mem0

Finally, add some extra memory and check the memslots
(qemu) qom-set vmem0 requested-size 4G
(qemu) info mtree
...
    0000000100000000-00000002ffffffff (prio 0, i/o): virtio-mem
      0000000100000000-000000013fffffff (prio 0, ram): alias memslot-0
@mem0 0000000000000000-000000003fffffff
      0000000140000000-000000017fffffff (prio 0, ram): alias memslot-1
@mem0 0000000040000000-000000007fffffff
      0000000180000000-00000001bfffffff (prio 0, ram): alias memslot-2
@mem0 0000000080000000-00000000bfffffff
      00000001c0000000-00000001ffffffff (prio 0, ram): alias memslot-3
@mem0 00000000c0000000-00000000ffffffff

Tested-by: Mario Casquero <mcasquer@redhat.com>


On Tue, Oct 8, 2024 at 12:57 PM David Hildenbrand <david@redhat.com> wrote:
>
> Based on current master.
>
> There is really not much left to do on s390x, because virtio-mem already
> implements most things we need today (e.g., early-migration,
> unplugged-inaccessible). The biggest part of this series is just doing what
> we do with virtio-pci, wiring it up in the machine hotplug handler and ...
> well, messing with the physical memory layout where we can now exceed
> initial RAM size and have sparsity (memory holes).
>
> I tested a lot of things, including:
>  * Memory hotplug/unplug
>  * Device hotplug/unplug
>  * System resets / reboots
>  * Migrate to/from file (including storage attributes under KVM)
>  * Basic live migration
>  * Basic postcopy live migration
>
> More details on how to use it on s390x -- which is pretty much how
> we use it on other architectures, except
> s/virtio-mem-pci/virtio-mem-ccw/ --- is in the last patch.
>
> This series introduces a new diag(500) "STORAGE LIMIT" subcode that will
> be documented in the kernel and at [2] once this+kernel part go upstream.
>
> There are not many s390x-specific virtio-mem future work items, except:
> * Storage attribute migration might be improved
> * We might want to reset storage attributes of unplugged memory
>   (might or might not be required for upcoming page table reclaim in
>    Linux; TBD)
>
> The Linux driver is available at [3].
>
> [1] https://lkml.kernel.org/r/20240906101658.514470-1-pbonzini@redhat.com
> [2] https://gitlab.com/davidhildenbrand/s390x-os-virt-spec
> [3] https://lkml.kernel.org/r/20240910191541.2179655-6-david@redhat.com
>
> v1 -> v2:
> * "s390x/s390-virtio-hcall: prepare for more diag500 hypercalls"
>  - Turn handle_diag_500() into a void function
>  - Inject PGM_SPECIFICATION from handle_diag_500() directly
> * "s390x/s390-virtio-ccw: move setting the maximum guest size from sclp to
>    machine code"
>  - Move code to a new function to make further changes easier
>  - Adjust s390_pv_vm_try_disable_async() to stay in sync with the
>    maxram->ram change
> * "s390x: introduce s390_get_memory_limit()"
>  - Store limit in machine
>  - Move s390_set_memory_limit() from target code into machine code
> * "s390x/s390-hypercall: introduce DIAG500 STORAGE_LIMIT"
>  - Move handling into a separate function now that we lookup the machine
> * "s390x/s390-stattrib-kvm: prepare for memory devices and sparse memory
>    layouts"
>  - Adjust to s390_get_memory_limit() changes
> * "s390x/s390-skeys: prepare for memory devices"
>  - Adjust to s390_get_memory_limit() changes
> * "s390x/pv: prepare for memory devices"
>  - Use s390_get_memory_limit()
> * "s390x: remember the maximum page size"
>  - Store it in the machine
>  - Move s390_set_max_pagesize() from target code into machine code
>  - No need for s390_get_max_pagesize()
> * "s390x/virtio-ccw: add support for virtio based memory devices"
>  - Move machine wire-up code from virtio-mem patch into this patch
>  - Add stubs to make compilation without virtio-mem work
> * "s390x: virtio-mem support"
>  - Move machine write-up code to previous patch
>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Thomas Huth <thuth@redhat.com>
> Cc: Halil Pasic <pasic@linux.ibm.com>
> Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
> Cc: Eric Farman <farman@linux.ibm.com>
> Cc: Richard Henderson <richard.henderson@linaro.org>
> Cc: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
> Cc: Heiko Carstens <hca@linux.ibm.com>
> Cc: Ilya Leoshkevich <iii@linux.ibm.com>
> Cc: Janosch Frank <frankja@linux.ibm.com>
> Cc: "Michael S. Tsirkin" <mst@redhat.com>
> Cc: Cornelia Huck <cohuck@redhat.com>
>
> David Hildenbrand (14):
>   s390x/s390-virtio-ccw: don't crash on weird RAM sizes
>   s390x/s390-virtio-hcall: remove hypercall registration mechanism
>   s390x/s390-virtio-hcall: prepare for more diag500 hypercalls
>   s390x: rename s390-virtio-hcall* to s390-hypercall*
>   s390x/s390-virtio-ccw: move setting the maximum guest size from sclp
>     to machine code
>   s390x: introduce s390_get_memory_limit()
>   s390x/s390-hypercall: introduce DIAG500 STORAGE_LIMIT
>   s390x/s390-stattrib-kvm: prepare for memory devices and sparse memory
>     layouts
>   s390x/s390-skeys: prepare for memory devices
>   s390x/s390-virtio-ccw: prepare for memory devices
>   s390x/pv: prepare for memory devices
>   s390x: remember the maximum page size
>   s390x/virtio-ccw: add support for virtio based memory devices
>   s390x: virtio-mem support
>
>  MAINTAINERS                        |   5 +
>  hw/s390x/Kconfig                   |   1 +
>  hw/s390x/meson.build               |   6 +-
>  hw/s390x/s390-hypercall.c          |  85 +++++++++++
>  hw/s390x/s390-hypercall.h          |  25 ++++
>  hw/s390x/s390-skeys.c              |   6 +-
>  hw/s390x/s390-stattrib-kvm.c       |  67 ++++++---
>  hw/s390x/s390-virtio-ccw.c         | 165 ++++++++++++++-------
>  hw/s390x/s390-virtio-hcall.c       |  41 ------
>  hw/s390x/s390-virtio-hcall.h       |  25 ----
>  hw/s390x/sclp.c                    |  17 +--
>  hw/s390x/virtio-ccw-md-stubs.c     |  24 +++
>  hw/s390x/virtio-ccw-md.c           | 153 +++++++++++++++++++
>  hw/s390x/virtio-ccw-md.h           |  44 ++++++
>  hw/s390x/virtio-ccw-mem.c          | 226 +++++++++++++++++++++++++++++
>  hw/s390x/virtio-ccw-mem.h          |  34 +++++
>  hw/virtio/Kconfig                  |   1 +
>  hw/virtio/virtio-mem.c             |   4 +-
>  include/hw/s390x/s390-virtio-ccw.h |   4 +
>  target/s390x/cpu-sysemu.c          |  15 --
>  target/s390x/cpu.h                 |   2 -
>  target/s390x/kvm/kvm.c             |  18 +--
>  target/s390x/kvm/pv.c              |   2 +-
>  target/s390x/tcg/misc_helper.c     |   7 +-
>  24 files changed, 782 insertions(+), 195 deletions(-)
>  create mode 100644 hw/s390x/s390-hypercall.c
>  create mode 100644 hw/s390x/s390-hypercall.h
>  delete mode 100644 hw/s390x/s390-virtio-hcall.c
>  delete mode 100644 hw/s390x/s390-virtio-hcall.h
>  create mode 100644 hw/s390x/virtio-ccw-md-stubs.c
>  create mode 100644 hw/s390x/virtio-ccw-md.c
>  create mode 100644 hw/s390x/virtio-ccw-md.h
>  create mode 100644 hw/s390x/virtio-ccw-mem.c
>  create mode 100644 hw/s390x/virtio-ccw-mem.h
>
> --
> 2.46.1
>
>
Re: [PATCH v2 00/14] s390x: virtio-mem support
Posted by David Hildenbrand 1 week, 3 days ago
On 08.10.24 12:54, David Hildenbrand wrote:
> Based on current master.
> 
> There is really not much left to do on s390x, because virtio-mem already
> implements most things we need today (e.g., early-migration,
> unplugged-inaccessible). The biggest part of this series is just doing what
> we do with virtio-pci, wiring it up in the machine hotplug handler and ...
> well, messing with the physical memory layout where we can now exceed
> initial RAM size and have sparsity (memory holes).
> 
> I tested a lot of things, including:
>   * Memory hotplug/unplug
>   * Device hotplug/unplug
>   * System resets / reboots
>   * Migrate to/from file (including storage attributes under KVM)
>   * Basic live migration
>   * Basic postcopy live migration
> 
> More details on how to use it on s390x -- which is pretty much how
> we use it on other architectures, except
> s/virtio-mem-pci/virtio-mem-ccw/ --- is in the last patch.
> 
> This series introduces a new diag(500) "STORAGE LIMIT" subcode that will
> be documented in the kernel and at [2] once this+kernel part go upstream.
> 
> There are not many s390x-specific virtio-mem future work items, except:
> * Storage attribute migration might be improved
> * We might want to reset storage attributes of unplugged memory
>    (might or might not be required for upcoming page table reclaim in
>     Linux; TBD)
> 
> The Linux driver is available at [3].
> 
> [1] https://lkml.kernel.org/r/20240906101658.514470-1-pbonzini@redhat.com
> [2] https://gitlab.com/davidhildenbrand/s390x-os-virt-spec
> [3] https://lkml.kernel.org/r/20240910191541.2179655-6-david@redhat.com

Gentle ping (and thanks to Thomas for the review!).

I assume the kernel portion will go upstream in the next merge window. 
I'd like get the QEMU parts merged soon after that.

9.2 is going to get released in roughly one month, so there is still time.

@Thomas, this is mostly s390x stuff, so I guess it should go through the 
s390x tree? But I could also take this through my "memory devices" tree.

-- 
Cheers,

David / dhildenb