This series is based on:
    [PATCH v2] virtio: kconfig: memory devices are PCI only [1]

I finally found the time (IOW forced myself) to finish virtio-mem
support on s390x. The last RFC was from 2020, so I won't talk about
what changed -- a lot changed in the meantime :)

There is really not much left to do on s390x, because virtio-mem already
implements most things we need today (e.g., early-migration,
unplugged-inaccessible). The biggest part of this series is just doing what
we do with virtio-pci, wiring it up in the machine hotplug handler and ...
well, messing with the physical memory layout where we can now exceed
initial RAM size and have sparsity (memory holes).

I tested a lot of things, including:
 * Memory hotplug/unplug
 * Device hotplug/unplug
 * System resets / reboots
 * Migrate to/from file (including storage attributes under KVM)
 * Basic live migration
 * Basic postcopy live migration

More details on how to use it on s390x -- which is pretty much how
we use it on other architectures, except
s/virtio-mem-pci/virtio-mem-ccw/ --- is in the last patch.

This series introduces a new diag(500) "STORAGE LIMIT" subcode that will
be documented at [2] once this+kernel part go upstream.

There are not many s390x-specific virtio-mem future work items, except:
* Storage attribute migration might be improved
* We might want to reset storage attributes of unplugged memory
  (might or might not be required for upcoming page table reclaim in
   Linux; TBD)

I'll send out the kernel driver bits soon.

[1] https://lkml.kernel.org/r/20240906101658.514470-1-pbonzini@redhat.com
[2] https://gitlab.com/davidhildenbrand/s390x-os-virt-spec

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Thomas Huth <thuth@redhat.com>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Eric Farman <farman@linux.ibm.com>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: David Hildenbrand <david@redhat.com>
Cc: Ilya Leoshkevich <iii@linux.ibm.com>
Cc: Janosch Frank <frankja@linux.ibm.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Cornelia Huck <cohuck@redhat.com>

David Hildenbrand (14):
  s390x/s390-virtio-ccw: don't crash on weird RAM sizes
  s390x/s390-virtio-hcall: remove hypercall registration mechanism
  s390x/s390-virtio-hcall: prepare for more diag500 hypercalls
  s390x: rename s390-virtio-hcall* to s390-hypercall*
  s390x/s390-virtio-ccw: move setting the maximum guest size from sclp
    to machine code
  s390x: introduce s390_get_memory_limit()
  s390x/s390-hypercall: introduce DIAG500 STORAGE_LIMIT
  s390x/s390-stattrib-kvm: prepare memory devices and sparse memory
    layouts
  s390x/s390-skeys: prepare for memory devices
  s390x/pv: check initial, not maximum RAM size
  s390x/s390-virtio-ccw: prepare for memory devices
  s390x: introduce s390_get_max_pagesize()
  s390x/virtio-ccw: add support for virtio based memory devices
  s390x: virtio-mem support

 MAINTAINERS                    |   4 +
 hw/s390x/Kconfig               |   1 +
 hw/s390x/meson.build           |   4 +-
 hw/s390x/s390-hypercall.c      |  77 +++++++++++
 hw/s390x/s390-hypercall.h      |  25 ++++
 hw/s390x/s390-skeys.c          |   4 +-
 hw/s390x/s390-stattrib-kvm.c   |  63 +++++----
 hw/s390x/s390-virtio-ccw.c     | 143 +++++++++++++--------
 hw/s390x/s390-virtio-hcall.c   |  41 ------
 hw/s390x/s390-virtio-hcall.h   |  25 ----
 hw/s390x/sclp.c                |  17 +--
 hw/s390x/virtio-ccw-md.c       | 153 ++++++++++++++++++++++
 hw/s390x/virtio-ccw-md.h       |  44 +++++++
 hw/s390x/virtio-ccw-mem.c      | 226 +++++++++++++++++++++++++++++++++
 hw/s390x/virtio-ccw-mem.h      |  34 +++++
 hw/virtio/Kconfig              |   1 +
 hw/virtio/virtio-mem.c         |   4 +-
 target/s390x/cpu-sysemu.c      |  35 ++++-
 target/s390x/cpu.h             |   2 +
 target/s390x/kvm/kvm.c         |  12 +-
 target/s390x/kvm/pv.c          |   2 +-
 target/s390x/tcg/misc_helper.c |   6 +-
 22 files changed, 746 insertions(+), 177 deletions(-)
 create mode 100644 hw/s390x/s390-hypercall.c
 create mode 100644 hw/s390x/s390-hypercall.h
 delete mode 100644 hw/s390x/s390-virtio-hcall.c
 delete mode 100644 hw/s390x/s390-virtio-hcall.h
 create mode 100644 hw/s390x/virtio-ccw-md.c
 create mode 100644 hw/s390x/virtio-ccw-md.h
 create mode 100644 hw/s390x/virtio-ccw-mem.c
 create mode 100644 hw/s390x/virtio-ccw-mem.h

-- 
2.46.0

On 9/10/24 7:57 PM, David Hildenbrand wrote:
> This series introduces a new diag(500) "STORAGE LIMIT" subcode that will
> be documented at [2] once this+kernel part go upstream.

Why not in Documentation/virt/kvm/s390/?
s390-diag.rst is very similar already.

I'd rather have it in a shared and bigger repo than in your personal 
gitlab one. Maybe there's a space somewhere in QEMU or the Virtio team's 
repos that would be a good fit if the kernel's docu isn't the right place?

On 11.09.24 13:49, Janosch Frank wrote:
> On 9/10/24 7:57 PM, David Hildenbrand wrote:
>> This series introduces a new diag(500) "STORAGE LIMIT" subcode that will
>> be documented at [2] once this+kernel part go upstream.
> 
> Why not in Documentation/virt/kvm/s390/?
> s390-diag.rst is very similar already.

I can document it anywhere people fancy, really I don't care. But this 
has some history. The last time we discussed it [1] there was:

"Regardless what we end up with, this needs to be specified
somewhere(tm)." from Conny

"It must be well defined and easy to find also for kernel developers
who actually have to care about memory detection code :)" from Heiko.

And then

"the kernel's s390-diag.rst should also point to it ... Nothing as 
complicated as an OASIS spec, but maybe a git??b project?" from Conny

> 
> I'd rather have it in a shared and bigger repo than in your personal
> gitlab one. Maybe there's a space somewhere in QEMU or the Virtio team's
> repos that would be a good fit if the kernel's docu isn't the right place?

At this point, outside of kernel/QEMU feels like the right thing to do. 
Conny is already a co-maintainer of my "personal" (;)) gitlab.


And now I realize that I CCed Heiko on the Linux series but not the QEMU 
series. My bad.

[1] https://lore.kernel.org/all/20200727114819.3f816010.cohuck@redhat.com/

-- 
Cheers,

David / dhildenb

> > 
> > I'd rather have it in a shared and bigger repo than in your personal
> > gitlab one. Maybe there's a space somewhere in QEMU or the Virtio team's
> > repos that would be a good fit if the kernel's docu isn't the right place?
> 
> At this point, outside of kernel/QEMU feels like the right thing to do.
> Conny is already a co-maintainer of my "personal" (;)) gitlab.
> 
> 
> And now I realize that I CCed Heiko on the Linux series but not the QEMU
> series. My bad.
> 
> [1] https://lore.kernel.org/all/20200727114819.3f816010.cohuck@redhat.com/


No prob. Or if you want it in virtio spec, that's also fine.

On Wed, Sep 11 2024, "Michael S. Tsirkin" <mst@redhat.com> wrote:

>> > 
>> > I'd rather have it in a shared and bigger repo than in your personal
>> > gitlab one. Maybe there's a space somewhere in QEMU or the Virtio team's
>> > repos that would be a good fit if the kernel's docu isn't the right place?
>> 
>> At this point, outside of kernel/QEMU feels like the right thing to do.
>> Conny is already a co-maintainer of my "personal" (;)) gitlab.
>> 
>> 
>> And now I realize that I CCed Heiko on the Linux series but not the QEMU
>> series. My bad.
>> 
>> [1] https://lore.kernel.org/all/20200727114819.3f816010.cohuck@redhat.com/
>
>
> No prob. Or if you want it in virtio spec, that's also fine.

The virtio spec makes sense for documenting things needed to implement
it, but what I liked about the gitlab project is that it tries to go
into more s390-specific aspects (that feel a bit out of scope for the
virtio spec), and that it provides a place to document non-virtio
related interfaces.

Anyway, if we want to proceed with the gitlab project, would it make
sense to create an org for it, so that it doesn't look like David's
personal project? In any case, while I'm happy to stay on, I'm not that
involved with s390 anymore, and it might make sense to add more people
to it.

On 11.09.24 17:38, Cornelia Huck wrote:
> On Wed, Sep 11 2024, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
>>>>
>>>> I'd rather have it in a shared and bigger repo than in your personal
>>>> gitlab one. Maybe there's a space somewhere in QEMU or the Virtio team's
>>>> repos that would be a good fit if the kernel's docu isn't the right place?
>>>
>>> At this point, outside of kernel/QEMU feels like the right thing to do.
>>> Conny is already a co-maintainer of my "personal" (;)) gitlab.
>>>
>>>
>>> And now I realize that I CCed Heiko on the Linux series but not the QEMU
>>> series. My bad.
>>>
>>> [1] https://lore.kernel.org/all/20200727114819.3f816010.cohuck@redhat.com/
>>
>>
>> No prob. Or if you want it in virtio spec, that's also fine.
> 
> The virtio spec makes sense for documenting things needed to implement
> it, but what I liked about the gitlab project is that it tries to go
> into more s390-specific aspects (that feel a bit out of scope for the
> virtio spec), and that it provides a place to document non-virtio
> related interfaces.
> 
> Anyway, if we want to proceed with the gitlab project, would it make
> sense to create an org for it, so that it doesn't look like David's
> personal project?

I had that in mind as well, I can move the project.

> In any case, while I'm happy to stay on, I'm not that
> involved with s390 anymore, and it might make sense to add more people
> to it.

Indeed!

-- 
Cheers,

David / dhildenb

On Wed, 11 Sep 2024 21:09:27 +0200
David Hildenbrand <david@redhat.com> wrote:

> > Anyway, if we want to proceed with the gitlab project, would it make
> > sense to create an org for it, so that it doesn't look like David's
> > personal project?

Frankly, I would prefer making Documentation/virt/kvm/s390/s390-diag.rst
the authoritative documentation on DIAGs.

My train of thought is DIAG 500 is a KVM thing, and KVM is a linux
kernel thing, so it just feels right for the documentatio to
live within the linux source tree.

I may have missed some of the discussion: what were the benefits
of having this in its separate project/repository?

Regards,
Halil

On 27.09.24 20:20, Halil Pasic wrote:
> On Wed, 11 Sep 2024 21:09:27 +0200
> David Hildenbrand <david@redhat.com> wrote:
> 
>>> Anyway, if we want to proceed with the gitlab project, would it make
>>> sense to create an org for it, so that it doesn't look like David's
>>> personal project?
> 
> Frankly, I would prefer making Documentation/virt/kvm/s390/s390-diag.rst
> the authoritative documentation on DIAGs.
> 
> My train of thought is DIAG 500 is a KVM thing, and KVM is a linux
> kernel thing, so it just feels right for the documentatio to
> live within the linux source tree.

QEMU/TCG is the proof that KVM is not necessarily involved.

Are you sure that no other OS out there besides Linux implements virtio 
on s390x, or would want to implement it? :)

> 
> I may have missed some of the discussion: what were the benefits
> of having this in its separate project/repository?

Having it independent of the implementation.

-- 
Cheers,

David / dhildenb

On Fri, 27 Sep 2024 20:29:19 +0200
David Hildenbrand <david@redhat.com> wrote:

> On 27.09.24 20:20, Halil Pasic wrote:
> > On Wed, 11 Sep 2024 21:09:27 +0200
> > David Hildenbrand <david@redhat.com> wrote:
> >   
> >>> Anyway, if we want to proceed with the gitlab project, would it make
> >>> sense to create an org for it, so that it doesn't look like David's
> >>> personal project?  
> > 
> > Frankly, I would prefer making Documentation/virt/kvm/s390/s390-diag.rst
> > the authoritative documentation on DIAGs.
> > 
> > My train of thought is DIAG 500 is a KVM thing, and KVM is a linux
> > kernel thing, so it just feels right for the documentatio to
> > live within the linux source tree.  
> 
> QEMU/TCG is the proof that KVM is not necessarily involved.
> 
> Are you sure that no other OS out there besides Linux implements virtio 
> on s390x, or would want to implement it? :)
>

As Christian has pointed out in another thread DIAG 500 is documented
as the KVM hypervisor call, and that made me argue it is a KVM thing.

You are right KVM is not necessarily involved, and neither is QEMU. For
me it is not about the components involved in the visualization, but
about the people, projects and governance.

IMHO this is basically extending the s390 architecture. We are guaranteed
to not collide with the Architecture because DIAG 500 is reserved for
KVM as a project I guess.

> > 
> > I may have missed some of the discussion: what were the benefits
> > of having this in its separate project/repository?  
> 
> Having it independent of the implementation.
> 

That is a valid point. But IMHO the benefit of having this independent,
does not justify the churn of having a separate project with its
own governance, and communication infrastructure. And I suppose for an
open(?) specification, one would need those things.

No strong opinions though. If Christian, Janosch and Claudio are in
favor of a separate "Specifications for open-source virtualization on
s390x (IBM z Systems)" project, I'm fine with it as well.

Regards,
Halil

On 30.09.24 23:49, Halil Pasic wrote:
> On Fri, 27 Sep 2024 20:29:19 +0200
> David Hildenbrand <david@redhat.com> wrote:
> 
>> On 27.09.24 20:20, Halil Pasic wrote:
>>> On Wed, 11 Sep 2024 21:09:27 +0200
>>> David Hildenbrand <david@redhat.com> wrote:
>>>    
>>>>> Anyway, if we want to proceed with the gitlab project, would it make
>>>>> sense to create an org for it, so that it doesn't look like David's
>>>>> personal project?
>>>
>>> Frankly, I would prefer making Documentation/virt/kvm/s390/s390-diag.rst
>>> the authoritative documentation on DIAGs.
>>>
>>> My train of thought is DIAG 500 is a KVM thing, and KVM is a linux
>>> kernel thing, so it just feels right for the documentatio to
>>> live within the linux source tree.
>>
>> QEMU/TCG is the proof that KVM is not necessarily involved.
>>
>> Are you sure that no other OS out there besides Linux implements virtio
>> on s390x, or would want to implement it? :)
>>
> 
> As Christian has pointed out in another thread DIAG 500 is documented
> as the KVM hypervisor call, and that made me argue it is a KVM thing.
> 
> You are right KVM is not necessarily involved, and neither is QEMU. For
> me it is not about the components involved in the visualization, but
> about the people, projects and governance.
> 
> IMHO this is basically extending the s390 architecture. We are guaranteed
> to not collide with the Architecture because DIAG 500 is reserved for
> KVM as a project I guess.

That's my understanding. I assume because the CCW virtio machine started 
out as KVM-only, documenting that it is "KVM ONLY" may be because of 
historical reasons.

> 
>>>
>>> I may have missed some of the discussion: what were the benefits
>>> of having this in its separate project/repository?
>>
>> Having it independent of the implementation.
>>
> 
> That is a valid point. But IMHO the benefit of having this independent,
> does not justify the churn of having a separate project with its
> own governance, and communication infrastructure. And I suppose for an
> open(?) specification, one would need those things.

I don't see the need to bring in all that bureaucracy. The original idea 
was simple: if QEMU/TCG or QEMU/KVM implement a hypercall (IOW: it was 
acked by the s390x maintainers), we document it somewhere.

Implementing something in QEMU and then modifying a KVM document in the 
kernel tree sounded odd.

It is a valid question to ask: what if any other hypervisor 
(cloud-hypervisor etc.) would want to implement a custom diag500 
hypercall, and who would ack it. But I don't really think that we would 
have to sort this out this at this point in time.

> 
> No strong opinions though. If Christian, Janosch and Claudio are in
> favor of a separate "Specifications for open-source virtualization on
> s390x (IBM z Systems)" project, I'm fine with it as well.

I'm more than happy if we don't need that. As said, I'm happy to 
document wherever people tell me to document.

4 years ago we thought that having a separate repository was a good 
idea, maybe it is no longer. In that case, s390x mainters please let me 
know what to do :)

-- 
Cheers,

David / dhildenb

On 10/1/24 10:54 AM, David Hildenbrand wrote:
> On 30.09.24 23:49, Halil Pasic wrote:
>> On Fri, 27 Sep 2024 20:29:19 +0200
>> David Hildenbrand <david@redhat.com> wrote:
>>
>>> On 27.09.24 20:20, Halil Pasic wrote:
>>>> On Wed, 11 Sep 2024 21:09:27 +0200
>>>> David Hildenbrand <david@redhat.com> wrote:

[...]

>> That is a valid point. But IMHO the benefit of having this independent,
>> does not justify the churn of having a separate project with its
>> own governance, and communication infrastructure. And I suppose for an
>> open(?) specification, one would need those things.
> 
> I don't see the need to bring in all that bureaucracy. The original idea
> was simple: if QEMU/TCG or QEMU/KVM implement a hypercall (IOW: it was
> acked by the s390x maintainers), we document it somewhere.
> 
> Implementing something in QEMU and then modifying a KVM document in the
> kernel tree sounded odd.
> 
> It is a valid question to ask: what if any other hypervisor
> (cloud-hypervisor etc.) would want to implement a custom diag500
> hypercall, and who would ack it. But I don't really think that we would
> have to sort this out this at this point in time.
> 
>>
>> No strong opinions though. If Christian, Janosch and Claudio are in
>> favor of a separate "Specifications for open-source virtualization on
>> s390x (IBM z Systems)" project, I'm fine with it as well.
> 
> I'm more than happy if we don't need that. As said, I'm happy to
> document wherever people tell me to document.
> 
> 4 years ago we thought that having a separate repository was a good
> idea, maybe it is no longer. In that case, s390x mainters please let me
> know what to do :)
> 

I'd like to at least have a partial documentation in the kernel if not a 
full one. You can add a link to your repo to that.

Even if they go out of sync I'd rather have documentation where I'd 
expect it (Linux) than only the repo. IMHO duplication isn't a gigantic 
issue here.

We also have an internal space where KVM architecture is being stored 
(currently a handful of documents) and we'll store it there as well, 
including a link to the repo.

On 02.10.24 11:04, Janosch Frank wrote:
> On 10/1/24 10:54 AM, David Hildenbrand wrote:
>> On 30.09.24 23:49, Halil Pasic wrote:
>>> On Fri, 27 Sep 2024 20:29:19 +0200
>>> David Hildenbrand <david@redhat.com> wrote:
>>>
>>>> On 27.09.24 20:20, Halil Pasic wrote:
>>>>> On Wed, 11 Sep 2024 21:09:27 +0200
>>>>> David Hildenbrand <david@redhat.com> wrote:
> 
> [...]
> 
>>> That is a valid point. But IMHO the benefit of having this independent,
>>> does not justify the churn of having a separate project with its
>>> own governance, and communication infrastructure. And I suppose for an
>>> open(?) specification, one would need those things.
>>
>> I don't see the need to bring in all that bureaucracy. The original idea
>> was simple: if QEMU/TCG or QEMU/KVM implement a hypercall (IOW: it was
>> acked by the s390x maintainers), we document it somewhere.
>>
>> Implementing something in QEMU and then modifying a KVM document in the
>> kernel tree sounded odd.
>>
>> It is a valid question to ask: what if any other hypervisor
>> (cloud-hypervisor etc.) would want to implement a custom diag500
>> hypercall, and who would ack it. But I don't really think that we would
>> have to sort this out this at this point in time.
>>
>>>
>>> No strong opinions though. If Christian, Janosch and Claudio are in
>>> favor of a separate "Specifications for open-source virtualization on
>>> s390x (IBM z Systems)" project, I'm fine with it as well.
>>
>> I'm more than happy if we don't need that. As said, I'm happy to
>> document wherever people tell me to document.
>>
>> 4 years ago we thought that having a separate repository was a good
>> idea, maybe it is no longer. In that case, s390x mainters please let me
>> know what to do :)
>>
> 
> I'd like to at least have a partial documentation in the kernel if not a
> full one. You can add a link to your repo to that.

Makes sense.

> 
> Even if they go out of sync I'd rather have documentation where I'd
> expect it (Linux) than only the repo. IMHO duplication isn't a gigantic
> issue here.

Agreed.

> 
> We also have an internal space where KVM architecture is being stored
> (currently a handful of documents) and we'll store it there as well,
> including a link to the repo.

Perfect. If there is a way to enable that repository, that would be 
great. For the time being I'll document it there as well.

Thanks!

-- 
Cheers,

David / dhildenb

On Tue, Sep 10, 2024 at 07:57:55PM +0200, David Hildenbrand wrote:
> This series is based on:
>     [PATCH v2] virtio: kconfig: memory devices are PCI only [1]
> 
> I finally found the time (IOW forced myself) to finish virtio-mem
> support on s390x. The last RFC was from 2020, so I won't talk about
> what changed -- a lot changed in the meantime :)
> 
> There is really not much left to do on s390x, because virtio-mem already
> implements most things we need today (e.g., early-migration,
> unplugged-inaccessible). The biggest part of this series is just doing what
> we do with virtio-pci, wiring it up in the machine hotplug handler and ...
> well, messing with the physical memory layout where we can now exceed
> initial RAM size and have sparsity (memory holes).
> 
> I tested a lot of things, including:
>  * Memory hotplug/unplug
>  * Device hotplug/unplug
>  * System resets / reboots
>  * Migrate to/from file (including storage attributes under KVM)
>  * Basic live migration
>  * Basic postcopy live migration
> 
> More details on how to use it on s390x -- which is pretty much how
> we use it on other architectures, except
> s/virtio-mem-pci/virtio-mem-ccw/ --- is in the last patch.
> 
> This series introduces a new diag(500) "STORAGE LIMIT" subcode that will
> be documented at [2] once this+kernel part go upstream.
> 
> There are not many s390x-specific virtio-mem future work items, except:
> * Storage attribute migration might be improved
> * We might want to reset storage attributes of unplugged memory
>   (might or might not be required for upcoming page table reclaim in
>    Linux; TBD)


I don't see anything needing virtio specific here, let me know if
I missed anything.
A quick look is fine so I guess you can add
Acked-by: Michael S. Tsirkin <mst@redhat.com>



> I'll send out the kernel driver bits soon.
> 
> [1] https://lkml.kernel.org/r/20240906101658.514470-1-pbonzini@redhat.com
> [2] https://gitlab.com/davidhildenbrand/s390x-os-virt-spec
> 
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Thomas Huth <thuth@redhat.com>
> Cc: Halil Pasic <pasic@linux.ibm.com>
> Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
> Cc: Eric Farman <farman@linux.ibm.com>
> Cc: Richard Henderson <richard.henderson@linaro.org>
> Cc: David Hildenbrand <david@redhat.com>
> Cc: Ilya Leoshkevich <iii@linux.ibm.com>
> Cc: Janosch Frank <frankja@linux.ibm.com>
> Cc: "Michael S. Tsirkin" <mst@redhat.com>
> Cc: Cornelia Huck <cohuck@redhat.com>
> 
> David Hildenbrand (14):
>   s390x/s390-virtio-ccw: don't crash on weird RAM sizes
>   s390x/s390-virtio-hcall: remove hypercall registration mechanism
>   s390x/s390-virtio-hcall: prepare for more diag500 hypercalls
>   s390x: rename s390-virtio-hcall* to s390-hypercall*
>   s390x/s390-virtio-ccw: move setting the maximum guest size from sclp
>     to machine code
>   s390x: introduce s390_get_memory_limit()
>   s390x/s390-hypercall: introduce DIAG500 STORAGE_LIMIT
>   s390x/s390-stattrib-kvm: prepare memory devices and sparse memory
>     layouts
>   s390x/s390-skeys: prepare for memory devices
>   s390x/pv: check initial, not maximum RAM size
>   s390x/s390-virtio-ccw: prepare for memory devices
>   s390x: introduce s390_get_max_pagesize()
>   s390x/virtio-ccw: add support for virtio based memory devices
>   s390x: virtio-mem support
> 
>  MAINTAINERS                    |   4 +
>  hw/s390x/Kconfig               |   1 +
>  hw/s390x/meson.build           |   4 +-
>  hw/s390x/s390-hypercall.c      |  77 +++++++++++
>  hw/s390x/s390-hypercall.h      |  25 ++++
>  hw/s390x/s390-skeys.c          |   4 +-
>  hw/s390x/s390-stattrib-kvm.c   |  63 +++++----
>  hw/s390x/s390-virtio-ccw.c     | 143 +++++++++++++--------
>  hw/s390x/s390-virtio-hcall.c   |  41 ------
>  hw/s390x/s390-virtio-hcall.h   |  25 ----
>  hw/s390x/sclp.c                |  17 +--
>  hw/s390x/virtio-ccw-md.c       | 153 ++++++++++++++++++++++
>  hw/s390x/virtio-ccw-md.h       |  44 +++++++
>  hw/s390x/virtio-ccw-mem.c      | 226 +++++++++++++++++++++++++++++++++
>  hw/s390x/virtio-ccw-mem.h      |  34 +++++
>  hw/virtio/Kconfig              |   1 +
>  hw/virtio/virtio-mem.c         |   4 +-
>  target/s390x/cpu-sysemu.c      |  35 ++++-
>  target/s390x/cpu.h             |   2 +
>  target/s390x/kvm/kvm.c         |  12 +-
>  target/s390x/kvm/pv.c          |   2 +-
>  target/s390x/tcg/misc_helper.c |   6 +-
>  22 files changed, 746 insertions(+), 177 deletions(-)
>  create mode 100644 hw/s390x/s390-hypercall.c
>  create mode 100644 hw/s390x/s390-hypercall.h
>  delete mode 100644 hw/s390x/s390-virtio-hcall.c
>  delete mode 100644 hw/s390x/s390-virtio-hcall.h
>  create mode 100644 hw/s390x/virtio-ccw-md.c
>  create mode 100644 hw/s390x/virtio-ccw-md.h
>  create mode 100644 hw/s390x/virtio-ccw-mem.c
>  create mode 100644 hw/s390x/virtio-ccw-mem.h
> 
> -- 
> 2.46.0

On 10.09.24 20:33, Michael S. Tsirkin wrote:
> On Tue, Sep 10, 2024 at 07:57:55PM +0200, David Hildenbrand wrote:
>> This series is based on:
>>      [PATCH v2] virtio: kconfig: memory devices are PCI only [1]
>>
>> I finally found the time (IOW forced myself) to finish virtio-mem
>> support on s390x. The last RFC was from 2020, so I won't talk about
>> what changed -- a lot changed in the meantime :)
>>
>> There is really not much left to do on s390x, because virtio-mem already
>> implements most things we need today (e.g., early-migration,
>> unplugged-inaccessible). The biggest part of this series is just doing what
>> we do with virtio-pci, wiring it up in the machine hotplug handler and ...
>> well, messing with the physical memory layout where we can now exceed
>> initial RAM size and have sparsity (memory holes).
>>
>> I tested a lot of things, including:
>>   * Memory hotplug/unplug
>>   * Device hotplug/unplug
>>   * System resets / reboots
>>   * Migrate to/from file (including storage attributes under KVM)
>>   * Basic live migration
>>   * Basic postcopy live migration
>>
>> More details on how to use it on s390x -- which is pretty much how
>> we use it on other architectures, except
>> s/virtio-mem-pci/virtio-mem-ccw/ --- is in the last patch.
>>
>> This series introduces a new diag(500) "STORAGE LIMIT" subcode that will
>> be documented at [2] once this+kernel part go upstream.
>>
>> There are not many s390x-specific virtio-mem future work items, except:
>> * Storage attribute migration might be improved
>> * We might want to reset storage attributes of unplugged memory
>>    (might or might not be required for upcoming page table reclaim in
>>     Linux; TBD)
> 
> 
> I don't see anything needing virtio specific here, let me know if
> I missed anything.

No, it's really just wiring up virtio-mem.

> A quick look is fine so I guess you can add
> Acked-by: Michael S. Tsirkin <mst@redhat.com>

Thanks!

-- 
Cheers,

David / dhildenb