From: Igor Kotrasinski <i.kotrasinsk@partner.samsung.com>

This patchset adds a "memory exposing" device that allows two QEMU
instances to share arbitrary memory regions. Unlike ivshmem, it does not
create a new region of memory that's shared between VMs, but instead
allows one VM to access any memory region of the other VM we choose to
share.

The motivation for this device is a sort of ARM Trustzone "emulation",
where a rich system running on one machine (e.g. x86_64 linux) is able
to perform SMCs to a trusted system running on another (e.g. OpTEE on
ARM). With a device that allows sharing arbitrary memory between VMs,
this can be achieved with minimal changes to the trusted system and its
linux driver while allowing the rich system to run on a speedier x86
emulator. I prepared additional patches for linux, OpTEE OS and OpTEE
build system as a PoC that such emulation works and passes OpTEE tests;
I'm not sure what would be the best way to share them.

This patchset is my first foray into QEMU source code and I'm certain
it's not yet ready to be merged in. I'm not sure whether memory sharing
code has any race conditions or breaks rules of working with memory
regions, or if having VMs communicate synchronously via chardevs is the
right way to do it. I do believe the basic idea for sharing memory
regions is sound and that it could be useful for inter-VM communication.

Igor Kotrasinski (9):
  memory: Add function for finding flat memory ranges
  memory: Support mmap offset for fd-backed memory regions
  memory: Hack - use shared memory when possible
  hw/misc/memexpose: Add documentation
  hw/misc/memexpose: Add core memexpose files
  hw/misc/memexpose: Add memexpose pci device
  hw/misc/memexpose: Add memexpose memory region device
  hw/misc/memexpose: Add simple tests
  hw/arm/virt: Hack in support for memexpose device

 Kconfig.host                            |   3 +
 MAINTAINERS                             |  12 +
 Makefile                                |   1 +
 backends/hostmem-memfd.c                |   2 +-
 configure                               |   8 +
 docs/specs/memexpose-spec.txt           | 168 +++++++++
 exec.c                                  |  10 +-
 hw/arm/virt.c                           | 110 +++++-
 hw/core/numa.c                          |   4 +-
 hw/mem/Kconfig                          |   3 +
 hw/misc/Makefile.objs                   |   1 +
 hw/misc/ivshmem.c                       |   3 +-
 hw/misc/memexpose/Makefile.objs         |   4 +
 hw/misc/memexpose/memexpose-core.c      | 630 ++++++++++++++++++++++++++++++++
 hw/misc/memexpose/memexpose-core.h      | 109 ++++++
 hw/misc/memexpose/memexpose-memregion.c | 142 +++++++
 hw/misc/memexpose/memexpose-memregion.h |  41 +++
 hw/misc/memexpose/memexpose-msg.c       | 261 +++++++++++++
 hw/misc/memexpose/memexpose-msg.h       | 161 ++++++++
 hw/misc/memexpose/memexpose-pci.c       | 218 +++++++++++
 include/exec/memory.h                   |  20 +
 include/exec/ram_addr.h                 |   2 +-
 include/hw/arm/virt.h                   |   5 +
 include/qemu/mmap-alloc.h               |   1 +
 memory.c                                |  82 ++++-
 tests/qtest/Makefile.include            |   2 +
 tests/qtest/memexpose-test.c            | 364 ++++++++++++++++++
 util/mmap-alloc.c                       |   7 +-
 util/oslib-posix.c                      |   2 +-
 29 files changed, 2360 insertions(+), 16 deletions(-)
 create mode 100644 docs/specs/memexpose-spec.txt
 create mode 100644 hw/misc/memexpose/Makefile.objs
 create mode 100644 hw/misc/memexpose/memexpose-core.c
 create mode 100644 hw/misc/memexpose/memexpose-core.h
 create mode 100644 hw/misc/memexpose/memexpose-memregion.c
 create mode 100644 hw/misc/memexpose/memexpose-memregion.h
 create mode 100644 hw/misc/memexpose/memexpose-msg.c
 create mode 100644 hw/misc/memexpose/memexpose-msg.h
 create mode 100644 hw/misc/memexpose/memexpose-pci.c
 create mode 100644 tests/qtest/memexpose-test.c

-- 
2.7.4

On Tue, Feb 04, 2020 at 12:30:42PM +0100, i.kotrasinsk@partner.samsung.com wrote:
> From: Igor Kotrasinski <i.kotrasinsk@partner.samsung.com>
> 
> This patchset adds a "memory exposing" device that allows two QEMU
> instances to share arbitrary memory regions. Unlike ivshmem, it does not
> create a new region of memory that's shared between VMs, but instead
> allows one VM to access any memory region of the other VM we choose to
> share.
> 
> The motivation for this device is a sort of ARM Trustzone "emulation",
> where a rich system running on one machine (e.g. x86_64 linux) is able
> to perform SMCs to a trusted system running on another (e.g. OpTEE on
> ARM). With a device that allows sharing arbitrary memory between VMs,
> this can be achieved with minimal changes to the trusted system and its
> linux driver while allowing the rich system to run on a speedier x86
> emulator. I prepared additional patches for linux, OpTEE OS and OpTEE
> build system as a PoC that such emulation works and passes OpTEE tests;
> I'm not sure what would be the best way to share them.
> 
> This patchset is my first foray into QEMU source code and I'm certain
> it's not yet ready to be merged in. I'm not sure whether memory sharing
> code has any race conditions or breaks rules of working with memory
> regions, or if having VMs communicate synchronously via chardevs is the
> right way to do it. I do believe the basic idea for sharing memory
> regions is sound and that it could be useful for inter-VM communication.

Hi,
Without having looked into the patches yet, I'm already wondering if you
can use the existing -object
memory-backend-file,size=512M,mem-path=/my/shared/mem feature for your
use case?

That's the existing mechanism for fully sharing guest RAM and if you
want to share all of memory then maybe a device is not necessary - just
share the memory.

Stefan

> Igor Kotrasinski (9):
>   memory: Add function for finding flat memory ranges
>   memory: Support mmap offset for fd-backed memory regions
>   memory: Hack - use shared memory when possible
>   hw/misc/memexpose: Add documentation
>   hw/misc/memexpose: Add core memexpose files
>   hw/misc/memexpose: Add memexpose pci device
>   hw/misc/memexpose: Add memexpose memory region device
>   hw/misc/memexpose: Add simple tests
>   hw/arm/virt: Hack in support for memexpose device
> 
>  Kconfig.host                            |   3 +
>  MAINTAINERS                             |  12 +
>  Makefile                                |   1 +
>  backends/hostmem-memfd.c                |   2 +-
>  configure                               |   8 +
>  docs/specs/memexpose-spec.txt           | 168 +++++++++
>  exec.c                                  |  10 +-
>  hw/arm/virt.c                           | 110 +++++-
>  hw/core/numa.c                          |   4 +-
>  hw/mem/Kconfig                          |   3 +
>  hw/misc/Makefile.objs                   |   1 +
>  hw/misc/ivshmem.c                       |   3 +-
>  hw/misc/memexpose/Makefile.objs         |   4 +
>  hw/misc/memexpose/memexpose-core.c      | 630 ++++++++++++++++++++++++++++++++
>  hw/misc/memexpose/memexpose-core.h      | 109 ++++++
>  hw/misc/memexpose/memexpose-memregion.c | 142 +++++++
>  hw/misc/memexpose/memexpose-memregion.h |  41 +++
>  hw/misc/memexpose/memexpose-msg.c       | 261 +++++++++++++
>  hw/misc/memexpose/memexpose-msg.h       | 161 ++++++++
>  hw/misc/memexpose/memexpose-pci.c       | 218 +++++++++++
>  include/exec/memory.h                   |  20 +
>  include/exec/ram_addr.h                 |   2 +-
>  include/hw/arm/virt.h                   |   5 +
>  include/qemu/mmap-alloc.h               |   1 +
>  memory.c                                |  82 ++++-
>  tests/qtest/Makefile.include            |   2 +
>  tests/qtest/memexpose-test.c            | 364 ++++++++++++++++++
>  util/mmap-alloc.c                       |   7 +-
>  util/oslib-posix.c                      |   2 +-
>  29 files changed, 2360 insertions(+), 16 deletions(-)
>  create mode 100644 docs/specs/memexpose-spec.txt
>  create mode 100644 hw/misc/memexpose/Makefile.objs
>  create mode 100644 hw/misc/memexpose/memexpose-core.c
>  create mode 100644 hw/misc/memexpose/memexpose-core.h
>  create mode 100644 hw/misc/memexpose/memexpose-memregion.c
>  create mode 100644 hw/misc/memexpose/memexpose-memregion.h
>  create mode 100644 hw/misc/memexpose/memexpose-msg.c
>  create mode 100644 hw/misc/memexpose/memexpose-msg.h
>  create mode 100644 hw/misc/memexpose/memexpose-pci.c
>  create mode 100644 tests/qtest/memexpose-test.c
> 
> -- 
> 2.7.4
> 
> 

On 05.02.20 15:39, Stefan Hajnoczi wrote:
> On Tue, Feb 04, 2020 at 12:30:42PM +0100, i.kotrasinsk@partner.samsung.com wrote:
>> From: Igor Kotrasinski <i.kotrasinsk@partner.samsung.com>
>>
>> This patchset adds a "memory exposing" device that allows two QEMU
>> instances to share arbitrary memory regions. Unlike ivshmem, it does not
>> create a new region of memory that's shared between VMs, but instead
>> allows one VM to access any memory region of the other VM we choose to
>> share.
>>
>> The motivation for this device is a sort of ARM Trustzone "emulation",
>> where a rich system running on one machine (e.g. x86_64 linux) is able
>> to perform SMCs to a trusted system running on another (e.g. OpTEE on
>> ARM). With a device that allows sharing arbitrary memory between VMs,
>> this can be achieved with minimal changes to the trusted system and its
>> linux driver while allowing the rich system to run on a speedier x86
>> emulator. I prepared additional patches for linux, OpTEE OS and OpTEE
>> build system as a PoC that such emulation works and passes OpTEE tests;
>> I'm not sure what would be the best way to share them.
>>
>> This patchset is my first foray into QEMU source code and I'm certain
>> it's not yet ready to be merged in. I'm not sure whether memory sharing
>> code has any race conditions or breaks rules of working with memory
>> regions, or if having VMs communicate synchronously via chardevs is the
>> right way to do it. I do believe the basic idea for sharing memory
>> regions is sound and that it could be useful for inter-VM communication.
> 
> Hi,
> Without having looked into the patches yet, I'm already wondering if you
> can use the existing -object
> memory-backend-file,size=512M,mem-path=/my/shared/mem feature for your
> use case?
> 
> That's the existing mechanism for fully sharing guest RAM and if you
> want to share all of memory then maybe a device is not necessary - just
> share the memory.

I suspect it's about sharing that memory in a discoverable way. Maybe it 
is also about the signalling channel defined in the device.

OTOH, when it's really about sharing everything, even bidirectional, 
that rather looks like a pragmatic shortcut, not a generic model.

The patches should clarify their use case a bit further, I think. The 
title suggests a generic sharing solution, but my impression is that it 
rather caters a specific case under specific boundary conditions.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

On 2/5/20 3:49 PM, Jan Kiszka wrote:
> On 05.02.20 15:39, Stefan Hajnoczi wrote:
>> On Tue, Feb 04, 2020 at 12:30:42PM +0100, 
>> i.kotrasinsk@partner.samsung.com wrote:
>>> From: Igor Kotrasinski <i.kotrasinsk@partner.samsung.com>
>>>
>>> This patchset adds a "memory exposing" device that allows two QEMU
>>> instances to share arbitrary memory regions. Unlike ivshmem, it does not
>>> create a new region of memory that's shared between VMs, but instead
>>> allows one VM to access any memory region of the other VM we choose to
>>> share.
>>>
>>> The motivation for this device is a sort of ARM Trustzone "emulation",
>>> where a rich system running on one machine (e.g. x86_64 linux) is able
>>> to perform SMCs to a trusted system running on another (e.g. OpTEE on
>>> ARM). With a device that allows sharing arbitrary memory between VMs,
>>> this can be achieved with minimal changes to the trusted system and its
>>> linux driver while allowing the rich system to run on a speedier x86
>>> emulator. I prepared additional patches for linux, OpTEE OS and OpTEE
>>> build system as a PoC that such emulation works and passes OpTEE tests;
>>> I'm not sure what would be the best way to share them.
>>>
>>> This patchset is my first foray into QEMU source code and I'm certain
>>> it's not yet ready to be merged in. I'm not sure whether memory sharing
>>> code has any race conditions or breaks rules of working with memory
>>> regions, or if having VMs communicate synchronously via chardevs is the
>>> right way to do it. I do believe the basic idea for sharing memory
>>> regions is sound and that it could be useful for inter-VM communication.
>>
>> Hi,
>> Without having looked into the patches yet, I'm already wondering if you
>> can use the existing -object
>> memory-backend-file,size=512M,mem-path=/my/shared/mem feature for your
>> use case?
>>
>> That's the existing mechanism for fully sharing guest RAM and if you
>> want to share all of memory then maybe a device is not necessary - just
>> share the memory.

That option adds memory in addition to the memory allocated with the 
'-m' flag, doesn't it? I looked into that option, and it seemed to me 
you can't back all memory this way.

Apart from that, the only advantage my solution has is that it's aware 
of any memory overlaying the memory-backed regions (i.e. memory backed 
by accessor functions). Maybe I don't need this for my use case, I'd 
have to test that.

> 
> I suspect it's about sharing that memory in a discoverable way. Maybe it 
> is also about the signalling channel defined in the device.
> 
> OTOH, when it's really about sharing everything, even bidirectional, 
> that rather looks like a pragmatic shortcut, not a generic model.
> 
> The patches should clarify their use case a bit further, I think. The 
> title suggests a generic sharing solution, but my impression is that it 
> rather caters a specific case under specific boundary conditions.
> 
> Jan
> 

The solution does stem from a specific use case, the ARM Trustzone 
forwarding described in the cover letter. Normally both OSes can pass 
data around by sharing physical addresses (potentially from anywhere in 
memory), so giving VMs an ability to access one another's memory no 
matter how it's backed allows for minimal trusted OS modification, just 
offsetting physical addresses. The interrupt functionality also reflects 
this, it's intended to pass around SMC data.

I realize that this kind of total memory sharing couples the two VMs 
tightly - this is why I'm asking for comments on this, perhaps there's a 
better solution for this specific scenario.

For what it's worth, the extent of this sharing can be reduced by using 
a limited MemoryRegion like it's done for secure and non-secure memory 
views on ARM.

Igor

On Fri, 7 Feb 2020 10:00:50 +0100
Igor Kotrasiński <i.kotrasinsk@partner.samsung.com> wrote:

> On 2/5/20 3:49 PM, Jan Kiszka wrote:
> > On 05.02.20 15:39, Stefan Hajnoczi wrote:  
> >> On Tue, Feb 04, 2020 at 12:30:42PM +0100, 
> >> i.kotrasinsk@partner.samsung.com wrote:  
> >>> From: Igor Kotrasinski <i.kotrasinsk@partner.samsung.com>
> >>>
> >>> This patchset adds a "memory exposing" device that allows two QEMU
> >>> instances to share arbitrary memory regions. Unlike ivshmem, it does not
> >>> create a new region of memory that's shared between VMs, but instead
> >>> allows one VM to access any memory region of the other VM we choose to
> >>> share.
> >>>
> >>> The motivation for this device is a sort of ARM Trustzone "emulation",
> >>> where a rich system running on one machine (e.g. x86_64 linux) is able
> >>> to perform SMCs to a trusted system running on another (e.g. OpTEE on
> >>> ARM). With a device that allows sharing arbitrary memory between VMs,
> >>> this can be achieved with minimal changes to the trusted system and its
> >>> linux driver while allowing the rich system to run on a speedier x86
> >>> emulator. I prepared additional patches for linux, OpTEE OS and OpTEE
> >>> build system as a PoC that such emulation works and passes OpTEE tests;
> >>> I'm not sure what would be the best way to share them.
> >>>
> >>> This patchset is my first foray into QEMU source code and I'm certain
> >>> it's not yet ready to be merged in. I'm not sure whether memory sharing
> >>> code has any race conditions or breaks rules of working with memory
> >>> regions, or if having VMs communicate synchronously via chardevs is the
> >>> right way to do it. I do believe the basic idea for sharing memory
> >>> regions is sound and that it could be useful for inter-VM communication.  
> >>
> >> Hi,
> >> Without having looked into the patches yet, I'm already wondering if you
> >> can use the existing -object
> >> memory-backend-file,size=512M,mem-path=/my/shared/mem feature for your
> >> use case?
> >>
> >> That's the existing mechanism for fully sharing guest RAM and if you
> >> want to share all of memory then maybe a device is not necessary - just
> >> share the memory.  
> 
> That option adds memory in addition to the memory allocated with the 
> '-m' flag, doesn't it? I looked into that option, and it seemed to me 
> you can't back all memory this way.
with current QEMU you play with memory sharing using numa workaround

-m 512 \
-object memory-backend-file,id=mem,size=512M,mem-path=/my/shared/mem feature,share=on \
-numa node,memdev=mem

also on the list there is series that allows to share main ram
without numa workaround, see
  "[PATCH v4 00/80] refactor main RAM allocation to use hostmem backend"

with it applied you can share main RAM with following CLI:

-object memory-backend-file,id=mem,size=512M,mem-path=/my/shared/mem feature,share=on \
-m 512 \
-M virt,memory-backend=mem

> Apart from that, the only advantage my solution has is that it's aware 
> of any memory overlaying the memory-backed regions (i.e. memory backed 
> by accessor functions). Maybe I don't need this for my use case, I'd 
> have to test that.
> 
> > 
> > I suspect it's about sharing that memory in a discoverable way. Maybe it 
> > is also about the signalling channel defined in the device.
> > 
> > OTOH, when it's really about sharing everything, even bidirectional, 
> > that rather looks like a pragmatic shortcut, not a generic model.
> > 
> > The patches should clarify their use case a bit further, I think. The 
> > title suggests a generic sharing solution, but my impression is that it 
> > rather caters a specific case under specific boundary conditions.
> > 
> > Jan
> >   
> 
> The solution does stem from a specific use case, the ARM Trustzone 
> forwarding described in the cover letter. Normally both OSes can pass 
> data around by sharing physical addresses (potentially from anywhere in 
> memory), so giving VMs an ability to access one another's memory no 
> matter how it's backed allows for minimal trusted OS modification, just 
> offsetting physical addresses. The interrupt functionality also reflects 
> this, it's intended to pass around SMC data.
> 
> I realize that this kind of total memory sharing couples the two VMs 
> tightly - this is why I'm asking for comments on this, perhaps there's a 
> better solution for this specific scenario.
> 
> For what it's worth, the extent of this sharing can be reduced by using 
> a limited MemoryRegion like it's done for secure and non-secure memory 
> views on ARM.
> 
> Igor
> 

On Fri, Feb 07, 2020 at 11:04:03AM +0100, Igor Mammedov wrote:
> On Fri, 7 Feb 2020 10:00:50 +0100
> Igor Kotrasiński <i.kotrasinsk@partner.samsung.com> wrote:
> 
> > On 2/5/20 3:49 PM, Jan Kiszka wrote:
> > > On 05.02.20 15:39, Stefan Hajnoczi wrote:  
> > >> On Tue, Feb 04, 2020 at 12:30:42PM +0100, 
> > >> i.kotrasinsk@partner.samsung.com wrote:  
> > >>> From: Igor Kotrasinski <i.kotrasinsk@partner.samsung.com>
> > >>>
> > >>> This patchset adds a "memory exposing" device that allows two QEMU
> > >>> instances to share arbitrary memory regions. Unlike ivshmem, it does not
> > >>> create a new region of memory that's shared between VMs, but instead
> > >>> allows one VM to access any memory region of the other VM we choose to
> > >>> share.
> > >>>
> > >>> The motivation for this device is a sort of ARM Trustzone "emulation",
> > >>> where a rich system running on one machine (e.g. x86_64 linux) is able
> > >>> to perform SMCs to a trusted system running on another (e.g. OpTEE on
> > >>> ARM). With a device that allows sharing arbitrary memory between VMs,
> > >>> this can be achieved with minimal changes to the trusted system and its
> > >>> linux driver while allowing the rich system to run on a speedier x86
> > >>> emulator. I prepared additional patches for linux, OpTEE OS and OpTEE
> > >>> build system as a PoC that such emulation works and passes OpTEE tests;
> > >>> I'm not sure what would be the best way to share them.
> > >>>
> > >>> This patchset is my first foray into QEMU source code and I'm certain
> > >>> it's not yet ready to be merged in. I'm not sure whether memory sharing
> > >>> code has any race conditions or breaks rules of working with memory
> > >>> regions, or if having VMs communicate synchronously via chardevs is the
> > >>> right way to do it. I do believe the basic idea for sharing memory
> > >>> regions is sound and that it could be useful for inter-VM communication.  
> > >>
> > >> Hi,
> > >> Without having looked into the patches yet, I'm already wondering if you
> > >> can use the existing -object
> > >> memory-backend-file,size=512M,mem-path=/my/shared/mem feature for your
> > >> use case?
> > >>
> > >> That's the existing mechanism for fully sharing guest RAM and if you
> > >> want to share all of memory then maybe a device is not necessary - just
> > >> share the memory.  
> > 
> > That option adds memory in addition to the memory allocated with the 
> > '-m' flag, doesn't it? I looked into that option, and it seemed to me 
> > you can't back all memory this way.
> with current QEMU you play with memory sharing using numa workaround
> 
> -m 512 \
> -object memory-backend-file,id=mem,size=512M,mem-path=/my/shared/mem feature,share=on \
> -numa node,memdev=mem
> 
> also on the list there is series that allows to share main ram
> without numa workaround, see
>   "[PATCH v4 00/80] refactor main RAM allocation to use hostmem backend"
> 
> with it applied you can share main RAM with following CLI:
> 
> -object memory-backend-file,id=mem,size=512M,mem-path=/my/shared/mem feature,share=on \
> -m 512 \
> -M virt,memory-backend=mem

Nice!  That takes care of memory.

If signalling (e.g. a notification interrupt) is necessary then a
mechanism is still needed for that.  I don't know enough about TrustZone
to suggest an appropriate way of doing it with existing QEMU features.
Maybe Peter understands?

Stefan

On 2/7/20 5:33 PM, Stefan Hajnoczi wrote:
> On Fri, Feb 07, 2020 at 11:04:03AM +0100, Igor Mammedov wrote:
>> On Fri, 7 Feb 2020 10:00:50 +0100
>> Igor Kotrasiński <i.kotrasinsk@partner.samsung.com> wrote:
>>
>>> On 2/5/20 3:49 PM, Jan Kiszka wrote:
>>>> On 05.02.20 15:39, Stefan Hajnoczi wrote:
>>>>> On Tue, Feb 04, 2020 at 12:30:42PM +0100,
>>>>> i.kotrasinsk@partner.samsung.com wrote:
>>>>>> From: Igor Kotrasinski <i.kotrasinsk@partner.samsung.com>
>>>>>>
>>>>>> This patchset adds a "memory exposing" device that allows two QEMU
>>>>>> instances to share arbitrary memory regions. Unlike ivshmem, it does not
>>>>>> create a new region of memory that's shared between VMs, but instead
>>>>>> allows one VM to access any memory region of the other VM we choose to
>>>>>> share.
>>>>>>
>>>>>> The motivation for this device is a sort of ARM Trustzone "emulation",
>>>>>> where a rich system running on one machine (e.g. x86_64 linux) is able
>>>>>> to perform SMCs to a trusted system running on another (e.g. OpTEE on
>>>>>> ARM). With a device that allows sharing arbitrary memory between VMs,
>>>>>> this can be achieved with minimal changes to the trusted system and its
>>>>>> linux driver while allowing the rich system to run on a speedier x86
>>>>>> emulator. I prepared additional patches for linux, OpTEE OS and OpTEE
>>>>>> build system as a PoC that such emulation works and passes OpTEE tests;
>>>>>> I'm not sure what would be the best way to share them.
>>>>>>
>>>>>> This patchset is my first foray into QEMU source code and I'm certain
>>>>>> it's not yet ready to be merged in. I'm not sure whether memory sharing
>>>>>> code has any race conditions or breaks rules of working with memory
>>>>>> regions, or if having VMs communicate synchronously via chardevs is the
>>>>>> right way to do it. I do believe the basic idea for sharing memory
>>>>>> regions is sound and that it could be useful for inter-VM communication.
>>>>>
>>>>> Hi,
>>>>> Without having looked into the patches yet, I'm already wondering if you
>>>>> can use the existing -object
>>>>> memory-backend-file,size=512M,mem-path=/my/shared/mem feature for your
>>>>> use case?
>>>>>
>>>>> That's the existing mechanism for fully sharing guest RAM and if you
>>>>> want to share all of memory then maybe a device is not necessary - just
>>>>> share the memory.
>>>
>>> That option adds memory in addition to the memory allocated with the
>>> '-m' flag, doesn't it? I looked into that option, and it seemed to me
>>> you can't back all memory this way.
>> with current QEMU you play with memory sharing using numa workaround
>>
>> -m 512 \
>> -object memory-backend-file,id=mem,size=512M,mem-path=/my/shared/mem feature,share=on \
>> -numa node,memdev=mem
>>
>> also on the list there is series that allows to share main ram
>> without numa workaround, see
>>    "[PATCH v4 00/80] refactor main RAM allocation to use hostmem backend"
>>
>> with it applied you can share main RAM with following CLI:
>>
>> -object memory-backend-file,id=mem,size=512M,mem-path=/my/shared/mem feature,share=on \
>> -m 512 \
>> -M virt,memory-backend=mem
> 
> Nice!  That takes care of memory.

After a bit of hacking to map the shared RAM instead of communicating 
via socket I can confirm - I can run OpTEE this way, and it passes 
tests. My solution is *technically* more accurate since it is aware of 
memory subregions and completely independent from memory backend setup, 
but with my use case satisfied already, I don't think it's of use to anyone.

> 
> If signalling (e.g. a notification interrupt) is necessary then a
> mechanism is still needed for that.  I don't know enough about TrustZone
> to suggest an appropriate way of doing it with existing QEMU features.
> Maybe Peter understands?
> 

Any signalling mechanism that can pass data along with it (e.g. ivshmem 
with its shared memory) will suffice.

Igor

On Mon, Feb 10, 2020 at 02:01:48PM +0100, Igor Kotrasiński wrote:
> On 2/7/20 5:33 PM, Stefan Hajnoczi wrote:
> > On Fri, Feb 07, 2020 at 11:04:03AM +0100, Igor Mammedov wrote:
> >> On Fri, 7 Feb 2020 10:00:50 +0100
> >> Igor Kotrasiński <i.kotrasinsk@partner.samsung.com> wrote:
> >>
> >>> On 2/5/20 3:49 PM, Jan Kiszka wrote:
> >>>> On 05.02.20 15:39, Stefan Hajnoczi wrote:
> >>>>> On Tue, Feb 04, 2020 at 12:30:42PM +0100,
> >>>>> i.kotrasinsk@partner.samsung.com wrote:
> >>>>>> From: Igor Kotrasinski <i.kotrasinsk@partner.samsung.com>
> >>>>>>
> >>>>>> This patchset adds a "memory exposing" device that allows two QEMU
> >>>>>> instances to share arbitrary memory regions. Unlike ivshmem, it does not
> >>>>>> create a new region of memory that's shared between VMs, but instead
> >>>>>> allows one VM to access any memory region of the other VM we choose to
> >>>>>> share.
> >>>>>>
> >>>>>> The motivation for this device is a sort of ARM Trustzone "emulation",
> >>>>>> where a rich system running on one machine (e.g. x86_64 linux) is able
> >>>>>> to perform SMCs to a trusted system running on another (e.g. OpTEE on
> >>>>>> ARM). With a device that allows sharing arbitrary memory between VMs,
> >>>>>> this can be achieved with minimal changes to the trusted system and its
> >>>>>> linux driver while allowing the rich system to run on a speedier x86
> >>>>>> emulator. I prepared additional patches for linux, OpTEE OS and OpTEE
> >>>>>> build system as a PoC that such emulation works and passes OpTEE tests;
> >>>>>> I'm not sure what would be the best way to share them.
> >>>>>>
> >>>>>> This patchset is my first foray into QEMU source code and I'm certain
> >>>>>> it's not yet ready to be merged in. I'm not sure whether memory sharing
> >>>>>> code has any race conditions or breaks rules of working with memory
> >>>>>> regions, or if having VMs communicate synchronously via chardevs is the
> >>>>>> right way to do it. I do believe the basic idea for sharing memory
> >>>>>> regions is sound and that it could be useful for inter-VM communication.
> >>>>>
> >>>>> Hi,
> >>>>> Without having looked into the patches yet, I'm already wondering if you
> >>>>> can use the existing -object
> >>>>> memory-backend-file,size=512M,mem-path=/my/shared/mem feature for your
> >>>>> use case?
> >>>>>
> >>>>> That's the existing mechanism for fully sharing guest RAM and if you
> >>>>> want to share all of memory then maybe a device is not necessary - just
> >>>>> share the memory.
> >>>
> >>> That option adds memory in addition to the memory allocated with the
> >>> '-m' flag, doesn't it? I looked into that option, and it seemed to me
> >>> you can't back all memory this way.
> >> with current QEMU you play with memory sharing using numa workaround
> >>
> >> -m 512 \
> >> -object memory-backend-file,id=mem,size=512M,mem-path=/my/shared/mem feature,share=on \
> >> -numa node,memdev=mem
> >>
> >> also on the list there is series that allows to share main ram
> >> without numa workaround, see
> >>    "[PATCH v4 00/80] refactor main RAM allocation to use hostmem backend"
> >>
> >> with it applied you can share main RAM with following CLI:
> >>
> >> -object memory-backend-file,id=mem,size=512M,mem-path=/my/shared/mem feature,share=on \
> >> -m 512 \
> >> -M virt,memory-backend=mem
> > 
> > Nice!  That takes care of memory.
> 
> After a bit of hacking to map the shared RAM instead of communicating 
> via socket I can confirm - I can run OpTEE this way, and it passes 
> tests. My solution is *technically* more accurate since it is aware of 
> memory subregions and completely independent from memory backend setup, 
> but with my use case satisfied already, I don't think it's of use to anyone.

Great!

Stefan

On 10.02.2020 14:01, Igor Kotrasiński wrote:
> On 2/7/20 5:33 PM, Stefan Hajnoczi wrote:
>> On Fri, Feb 07, 2020 at 11:04:03AM +0100, Igor Mammedov wrote:
>>> On Fri, 7 Feb 2020 10:00:50 +0100
>>> Igor Kotrasiński <i.kotrasinsk@partner.samsung.com> wrote:
>>>
>>>> On 2/5/20 3:49 PM, Jan Kiszka wrote:
>>>>> On 05.02.20 15:39, Stefan Hajnoczi wrote:
>>>>>> On Tue, Feb 04, 2020 at 12:30:42PM +0100,
>>>>>> i.kotrasinsk@partner.samsung.com wrote:
>>>>>>> From: Igor Kotrasinski <i.kotrasinsk@partner.samsung.com>
>>>>>>>
>>>>>>> This patchset adds a "memory exposing" device that allows two QEMU
>>>>>>> instances to share arbitrary memory regions. Unlike ivshmem, it 
>>>>>>> does not
>>>>>>> create a new region of memory that's shared between VMs, but instead
>>>>>>> allows one VM to access any memory region of the other VM we 
>>>>>>> choose to
>>>>>>> share.
>>>>>>>
>>>>>>> The motivation for this device is a sort of ARM Trustzone 
>>>>>>> "emulation",
>>>>>>> where a rich system running on one machine (e.g. x86_64 linux) is 
>>>>>>> able
>>>>>>> to perform SMCs to a trusted system running on another (e.g. 
>>>>>>> OpTEE on
>>>>>>> ARM). With a device that allows sharing arbitrary memory between 
>>>>>>> VMs,
>>>>>>> this can be achieved with minimal changes to the trusted system 
>>>>>>> and its
>>>>>>> linux driver while allowing the rich system to run on a speedier x86
>>>>>>> emulator. I prepared additional patches for linux, OpTEE OS and 
>>>>>>> OpTEE
>>>>>>> build system as a PoC that such emulation works and passes OpTEE 
>>>>>>> tests;
>>>>>>> I'm not sure what would be the best way to share them.
>>>>>>>
>>>>>>> This patchset is my first foray into QEMU source code and I'm 
>>>>>>> certain
>>>>>>> it's not yet ready to be merged in. I'm not sure whether memory 
>>>>>>> sharing
>>>>>>> code has any race conditions or breaks rules of working with memory
>>>>>>> regions, or if having VMs communicate synchronously via chardevs 
>>>>>>> is the
>>>>>>> right way to do it. I do believe the basic idea for sharing memory
>>>>>>> regions is sound and that it could be useful for inter-VM 
>>>>>>> communication.
>>>>>>
>>>>>> Hi,
>>>>>> Without having looked into the patches yet, I'm already wondering 
>>>>>> if you
>>>>>> can use the existing -object
>>>>>> memory-backend-file,size=512M,mem-path=/my/shared/mem feature for 
>>>>>> your
>>>>>> use case?
>>>>>>
>>>>>> That's the existing mechanism for fully sharing guest RAM and if you
>>>>>> want to share all of memory then maybe a device is not necessary - 
>>>>>> just
>>>>>> share the memory.
>>>>
>>>> That option adds memory in addition to the memory allocated with the
>>>> '-m' flag, doesn't it? I looked into that option, and it seemed to me
>>>> you can't back all memory this way.
>>> with current QEMU you play with memory sharing using numa workaround
>>>
>>> -m 512 \
>>> -object memory-backend-file,id=mem,size=512M,mem-path=/my/shared/mem 
>>> feature,share=on \
>>> -numa node,memdev=mem
>>>
>>> also on the list there is series that allows to share main ram
>>> without numa workaround, see
>>>    "[PATCH v4 00/80] refactor main RAM allocation to use hostmem 
>>> backend"
>>>
>>> with it applied you can share main RAM with following CLI:
>>>
>>> -object memory-backend-file,id=mem,size=512M,mem-path=/my/shared/mem 
>>> feature,share=on \
>>> -m 512 \
>>> -M virt,memory-backend=mem
>>
>> Nice!  That takes care of memory.
> 
> After a bit of hacking to map the shared RAM instead of communicating 
> via socket I can confirm - I can run OpTEE this way, and it passes 
> tests. My solution is *technically* more accurate since it is aware of 
> memory subregions and completely independent from memory backend setup, 
> but with my use case satisfied already, I don't think it's of use to 
> anyone.
> 

After a long while hacking QEMU to achieve 1-to-1 memory mapping between 
machines, I realized that I wasn't completely right here. I can share 
main memory from both machines, but for the arm machine that's only 
non-secure memory. Secure memory (VIRT_SECURE_MEM) is always allocated 
with memory_region_init_ram(), I don't know if other secure memory 
regions (e.g. VIRT_FLASH) might need to be shared as well.

This can probably be solved by allowing these regions to use file-backed 
memory when configured to do so, then mapping these files in the other 
machine at correct offsets.

Another solution would be memory sharing in this patchset. If we strip 
away interrupts, PCI stuff and io memory region support, it amounts to 
basically the same thing - a mechanism for accessing shareable memory in 
the other machine in a discoverable way.

>>
>> If signalling (e.g. a notification interrupt) is necessary then a
>> mechanism is still needed for that.  I don't know enough about TrustZone
>> to suggest an appropriate way of doing it with existing QEMU features.
>> Maybe Peter understands?
>>
> 
> Any signalling mechanism that can pass data along with it (e.g. ivshmem 
> with its shared memory) will suffice.

Igor

Patchew URL: https://patchew.org/QEMU/1580815851-28887-1-git-send-email-i.kotrasinsk@partner.samsung.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [RFC 0/9] Add an interVM memory sharing device
Message-id: 1580815851-28887-1-git-send-email-i.kotrasinsk@partner.samsung.com
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 - [tag update]      patchew/20200204105142.21845-1-alex.bennee@linaro.org -> patchew/20200204105142.21845-1-alex.bennee@linaro.org
 - [tag update]      patchew/20200204110501.10731-1-dgilbert@redhat.com -> patchew/20200204110501.10731-1-dgilbert@redhat.com
Switched to a new branch 'test'
4b2f7b1 hw/arm/virt: Hack in support for memexpose device
bea4faf hw/misc/memexpose: Add simple tests
b9a53fc hw/misc/memexpose: Add memexpose memory region device
8d2c64f hw/misc/memexpose: Add memexpose pci device
6a6d45a hw/misc/memexpose: Add core memexpose files
c35aef7 hw/misc/memexpose: Add documentation
d6e7169 memory: Hack - use shared memory when possible
6f6a337 memory: Support mmap offset for fd-backed memory regions
fe515a9 memory: Add function for finding flat memory ranges

=== OUTPUT BEGIN ===
1/9 Checking commit fe515a937f89 (memory: Add function for finding flat memory ranges)
2/9 Checking commit 6f6a337b8e94 (memory: Support mmap offset for fd-backed memory regions)
3/9 Checking commit d6e716993be8 (memory: Hack - use shared memory when possible)
4/9 Checking commit c35aef77ec71 (hw/misc/memexpose: Add documentation)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#11: 
new file mode 100644

ERROR: code blocks in documentation should have empty lines with exactly 4 columns of whitespace
#45: FILE: docs/specs/memexpose-spec.txt:30:
+ $

total: 1 errors, 1 warnings, 168 lines checked

Patch 4/9 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

5/9 Checking commit 6a6d45a15b43 (hw/misc/memexpose: Add core memexpose files)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#115: 
new file mode 100644

total: 0 errors, 1 warnings, 1235 lines checked

Patch 5/9 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
6/9 Checking commit 8d2c64f273a3 (hw/misc/memexpose: Add memexpose pci device)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#31: 
new file mode 100644

total: 0 errors, 1 warnings, 228 lines checked

Patch 6/9 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
7/9 Checking commit b9a53fc1ef8b (hw/misc/memexpose: Add memexpose memory region device)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#33: 
new file mode 100644

total: 0 errors, 1 warnings, 195 lines checked

Patch 7/9 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
8/9 Checking commit bea4faff4da5 (hw/misc/memexpose: Add simple tests)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#43: 
new file mode 100644

total: 0 errors, 1 warnings, 385 lines checked

Patch 8/9 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
9/9 Checking commit 4b2f7b165800 (hw/arm/virt: Hack in support for memexpose device)
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/1580815851-28887-1-git-send-email-i.kotrasinsk@partner.samsung.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Patchew URL: https://patchew.org/QEMU/1580815851-28887-1-git-send-email-i.kotrasinsk@partner.samsung.com/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      block/blkreplay.o
  CC      block/parallels.o
  CC      block/blklogwrites.o
/tmp/qemu-test/src/docs/../include/exec/memory.h:923: warning: Function parameter or member 'mmap_offset' not described in 'memory_region_init_ram_from_fd'

Warning, treated as error:
/tmp/qemu-test/src/docs/../include/exec/memory.h:1923:Unexpected indentation.
  CC      block/block-backend.o
  CC      block/snapshot.o
---
  CC      block/create.o
  CC      block/throttle-groups.o
  CC      block/nbd.o
make: *** [Makefile:1043: docs/devel/index.html] Error 2
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 662, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=4e0ce4bc006546c6a7e47701712aa571', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-gsa7_m4x/src/docker-src.2020-02-04-07.14.22.20243:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=4e0ce4bc006546c6a7e47701712aa571
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-gsa7_m4x/src'
make: *** [docker-run-test-mingw@fedora] Error 2

real    1m55.421s
user    0m8.478s


The full log is available at
http://patchew.org/logs/1580815851-28887-1-git-send-email-i.kotrasinsk@partner.samsung.com/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com