From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

You can find an initial discussion at [1]-[7].

The extended region (safe range) is a region of guest physical address space
which is unused and could be safely used to create grant/foreign mappings instead
of wasting real RAM pages from the domain memory for establishing these mappings.

The extended regions are chosen at the domain creation time and advertised
to it via "reg" property under hypervisor node in the guest device-tree
(the indexes for extended regions are 1...N).
No device tree bindings update is needed, guest infers the presense of extended
regions from the number of regions in "reg" property.
New compatible/property will be needed (but only after this patch [8] or alternative
goes in) to indicate that "region 0 is safe to use". Until this patch is merged it is
not safe to use extended regions for the grant table space.

The extended regions are calculated differently for direct mapped Dom0 (with and without
IOMMU) and non-direct mapped DomUs.

Please note the following limitations:
- The extended region feature is only supported for 64-bit domain currently.
- The ACPI case is not covered.

Please note that support for Dom0 was already committed, so these patches are remaining DomU bits.

Xen patch series is also available at [9]. The corresponding Linux patch series is at [10]
for now (last 4 patches).

Tested on Renesas Salvator-X board + H3 ES3.0 SoC (Arm64) with updated virtio-disk backend [11]
running in Dom0 (256MB RAM) and DomD (2GB RAM). In both cases the backend pre-maps DomU memory
which is 3GB. All foreign memory gets mapped into extended regions (so the amount of RAM in
the backend domain is not reduced). No issues were observed.

[1] https://lore.kernel.org/xen-devel/1627489110-25633-1-git-send-email-olekstysh@gmail.com/
[2] https://lore.kernel.org/xen-devel/1631034578-12598-1-git-send-email-olekstysh@gmail.com/
[3] https://lore.kernel.org/xen-devel/1631297924-8658-1-git-send-email-olekstysh@gmail.com/
[4] https://lore.kernel.org/xen-devel/1632437334-12015-1-git-send-email-olekstysh@gmail.com/
[5] https://lore.kernel.org/xen-devel/1632955927-27911-1-git-send-email-olekstysh@gmail.com/
[6] https://lore.kernel.org/xen-devel/1633519346-3686-1-git-send-email-olekstysh@gmail.com/
[7] https://lore.kernel.org/xen-devel/1633974539-7380-1-git-send-email-olekstysh@gmail.com/
[8] https://lore.kernel.org/xen-devel/1632425551-18910-1-git-send-email-olekstysh@gmail.com/
[9] https://github.com/otyshchenko1/xen/commits/map_opt_ml8
[10] https://github.com/otyshchenko1/linux/commits/map_opt_ml4
[11] https://github.com/otyshchenko1/virtio-disk/commits/map_opt_next

Oleksandr Tyshchenko (2):
  xen/arm: Introduce gpaddr_bits field to struct
    xen_domctl_getdomaininfo
  libxl/arm: Add handling of extended regions for DomU

 tools/include/libxl.h            |   8 +++
 tools/include/xenctrl.h          |   1 +
 tools/libs/ctrl/xc_domain.c      |   1 +
 tools/libs/light/libxl_arm.c     | 106 +++++++++++++++++++++++++++++++++++++--
 tools/libs/light/libxl_domain.c  |   1 +
 tools/libs/light/libxl_types.idl |   1 +
 xen/arch/arm/domctl.c            |   2 +
 xen/arch/x86/domctl.c            |   1 +
 xen/common/domctl.c              |   4 +-
 xen/common/sysctl.c              |   2 +-
 xen/include/public/arch-arm.h    |   5 ++
 xen/include/public/domctl.h      |   3 ++
 xen/include/public/sysctl.h      |   2 +-
 13 files changed, 128 insertions(+), 9 deletions(-)

-- 
2.7.4

Oleksandr Tyshchenko writes ("[PATCH V7 0/2] Add handling of extended regions (safe ranges) on Arm (Was "xen/memory: Introduce a hypercall to provide unallocated space")"):
> You can find an initial discussion at [1]-[7].
> 
> The extended region (safe range) is a region of guest physical address space
> which is unused and could be safely used to create grant/foreign mappings instead
> of wasting real RAM pages from the domain memory for establishing these mappings.

Thanks.

This patch has all the required acks, but I was aware of an
outstanding concern from Andrew, as set out in his most
recent mail on the subject:
  Subject: Re: [RFC PATCH 1/3] xen: Introduce "gpaddr_bits" field to XEN_SYSCTL_physinfo
  Date: Tue, 7 Sep 2021 18:35:47 +0100
  Message-ID: <973f5344-aa10-3ad6-ff02-ad5f358ad279@citrix.com>

I think it would be within the process to just commit the patch now,
but I thought it best to check as best I could that we weren't missing
anything.  The process is supposed to support our continuing
development and also our quality, so I aim to do those things.

I reviewed that mail and had a conversation with Julien about it on
irc.  My understanding is that Julien and Oleksandr's intent is that
Andrew's concerns have been addressed, although we don't have a
confirmation of that from Andrew.

In particular, I wanted to convince myself that if in fact there was
still a problem, we hadn't made a problem for ourselves with the API
here.

The new hypercalls are in unstable interfaces, so if we need to change
them in a future version (eg to make ARM migration work) that's OK.
Julien tells me that he doesn't believe there to be any impact on the
(x86) migration stream right now.

There is a new libxl stable interface.  But I think it is
inoffensive.  In particular, basically any mechanism to do this would
have that API.  And that doesn't seem to touch on the implementation
issues described by Andrew.

Therefore, I think (i) we have tried to address the issues (ii) any
reminaing problems can be dealt with as followups, without trouble.

So I have just pushed these two.

Thanks,
Ian.

On 14.10.21 17:28, Ian Jackson wrote:

Hi Ian

> Oleksandr Tyshchenko writes ("[PATCH V7 0/2] Add handling of extended regions (safe ranges) on Arm (Was "xen/memory: Introduce a hypercall to provide unallocated space")"):
>> You can find an initial discussion at [1]-[7].
>>
>> The extended region (safe range) is a region of guest physical address space
>> which is unused and could be safely used to create grant/foreign mappings instead
>> of wasting real RAM pages from the domain memory for establishing these mappings.
> Thanks.
>
> This patch has all the required acks, but I was aware of an
> outstanding concern from Andrew, as set out in his most
> recent mail on the subject:
>    Subject: Re: [RFC PATCH 1/3] xen: Introduce "gpaddr_bits" field to XEN_SYSCTL_physinfo
>    Date: Tue, 7 Sep 2021 18:35:47 +0100
>    Message-ID: <973f5344-aa10-3ad6-ff02-ad5f358ad279@citrix.com>
>
> I think it would be within the process to just commit the patch now,
> but I thought it best to check as best I could that we weren't missing
> anything.  The process is supposed to support our continuing
> development and also our quality, so I aim to do those things.
>
> I reviewed that mail and had a conversation with Julien about it on
> irc.  My understanding is that Julien and Oleksandr's intent is that
> Andrew's concerns have been addressed, although we don't have a
> confirmation of that from Andrew.
>
> In particular, I wanted to convince myself that if in fact there was
> still a problem, we hadn't made a problem for ourselves with the API
> here.
>
> The new hypercalls are in unstable interfaces, so if we need to change
> them in a future version (eg to make ARM migration work) that's OK.
> Julien tells me that he doesn't believe there to be any impact on the
> (x86) migration stream right now.
>
> There is a new libxl stable interface.  But I think it is
> inoffensive.  In particular, basically any mechanism to do this would
> have that API.  And that doesn't seem to touch on the implementation
> issues described by Andrew.
>
> Therefore, I think (i) we have tried to address the issues (ii) any
> reminaing problems can be dealt with as followups, without trouble.

Completely agree with both statements.


>
> So I have just pushed these two.

Thanks!


> Thanks,
> Ian.

-- 
Regards,

Oleksandr Tyshchenko

Hello all


On 14.10.21 14:40, Oleksandr Tyshchenko wrote:
> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
>
> You can find an initial discussion at [1]-[7].
>
> The extended region (safe range) is a region of guest physical address space
> which is unused and could be safely used to create grant/foreign mappings instead
> of wasting real RAM pages from the domain memory for establishing these mappings.
>
> The extended regions are chosen at the domain creation time and advertised
> to it via "reg" property under hypervisor node in the guest device-tree
> (the indexes for extended regions are 1...N).
> No device tree bindings update is needed, guest infers the presense of extended
> regions from the number of regions in "reg" property.
> New compatible/property will be needed (but only after this patch [8] or alternative
> goes in) to indicate that "region 0 is safe to use". Until this patch is merged it is
> not safe to use extended regions for the grant table space.
>
> The extended regions are calculated differently for direct mapped Dom0 (with and without
> IOMMU) and non-direct mapped DomUs.
>
> Please note the following limitations:
> - The extended region feature is only supported for 64-bit domain currently.
> - The ACPI case is not covered.
>
> Please note that support for Dom0 was already committed, so these patches are remaining DomU bits.
>
> Xen patch series is also available at [9]. The corresponding Linux patch series is at [10]
> for now (last 4 patches).

So, all Xen changes are already committed (thanks!) and I will start 
trying to push Linux changes. I believe that with your help in review, 
it will be possible to finish this enabling work.


[snip]

-- 
Regards,

Oleksandr Tyshchenko