[PATCH v3 00/10] direct-map memory map

Penny Zheng posted 10 patches 2 weeks, 5 days ago
Test gitlab-ci failed
Patches applied successfully (tree, apply log)
git fetch https://gitlab.com/xen-project/patchew/xen tags/patchew/20211116063155.901183-1-penny.zheng@arm.com
docs/misc/arm/device-tree/booting.txt |   6 +
docs/misc/arm/passthrough-noiommu.txt |  52 +++++
xen/arch/arm/domain.c                 |   3 +-
xen/arch/arm/domain_build.c           | 310 +++++++++++++++++++++-----
xen/arch/arm/vgic-v2.c                |  31 ++-
xen/arch/arm/vgic-v3.c                |  29 ++-
xen/arch/arm/vgic/vgic-v2.c           |  31 ++-
xen/arch/arm/vpl011.c                 |  60 ++++-
xen/arch/x86/domain.c                 |   6 +
xen/common/domain.c                   |   3 +-
xen/include/asm-arm/domain.h          |  11 +-
xen/include/asm-arm/new_vgic.h        |  10 +
xen/include/asm-arm/vgic.h            |  11 +
xen/include/asm-arm/vpl011.h          |   2 +
xen/include/public/domctl.h           |   4 +
xen/include/xen/domain.h              |   3 +
16 files changed, 471 insertions(+), 101 deletions(-)
create mode 100644 docs/misc/arm/passthrough-noiommu.txt

[PATCH v3 00/10] direct-map memory map

Posted by Penny Zheng 2 weeks, 5 days ago
Cases where domU needs direct-map memory map:
  * IOMMU not present in the system.
  * IOMMU disabled if it doesn't cover a specific device and all the guests
are trusted. Thinking a mixed scenario, where a few devices with IOMMU and
a few without, then guest DMA security still could not be totally guaranteed.
So users may want to disable the IOMMU, to at least gain some performance
improvement from IOMMU disabled.
  * IOMMU disabled as a workaround when it doesn't have enough bandwidth.
To be specific, in a few extreme situation, when multiple devices do DMA
concurrently, these requests may exceed IOMMU's transmission capacity.
  * IOMMU disabled when it adds too much latency on DMA. For example,
TLB may be missing in some IOMMU hardware, which may bring latency in DMA
progress, so users may want to disable it in some realtime scenario.
  * Guest OS relies on the host memory layout

"direct-map" property shall be added under the appropriate domain node,
when users requesting direct-map memory mapping for the domain.

Right now, direct-map is only supported when domain on Static Allocation,
that is, "xen,static-mem" is also necessary in the domain configuration.

Looking into related [design link](
https://lists.xenproject.org/archives/html/xen-devel/2021-05/msg00882.html)
for more details.

The whole design is about Static Allocation and direct-map, and this
Patch Serie only covers parts of it, which are direct-map memory map.
Other features will be delievered through different patch series.

See https://lists.xenproject.org/archives/html/xen-devel/2021-09/msg00855.html
for Domain on Static Allocation.

This patch serie is based on
https://lists.xenproject.org/archives/html/xen-devel/2021-10/msg00822.html
---
v3 changes:
- move flag XEN_DOMCTL_CDF_INTERNAL_directmap back to xen/include/xen/domain.h,
to let it be only available for domain created by XEN.
- name it with extra "INTERNAL" and add comments to warn developers not
to accidently use its bitfield when introducing new XEN_DOMCTL_CDF_xxx flag.
- reject this flag in x86'es arch_sanitise_domain_config()
- add ASSERT_UNREACHABLE to catch any misuse in allocate_static_memory()
and allocate_static_memory_11()
- add another check of validating flag XEN_DOMCTL_CDF_INTERNAL_directmap only
when CONFIG_STATIC_MEMORY is set.
- simply map the CPU interface at the GPA vgic_v2_hw.cbase
- drop 'cells += (GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS)'
- rename 'is_domain_use_host_layout()' to 'domain_use_host_layout()'
---
v2 changes:
- remove the introduce of internal flag
- Refine is_domain_direct_mapped to check whether the flag
XEN_DOMCTL_CDF_directmap is set
- reword "1:1 direct-map" to just "direct-map"
- split the common codes into two helpers: parse_static_mem_prop and
acquire_static_memory_bank to deduce complexity.
- introduce a new helper allocate_static_memory_11 for allocating static
memory for direct-map guests
- remove panic action since it is fine to assign a non-DMA capable device when
IOMMU and direct-map both off
- remove redistributor accessor
- introduce new helper "is_domain_use_host_layout()"
- explain why vpl011 initialization before creating its device tree node
- error out if the domain is direct-mapped and the IRQ is not found
- harden the code and add a check/comment when the hardware UART region
is smaller than CUEST_VPL011_SIZE.

Penny Zheng (4):
  xen/arm: introduce new helper parse_static_mem_prop and ...
  xen/arm: introduce direct-map for domUs
  xen/arm: add ASSERT_UNREACHABLE in allocate_static_memory
  xen/arm: gate make_gicv3_domU_node with CONFIG_GICV3

Stefano Stabellini (6):
  xen: introduce XEN_DOMCTL_CDF_INTERNAL_directmap
  xen/arm: avoid setting XEN_DOMCTL_CDF_iommu when IOMMU off
  xen/arm: if direct-map domain use native addresses for GICv2
  xen/arm: if direct-map domain use native addresses for GICv3
  xen/arm: if direct-map domain use native UART address and IRQ ...
  xen/docs: Document how to do passthrough without IOMMU

 docs/misc/arm/device-tree/booting.txt |   6 +
 docs/misc/arm/passthrough-noiommu.txt |  52 +++++
 xen/arch/arm/domain.c                 |   3 +-
 xen/arch/arm/domain_build.c           | 310 +++++++++++++++++++++-----
 xen/arch/arm/vgic-v2.c                |  31 ++-
 xen/arch/arm/vgic-v3.c                |  29 ++-
 xen/arch/arm/vgic/vgic-v2.c           |  31 ++-
 xen/arch/arm/vpl011.c                 |  60 ++++-
 xen/arch/x86/domain.c                 |   6 +
 xen/common/domain.c                   |   3 +-
 xen/include/asm-arm/domain.h          |  11 +-
 xen/include/asm-arm/new_vgic.h        |  10 +
 xen/include/asm-arm/vgic.h            |  11 +
 xen/include/asm-arm/vpl011.h          |   2 +
 xen/include/public/domctl.h           |   4 +
 xen/include/xen/domain.h              |   3 +
 16 files changed, 471 insertions(+), 101 deletions(-)
 create mode 100644 docs/misc/arm/passthrough-noiommu.txt

-- 
2.25.1


Re: [PATCH v3 00/10] direct-map memory map

Posted by Stefano Stabellini 2 weeks, 4 days ago
On Tue, 16 Nov 2021, Penny Zheng wrote:
> Cases where domU needs direct-map memory map:
>   * IOMMU not present in the system.
>   * IOMMU disabled if it doesn't cover a specific device and all the guests
> are trusted. Thinking a mixed scenario, where a few devices with IOMMU and
> a few without, then guest DMA security still could not be totally guaranteed.
> So users may want to disable the IOMMU, to at least gain some performance
> improvement from IOMMU disabled.
>   * IOMMU disabled as a workaround when it doesn't have enough bandwidth.
> To be specific, in a few extreme situation, when multiple devices do DMA
> concurrently, these requests may exceed IOMMU's transmission capacity.
>   * IOMMU disabled when it adds too much latency on DMA. For example,
> TLB may be missing in some IOMMU hardware, which may bring latency in DMA
> progress, so users may want to disable it in some realtime scenario.
>   * Guest OS relies on the host memory layout
> 
> "direct-map" property shall be added under the appropriate domain node,
> when users requesting direct-map memory mapping for the domain.
> 
> Right now, direct-map is only supported when domain on Static Allocation,
> that is, "xen,static-mem" is also necessary in the domain configuration.
> 
> Looking into related [design link](
> https://lists.xenproject.org/archives/html/xen-devel/2021-05/msg00882.html)
> for more details.
> 
> The whole design is about Static Allocation and direct-map, and this
> Patch Serie only covers parts of it, which are direct-map memory map.
> Other features will be delievered through different patch series.
> 
> See https://lists.xenproject.org/archives/html/xen-devel/2021-09/msg00855.html
> for Domain on Static Allocation.
> 
> This patch serie is based on
> https://lists.xenproject.org/archives/html/xen-devel/2021-10/msg00822.html

I haven't had a chance to review the series but I wanted to say that I
tested it successfully both with and without direct-map, so:

Tested-by: Stefano Stabellini <sstabellini@kernel.org>

Re: [PATCH v3 00/10] direct-map memory map

Posted by Jan Beulich 2 weeks, 5 days ago
On 16.11.2021 07:31, Penny Zheng wrote:
> v3 changes:
> - move flag XEN_DOMCTL_CDF_INTERNAL_directmap back to xen/include/xen/domain.h,
> to let it be only available for domain created by XEN.
> - name it with extra "INTERNAL" and add comments to warn developers not
> to accidently use its bitfield when introducing new XEN_DOMCTL_CDF_xxx flag.

FTR I continue to object to the hijacking of a public interface bit for
this purpose. Not to the degree of nak-ing the change, but still.

Jan


Re: [PATCH v3 00/10] direct-map memory map

Posted by Julien Grall 2 weeks, 5 days ago
Hi Jan,

On 16/11/2021 07:49, Jan Beulich wrote:
> On 16.11.2021 07:31, Penny Zheng wrote:
>> v3 changes:
>> - move flag XEN_DOMCTL_CDF_INTERNAL_directmap back to xen/include/xen/domain.h,
>> to let it be only available for domain created by XEN.
>> - name it with extra "INTERNAL" and add comments to warn developers not
>> to accidently use its bitfield when introducing new XEN_DOMCTL_CDF_xxx flag.
> 
> FTR I continue to object to the hijacking of a public interface bit for
> this purpose. Not to the degree of nak-ing the change, but still.

I remember this discussion in v1 and I am a bit confused why this was 
re-introduced. Looking at the thread, I think you and I were happy with 
the following approach:

   1) Switch the last parameter of domain_create() (i.e. bool is_priv) 
to flags.
   2) Define a bit in the parameter to indicate whether the domain will 
be direct-mapped.

Cheers,

-- 
Julien Grall

Re: [PATCH v3 00/10] direct-map memory map

Posted by Jan Beulich 2 weeks, 5 days ago
On 16.11.2021 10:44, Julien Grall wrote:
> On 16/11/2021 07:49, Jan Beulich wrote:
>> On 16.11.2021 07:31, Penny Zheng wrote:
>>> v3 changes:
>>> - move flag XEN_DOMCTL_CDF_INTERNAL_directmap back to xen/include/xen/domain.h,
>>> to let it be only available for domain created by XEN.
>>> - name it with extra "INTERNAL" and add comments to warn developers not
>>> to accidently use its bitfield when introducing new XEN_DOMCTL_CDF_xxx flag.
>>
>> FTR I continue to object to the hijacking of a public interface bit for
>> this purpose. Not to the degree of nak-ing the change, but still.
> 
> I remember this discussion in v1 and I am a bit confused why this was 
> re-introduced. Looking at the thread, I think you and I were happy with 
> the following approach:
> 
>    1) Switch the last parameter of domain_create() (i.e. bool is_priv) 
> to flags.
>    2) Define a bit in the parameter to indicate whether the domain will 
> be direct-mapped.

Indeed, that's how I too would prefer this to be dealt with.

Jan