Shared caches in multi-core CPU architectures represent a problem for
predictability of memory access latency. This jeopardizes applicability
of many Arm platform in real-time critical and mixed-criticality
scenarios. We introduce support for cache partitioning with page
coloring, a transparent software technique that enables isolation
between domains and Xen, and thus avoids cache interference.

When creating a domain, a simple syntax (e.g. `0-3` or `4-11`) allows
the user to define assignments of cache partitions ids, called colors,
where assigning different colors guarantees no mutual eviction on cache
will ever happen. This instructs the Xen memory allocator to provide
the i-th color assignee only with pages that maps to color i, i.e. that
are indexed in the i-th cache partition.

The proposed implementation supports the dom0less feature.
The proposed implementation doesn't support the static-mem feature.
The solution has been tested in several scenarios, including Xilinx Zynq
MPSoCs.

Open points:
- Michal found some problem here
https://patchew.org/Xen/20230123154735.74832-1-carlo.nonato@minervasys.tech/20230123154735.74832-4-carlo.nonato@minervasys.tech/#a7a06a26-ae79-402c-96a4-a1ebfe8b5862@amd.com
  but I havent fully understood it. In the meantime I want to advance with v6,
  so I hope we can continue the discussion here.
- Default configuration. After a few tries with a domain default configuration
  that included all colors not already in use by Xen, I went back with having
  all available colors indiscriminately. The problem is that if Xen has all the
  colors then domains can't be created with the default configuration.
  In general we don't want to impose any rule on the user so, the original
  default is preferred since it allows users to try coloring without too much
  headache, even if this means that they are using a suboptimal configuration
  that includes shared colors.
- I don't know how to split #14 in smaller patches. I hope also not to have
  misunderstood any comment there.

Carlo Nonato (14):
  xen/common: add cache coloring common code
  xen/arm: add initial support for LLC coloring on arm64
  xen/arm: permit non direct-mapped Dom0 construction
  xen/arm: add Dom0 cache coloring support
  xen: extend domctl interface for cache coloring
  tools: add support for cache coloring configuration
  xen/arm: add support for cache coloring configuration via device-tree
  xen/page_alloc: introduce init_free_page_fields() helper
  xen/page_alloc: introduce preserved page flags macro
  xen: add cache coloring allocator for domains
  xen/arm: use domain memory to allocate p2m page tables
  xen/arm: make consider_modules() available for xen relocation
  xen/arm: add cache coloring support for Xen
  support: add cache coloring arm64 experimental feature

Luca Miccio (1):
  xen/arm: add Xen cache colors command line parameter

 SUPPORT.md                              |   7 +
 docs/man/xl.cfg.5.pod.in                |  10 +
 docs/misc/arm/device-tree/booting.txt   |   4 +
 docs/misc/cache-coloring.rst            | 210 ++++++++++++++
 docs/misc/xen-command-line.pandoc       |  60 ++++
 tools/include/libxl.h                   |   5 +
 tools/include/xenctrl.h                 |   9 +
 tools/libs/ctrl/xc_domain.c             |  34 +++
 tools/libs/light/libxl_create.c         |   9 +
 tools/libs/light/libxl_types.idl        |   1 +
 tools/xl/xl_parse.c                     |  38 ++-
 xen/arch/Kconfig                        |  25 ++
 xen/arch/arm/Kconfig                    |   1 +
 xen/arch/arm/Makefile                   |   1 +
 xen/arch/arm/alternative.c              |  30 +-
 xen/arch/arm/arm32/mmu/mm.c             |  93 +------
 xen/arch/arm/arm64/mmu/head.S           |  57 +++-
 xen/arch/arm/arm64/mmu/mm.c             |  28 +-
 xen/arch/arm/dom0less-build.c           |  60 ++--
 xen/arch/arm/domain_build.c             |  60 +++-
 xen/arch/arm/include/asm/domain_build.h |   1 +
 xen/arch/arm/include/asm/mm.h           |   5 +
 xen/arch/arm/include/asm/mmu/layout.h   |   3 +
 xen/arch/arm/include/asm/processor.h    |  16 ++
 xen/arch/arm/include/asm/setup.h        |   3 +
 xen/arch/arm/llc-coloring.c             | 134 +++++++++
 xen/arch/arm/mmu/p2m.c                  |   4 +-
 xen/arch/arm/mmu/setup.c                |  97 ++++++-
 xen/arch/arm/setup.c                    | 100 ++++++-
 xen/common/Kconfig                      |   3 +
 xen/common/Makefile                     |   1 +
 xen/common/domain.c                     |   3 +
 xen/common/domctl.c                     |  11 +
 xen/common/keyhandler.c                 |   3 +
 xen/common/llc-coloring.c               | 350 ++++++++++++++++++++++++
 xen/common/page_alloc.c                 | 214 ++++++++++++++-
 xen/include/public/domctl.h             |   9 +
 xen/include/xen/llc-coloring.h          |  62 +++++
 xen/include/xen/sched.h                 |   5 +
 39 files changed, 1593 insertions(+), 173 deletions(-)
 create mode 100644 docs/misc/cache-coloring.rst
 create mode 100644 xen/arch/arm/llc-coloring.c
 create mode 100644 xen/common/llc-coloring.c
 create mode 100644 xen/include/xen/llc-coloring.h

-- 
2.34.1

Hi Carlo,

On 29/01/2024 18:17, Carlo Nonato wrote:
> 
> 
> Shared caches in multi-core CPU architectures represent a problem for
> predictability of memory access latency. This jeopardizes applicability
> of many Arm platform in real-time critical and mixed-criticality
> scenarios. We introduce support for cache partitioning with page
> coloring, a transparent software technique that enables isolation
> between domains and Xen, and thus avoids cache interference.
> 
> When creating a domain, a simple syntax (e.g. `0-3` or `4-11`) allows
> the user to define assignments of cache partitions ids, called colors,
> where assigning different colors guarantees no mutual eviction on cache
> will ever happen. This instructs the Xen memory allocator to provide
> the i-th color assignee only with pages that maps to color i, i.e. that
> are indexed in the i-th cache partition.
> 
> The proposed implementation supports the dom0less feature.
> The proposed implementation doesn't support the static-mem feature.
> The solution has been tested in several scenarios, including Xilinx Zynq
> MPSoCs.
> 
> Open points:
> - Michal found some problem here
> https://patchew.org/Xen/20230123154735.74832-1-carlo.nonato@minervasys.tech/20230123154735.74832-4-carlo.nonato@minervasys.tech/#a7a06a26-ae79-402c-96a4-a1ebfe8b5862@amd.com
>   but I havent fully understood it. In the meantime I want to advance with v6,
>   so I hope we can continue the discussion here.
The problem is that when LLC coloring is enabled, you use allocate_memory() for hwdom, just like for any
other domain, so it will get assigned a VA range from a typical Xen guest memory map (i.e. GUEST_RAM{0,1}_{BASE,SIZE}).
This can result in memory conflicts given that the HW resources are mapped 1:1 to it (MMIO, reserved memory regions).
Instead, for hwdom we should use the host memory layout to prevent these conflicts. A good example is find_unallocated_memory().
You need to:
 - fetch available RAM,
 - remove reserved-memory regions,
 - report ranges (+aligning the base and skipping banks that are not reasonable big)
This will give you a list of memory regions you can then use to pass to allocate_bank_memory().
The problem, as always, is to determine the size of the first region so that is is sufficiently
large to keep kernel+dtb+initrd in relatively close proximity.

~Michal