Shared caches in multi-core CPU architectures represent a problem for predictability of memory access latency. This jeopardizes applicability of many Arm platform in real-time critical and mixed-criticality scenarios. We introduce support for cache partitioning with page coloring, a transparent software technique that enables isolation between domains and Xen, and thus avoids cache interference. When creating a domain, a simple syntax (e.g. `0-3` or `4-11`) allows the user to define assignments of cache partitions ids, called colors, where assigning different colors guarantees no mutual eviction on cache will ever happen. This instructs the Xen memory allocator to provide the i-th color assignee only with pages that maps to color i, i.e. that are indexed in the i-th cache partition. The proposed implementation supports the dom0less experimental feature. The solution has been tested in several scenarios, including Xilinx Zynq MPSoCs. Overview of implementation and commits structure ------------------------------------------------ - Coloring support is added for dom0 and domU by defining the core logic, as well as the hardware inspection functionalities used for getting needed coloring information [4-17]. - A new memory page allocator that implement the cache coloring mechanism is introduced. The allocation algorithm follows the given coloring scheme specified for each domain, and maximizes contiguity in the page selection [18-21]. - Coloring support is added to Xen .text region [22-29], as well as to dom0less domains [30]. - Extensive documentation details the technique and gently explains usage [33-36]. Known limitations ----------------- - We need to bring back [1-3] the relocation feature in order to move Xen memory to a colored space where the hypervisor could be isolated from VMs interference. - When cache coloring is used, static memory assignment is disabled to avoid incompatibility. [31] - Due to assert failure [32], the number of supported colors is currently limited at 64, which should be satisfactory for most chips. In particular, the problem lies in the cache coloring configuration data structure that belongs to each domain. We are aware that this is not a clean solution but we hope that this could be discussed and solved within this pull request. Acknowledgements ---------------- This work is sponsored by Xilinx Inc., and supported by University of Modena and Reggio Emilia and Minerva Systems. *** Luca Miccio (36): Revert "xen/arm: setup: Add Xen as boot module before printing all boot modules" Revert "xen/arm: mm: Initialize page-tables earlier" xen/arm: restore xen_paddr argument in setup_pagetables xen/arm: add parsing function for cache coloring configuration xen/arm: compute LLC way size by hardware inspection xen/arm: add coloring basic initialization xen/arm: add coloring data to domains xen/arm: add colored flag to page struct xen/arch: add default colors selection function xen/arch: check color selection function xen/include: define hypercall parameter for coloring xen/arm: initialize cache coloring data for Dom0/U xen/arm: A domain is not direct mapped when coloring is enabled xen/arch: add dump coloring info for domains tools: add support for cache coloring configuration xen/color alloc: implement color_from_page for ARM64 xen/arm: add get_max_color function Alloc: introduce page_list_for_each_reverse xen/arch: introduce cache-coloring allocator xen/common: introduce buddy required reservation xen/common: add colored allocator initialization xen/arch: init cache coloring conf for Xen xen/arch: coloring: manually calculate Xen physical addresses xen/arm: enable consider_modules for coloring xen/arm: bring back get_xen_paddr xen/arm: add argument to remove_early_mappings xen/arch: add coloring support for Xen xen/arm: introduce xen_map_text_rw xen/arm: add dump function for coloring info xen/arm: add coloring support to dom0less Disable coloring if static memory support is selected xen/arm: reduce the number of supported colors doc, xen-command-line: introduce coloring options doc, xl.cfg: introduce coloring configuration option doc, device-tree: introduce 'colors' property doc, arm: add usage documentation for cache coloring support docs/man/xl.cfg.5.pod.in | 14 + docs/misc/arm/cache_coloring.rst | 191 +++++++++++ docs/misc/arm/device-tree/booting.txt | 3 + docs/misc/xen-command-line.pandoc | 51 ++- tools/libs/light/libxl_arm.c | 11 + tools/libs/light/libxl_types.idl | 1 + tools/xl/xl_parse.c | 59 +++- xen/arch/arm/Kconfig | 6 + xen/arch/arm/Makefile | 2 +- xen/arch/arm/alternative.c | 8 +- xen/arch/arm/coloring.c | 469 ++++++++++++++++++++++++++ xen/arch/arm/domain.c | 56 +++ xen/arch/arm/domain_build.c | 42 ++- xen/arch/arm/include/asm/coloring.h | 98 ++++++ xen/arch/arm/include/asm/mm.h | 18 +- xen/arch/arm/mm.c | 245 +++++++++++++- xen/arch/arm/psci.c | 4 +- xen/arch/arm/setup.c | 94 +++++- xen/arch/arm/smpboot.c | 19 +- xen/common/page_alloc.c | 321 +++++++++++++++++- xen/common/vmap.c | 4 +- xen/include/public/arch-arm.h | 8 + xen/include/xen/mm.h | 7 + xen/include/xen/sched.h | 4 + xen/include/xen/vmap.h | 2 + 25 files changed, 1689 insertions(+), 48 deletions(-) create mode 100644 docs/misc/arm/cache_coloring.rst create mode 100644 xen/arch/arm/coloring.c create mode 100644 xen/arch/arm/include/asm/coloring.h -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> This reverts commit 48fb2a9deba11ee48dde21c5c1aa93b4d4e1043b. --- xen/arch/arm/setup.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/setup.c +++ b/xen/arch/arm/setup.c @@ -XXX,XX +XXX,XX @@ void __init start_xen(unsigned long boot_phys_offset, "Please check your bootloader.\n", fdt_paddr); - /* Register Xen's load address as a boot module. */ - xen_bootmodule = add_boot_module(BOOTMOD_XEN, - (paddr_t)(uintptr_t)(_start + boot_phys_offset), - (paddr_t)(uintptr_t)(_end - _start), false); - BUG_ON(!xen_bootmodule); - fdt_size = boot_fdt_info(device_tree_flattened, fdt_paddr); cmdline = boot_fdt_cmdline(device_tree_flattened); printk("Command line: %s\n", cmdline); cmdline_parse(cmdline); + /* Register Xen's load address as a boot module. */ + xen_bootmodule = add_boot_module(BOOTMOD_XEN, + (paddr_t)(uintptr_t)(_start + boot_phys_offset), + (paddr_t)(uintptr_t)(_end - _start + 1), false); + BUG_ON(!xen_bootmodule); + setup_mm(); /* Parse the ACPI tables for possible boot-time configuration */ -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> This reverts commit 3a5d341681af650825bbe3bee9be5d187da35080. The coloring support will be configurable within the Xen command line but it will be initialized before the page-tables; this is necessary for coloring the hypervisor itself beacuse we will create a specific mapping for it that could be configured using some options. In order to parse all the needed information from the device tree, we need to revert the above commit and restore the previous order for page-tables initialization. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> --- xen/arch/arm/mm.c | 11 +++++++++-- xen/arch/arm/setup.c | 4 ++-- 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/mm.c +++ b/xen/arch/arm/mm.c @@ -XXX,XX +XXX,XX @@ static inline lpae_t pte_of_xenaddr(vaddr_t va) return mfn_to_xen_entry(maddr_to_mfn(ma), MT_NORMAL); } +/* Map the FDT in the early boot page table */ void * __init early_fdt_map(paddr_t fdt_paddr) { /* We are using 2MB superpage for mapping the FDT */ @@ -XXX,XX +XXX,XX @@ void * __init early_fdt_map(paddr_t fdt_paddr) /* The FDT is mapped using 2MB superpage */ BUILD_BUG_ON(BOOT_FDT_VIRT_START % SZ_2M); - create_mappings(xen_second, BOOT_FDT_VIRT_START, paddr_to_pfn(base_paddr), + create_mappings(boot_second, BOOT_FDT_VIRT_START, paddr_to_pfn(base_paddr), SZ_2M >> PAGE_SHIFT, SZ_2M); offset = fdt_paddr % SECOND_SIZE; @@ -XXX,XX +XXX,XX @@ void * __init early_fdt_map(paddr_t fdt_paddr) if ( (offset + size) > SZ_2M ) { - create_mappings(xen_second, BOOT_FDT_VIRT_START + SZ_2M, + create_mappings(boot_second, BOOT_FDT_VIRT_START + SZ_2M, paddr_to_pfn(base_paddr + SZ_2M), SZ_2M >> PAGE_SHIFT, SZ_2M); } @@ -XXX,XX +XXX,XX @@ void __init setup_pagetables(unsigned long boot_phys_offset) pte.pt.table = 1; xen_second[second_table_offset(FIXMAP_ADDR(0))] = pte; + /* ... DTB */ + pte = boot_second[second_table_offset(BOOT_FDT_VIRT_START)]; + xen_second[second_table_offset(BOOT_FDT_VIRT_START)] = pte; + pte = boot_second[second_table_offset(BOOT_FDT_VIRT_START + SZ_2M)]; + xen_second[second_table_offset(BOOT_FDT_VIRT_START + SZ_2M)] = pte; + #ifdef CONFIG_ARM_64 ttbr = (uintptr_t) xen_pgtable + phys_offset; #else diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/setup.c +++ b/xen/arch/arm/setup.c @@ -XXX,XX +XXX,XX @@ void __init start_xen(unsigned long boot_phys_offset, /* Initialize traps early allow us to get backtrace when an error occurred */ init_traps(); - setup_pagetables(boot_phys_offset); - smp_clear_cpu_maps(); device_tree_flattened = early_fdt_map(fdt_paddr); @@ -XXX,XX +XXX,XX @@ void __init start_xen(unsigned long boot_phys_offset, (paddr_t)(uintptr_t)(_end - _start + 1), false); BUG_ON(!xen_bootmodule); + setup_pagetables(boot_phys_offset); + setup_mm(); /* Parse the ACPI tables for possible boot-time configuration */ -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Coloring support will re-enable part of the Xen relocation since the underlying idea is to "relocate using coloring" for the hypervisors itself. We setup a target region that will be used exclusively from Xen and it will be mapped using the coloring configuration of the hypervisor. Part of the relocation we need to bring back is the usage of xen_paddr variable that will tell us the physical start address where Xen is located. Add this variable to the setup_pagetables function and set it properly when coloring is not enabled. Later on it will be initialized accordingly whether the coloring support is enabled or not. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> --- xen/arch/arm/include/asm/mm.h | 2 +- xen/arch/arm/mm.c | 2 +- xen/arch/arm/setup.c | 6 +++--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/xen/arch/arm/include/asm/mm.h b/xen/arch/arm/include/asm/mm.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/mm.h +++ b/xen/arch/arm/include/asm/mm.h @@ -XXX,XX +XXX,XX @@ extern unsigned long total_pages; #define PDX_GROUP_SHIFT SECOND_SHIFT /* Boot-time pagetable setup */ -extern void setup_pagetables(unsigned long boot_phys_offset); +extern void setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr); /* Map FDT in boot pagetable */ extern void *early_fdt_map(paddr_t fdt_paddr); /* Remove early mappings */ diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/mm.c +++ b/xen/arch/arm/mm.c @@ -XXX,XX +XXX,XX @@ static void clear_table(void *table) /* Boot-time pagetable setup. * Changes here may need matching changes in head.S */ -void __init setup_pagetables(unsigned long boot_phys_offset) +void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr) { uint64_t ttbr; lpae_t pte, *p; diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/setup.c +++ b/xen/arch/arm/setup.c @@ -XXX,XX +XXX,XX @@ void __init start_xen(unsigned long boot_phys_offset, struct bootmodule *xen_bootmodule; struct domain *d; int rc; + paddr_t xen_paddr = (paddr_t)(_start + boot_phys_offset); dcache_line_bytes = read_dcache_line_bytes(); @@ -XXX,XX +XXX,XX @@ void __init start_xen(unsigned long boot_phys_offset, cmdline_parse(cmdline); /* Register Xen's load address as a boot module. */ - xen_bootmodule = add_boot_module(BOOTMOD_XEN, - (paddr_t)(uintptr_t)(_start + boot_phys_offset), + xen_bootmodule = add_boot_module(BOOTMOD_XEN, xen_paddr, (paddr_t)(uintptr_t)(_end - _start + 1), false); BUG_ON(!xen_bootmodule); - setup_pagetables(boot_phys_offset); + setup_pagetables(boot_phys_offset, xen_paddr); setup_mm(); -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Add three new bootargs allowing configuration of cache coloring support for Xen: - way_size: The size of a LLC way in bytes. This value is mainly used to calculate the maximum available colors on the platform. - dom0_colors: The coloring configuration for Dom0, which also acts as default configuration for any DomU without an explicit configuration. - xen_colors: The coloring configuration for the Xen hypervisor itself. A cache coloring configuration consists of a selection of colors to be assigned to a VM or to the hypervisor. It is represented by a set of ranges. Add a common function that parses a string with a comma-separated set of hyphen-separated ranges like "0-7,15-16" and returns both: the number of chosen colors, and an array containing their ids. Currently we support platforms with up to 128 colors. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com> --- xen/arch/arm/Kconfig | 5 ++ xen/arch/arm/Makefile | 2 +- xen/arch/arm/coloring.c | 131 ++++++++++++++++++++++++++++ xen/arch/arm/include/asm/coloring.h | 28 ++++++ 4 files changed, 165 insertions(+), 1 deletion(-) create mode 100644 xen/arch/arm/coloring.c create mode 100644 xen/arch/arm/include/asm/coloring.h diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/Kconfig +++ b/xen/arch/arm/Kconfig @@ -XXX,XX +XXX,XX @@ config HARDEN_BRANCH_PREDICTOR If unsure, say Y. +config COLORING + bool "L2 cache coloring" + default n + depends on ARM_64 + config TEE bool "Enable TEE mediators support (UNSUPPORTED)" if UNSUPPORTED default n diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/Makefile +++ b/xen/arch/arm/Makefile @@ -XXX,XX +XXX,XX @@ obj-$(CONFIG_SBSA_VUART_CONSOLE) += vpl011.o obj-y += vsmc.o obj-y += vpsci.o obj-y += vuart.o - +obj-$(CONFIG_COLORING) += coloring.o extra-y += xen.lds #obj-bin-y += ....o diff --git a/xen/arch/arm/coloring.c b/xen/arch/arm/coloring.c new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/xen/arch/arm/coloring.c @@ -XXX,XX +XXX,XX @@ +/* + * xen/arch/arm/coloring.c + * + * Coloring support for ARM + * + * Copyright (C) 2019 Xilinx Inc. + * + * Authors: + * Luca Miccio <lucmiccio@gmail.com> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see <http://www.gnu.org/licenses/>. + */ +#include <xen/init.h> +#include <xen/types.h> +#include <xen/lib.h> +#include <xen/errno.h> +#include <xen/param.h> +#include <asm/coloring.h> + +/* Number of color(s) assigned to Xen */ +static uint32_t xen_col_num; +/* Coloring configuration of Xen as bitmask */ +static uint32_t xen_col_mask[MAX_COLORS_CELLS]; + +/* Number of color(s) assigned to Dom0 */ +static uint32_t dom0_col_num; +/* Coloring configuration of Dom0 as bitmask */ +static uint32_t dom0_col_mask[MAX_COLORS_CELLS]; + +static uint64_t way_size; + +/************************* + * PARSING COLORING BOOTARGS + */ + +/* + * Parse the coloring configuration given in the buf string, following the + * syntax below, and store the number of colors and a corresponding mask in + * the last two given pointers. + * + * COLOR_CONFIGURATION ::= RANGE,...,RANGE + * RANGE ::= COLOR-COLOR + * + * Example: "2-6,15-16" represents the set of colors: 2,3,4,5,6,15,16. + */ +static int parse_color_config( + const char *buf, uint32_t *col_mask, uint32_t *col_num) +{ + int start, end, i; + const char* s = buf; + unsigned int offset; + + if ( !col_mask || !col_num ) + return -EINVAL; + + *col_num = 0; + for ( i = 0; i < MAX_COLORS_CELLS; i++ ) + col_mask[i] = 0; + + while ( *s != '\0' ) + { + if ( *s != ',' ) + { + start = simple_strtoul(s, &s, 0); + + /* Ranges are hyphen-separated */ + if ( *s != '-' ) + goto fail; + s++; + + end = simple_strtoul(s, &s, 0); + + for ( i = start; i <= end; i++ ) + { + offset = i / 32; + if ( offset > MAX_COLORS_CELLS ) + goto fail; + + if ( !(col_mask[offset] & (1 << i % 32)) ) + *col_num += 1; + col_mask[offset] |= (1 << i % 32); + } + } + else + s++; + } + + return *s ? -EINVAL : 0; +fail: + return -EINVAL; +} + +static int __init parse_way_size(const char *s) +{ + way_size = simple_strtoull(s, &s, 0); + + return *s ? -EINVAL : 0; +} +custom_param("way_size", parse_way_size); + +static int __init parse_dom0_colors(const char *s) +{ + return parse_color_config(s, dom0_col_mask, &dom0_col_num); +} +custom_param("dom0_colors", parse_dom0_colors); + +static int __init parse_xen_colors(const char *s) +{ + return parse_color_config(s, xen_col_mask, &xen_col_num); +} +custom_param("xen_colors", parse_xen_colors); + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/arch/arm/include/asm/coloring.h b/xen/arch/arm/include/asm/coloring.h new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/xen/arch/arm/include/asm/coloring.h @@ -XXX,XX +XXX,XX @@ +/* + * xen/arm/include/asm/coloring.h + * + * Coloring support for ARM + * + * Copyright (C) 2019 Xilinx Inc. + * + * Authors: + * Luca Miccio <lucmiccio@gmail.com> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see <http://www.gnu.org/licenses/>. + */ +#ifndef __ASM_ARM_COLORING_H__ +#define __ASM_ARM_COLORING_H__ + +#define MAX_COLORS_CELLS 4 + +#endif /* !__ASM_ARM_COLORING_H__ */ -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> The size of the LLC way is a crucial parameter for the cache coloring support, since it determines the maximum number of available colors on the platform. This parameter can currently be retrieved only from the way_size bootarg and it is prone to misconfiguration nullifying the coloring mechanism and breaking cache isolation. Add an alternative and more safe method to retrieve the way size by directly asking the hardware, namely using CCSIDR_EL1 and CSSELR_EL1 registers. This method has to check also if at least L2 is implemented in the hardware since there are scenarios where only L1 cache is availble, e.g, QEMU. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com> --- xen/arch/arm/coloring.c | 76 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/xen/arch/arm/coloring.c b/xen/arch/arm/coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/coloring.c +++ b/xen/arch/arm/coloring.c @@ -XXX,XX +XXX,XX @@ #include <xen/lib.h> #include <xen/errno.h> #include <xen/param.h> + +#include <asm/sysregs.h> #include <asm/coloring.h> +#include <asm/io.h> /* Number of color(s) assigned to Xen */ static uint32_t xen_col_num; @@ -XXX,XX +XXX,XX @@ static uint32_t dom0_col_mask[MAX_COLORS_CELLS]; static uint64_t way_size; +#define CTR_LINESIZE_MASK 0x7 +#define CTR_SIZE_SHIFT 13 +#define CTR_SIZE_MASK 0x3FFF +#define CTR_SELECT_L2 1 << 1 +#define CTR_SELECT_L3 1 << 2 +#define CTR_CTYPEn_MASK 0x7 +#define CTR_CTYPE2_SHIFT 3 +#define CTR_CTYPE3_SHIFT 6 +#define CTR_LLC_ON 1 << 2 +#define CTR_LOC_SHIFT 24 +#define CTR_LOC_MASK 0x7 +#define CTR_LOC_L2 1 << 1 +#define CTR_LOC_NOT_IMPLEMENTED 1 << 0 + + +/* Return the way size of last level cache by asking the hardware */ +static uint64_t get_llc_way_size(void) +{ + uint32_t cache_sel = READ_SYSREG64(CSSELR_EL1); + uint32_t cache_global_info = READ_SYSREG64(CLIDR_EL1); + uint32_t cache_info; + uint32_t cache_line_size; + uint32_t cache_set_num; + uint32_t cache_sel_tmp; + + printk(XENLOG_INFO "Get information on LLC\n"); + printk(XENLOG_INFO "Cache CLIDR_EL1: 0x%"PRIx32"\n", cache_global_info); + + /* Check if at least L2 is implemented */ + if ( ((cache_global_info >> CTR_LOC_SHIFT) & CTR_LOC_MASK) + == CTR_LOC_NOT_IMPLEMENTED ) + { + printk(XENLOG_ERR "ERROR: L2 Cache not implemented\n"); + return 0; + } + + /* Save old value of CSSELR_EL1 */ + cache_sel_tmp = cache_sel; + + /* Get LLC index */ + if ( ((cache_global_info >> CTR_CTYPE2_SHIFT) & CTR_CTYPEn_MASK) + == CTR_LLC_ON ) + cache_sel = CTR_SELECT_L2; + else + cache_sel = CTR_SELECT_L3; + + printk(XENLOG_INFO "LLC selection: %u\n", cache_sel); + /* Select the correct LLC in CSSELR_EL1 */ + WRITE_SYSREG64(cache_sel, CSSELR_EL1); + + /* Ensure write */ + isb(); + + /* Get info about the LLC */ + cache_info = READ_SYSREG64(CCSIDR_EL1); + + /* ARM TRM: (Log2(Number of bytes in cache line)) - 4. */ + cache_line_size = 1 << ((cache_info & CTR_LINESIZE_MASK) + 4); + /* ARM TRM: (Number of sets in cache) - 1 */ + cache_set_num = ((cache_info >> CTR_SIZE_SHIFT) & CTR_SIZE_MASK) + 1; + + printk(XENLOG_INFO "Cache line size: %u bytes\n", cache_line_size); + printk(XENLOG_INFO "Cache sets num: %u\n", cache_set_num); + + /* Restore value in CSSELR_EL1 */ + WRITE_SYSREG64(cache_sel_tmp, CSSELR_EL1); + + /* Ensure write */ + isb(); + + return (cache_line_size * cache_set_num); +} + /************************* * PARSING COLORING BOOTARGS */ -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Introduce a first and simple initialization function for the cache coloring support. A helper function computes 'addr_col_mask', the platform-dependent bitmask asserting the bits in memory addresses that can be used for the coloring mechanism. This, in turn is used to determine the total amount of available colors. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- xen/arch/arm/coloring.c | 83 +++++++++++++++++++++++++++++ xen/arch/arm/include/asm/coloring.h | 8 +++ xen/arch/arm/setup.c | 4 ++ 3 files changed, 95 insertions(+) diff --git a/xen/arch/arm/coloring.c b/xen/arch/arm/coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/coloring.c +++ b/xen/arch/arm/coloring.c @@ -XXX,XX +XXX,XX @@ static uint32_t xen_col_mask[MAX_COLORS_CELLS]; static uint32_t dom0_col_num; /* Coloring configuration of Dom0 as bitmask */ static uint32_t dom0_col_mask[MAX_COLORS_CELLS]; +/* Maximum number of available color(s) */ +static uint32_t max_col_num; +/* Maximum available coloring configuration as bitmask */ +static uint32_t max_col_mask[MAX_COLORS_CELLS]; static uint64_t way_size; +static uint64_t addr_col_mask; #define CTR_LINESIZE_MASK 0x7 #define CTR_SIZE_SHIFT 13 @@ -XXX,XX +XXX,XX @@ static uint64_t get_llc_way_size(void) return (cache_line_size * cache_set_num); } +/* + * Return the coloring mask based on the value of @param llc_way_size. + * This mask represents the bits in the address that can be used + * for defining available colors. + * + * @param llc_way_size Last level cache way size. + * @return unsigned long The coloring bitmask. + */ +static __init uint64_t calculate_addr_col_mask(uint64_t llc_way_size) +{ + uint64_t addr_col_mask = 0; + unsigned int i; + unsigned int low_idx, high_idx; + + low_idx = PAGE_SHIFT; + high_idx = get_count_order(llc_way_size) - 1; + + for ( i = low_idx; i <= high_idx; i++ ) + addr_col_mask |= (1 << i); + + return addr_col_mask; +} + +bool __init coloring_init(void) +{ + int i; + + printk(XENLOG_INFO "Initialize XEN coloring: \n"); + /* + * If the way size is not provided by the configuration, try to get + * this information from hardware. + */ + if ( !way_size ) + { + way_size = get_llc_way_size(); + + if ( !way_size ) + { + printk(XENLOG_ERR "ERROR: way size is null\n"); + return false; + } + } + + addr_col_mask = calculate_addr_col_mask(way_size); + if ( !addr_col_mask ) + { + printk(XENLOG_ERR "ERROR: addr_col_mask is null\n"); + return false; + } + + max_col_num = ((addr_col_mask >> PAGE_SHIFT) + 1); + + /* + * If the user or the platform itself provide a way_size + * configuration that corresponds to a number of max. + * colors greater than the one we support, we cannot + * continue. So the check on offset value is necessary. + */ + if ( max_col_num > 32 * MAX_COLORS_CELLS ) + { + printk(XENLOG_ERR "ERROR: max. color value not supported\n"); + return false; + } + + for ( i = 0; i < max_col_num; i++ ) + { + unsigned int offset = i / 32; + + max_col_mask[offset] |= (1 << i % 32); + } + + printk(XENLOG_INFO "Way size: 0x%"PRIx64"\n", way_size); + printk(XENLOG_INFO "Color bits in address: 0x%"PRIx64"\n", addr_col_mask); + printk(XENLOG_INFO "Max number of colors: %u\n", max_col_num); + + return true; +} + /************************* * PARSING COLORING BOOTARGS */ diff --git a/xen/arch/arm/include/asm/coloring.h b/xen/arch/arm/include/asm/coloring.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/coloring.h +++ b/xen/arch/arm/include/asm/coloring.h @@ -XXX,XX +XXX,XX @@ #define MAX_COLORS_CELLS 4 +#ifdef CONFIG_COLORING +bool __init coloring_init(void); +#else /* !CONFIG_COLORING */ +static inline bool __init coloring_init(void) +{ + return true; +} +#endif /* CONFIG_COLORING */ #endif /* !__ASM_ARM_COLORING_H__ */ diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/setup.c +++ b/xen/arch/arm/setup.c @@ -XXX,XX +XXX,XX @@ #include <asm/setup.h> #include <xsm/xsm.h> #include <asm/acpi.h> +#include <asm/coloring.h> struct bootinfo __initdata bootinfo; @@ -XXX,XX +XXX,XX @@ void __init start_xen(unsigned long boot_phys_offset, printk("Command line: %s\n", cmdline); cmdline_parse(cmdline); + if ( !coloring_init() ) + panic("Xen Coloring support: setup failed\n"); + /* Register Xen's load address as a boot module. */ xen_bootmodule = add_boot_module(BOOTMOD_XEN, xen_paddr, (paddr_t)(uintptr_t)(_end - _start + 1), false); -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> We want to be able to associate an assignment of cache colors to each domain. Add a configurable-length array containing a set of color indices in the domain data. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- xen/include/xen/sched.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index XXXXXXX..XXXXXXX 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -XXX,XX +XXX,XX @@ struct domain atomic_t shr_pages; /* shared pages */ atomic_t paged_pages; /* paged-out pages */ + /* Coloring. */ + uint32_t *colors; + uint32_t max_colors; + /* Scheduling. */ void *sched_priv; /* scheduler-specific data */ struct sched_unit *sched_unit_list; -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> A new allocator enforcing a cache-coloring configuration is going to be introduced. We thus need to distinguish the memory pages assigned to, and managed by, such colored allocator from the ordinary buddy allocator's ones. Add a color flag to the page structure. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- xen/arch/arm/include/asm/mm.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/xen/arch/arm/include/asm/mm.h b/xen/arch/arm/include/asm/mm.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/mm.h +++ b/xen/arch/arm/include/asm/mm.h @@ -XXX,XX +XXX,XX @@ struct page_info */ u32 tlbflush_timestamp; }; + + /* Is page managed by the cache-colored allocator? */ + bool colored; + u64 pad; }; -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> When cache coloring support is enabled, a color assignment is needed for every domain. Introduce a function computing a default configuration with a safe and common value -- the dom0 color selection. Do not access directly the array of color indices of dom0. Instead make use of the dom0 color configuration as a bitmask. Add a helper function that converts the color configuration bitmask into the indices array. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- xen/arch/arm/coloring.c | 36 +++++++++++++++++++++++++++++ xen/arch/arm/include/asm/coloring.h | 7 ++++++ 2 files changed, 43 insertions(+) diff --git a/xen/arch/arm/coloring.c b/xen/arch/arm/coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/coloring.c +++ b/xen/arch/arm/coloring.c @@ -XXX,XX +XXX,XX @@ static __init uint64_t calculate_addr_col_mask(uint64_t llc_way_size) return addr_col_mask; } +static int copy_mask_to_list( + uint32_t *col_mask, uint32_t *col_list, uint64_t col_num) +{ + unsigned int i, k, c; + + if ( !col_list ) + return -EINVAL; + + for ( i = 0, k = 0; i < MAX_COLORS_CELLS; i++ ) + for ( c = 0; k < col_num && c < 32; c++ ) + if ( col_mask[i] & (1 << (c + (i*32))) ) + col_list[k++] = c + (i * 32); + + return 0; +} + +uint32_t *setup_default_colors(uint32_t *col_num) +{ + uint32_t *col_list; + + if ( dom0_col_num ) + { + *col_num = dom0_col_num; + col_list = xzalloc_array(uint32_t, dom0_col_num); + if ( !col_list ) + { + printk(XENLOG_ERR "setup_default_colors: Alloc failed\n"); + return NULL; + } + copy_mask_to_list(dom0_col_mask, col_list, dom0_col_num); + return col_list; + } + + return NULL; +} + bool __init coloring_init(void) { int i; diff --git a/xen/arch/arm/include/asm/coloring.h b/xen/arch/arm/include/asm/coloring.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/coloring.h +++ b/xen/arch/arm/include/asm/coloring.h @@ -XXX,XX +XXX,XX @@ #ifdef CONFIG_COLORING bool __init coloring_init(void); + +/* + * Return an array with default colors selection and store the number of + * colors in @param col_num. The array selection will be equal to the dom0 + * color configuration. + */ +uint32_t *setup_default_colors(uint32_t *col_num); #else /* !CONFIG_COLORING */ static inline bool __init coloring_init(void) { -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Dom0 color configuration is parsed in the Xen command line. Add an helper function to check the user selection. If no configuration is provided by the user, all the available colors supported by the hardware will be assigned to dom0. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- xen/arch/arm/coloring.c | 17 +++++++++++++++++ xen/arch/arm/include/asm/coloring.h | 8 ++++++++ 2 files changed, 25 insertions(+) diff --git a/xen/arch/arm/coloring.c b/xen/arch/arm/coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/coloring.c +++ b/xen/arch/arm/coloring.c @@ -XXX,XX +XXX,XX @@ uint32_t *setup_default_colors(uint32_t *col_num) return NULL; } +bool check_domain_colors(struct domain *d) +{ + int i; + bool ret = false; + + if ( !d ) + return ret; + + if ( d->max_colors > max_col_num ) + return ret; + + for ( i = 0; i < d->max_colors; i++ ) + ret |= (d->colors[i] > (max_col_num - 1)); + + return !ret; +} + bool __init coloring_init(void) { int i; diff --git a/xen/arch/arm/include/asm/coloring.h b/xen/arch/arm/include/asm/coloring.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/coloring.h +++ b/xen/arch/arm/include/asm/coloring.h @@ -XXX,XX +XXX,XX @@ #define MAX_COLORS_CELLS 4 #ifdef CONFIG_COLORING +#include <xen/sched.h> + bool __init coloring_init(void); +/* + * Check colors of a given domain. + * Return true if check passed, false otherwise. + */ +bool check_domain_colors(struct domain *d); + /* * Return an array with default colors selection and store the number of * colors in @param col_num. The array selection will be equal to the dom0 -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> During domU creation process the colors selection has to be passed to the Xen hypercall. This is generally done using what Xen calls GUEST_HANDLE_PARAMS. In this case a simple bitmask for the coloring configuration suffices. Currently the maximum amount of supported colors is 128. Add a new parameter that allows us to pass both the colors bitmask and the number of elements in it. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com> --- xen/arch/arm/include/asm/coloring.h | 2 -- xen/include/public/arch-arm.h | 8 ++++++++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/xen/arch/arm/include/asm/coloring.h b/xen/arch/arm/include/asm/coloring.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/coloring.h +++ b/xen/arch/arm/include/asm/coloring.h @@ -XXX,XX +XXX,XX @@ #ifndef __ASM_ARM_COLORING_H__ #define __ASM_ARM_COLORING_H__ -#define MAX_COLORS_CELLS 4 - #ifdef CONFIG_COLORING #include <xen/sched.h> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h index XXXXXXX..XXXXXXX 100644 --- a/xen/include/public/arch-arm.h +++ b/xen/include/public/arch-arm.h @@ -XXX,XX +XXX,XX @@ struct vcpu_guest_context { typedef struct vcpu_guest_context vcpu_guest_context_t; DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t); +#define MAX_COLORS_CELLS 4 +struct color_guest_config { + uint32_t max_colors; + uint32_t colors[MAX_COLORS_CELLS]; +}; + /* * struct xen_arch_domainconfig's ABI is covered by * XEN_DOMCTL_INTERFACE_VERSION. @@ -XXX,XX +XXX,XX @@ struct xen_arch_domainconfig { * */ uint32_t clock_frequency; + /* IN */ + struct color_guest_config colors; }; #endif /* __XEN__ || __XEN_TOOLS__ */ -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Initialize cache coloring configuration during domain creation. If no colors assignment is provided by the user, use the default one. The default configuration is the one assigned to Dom0. The latter is configured as a standard domain with default configuration. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- xen/arch/arm/domain.c | 53 +++++++++++++++++++++++++++++++++++++ xen/arch/arm/domain_build.c | 5 +++- 2 files changed, 57 insertions(+), 1 deletion(-) diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/domain.c +++ b/xen/arch/arm/domain.c @@ -XXX,XX +XXX,XX @@ #include <asm/vfp.h> #include <asm/vgic.h> #include <asm/vtimer.h> +#include <asm/coloring.h> #include "vpci.h" #include "vuart.h" @@ -XXX,XX +XXX,XX @@ int arch_domain_create(struct domain *d, if ( (rc = domain_vpci_init(d)) != 0 ) goto fail; + d->max_colors = 0; +#ifdef CONFIG_COLORING + /* Setup domain colors */ + if ( !config->arch.colors.max_colors ) + { + if ( !is_hardware_domain(d) ) + printk(XENLOG_INFO "Color configuration not found for dom%u, using default\n", + d->domain_id); + d->colors = setup_default_colors(&d->max_colors); + if ( !d->colors ) + { + rc = -ENOMEM; + printk(XENLOG_ERR "Color array allocation failed for dom%u\n", + d->domain_id); + goto fail; + } + } + else + { + int i, k; + + d->colors = xzalloc_array(uint32_t, config->arch.colors.max_colors); + if ( !d->colors ) + { + rc = -ENOMEM; + printk(XENLOG_ERR "Failed to alloc colors for dom%u\n", + d->domain_id); + goto fail; + } + + d->max_colors = config->arch.colors.max_colors; + for ( i = 0, k = 0; + k < d->max_colors && i < sizeof(config->arch.colors.colors) * 8; + i++ ) + { + if ( config->arch.colors.colors[i / 32] & (1 << (i % 32)) ) + d->colors[k++] = i; + } + } + + printk("Dom%u colors: [ ", d->domain_id); + for ( int i = 0; i < d->max_colors; i++ ) + printk("%u ", d->colors[i]); + printk("]\n"); + + if ( !check_domain_colors(d) ) + { + rc = -EINVAL; + printk(XENLOG_ERR "Failed to check colors for dom%u\n", d->domain_id); + goto fail; + } +#endif return 0; fail: diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/domain_build.c +++ b/xen/arch/arm/domain_build.c @@ -XXX,XX +XXX,XX @@ void __init create_dom0(void) printk(XENLOG_WARNING "Maximum number of vGIC IRQs exceeded.\n"); dom0_cfg.arch.tee_type = tee_get_type(); dom0_cfg.max_vcpus = dom0_max_vcpus(); - +#ifdef CONFIG_COLORING + /* Colors are set after domain_create */ + dom0_cfg.arch.colors.max_colors = 0; +#endif if ( iommu_enabled ) dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu; -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Based on the intrinsic nature of cache coloring, it is trivial to state that each domain that is colored is also not direct mapped. Set the directmap variable to false when coloring is enabled. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- xen/arch/arm/domain.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/domain.c +++ b/xen/arch/arm/domain.c @@ -XXX,XX +XXX,XX @@ int arch_domain_create(struct domain *d, d->max_colors = 0; #ifdef CONFIG_COLORING + d->arch.directmap = false; + /* Setup domain colors */ if ( !config->arch.colors.max_colors ) { -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Print the color assignment for each domain when requested. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- xen/arch/arm/coloring.c | 12 ++++++++++++ xen/arch/arm/domain.c | 1 + xen/arch/arm/include/asm/coloring.h | 7 +++++++ 3 files changed, 20 insertions(+) diff --git a/xen/arch/arm/coloring.c b/xen/arch/arm/coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/coloring.c +++ b/xen/arch/arm/coloring.c @@ -XXX,XX +XXX,XX @@ static int __init parse_xen_colors(const char *s) } custom_param("xen_colors", parse_xen_colors); +void coloring_dump_info(struct domain *d) +{ + int i; + + printk("Domain %d has %u color(s) [ ", d->domain_id, d->max_colors); + for ( i = 0; i < d->max_colors; i++ ) + { + printk("%"PRIu32" ", d->colors[i]); + } + printk("]\n"); +} + /* * Local variables: * mode: C diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/domain.c +++ b/xen/arch/arm/domain.c @@ -XXX,XX +XXX,XX @@ int domain_relinquish_resources(struct domain *d) void arch_dump_domain_info(struct domain *d) { p2m_dump_info(d); + coloring_dump_info(d); } diff --git a/xen/arch/arm/include/asm/coloring.h b/xen/arch/arm/include/asm/coloring.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/coloring.h +++ b/xen/arch/arm/include/asm/coloring.h @@ -XXX,XX +XXX,XX @@ bool check_domain_colors(struct domain *d); * color configuration. */ uint32_t *setup_default_colors(uint32_t *col_num); + +void coloring_dump_info(struct domain *d); #else /* !CONFIG_COLORING */ static inline bool __init coloring_init(void) { return true; } + +static inline void coloring_dump_info(struct domain *d) +{ + return; +} #endif /* CONFIG_COLORING */ #endif /* !__ASM_ARM_COLORING_H__ */ -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Add a new "colors" parameter that defines the color assignment for a domain. The user can specify one or more color ranges using the same syntax as the command line color selection (e.g. 0-4). The parameter is defined as a list of strings that represent the color ranges. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com> --- tools/libs/light/libxl_arm.c | 11 ++++++ tools/libs/light/libxl_types.idl | 1 + tools/xl/xl_parse.c | 59 ++++++++++++++++++++++++++++++-- 3 files changed, 69 insertions(+), 2 deletions(-) diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c index XXXXXXX..XXXXXXX 100644 --- a/tools/libs/light/libxl_arm.c +++ b/tools/libs/light/libxl_arm.c @@ -XXX,XX +XXX,XX @@ int libxl__arch_domain_prepare_config(libxl__gc *gc, return ERROR_FAIL; } + config->arch.colors.max_colors = d_config->b_info.num_colors; + for (i = 0; i < sizeof(config->arch.colors.colors) / 4; i++) + config->arch.colors.colors[i] = 0; + for (i = 0; i < d_config->b_info.num_colors; i++) { + unsigned int j = d_config->b_info.colors[i] / 32; + if (j > sizeof(config->arch.colors.colors) / 4) + return ERROR_FAIL; + config->arch.colors.colors[j] |= (1 << (d_config->b_info.colors[i] % 32)); + } + LOG(DEBUG, "Setup domain colors"); + return 0; } diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl index XXXXXXX..XXXXXXX 100644 --- a/tools/libs/light/libxl_types.idl +++ b/tools/libs/light/libxl_types.idl @@ -XXX,XX +XXX,XX @@ libxl_domain_build_info = Struct("domain_build_info",[ ("ioports", Array(libxl_ioport_range, "num_ioports")), ("irqs", Array(uint32, "num_irqs")), ("iomem", Array(libxl_iomem_range, "num_iomem")), + ("colors", Array(uint32, "num_colors")), ("claim_mode", libxl_defbool), ("event_channels", uint32), ("kernel", string), diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c index XXXXXXX..XXXXXXX 100644 --- a/tools/xl/xl_parse.c +++ b/tools/xl/xl_parse.c @@ -XXX,XX +XXX,XX @@ void parse_config_data(const char *config_source, XLU_ConfigList *cpus, *vbds, *nics, *pcis, *cvfbs, *cpuids, *vtpms, *usbctrls, *usbdevs, *p9devs, *vdispls, *pvcallsifs_devs; XLU_ConfigList *channels, *ioports, *irqs, *iomem, *viridian, *dtdevs, - *mca_caps; - int num_ioports, num_irqs, num_iomem, num_cpus, num_viridian, num_mca_caps; + *mca_caps, *colors; + int num_ioports, num_irqs, num_iomem, num_cpus, num_viridian, num_mca_caps, + num_colors; int pci_power_mgmt = 0; int pci_msitranslate = 0; int pci_permissive = 0; @@ -XXX,XX +XXX,XX @@ void parse_config_data(const char *config_source, if (!xlu_cfg_get_long (config, "maxmem", &l, 0)) b_info->max_memkb = l * 1024; + if (!xlu_cfg_get_list(config, "colors", &colors, &num_colors, 0)) { + int ret, k, p, cur_index; + + b_info->num_colors = 0; + /* Get number of colors based on ranges */ + for (i = 0; i < num_colors; i++) { + uint32_t start, end; + + buf = xlu_cfg_get_listitem (colors, i); + if (!buf) { + fprintf(stderr, + "xl: Unable to get element %d in colors range list\n", i); + exit(1); + } + + ret = sscanf(buf, "%u-%u", &start, &end); + if (ret < 2) { + fprintf(stderr, + "xl: Invalid argument parsing colors range: %s\n", buf); + exit(1); + } + + if (start > end) { + fprintf(stderr, + "xl: invalid range: S:%u > E:%u \n", start,end); + exit(1); + } + + /* + * Alloc a first array and then increase its size with realloc based + * on the number of ranges + */ + + /* Check for overlaps */ + for (k = start; k <= end; k++) { + for (p = 0; p < b_info->num_colors; p++) + if(b_info->colors[p] == k) { + fprintf(stderr, + "xl: overlapped ranges not allowed\n"); + exit(1); + } + } + + cur_index = b_info->num_colors; + b_info->num_colors += (end - start) + 1; + b_info->colors = (uint32_t *)realloc(b_info->colors, + sizeof(*b_info->colors) * b_info->num_colors); + + for (k = start; cur_index < b_info->num_colors; + cur_index++, k++) + b_info->colors[cur_index] = k; + } + } + if (!xlu_cfg_get_long (config, "vcpus", &l, 0)) { vcpus = l; if (libxl_cpu_bitmap_alloc(ctx, &b_info->avail_vcpus, l)) { -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> The colored allocator should not make any assumptions on how a color is defined, since the definition may change depending on the architecture. Use a generic function "color_from_page" that returns the color id based on the page address. Add a definition for ARMv8 architectures. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- xen/arch/arm/coloring.c | 13 +++++++++++++ xen/arch/arm/include/asm/coloring.h | 7 +++++++ 2 files changed, 20 insertions(+) diff --git a/xen/arch/arm/coloring.c b/xen/arch/arm/coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/coloring.c +++ b/xen/arch/arm/coloring.c @@ -XXX,XX +XXX,XX @@ bool check_domain_colors(struct domain *d) return !ret; } +/* + * Compute color id from the page @param pg. + * Page size determines the lowest available bit, while add_col_mask is used to + * select the rest. + * + * @param pg Page address + * @return unsigned long Color id + */ +unsigned long color_from_page(struct page_info *pg) +{ + return ((addr_col_mask & page_to_maddr(pg)) >> PAGE_SHIFT); +} + bool __init coloring_init(void) { int i; diff --git a/xen/arch/arm/include/asm/coloring.h b/xen/arch/arm/include/asm/coloring.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/coloring.h +++ b/xen/arch/arm/include/asm/coloring.h @@ -XXX,XX +XXX,XX @@ bool check_domain_colors(struct domain *d); uint32_t *setup_default_colors(uint32_t *col_num); void coloring_dump_info(struct domain *d); + +/* + * Compute the color of the given page address. + * This function should change depending on the cache architecture + * specifications. + */ +unsigned long color_from_page(struct page_info *pg); #else /* !CONFIG_COLORING */ static inline bool __init coloring_init(void) { -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> In order to initialize the colored allocator data structure, the maximum amount of colors defined by the hardware has to be know. Add a helper function that returns this information. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> --- xen/arch/arm/coloring.c | 5 +++++ xen/arch/arm/include/asm/coloring.h | 8 ++++++++ 2 files changed, 13 insertions(+) diff --git a/xen/arch/arm/coloring.c b/xen/arch/arm/coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/coloring.c +++ b/xen/arch/arm/coloring.c @@ -XXX,XX +XXX,XX @@ unsigned long color_from_page(struct page_info *pg) return ((addr_col_mask & page_to_maddr(pg)) >> PAGE_SHIFT); } +uint32_t get_max_colors(void) +{ + return max_col_num; +} + bool __init coloring_init(void) { int i; diff --git a/xen/arch/arm/include/asm/coloring.h b/xen/arch/arm/include/asm/coloring.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/coloring.h +++ b/xen/arch/arm/include/asm/coloring.h @@ -XXX,XX +XXX,XX @@ void coloring_dump_info(struct domain *d); * specifications. */ unsigned long color_from_page(struct page_info *pg); + +/* Return the maximum available number of colors supported by the hardware */ +uint32_t get_max_colors(void); #else /* !CONFIG_COLORING */ static inline bool __init coloring_init(void) { @@ -XXX,XX +XXX,XX @@ static inline void coloring_dump_info(struct domain *d) { return; } + +static inline uint32_t get_max_colors(void) +{ + return 0; +} #endif /* CONFIG_COLORING */ #endif /* !__ASM_ARM_COLORING_H__ */ -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Luca Miccio <lucmiccio@gmail.com> --- xen/include/xen/mm.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h index XXXXXXX..XXXXXXX 100644 --- a/xen/include/xen/mm.h +++ b/xen/include/xen/mm.h @@ -XXX,XX +XXX,XX @@ page_list_splice(struct page_list_head *list, struct page_list_head *head) list_for_each_entry_safe(pos, tmp, head, list) # define page_list_for_each_safe_reverse(pos, tmp, head) \ list_for_each_entry_safe_reverse(pos, tmp, head, list) +# define page_list_for_each_reverse(pos, head) \ + list_for_each_entry_reverse(pos, head, list) #endif static inline unsigned int get_order_from_bytes(paddr_t size) -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Introduce a new memory page allocator that implement the cache coloring mechanism. The allocation algorithm follows the given coloring scheme specified for each guest, and maximizes contiguity in the page selection. Pages are stored by color in separated and address-ordered lists that are collectively called the colored heap. These lists will be populated by a simple initialisation function, which, for any available page, compute its color and insert it in the corresponding list. When a domain requests a page, the allocator take one from the subset of lists whose colors equal the domain configuration. It chooses the highest page element among the lasts elements of such lists. This ordering guarantees that contiguous pages are sequentially allocated, if this is made possible by a color assignment which includes adjacent ids. The allocator can handle only requests with order equals to 0 since the single color granularity is represented in memory by one page. A dump function is added to allow inspection of colored heap information. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- xen/common/page_alloc.c | 264 +++++++++++++++++++++++++++++++++++++++- xen/include/xen/mm.h | 5 + 2 files changed, 268 insertions(+), 1 deletion(-) diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index XXXXXXX..XXXXXXX 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -XXX,XX +XXX,XX @@ #define p2m_pod_offline_or_broken_hit(pg) 0 #define p2m_pod_offline_or_broken_replace(pg) BUG_ON(pg != NULL) #endif +#ifdef CONFIG_COLORING +#include <asm/coloring.h> +#endif #ifndef PGC_reserved #define PGC_reserved 0 @@ -XXX,XX +XXX,XX @@ mfn_t __init alloc_boot_pages(unsigned long nr_pfns, unsigned long pfn_align) +static DEFINE_SPINLOCK(heap_lock); + +#ifdef CONFIG_COLORING +/************************* + * COLORED SIDE-ALLOCATOR + * + * Pages are stored by their color in separated lists. Each list defines a color + * and it is initialized during end_boot_allocator, where each page's color + * is calculated and the page itself is put in the correct list. + * After initialization we have N list where N is the number of maximum + * available colors on the platform. + * All the lists' heads are stored as element in an array with size N-1 using + * the following schema: + * array[X] = head of color X, where X goes from 0 to N-1 + */ +typedef struct page_list_head color_list; +static color_list *color_heap; +static long total_avail_col_pages; +static u64 col_num_max; +static bool color_init_state = true; + +#define page_to_head(pg) (&color_heap[color_from_page(pg)]) +#define color_to_head(col) (&color_heap[col]) + +/* Add page in list in order depending on its physical address. */ +static void page_list_add_order(struct page_info *pg, struct list_head *head) +{ + struct page_info *pos; + + /* Add first page after head */ + if ( page_list_empty(head) ) + { + page_list_add(pg, head); + return; + } + + /* Add non-first page in list in ascending order */ + page_list_for_each_reverse(pos, head) + { + /* Get pg position */ + if ( page_to_maddr(pos) <= page_to_maddr(pg) ) + { + /* Insert pg between pos and pos->list.next */ + page_list_add(pg, &pos->list); + break; + } + + /* + * If pos is the first element it means that pg <= pos so we have + * to insert pg after head. + */ + if ( page_list_first(head) == pos ) + { + page_list_add(pg, head); + break; + } + } +} + +/* Alloc one page based on domain color configuration */ +static struct page_info *alloc_col_heap_page( + unsigned int memflags, struct domain *d) +{ + struct page_info *pg, *tmp; + bool need_tlbflush = false; + uint32_t cur_color; + uint32_t tlbflush_timestamp = 0; + uint32_t *colors = 0; + int max_colors; + int i; + + colors = d->colors; + max_colors = d->max_colors; + + spin_lock(&heap_lock); + + tmp = pg = NULL; + + /* Check for the first pg on non-empty list */ + for ( i = 0; i < max_colors; i++ ) + { + if ( !page_list_empty(color_to_head(colors[i])) ) + { + tmp = pg = page_list_last(color_to_head(colors[i])); + cur_color = d->colors[i]; + break; + } + } + + /* If all lists are empty, no requests can be satisfied */ + if ( !pg ) + { + spin_unlock(&heap_lock); + return NULL; + } + + /* Get the highest page from the lists compliant to the domain color(s) */ + for ( i += 1; i < max_colors; i++ ) + { + if ( page_list_empty(color_to_head(colors[i])) ) + { + printk(XENLOG_INFO "List empty\n"); + continue; + } + tmp = page_list_last(color_to_head(colors[i])); + if ( page_to_maddr(tmp) > page_to_maddr(pg) ) + { + pg = tmp; + cur_color = colors[i]; + } + } + + if ( !pg ) + { + spin_unlock(&heap_lock); + return NULL; + } + + pg->count_info = PGC_state_inuse; + + if ( !(memflags & MEMF_no_tlbflush) ) + accumulate_tlbflush(&need_tlbflush, pg, + &tlbflush_timestamp); + + /* Initialise fields which have other uses for free pages. */ + pg->u.inuse.type_info = 0; + page_set_owner(pg, NULL); + + flush_page_to_ram(mfn_x(page_to_mfn(pg)), + !(memflags & MEMF_no_icache_flush)); + + page_list_del(pg, page_to_head(pg)); + total_avail_col_pages--; + + spin_unlock(&heap_lock); + + if ( need_tlbflush ) + filtered_flush_tlb_mask(tlbflush_timestamp); + + return pg; +} + +struct page_info *alloc_col_domheap_page( + struct domain *d, unsigned int memflags) +{ + struct page_info *pg; + + ASSERT(!in_irq()); + + /* Get page based on color selection */ + pg = alloc_col_heap_page(memflags, d); + + if ( !pg ) + { + printk(XENLOG_INFO "ERROR: Colored Page is null\n"); + return NULL; + } + + /* Assign page to domain */ + if ( d && !(memflags & MEMF_no_owner) && + assign_page(pg, 0, d, memflags) ) + { + free_col_heap_page(pg); + return NULL; + } + + return pg; +} + +void free_col_heap_page(struct page_info *pg) +{ + /* This page is not a guest frame any more. */ + pg->count_info = PGC_state_free; + + page_set_owner(pg, NULL); + total_avail_col_pages++; + page_list_add_order( pg, page_to_head(pg) ); +} + +static inline void init_col_heap_pages(struct page_info *pg, unsigned long nr_pages) +{ + int i; + + if ( color_init_state ) + { + col_num_max = get_max_colors(); + color_heap = xmalloc_array(color_list, col_num_max); + BUG_ON(!color_heap); + + for ( i = 0; i < col_num_max; i++ ) + { + printk(XENLOG_INFO "Init list for color: %u\n", i); + INIT_PAGE_LIST_HEAD(&color_heap[i]); + } + + color_init_state = false; + } + + printk(XENLOG_INFO "Init color heap pages with %lu pages for a given size of 0x%"PRIx64"\n", + nr_pages, nr_pages * PAGE_SIZE); + printk(XENLOG_INFO "Paging starting from: 0x%"PRIx64"\n", page_to_maddr(pg)); + total_avail_col_pages += nr_pages; + + for ( i = 0; i < nr_pages; i++ ) + { + pg->colored = true; + page_list_add_order(pg, page_to_head(pg)); + pg++; + } +} + +static inline bool is_page_colored(struct page_info *pg) +{ + return pg->colored; +} + +static void dump_col_heap(unsigned char key) +{ + struct page_info *pg; + unsigned long size; + unsigned int i; + + printk("Colored heap info\n"); + for ( i = 0; i < col_num_max; i++ ) + { + printk("Heap[%u]: ", i); + size = 0; + page_list_for_each( pg, color_to_head(i) ) + { + BUG_ON(!(color_from_page(pg) == i)); + size++; + } + printk("%lu pages -> %lukB free\n", size, size << (PAGE_SHIFT - 10)); + } + + printk("Total number of pages: %lu\n", total_avail_col_pages); +} +#else /* !CONFIG_COLORING */ +#define init_col_heap_pages(x, y) init_heap_pages(x, y) + +inline struct page_info *alloc_col_domheap_page( + struct domain *d, unsigned int memflags) +{ + return NULL; +} + +inline void free_col_heap_page(struct page_info *pg) +{ + return; +} + +static inline bool is_page_colored(struct page_info *pg) +{ + return false; +} +#endif /* CONFIG_COLORING */ + /************************* * BINARY BUDDY ALLOCATOR */ @@ -XXX,XX +XXX,XX @@ static unsigned long node_need_scrub[MAX_NUMNODES]; static unsigned long *avail[MAX_NUMNODES]; static long total_avail_pages; -static DEFINE_SPINLOCK(heap_lock); static long outstanding_claims; /* total outstanding claims by all domains */ unsigned long domain_adjust_tot_pages(struct domain *d, long pages) @@ -XXX,XX +XXX,XX @@ static void cf_check dump_heap(unsigned char key) static __init int cf_check register_heap_trigger(void) { register_keyhandler('H', dump_heap, "dump heap info", 1); +#ifdef CONFIG_COLORING + register_keyhandler('c', dump_col_heap, "dump coloring heap info", 1); +#endif return 0; } __initcall(register_heap_trigger); diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h index XXXXXXX..XXXXXXX 100644 --- a/xen/include/xen/mm.h +++ b/xen/include/xen/mm.h @@ -XXX,XX +XXX,XX @@ unsigned int online_page(mfn_t mfn, uint32_t *status); int offline_page(mfn_t mfn, int broken, uint32_t *status); int query_page_offline(mfn_t mfn, uint32_t *status); +/* Colored suballocator. */ +struct page_info *alloc_col_domheap_page( + struct domain *d, unsigned int memflags); +void free_col_heap_page(struct page_info *pg); + void heap_init_late(void); int assign_pages( -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> When cache coloring is enabled, a certain amount of memory is reserved for buddy allocation because current coloring implementation does not support Xen heap memory. As of this commit, the colored allocator is used for dom0, domUs, while the buddy manages only Xen memory. The memory reserved to the buddy is thus lowered to a reasonably small value. Introduce a new variable that specifies the amount of memory reserved for the buddy allocator. The current default value will be enough even when we will add coloring for Xen in the following patches. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- xen/common/page_alloc.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index XXXXXXX..XXXXXXX 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -XXX,XX +XXX,XX @@ static bool __read_mostly scrub_debug; #define scrub_debug false #endif +#ifdef CONFIG_COLORING +/* Minimum size required for buddy allocator to work with colored one */ +unsigned long buddy_required_size __read_mostly = MB(64); +#else +unsigned long buddy_required_size __read_mostly = 0; +#endif + /* * Bit width of the DMA heap -- used to override NUMA-node-first. * allocation strategy, which can otherwise exhaust low memory. @@ -XXX,XX +XXX,XX @@ static void dump_col_heap(unsigned char key) printk("Total number of pages: %lu\n", total_avail_col_pages); } +static int __init parse_buddy_required_size(const char *s) +{ + buddy_required_size = simple_strtoull(s, &s, 0); + + return *s ? -EINVAL : 0; +} +custom_param("buddy_size", parse_buddy_required_size); #else /* !CONFIG_COLORING */ #define init_col_heap_pages(x, y) init_heap_pages(x, y) -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Initialize colored heap and allocator data structures. It is assumed that pages are given to the init function is in ascending order. To ensure that, pages are retrieved from bootmem_regions starting from the first one. Moreover, this allows quickly insertion of freed pages into the colored allocator's internal data structures -- sorted lists. If coloring is disabled, changing the free page order should not affect both performance and functionalities of the buddy allocator. Do not allocate Dom0 memory with direct mapping if colored is enabled. Signed-off-by: Luca Miccio <206497@studenti.unimore.it> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com> --- xen/arch/arm/domain_build.c | 7 +++++- xen/common/page_alloc.c | 43 +++++++++++++++++++++++++++++++------ 2 files changed, 42 insertions(+), 8 deletions(-) diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/domain_build.c +++ b/xen/arch/arm/domain_build.c @@ -XXX,XX +XXX,XX @@ static int __init construct_dom0(struct domain *d) /* type must be set before allocate_memory */ d->arch.type = kinfo.type; #endif - allocate_memory_11(d, &kinfo); +#ifdef CONFIG_COLORING + if ( d->max_colors ) + allocate_memory(d, &kinfo); + else +#endif + allocate_memory_11(d, &kinfo); find_gnttab_region(d, &kinfo); /* Map extra GIC MMIO, irqs and other hw stuffs to dom0. */ diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index XXXXXXX..XXXXXXX 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -XXX,XX +XXX,XX @@ void __init end_boot_allocator(void) break; } } - for ( i = nr_bootmem_regions; i-- > 0; ) + + for ( i = 0; i < nr_bootmem_regions; i++ ) { struct bootmem_region *r = &bootmem_region_list[i]; - if ( r->s < r->e ) - init_heap_pages(mfn_to_page(_mfn(r->s)), r->e - r->s); + + /* + * Find the first region that can fill the buddy allocator memory + * specified by buddy_required_size. + */ + if ( buddy_required_size && (r->e - r->s) > + PFN_DOWN(buddy_required_size) ) + { + init_heap_pages(mfn_to_page(_mfn(r->s)), + PFN_DOWN(buddy_required_size)); + + r->s += PFN_DOWN(buddy_required_size); + buddy_required_size = 0; + } + + init_col_heap_pages(mfn_to_page(_mfn(r->s)), r->e - r->s); } nr_bootmem_regions = 0; @@ -XXX,XX +XXX,XX @@ int assign_pages( page_set_owner(&pg[i], d); smp_wmb(); /* Domain pointer must be visible before updating refcnt. */ pg[i].count_info = - (pg[i].count_info & (PGC_extra | PGC_reserved)) | PGC_allocated | 1; + (pg[i].count_info & (PGC_extra | PGC_reserved)) | PGC_allocated | 1; - page_list_add_tail(&pg[i], page_to_list(d, &pg[i])); + if ( is_page_colored(pg) ) + page_list_add(&pg[i], page_to_list(d, &pg[i])); + else + page_list_add_tail(&pg[i], page_to_list(d, &pg[i])); } out: @@ -XXX,XX +XXX,XX @@ struct page_info *alloc_domheap_pages( unsigned int bits = memflags >> _MEMF_bits, zone_hi = NR_ZONES - 1; unsigned int dma_zone; + /* Only Dom0 and DomUs are supported for coloring */ + if ( d && d->max_colors > 0 ) + { + /* Colored allocation must be done on 0 order */ + if (order) + return NULL; + + return alloc_col_domheap_page(d, memflags); + } ASSERT(!in_irq()); bits = domain_clamp_alloc_bitsize(memflags & MEMF_no_owner ? NULL : d, @@ -XXX,XX +XXX,XX @@ void free_domheap_pages(struct page_info *pg, unsigned int order) scrub = 1; } - free_heap_pages(pg, order, scrub); - } + if ( is_page_colored(pg) ) + free_col_heap_page(pg); + else + free_heap_pages(pg, order, scrub);} if ( drop_dom_ref ) put_domain(d); -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Add initialization for Xen coloring data. By default, use the lowest color index available. Benchmarking the VM interrupt response time provides an estimation of LLC usage by Xen's most latency-critical runtime task. Results on Arm Cortex-A53 on Xilinx Zynq UltraScale+ XCZU9EG show that one color, which reserves 64 KiB of L2, is enough to attain best responsiveness. More colors are instead very likely to be needed on processors whose L1 cache is physically-indexed and physically-tagged, such as Cortex-A57. In such cases, coloring applies to L1 also, and there typically are two distinct L1-colors. Therefore, reserving only one color for Xen would senselessly partitions a cache memory that is already private, i.e. underutilize it. The default amount of Xen colors is thus set to one. Signed-off-by: Luca Miccio <206497@studenti.unimore.it> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- xen/arch/arm/coloring.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/xen/arch/arm/coloring.c b/xen/arch/arm/coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/coloring.c +++ b/xen/arch/arm/coloring.c @@ -XXX,XX +XXX,XX @@ #include <asm/coloring.h> #include <asm/io.h> +/* By default Xen uses the lowestmost color */ +#define XEN_COLOR_DEFAULT_MASK 0x0001 +#define XEN_COLOR_DEFAULT_NUM 1 +/* Current maximum useful colors */ +#define MAX_XEN_COLOR 128 + /* Number of color(s) assigned to Xen */ static uint32_t xen_col_num; /* Coloring configuration of Xen as bitmask */ static uint32_t xen_col_mask[MAX_COLORS_CELLS]; +/* Xen colors IDs */ +static uint32_t xen_col_list[MAX_XEN_COLOR]; /* Number of color(s) assigned to Dom0 */ static uint32_t dom0_col_num; @@ -XXX,XX +XXX,XX @@ uint32_t get_max_colors(void) bool __init coloring_init(void) { - int i; + int i, rc; printk(XENLOG_INFO "Initialize XEN coloring: \n"); /* @@ -XXX,XX +XXX,XX @@ bool __init coloring_init(void) printk(XENLOG_INFO "Color bits in address: 0x%"PRIx64"\n", addr_col_mask); printk(XENLOG_INFO "Max number of colors: %u\n", max_col_num); + if ( !xen_col_num ) + { + xen_col_mask[0] = XEN_COLOR_DEFAULT_MASK; + xen_col_num = XEN_COLOR_DEFAULT_NUM; + printk(XENLOG_WARNING "Xen color configuration not found. Using default\n"); + } + + printk(XENLOG_INFO "Xen color configuration: 0x%"PRIx32"%"PRIx32"%"PRIx32"%"PRIx32"\n", + xen_col_mask[3], xen_col_mask[2], xen_col_mask[1], xen_col_mask[0]); + rc = copy_mask_to_list(xen_col_mask, xen_col_list, xen_col_num); + + if ( rc ) + return false; + + for ( i = 0; i < xen_col_num; i++ ) + if ( xen_col_list[i] > (max_col_num - 1) ) + { + printk(XENLOG_ERR "ERROR: max. color value not supported\n"); + return false; + } + return true; } -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> During Xen coloring procedure, we need to manually calculate consecutive physical addresses that conform to the color selection. Add an helper function that does this operation. The latter will return the next address that conforms to Xen color selection. The next_colored function is architecture dependent and the provided implementation is for ARMv8. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- xen/arch/arm/coloring.c | 43 +++++++++++++++++++++++++++++ xen/arch/arm/include/asm/coloring.h | 14 ++++++++++ 2 files changed, 57 insertions(+) diff --git a/xen/arch/arm/coloring.c b/xen/arch/arm/coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/coloring.c +++ b/xen/arch/arm/coloring.c @@ -XXX,XX +XXX,XX @@ uint32_t get_max_colors(void) return max_col_num; } +paddr_t next_xen_colored(paddr_t phys) +{ + unsigned int i; + unsigned int col_next_number = 0; + unsigned int col_cur_number = (phys & addr_col_mask) >> PAGE_SHIFT; + int overrun = 0; + paddr_t ret; + + /* + * Check if address color conforms to Xen selection. If it does, return + * the address as is. + */ + for( i = 0; i < xen_col_num; i++) + if ( col_cur_number == xen_col_list[i] ) + return phys; + + /* Find next col */ + for( i = xen_col_num -1 ; i >= 0; i--) + { + if ( col_cur_number > xen_col_list[i]) + { + /* Need to start to first element and add a way_size */ + if ( i == (xen_col_num - 1) ) + { + col_next_number = xen_col_list[0]; + overrun = 1; + } + else + { + col_next_number = xen_col_list[i+1]; + overrun = 0; + } + break; + } + } + + /* Align phys to way_size */ + ret = phys - (PAGE_SIZE * col_cur_number); + /* Add the offset based on color selection*/ + ret += (PAGE_SIZE * (col_next_number)) + (way_size*overrun); + return ret; +} + bool __init coloring_init(void) { int i, rc; diff --git a/xen/arch/arm/include/asm/coloring.h b/xen/arch/arm/include/asm/coloring.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/coloring.h +++ b/xen/arch/arm/include/asm/coloring.h @@ -XXX,XX +XXX,XX @@ bool __init coloring_init(void); +/* + * Return physical page address that conforms to the colors selection + * given in col_selection_mask after @param phys. + * + * @param phys Physical address start. + * @param addr_col_mask Mask specifying the bits available for coloring. + * @param col_selection_mask Mask asserting the color bits to be used, + * must not be 0. + * + * @return The lowest physical page address being greater or equal than + * 'phys' and belonging to Xen color selection + */ +paddr_t next_xen_colored(paddr_t phys); + /* * Check colors of a given domain. * Return true if check passed, false otherwise. -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> In order to relocate Xen the function get_xen_paddr will be used in the following patches. The method has "consider_modules" as a prerequisite so it has to be enabled both for ARM32 and coloring. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> --- xen/arch/arm/setup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/setup.c +++ b/xen/arch/arm/setup.c @@ -XXX,XX +XXX,XX @@ static void * __init relocate_fdt(paddr_t dtb_paddr, size_t dtb_size) return fdt; } -#ifdef CONFIG_ARM_32 +#if defined (CONFIG_ARM_32) || (CONFIG_COLORING) /* * Returns the end address of the highest region in the range s..e * with required size and alignment that does not conflict with the -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> In order to efficiently coloring Xen, we need to relocate it and move the xen code to a unique memory region that will be marked as colored for Xen itself. This region will be out target region and it will be placed as high as possibile in RAM. To do that we need to use the old get_xen_paddr function that was part of the relocation feature. Moreover the size of the region we want to relocate is not equal to xen code size anymore because of coloring. In the worst case the target region must be greater than xen code size * avail. colors. However the get_xen_paddr assumes to handle a memory with size equals only to xen code region. Add a new "size" parameter to handle also the coloring case. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> Acked-by: Stefano Stabellini <stefano.stabellini@xilinx.com> --- xen/arch/arm/setup.c | 54 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/setup.c +++ b/xen/arch/arm/setup.c @@ -XXX,XX +XXX,XX @@ static paddr_t __init next_module(paddr_t s, paddr_t *end) return lowest; } +#ifdef CONFIG_COLORING +/** + * get_xen_paddr - get physical address to relocate Xen to + * + * Xen is relocated to as near to the top of RAM as possible and + * aligned to a XEN_PADDR_ALIGN boundary. + */ +static paddr_t __init get_xen_paddr(uint32_t xen_size) +{ + struct meminfo *mi = &bootinfo.mem; + paddr_t min_size; + paddr_t paddr = 0; + int i; + + min_size = (xen_size + (XEN_PADDR_ALIGN-1)) & ~(XEN_PADDR_ALIGN-1); + + /* Find the highest bank with enough space. */ + for ( i = 0; i < mi->nr_banks; i++ ) + { + const struct membank *bank = &mi->bank[i]; + paddr_t s, e; + + if ( bank->size >= min_size ) + { + e = consider_modules(bank->start, bank->start + bank->size, + min_size, XEN_PADDR_ALIGN, 0); + if ( !e ) + continue; + +#ifdef CONFIG_ARM_32 + /* Xen must be under 4GB */ + if ( e > 0x100000000ULL ) + e = 0x100000000ULL; + if ( e < bank->start ) + continue; +#endif + + s = e - min_size; + + if ( s > paddr ) + paddr = s; + } + } + + if ( !paddr ) + panic("Not enough memory to relocate Xen\n"); + + printk("Placing Xen at 0x%"PRIpaddr"-0x%"PRIpaddr"\n", + paddr, paddr + min_size); + + return paddr; +} +#endif + static void __init init_pdx(void) { paddr_t bank_start, bank_size, bank_end; -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Upcoming patches will need to remove temporary mappings created during Xen coloring process. The function remove_early_mappings does what we need but it is case-specific. Parametrize the function to avoid code replication. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> Acked-by: Stefano Stabellini <stefano.stabellini@xilinx.com> --- xen/arch/arm/include/asm/mm.h | 2 +- xen/arch/arm/mm.c | 8 ++++---- xen/arch/arm/setup.c | 3 ++- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/xen/arch/arm/include/asm/mm.h b/xen/arch/arm/include/asm/mm.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/mm.h +++ b/xen/arch/arm/include/asm/mm.h @@ -XXX,XX +XXX,XX @@ extern void setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr); /* Map FDT in boot pagetable */ extern void *early_fdt_map(paddr_t fdt_paddr); /* Remove early mappings */ -extern void remove_early_mappings(void); +extern void remove_early_mappings(unsigned long va, unsigned long size); /* Allocate and initialise pagetables for a secondary CPU. Sets init_ttbr to the * new page table */ extern int init_secondary_pagetables(int cpu); diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/mm.c +++ b/xen/arch/arm/mm.c @@ -XXX,XX +XXX,XX @@ void * __init early_fdt_map(paddr_t fdt_paddr) return fdt_virt; } -void __init remove_early_mappings(void) +void __init remove_early_mappings(unsigned long va, unsigned long size) { lpae_t pte = {0}; - write_pte(xen_second + second_table_offset(BOOT_FDT_VIRT_START), pte); - write_pte(xen_second + second_table_offset(BOOT_FDT_VIRT_START + SZ_2M), + write_pte(xen_second + second_table_offset(va), pte); + write_pte(xen_second + second_table_offset(va + size), pte); - flush_xen_tlb_range_va(BOOT_FDT_VIRT_START, BOOT_FDT_SLOT_SIZE); + flush_xen_tlb_range_va(va, size); } /* diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/setup.c +++ b/xen/arch/arm/setup.c @@ -XXX,XX +XXX,XX @@ #include <xen/libfdt/libfdt.h> #include <xen/acpi.h> #include <xen/warning.h> +#include <xen/sizes.h> #include <asm/alternative.h> #include <asm/page.h> #include <asm/current.h> @@ -XXX,XX +XXX,XX @@ void __init discard_initial_modules(void) mi->nr_mods = 0; - remove_early_mappings(); + remove_early_mappings(BOOT_FDT_VIRT_START, SZ_2M); } /* Relocate the FDT in Xen heap */ -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Introduce a new implementation of setup_pagetables that uses coloring logic in order to isolate Xen code using its color selection. Page tables construction is essentially copied, except for the xenmap table, where coloring logic is needed. Given the absence of a contiguous physical mapping, pointers to next level tables need to be manually calculated. Xen code is relocated in strided mode using the same coloring logic as the one in xenmap table by using a temporary colored mapping that will be destroyed after switching the TTBR register. Keep Xen text section mapped in the newly created pagetables. The boot process relies on computing needed physical addresses of Xen code by using a shift, but colored mapping is not linear and not easily computable. Therefore, the old Xen code is temporarily kept and used to boot secondary CPUs until they switch to the colored mapping, which is accessed using the handy macro virt_old. After the boot process, the old Xen code memory is reset and its mapping is destroyed. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- xen/arch/arm/include/asm/coloring.h | 13 ++ xen/arch/arm/include/asm/mm.h | 7 ++ xen/arch/arm/mm.c | 186 +++++++++++++++++++++++++++- xen/arch/arm/psci.c | 4 +- xen/arch/arm/setup.c | 21 +++- xen/arch/arm/smpboot.c | 19 ++- 6 files changed, 241 insertions(+), 9 deletions(-) diff --git a/xen/arch/arm/include/asm/coloring.h b/xen/arch/arm/include/asm/coloring.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/coloring.h +++ b/xen/arch/arm/include/asm/coloring.h @@ -XXX,XX +XXX,XX @@ #ifdef CONFIG_COLORING #include <xen/sched.h> +/* + * Amount of memory that we need to map in order to color Xen. The value + * depends on the maximum number of available colors of the hardware. The + * memory size is pessimistically calculated assuming only one color is used, + * which means that any pages belonging to any other color has to be skipped. + */ +#define XEN_COLOR_MAP_SIZE \ + ((((_end - _start) * get_max_colors())\ + + (XEN_PADDR_ALIGN-1)) & ~(XEN_PADDR_ALIGN-1)) +#define XEN_COLOR_MAP_SIZE_M (XEN_COLOR_MAP_SIZE >> 20) + bool __init coloring_init(void); /* @@ -XXX,XX +XXX,XX @@ unsigned long color_from_page(struct page_info *pg); /* Return the maximum available number of colors supported by the hardware */ uint32_t get_max_colors(void); #else /* !CONFIG_COLORING */ +#define XEN_COLOR_MAP_SIZE (_end - _start) + static inline bool __init coloring_init(void) { return true; diff --git a/xen/arch/arm/include/asm/mm.h b/xen/arch/arm/include/asm/mm.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/mm.h +++ b/xen/arch/arm/include/asm/mm.h @@ -XXX,XX +XXX,XX @@ void clear_and_clean_page(struct page_info *page); unsigned int arch_get_dma_bitsize(void); +#ifdef CONFIG_COLORING +#define virt_boot_xen(virt)\ + (vaddr_t)((virt - XEN_VIRT_START) + BOOT_RELOC_VIRT_START) +#else +#define virt_boot_xen(virt) virt +#endif + #endif /* __ARCH_ARM_MM__ */ /* * Local variables: diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/mm.c +++ b/xen/arch/arm/mm.c @@ -XXX,XX +XXX,XX @@ #include <xen/libfdt/libfdt.h> #include <asm/setup.h> +#include <asm/coloring.h> /* Override macros from asm/page.h to make them work with mfn_t */ #undef virt_to_mfn @@ -XXX,XX +XXX,XX @@ DEFINE_BOOT_PAGE_TABLE(boot_second_id); DEFINE_BOOT_PAGE_TABLE(boot_third_id); DEFINE_BOOT_PAGE_TABLE(boot_second); DEFINE_BOOT_PAGE_TABLE(boot_third); +#ifdef CONFIG_COLORING +DEFINE_BOOT_PAGE_TABLE(boot_colored_xen); +#endif /* Main runtime page tables */ @@ -XXX,XX +XXX,XX @@ static void clear_table(void *table) clean_and_invalidate_dcache_va_range(table, PAGE_SIZE); } +#ifdef CONFIG_COLORING +/* + * Translate a Xen (.text) virtual address to the colored physical one + * depending on the hypervisor configuration. + * N.B: this function must be used only when migrating from non colored to + * colored pagetables since it assumes to have the temporary mappings created + * during setup_pagetables that starts from BOOT_RELOC_VIRT_START. + * After the migration we have to use virt_to_maddr. + */ +static paddr_t virt_to_maddr_colored(vaddr_t virt) +{ + unsigned int va_offset; + + va_offset = virt - XEN_VIRT_START; + return __pa(BOOT_RELOC_VIRT_START + va_offset); +} + +static void __init coloring_temp_mappings(paddr_t xen_paddr, vaddr_t virt_start) +{ + int i; + lpae_t pte; + unsigned int xen_text_size = (_end - _start); + + xen_text_size = PAGE_ALIGN(xen_text_size); + + pte = mfn_to_xen_entry(maddr_to_mfn(__pa(boot_second)), MT_NORMAL); + pte.pt.table = 1; + boot_first[first_table_offset(virt_start)] = pte; + + pte = mfn_to_xen_entry(maddr_to_mfn(__pa(boot_colored_xen)), MT_NORMAL); + pte.pt.table = 1; + boot_second[second_table_offset(virt_start)] = pte; + + for ( i = 0; i < (xen_text_size/PAGE_SIZE); i++ ) + { + mfn_t mfn; + xen_paddr = next_xen_colored(xen_paddr); + mfn = maddr_to_mfn(xen_paddr); + pte = mfn_to_xen_entry(mfn, MT_NORMAL); + pte.pt.table = 1; /* 4k mappings always have this bit set */ + boot_colored_xen[i] = pte; + xen_paddr += PAGE_SIZE; + } + + flush_xen_tlb_local(); +} + +/* + * Boot-time pagetable setup with coloring support + * Changes here may need matching changes in head.S + * + * The process can be explained as follows: + * - Create a temporary colored mapping that conforms to Xen color selection. + * - Update all the pagetables links that point to the next level table(s): + * this process is crucial beacause the translation tables are not physically + * contiguous and we cannot calculate the physical addresses by using the + * standard method (physical offset). In order to get the correct physical + * address we use virt_to_maddr_colored that translates the virtual address + * into a physical one based on the Xen coloring configuration. + * - Copy Xen to the new location. + * - Update TTBR0_EL2 with the new root page table address. + */ +void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr) +{ + int i; + lpae_t pte, *p; + paddr_t pt_phys; + mfn_t pt_phys_mfn; + paddr_t _xen_paddr = xen_paddr; + + phys_offset = boot_phys_offset; + + ASSERT((_end - _start) < SECOND_SIZE); + /* Create temporary mappings */ + coloring_temp_mappings(xen_paddr, BOOT_RELOC_VIRT_START); + + /* Build pagetables links */ + p = (void *)xen_pgtable; + pt_phys = virt_to_maddr_colored((vaddr_t)xen_first); + pt_phys_mfn = maddr_to_mfn(pt_phys); + p[0] = mfn_to_xen_entry(pt_phys_mfn, MT_NORMAL); + p[0].pt.table = 1; + p[0].pt.xn = 0; + p = (void *)xen_first; + + for ( i = 0; i < 2; i++ ) + { + pt_phys = virt_to_maddr_colored((vaddr_t)(xen_second + i * LPAE_ENTRIES)); + pt_phys_mfn = maddr_to_mfn(pt_phys); + p[i] = mfn_to_xen_entry(pt_phys_mfn, MT_NORMAL); + p[i].pt.table = 1; + p[i].pt.xn = 0; + } + + for ( i = 0; i < LPAE_ENTRIES; i++ ) + { + mfn_t mfn; + vaddr_t va = XEN_VIRT_START + (i << PAGE_SHIFT); + _xen_paddr = next_xen_colored(_xen_paddr); + mfn = maddr_to_mfn(_xen_paddr); + if ( !is_kernel(va) ) + break; + pte = mfn_to_xen_entry(mfn, MT_NORMAL); + pte.pt.table = 1; /* 4k mappings always have this bit set */ + if ( is_kernel_text(va) || is_kernel_inittext(va) ) + { + pte.pt.xn = 0; + pte.pt.ro = 1; + } + if ( is_kernel_rodata(va) ) + pte.pt.ro = 1; + xen_xenmap[i] = pte; + _xen_paddr += PAGE_SIZE; + } + + /* Initialise xen second level entries ... */ + /* ... Xen's text etc */ + pt_phys = virt_to_maddr_colored((vaddr_t)(xen_xenmap)); + pt_phys_mfn = maddr_to_mfn(pt_phys); + pte = mfn_to_xen_entry(pt_phys_mfn, MT_NORMAL); + pte.pt.table = 1; + xen_second[second_table_offset(XEN_VIRT_START)] = pte; + + /* ... Fixmap */ + pt_phys = virt_to_maddr_colored((vaddr_t)(xen_fixmap)); + pt_phys_mfn = maddr_to_mfn(pt_phys); + pte = mfn_to_xen_entry(pt_phys_mfn, MT_NORMAL); + pte.pt.table = 1; + xen_second[second_table_offset(FIXMAP_ADDR(0))] = pte; + + /* ... DTB */ + pte = boot_second[second_table_offset(BOOT_FDT_VIRT_START)]; + xen_second[second_table_offset(BOOT_FDT_VIRT_START)] = pte; + pte = boot_second[second_table_offset(BOOT_FDT_VIRT_START + SZ_2M)]; + xen_second[second_table_offset(BOOT_FDT_VIRT_START + SZ_2M)] = pte; + + /* Update the value of init_ttbr */ + init_ttbr = virt_to_maddr_colored((vaddr_t)xen_pgtable); + clean_dcache(init_ttbr); + + /* Copy Xen to the new location */ + memcpy((void*)BOOT_RELOC_VIRT_START, + (const void*)XEN_VIRT_START, (_end - _start)); + clean_dcache_va_range((void*)BOOT_RELOC_VIRT_START, (_end - _start)); + + /* Change ttbr */ + switch_ttbr(init_ttbr); + + /* + * Keep mapped old Xen memory in a contiguous mapping + * for other cpus to boot. This mapping will also replace the + * one created at the beginning of setup_pagetables. + */ + create_mappings(xen_second, BOOT_RELOC_VIRT_START, + paddr_to_pfn(XEN_VIRT_START + phys_offset), + SZ_2M >> PAGE_SHIFT, SZ_2M); + + xen_pt_enforce_wnx(); +} +#else /* Boot-time pagetable setup. * Changes here may need matching changes in head.S */ void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr) @@ -XXX,XX +XXX,XX @@ void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr) per_cpu(xen_dommap, 0) = cpu0_dommap; #endif } +#endif /* !CONFIG_COLORING */ static void clear_boot_pagetables(void) { @@ -XXX,XX +XXX,XX @@ static void clear_boot_pagetables(void) #endif clear_table(boot_second); clear_table(boot_third); +#ifdef CONFIG_COLORING + clear_table(boot_colored_xen); +#endif } #ifdef CONFIG_ARM_64 @@ -XXX,XX +XXX,XX @@ int init_secondary_pagetables(int cpu) { clear_boot_pagetables(); + /* + * For coloring the value of the ttbr was already set up during + * setup_pagetables. + */ +#ifndef CONFIG_COLORING /* Set init_ttbr for this CPU coming up. All CPus share a single setof * pagetables, but rewrite it each time for consistency with 32 bit. */ init_ttbr = (uintptr_t) xen_pgtable + phys_offset; clean_dcache(init_ttbr); +#endif return 0; } #else @@ -XXX,XX +XXX,XX @@ void __init setup_xenheap_mappings(unsigned long base_mfn, else if ( xenheap_first_first_slot == -1) { /* Use xenheap_first_first to bootstrap the mappings */ - first = xenheap_first_first; + paddr_t phys_addr; + + /* + * At this stage is safe to use virt_to_maddr because Xen mapping + * is already in place. Using virt_to_maddr allows us to unify + * codepath with and without cache coloring enabled. + */ + phys_addr = virt_to_maddr((vaddr_t)xenheap_first_first); + pte = mfn_to_xen_entry(maddr_to_mfn(phys_addr),MT_NORMAL); - pte = pte_of_xenaddr((vaddr_t)xenheap_first_first); pte.pt.table = 1; write_pte(p, pte); + first = xenheap_first_first; xenheap_first_first_slot = slot; } else diff --git a/xen/arch/arm/psci.c b/xen/arch/arm/psci.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/psci.c +++ b/xen/arch/arm/psci.c @@ -XXX,XX +XXX,XX @@ #include <asm/cpufeature.h> #include <asm/psci.h> #include <asm/acpi.h> +#include <asm/coloring.h> /* * While a 64-bit OS can make calls with SMC32 calling conventions, for @@ -XXX,XX +XXX,XX @@ int call_psci_cpu_on(int cpu) { struct arm_smccc_res res; - arm_smccc_smc(psci_cpu_on_nr, cpu_logical_map(cpu), __pa(init_secondary), + arm_smccc_smc(psci_cpu_on_nr, cpu_logical_map(cpu), + __pa(virt_boot_xen((vaddr_t)init_secondary)), &res); return PSCI_RET(res); diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/setup.c +++ b/xen/arch/arm/setup.c @@ -XXX,XX +XXX,XX @@ void __init start_xen(unsigned long boot_phys_offset, struct domain *d; int rc; paddr_t xen_paddr = (paddr_t)(_start + boot_phys_offset); + uint32_t xen_size = (_end - _start); dcache_line_bytes = read_dcache_line_bytes(); @@ -XXX,XX +XXX,XX @@ void __init start_xen(unsigned long boot_phys_offset, if ( !coloring_init() ) panic("Xen Coloring support: setup failed\n"); + xen_size = XEN_COLOR_MAP_SIZE; +#ifdef CONFIG_COLORING + xen_paddr = get_xen_paddr(xen_size); +#endif + /* Register Xen's load address as a boot module. */ - xen_bootmodule = add_boot_module(BOOTMOD_XEN, xen_paddr, - (paddr_t)(uintptr_t)(_end - _start + 1), false); + xen_bootmodule = add_boot_module(BOOTMOD_XEN, xen_paddr, xen_size, false); BUG_ON(!xen_bootmodule); setup_pagetables(boot_phys_offset, xen_paddr); - setup_mm(); /* Parse the ACPI tables for possible boot-time configuration */ @@ -XXX,XX +XXX,XX @@ void __init start_xen(unsigned long boot_phys_offset, setup_virt_paging(); + /* + * This removal is useful if cache coloring is enabled but + * it should not affect non coloring configuration. + * The removal is done earlier than discard_initial_modules + * beacuse in do_initcalls there is the livepatch support + * setup which uses the virtual addresses starting from + * BOOT_RELOC_VIRT_START. + * Remove coloring mappings to expose a clear state to the + * livepatch module. + */ + remove_early_mappings(BOOT_RELOC_VIRT_START, SZ_2M); do_initcalls(); /* diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/smpboot.c +++ b/xen/arch/arm/smpboot.c @@ -XXX,XX +XXX,XX @@ int __cpu_up(unsigned int cpu) { int rc; s_time_t deadline; + vaddr_t *smp_up_cpu_addr; printk("Bringing up CPU%d\n", cpu); @@ -XXX,XX +XXX,XX @@ int __cpu_up(unsigned int cpu) /* Tell the remote CPU what its logical CPU ID is. */ init_data.cpuid = cpu; + /* + * If coloring is enabled, non-Master CPUs boot using the old Xen code. + * During the boot process each cpu is booted one after another using the + * smp_cpu_cpu variable. This variable is accessed in head.S using its + * physical address. + * That address is calculated using the physical offset of the old Xen + * code. With coloring we can not rely anymore on that offset. For this + * reason in order to boot the other cpus we rely on the old xen code that + * was mapped during tables setup in mm.c so that we can use the old physical + * offset and the old head.S code also. In order to modify the old Xen code + * we need to access it using the mapped done in color_xen. + */ + smp_up_cpu_addr = (vaddr_t *)virt_boot_xen((vaddr_t)&smp_up_cpu); + *smp_up_cpu_addr = cpu_logical_map(cpu); /* Open the gate for this CPU */ - smp_up_cpu = cpu_logical_map(cpu); - clean_dcache(smp_up_cpu); - + clean_dcache(*smp_up_cpu_addr); rc = arch_cpu_up(cpu); console_end_sync(); -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Introduce two new arm specific functions to temporarily map/unmap the Xen text read-write (the Xen text is mapped read-only by default by setup_pagetables): xen_map_text_rw and xen_unmap_text_rw. There is only one caller in the alternative framework. The non-colored implementation simply uses __vmap to do the mapping. In other words, there are no changes to the non-colored case. The colored implementation calculates Xen text physical addresses appropriately, according to the coloring configuration. Export vm_alloc because it is needed by the colored implementation of xen_map_text_rw. Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com> Signed-off-by: Luca Miccio <lucmiccio@gmail.com> --- xen/arch/arm/alternative.c | 8 ++------ xen/arch/arm/include/asm/mm.h | 3 +++ xen/arch/arm/mm.c | 38 +++++++++++++++++++++++++++++++++++ xen/common/vmap.c | 4 ++-- xen/include/xen/vmap.h | 2 ++ 5 files changed, 47 insertions(+), 8 deletions(-) diff --git a/xen/arch/arm/alternative.c b/xen/arch/arm/alternative.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/alternative.c +++ b/xen/arch/arm/alternative.c @@ -XXX,XX +XXX,XX @@ static int __apply_alternatives_multi_stop(void *unused) { int ret; struct alt_region region; - mfn_t xen_mfn = virt_to_mfn(_start); - paddr_t xen_size = _end - _start; - unsigned int xen_order = get_order_from_bytes(xen_size); void *xenmap; BUG_ON(patched); @@ -XXX,XX +XXX,XX @@ static int __apply_alternatives_multi_stop(void *unused) * The text and inittext section are read-only. So re-map Xen to * be able to patch the code. */ - xenmap = __vmap(&xen_mfn, 1U << xen_order, 1, 1, PAGE_HYPERVISOR, - VMAP_DEFAULT); + xenmap = xen_map_text_rw(); /* Re-mapping Xen is not expected to fail during boot. */ BUG_ON(!xenmap); @@ -XXX,XX +XXX,XX @@ static int __apply_alternatives_multi_stop(void *unused) /* The patching is not expected to fail during boot. */ BUG_ON(ret != 0); - vunmap(xenmap); + xen_unmap_text_rw(xenmap); /* Barriers provided by the cache flushing */ write_atomic(&patched, 1); diff --git a/xen/arch/arm/include/asm/mm.h b/xen/arch/arm/include/asm/mm.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/mm.h +++ b/xen/arch/arm/include/asm/mm.h @@ -XXX,XX +XXX,XX @@ extern void mmu_init_secondary_cpu(void); extern void setup_xenheap_mappings(unsigned long base_mfn, unsigned long nr_mfns); /* Map a frame table to cover physical addresses ps through pe */ extern void setup_frametable_mappings(paddr_t ps, paddr_t pe); +/* Create temporary Xen text read-write mapping */ +extern void *xen_map_text_rw(void); +extern void xen_unmap_text_rw(void *va); /* Map a 4k page in a fixmap entry */ extern void set_fixmap(unsigned map, mfn_t mfn, unsigned attributes); /* Remove a mapping from a fixmap entry */ diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/mm.c +++ b/xen/arch/arm/mm.c @@ -XXX,XX +XXX,XX @@ static void clear_table(void *table) } #ifdef CONFIG_COLORING +void* __init xen_map_text_rw(void) +{ + paddr_t xen_paddr = __pa(_start); + unsigned int xen_size = 1 << get_order_from_bytes(_end - _start); + void *va = vm_alloc(xen_size, 1, VMAP_DEFAULT); + unsigned long cur = (unsigned long)va; + mfn_t mfn_col; + unsigned int i; + + for ( i = 0; i < xen_size; i++, cur += PAGE_SIZE ) + { + xen_paddr = next_xen_colored(xen_paddr); + mfn_col = maddr_to_mfn(xen_paddr); + if ( map_pages_to_xen(cur, mfn_col, 1, PAGE_HYPERVISOR) ) + return NULL; + xen_paddr += PAGE_SIZE; + } + return va; +} + +void __init xen_unmap_text_rw(void *va) +{ + vunmap(va); +} + /* * Translate a Xen (.text) virtual address to the colored physical one * depending on the hypervisor configuration. @@ -XXX,XX +XXX,XX @@ void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr) xen_pt_enforce_wnx(); } #else +void* __init xen_map_text_rw(void) +{ + unsigned int xen_order = get_order_from_bytes(_end - _start); + mfn_t xen_mfn = virt_to_mfn(_start); + return __vmap(&xen_mfn, 1U << xen_order, 1, 1, PAGE_HYPERVISOR, + VMAP_DEFAULT); +} + +void __init xen_unmap_text_rw(void *va) +{ + vunmap(va); +} + /* Boot-time pagetable setup. * Changes here may need matching changes in head.S */ void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr) diff --git a/xen/common/vmap.c b/xen/common/vmap.c index XXXXXXX..XXXXXXX 100644 --- a/xen/common/vmap.c +++ b/xen/common/vmap.c @@ -XXX,XX +XXX,XX @@ void __init vm_init_type(enum vmap_region type, void *start, void *end) populate_pt_range(va, vm_low[type] - nr); } -static void *vm_alloc(unsigned int nr, unsigned int align, - enum vmap_region t) +void *vm_alloc(unsigned int nr, unsigned int align, + enum vmap_region t) { unsigned int start, bit; diff --git a/xen/include/xen/vmap.h b/xen/include/xen/vmap.h index XXXXXXX..XXXXXXX 100644 --- a/xen/include/xen/vmap.h +++ b/xen/include/xen/vmap.h @@ -XXX,XX +XXX,XX @@ enum vmap_region { void vm_init_type(enum vmap_region type, void *start, void *end); +void *vm_alloc(unsigned int nr, unsigned int align, + enum vmap_region t); void *__vmap(const mfn_t *mfn, unsigned int granularity, unsigned int nr, unsigned int align, unsigned int flags, enum vmap_region); void *vmap(const mfn_t *mfn, unsigned int nr); -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Display general information about coloring support both during boot and when requested by the user. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com> --- xen/arch/arm/coloring.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/xen/arch/arm/coloring.c b/xen/arch/arm/coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/coloring.c +++ b/xen/arch/arm/coloring.c @@ -XXX,XX +XXX,XX @@ #include <xen/types.h> #include <xen/lib.h> #include <xen/errno.h> +#include <xen/keyhandler.h> #include <xen/param.h> #include <asm/sysregs.h> @@ -XXX,XX +XXX,XX @@ void coloring_dump_info(struct domain *d) printk("]\n"); } +static void dump_coloring_info(unsigned char key) +{ + int i; + + printk("Coloring general information\n"); + printk("Way size: %"PRIu64"kB\n", way_size >> 10); + printk("Max. number of colors available: %"PRIu32"\n", max_col_num); + + printk("Xen color(s):\t["); + for ( i = 0; i < xen_col_num; i++ ) + printk(" %"PRIu32" ", xen_col_list[i]); + printk("]\n"); +} + +static __init int register_heap_trigger(void) +{ + register_keyhandler('C', dump_coloring_info, "dump coloring general info", 1); + + /* Also print general information once at boot */ + dump_coloring_info('C'); + return 0; +} +__initcall(register_heap_trigger); /* * Local variables: * mode: C -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Dom0less color assignment is performed via Device Tree with a new attribute "colors". In this case the color assignment is represented by a bitmask where it suffices to set all and only the bits having a position equal to the chosen colors, leaving unset all the others. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- xen/arch/arm/domain_build.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/domain_build.c +++ b/xen/arch/arm/domain_build.c @@ -XXX,XX +XXX,XX @@ #include <asm/setup.h> #include <asm/cpufeature.h> #include <asm/domain_build.h> +#include <asm/coloring.h> #include <xen/irq.h> #include <xen/grant_table.h> @@ -XXX,XX +XXX,XX @@ void __init create_domUs(void) { struct dt_device_node *node; const struct dt_device_node *chosen = dt_find_node_by_path("/chosen"); + u32 col_val; + const u32 *cells; + u32 len; + int cell, i, k; BUG_ON(chosen == NULL); dt_for_each_child_node(chosen, node) @@ -XXX,XX +XXX,XX @@ void __init create_domUs(void) vpl011_virq - 32 + 1); } + d_cfg.arch.colors.max_colors = 0; + memset(&d_cfg.arch.colors.colors, 0x0, sizeof(d_cfg.arch.colors.colors)); + + cells = dt_get_property(node, "colors", &len); + if ( cells != NULL && len > 0 ) + { + if ( !get_max_colors() ) + panic("Coloring requested but no colors configuration found!\n"); + + if ( len > sizeof(d_cfg.arch.colors.colors) ) + panic("Dom0less DomU color information is invalid\n"); + + for ( k = 0, cell = len/4 - 1; cell >= 0; cell--, k++ ) + { + col_val = be32_to_cpup(&cells[cell]); + if ( col_val ) + { + /* Calculate number of bit set */ + for ( i = 0; i < 32; i++) + if ( col_val & (1 << i) ) + d_cfg.arch.colors.max_colors++; + d_cfg.arch.colors.colors[k] = col_val; + } + } + } /* * The variable max_init_domid is initialized with zero, so here it's * very important to use the pre-increment operator to call -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Static memory assumes to have physically contiguous memory mapped to domains. This assumption cannot be made when coloring is enabled. These two features have to be mutually exclusive. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> --- xen/arch/arm/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/Kconfig +++ b/xen/arch/arm/Kconfig @@ -XXX,XX +XXX,XX @@ config HARDEN_BRANCH_PREDICTOR config COLORING bool "L2 cache coloring" default n + depends on !STATIC_MEMORY depends on ARM_64 config TEE -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Currently coloring supports breaks assertion in domctl.c:892 because of the data structure used for color configuration. Currently the array is set to support up to 128 colors. Lower the number of supported colors to 64 until a better solution is found. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> --- xen/include/public/arch-arm.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h index XXXXXXX..XXXXXXX 100644 --- a/xen/include/public/arch-arm.h +++ b/xen/include/public/arch-arm.h @@ -XXX,XX +XXX,XX @@ struct vcpu_guest_context { typedef struct vcpu_guest_context vcpu_guest_context_t; DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t); -#define MAX_COLORS_CELLS 4 +#define MAX_COLORS_CELLS 2 struct color_guest_config { uint32_t max_colors; uint32_t colors[MAX_COLORS_CELLS]; -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Four additional parameters in the Xen command line are used to define the underlying coloring policy, which is not directly configurable otherwise. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- docs/misc/xen-command-line.pandoc | 51 +++++++++++++++++++++++++++++-- 1 file changed, 49 insertions(+), 2 deletions(-) diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc index XXXXXXX..XXXXXXX 100644 --- a/docs/misc/xen-command-line.pandoc +++ b/docs/misc/xen-command-line.pandoc @@ -XXX,XX +XXX,XX @@ can be maintained with the pv-shim mechanism. cause Xen not to use Indirect Branch Tracking even when support is available in hardware. +### buddy\_size (arm64) +> `= <size in megabyte>` + +> Default: `64 MB` + +Amount of memory reserved for the buddy allocator when colored allocator is +active. This options is useful only if coloring support is enabled. +The colored allocator is meant as an alternative to the buddy allocator, +since its allocation policy is by definition incompatible with the +generic one. Since the Xen heap systems is not colored yet, we need to +support the coexistence of the two allocators for now. This parameter, which is +optional and for expert only, is used to set the amount of memory reserved to +the buddy allocator. + ### clocksource (x86) > `= pit | hpet | acpi | tsc` @@ -XXX,XX +XXX,XX @@ Controls for the dom0 IOMMU setup. Incorrect use of this option may result in a malfunctioning system. -### dom0_ioports_disable (x86) +### dom0\_colors (arm64) +> `= List of <integer>-<integer>` + +> Default: `All available colors` + +Specify dom0 color configuration. If the parameter is not set, all available +colors are chosen and the user is warned on Xen's serial console. This color +configuration acts also as the default one for all DomUs that do not have any +explicit color assignment in their configuration file. + +### dom0\_ioports\_disable (x86) > `= List of <hex>-<hex>` Specify a list of IO ports to be excluded from dom0 access. @@ -XXX,XX +XXX,XX @@ unknown NMIs will still be processed. Set the NMI watchdog timeout in seconds. Specifying `0` will turn off the watchdog. +### way\_size (arm64) +> `= <size in byte>` + +> Default: `Obtained from the hardware` + +Specify the way size of the Last Level Cache. This parameter is only useful with +coloring support enabled. It is an optional, expert-only parameter and it is +used to calculate what bits in the physical address can be used by the coloring +algorithm, and thus the maximum available colors on the platform. It can be +obtained by dividing the total LLC size by the number of associativity ways. +By default, the value is also automatically computed during coloring +initialization to avoid any kind of misconfiguration. For this reason, it is +highly recommended to use this boot argument with specific needs only. + ### x2apic (x86) > `= <boolean>` @@ -XXX,XX +XXX,XX @@ In the case that x2apic is in use, this option switches between physical and clustered mode. The default, given no hint from the **FADT**, is cluster mode. -### xenheap_megabytes (arm32) +### xen\_colors (arm64) +> `= List of <integer>-<integer>` + +> Default: `0-0: the lowermost color` + +Specify Xen color configuration. +Two colors are most likely needed on platforms where private caches are +physically indexed, e.g. the L1 instruction cache of the Arm Cortex-A57. + +### xenheap\_megabytes (arm32) > `= <size>` > Default: `0` (1/32 of RAM) -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> The color selection has to be specified in the configuration file of the virtual machine with the new parameter 'colors'. This parameter defines the colors to be assigned to that particular VM, expressed as a list of ranges. Add documentation for the new 'colors' parameter. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- docs/man/xl.cfg.5.pod.in | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in index XXXXXXX..XXXXXXX 100644 --- a/docs/man/xl.cfg.5.pod.in +++ b/docs/man/xl.cfg.5.pod.in @@ -XXX,XX +XXX,XX @@ Currently, only the "sbsa_uart" model is supported for ARM. =back +=over 4 + +=item B<colors=[ "COLORS_RANGE", "COLORS_RANGE", ...]> + +Specify the color configuration for the guest. B<COLORS_RANGE> is expressed +using colors numbers. The range starts always from 0 up to the maximum amount +of available colors. +The number of available colors depends on the LLC layout of the specific +platform and determines the maximum allowed value. This number can be either +calculated or read from the output given by the hypervisor during boot, if +DEBUG logging is enabled. + +=back + =head3 x86 =over 4 -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Dom0less uses device tree for DomUs when booting them without using Dom0. Add a new device tree property 'colors' that specifies the coloring configuration for DomUs when using Dom0less. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- docs/misc/arm/device-tree/booting.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/misc/arm/device-tree/booting.txt b/docs/misc/arm/device-tree/booting.txt index XXXXXXX..XXXXXXX 100644 --- a/docs/misc/arm/device-tree/booting.txt +++ b/docs/misc/arm/device-tree/booting.txt @@ -XXX,XX +XXX,XX @@ with the following properties: An integer specifying the number of vcpus to allocate to the guest. +- colors + A 64 bit bitmask specifying the color configuration for the guest. + - vpl011 An empty property to enable/disable a virtual pl011 for the guest to -- 2.30.2 From: Luca Miccio <lucmiccio@gmail.com> Add basic documentation that shows how cache coloring support can be used in Xen. It introduces the basic concepts behind cache coloring, defines the cache selection format, and explains how to assign colors to the supported domains: Dom0, DomUs and Xen itself. Known issues are also reported. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- docs/misc/arm/cache_coloring.rst | 191 +++++++++++++++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100644 docs/misc/arm/cache_coloring.rst diff --git a/docs/misc/arm/cache_coloring.rst b/docs/misc/arm/cache_coloring.rst new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/docs/misc/arm/cache_coloring.rst @@ -XXX,XX +XXX,XX @@ +Xen coloring support user's guide +================================= + +The cache coloring support in Xen allows to reserve last level cache partition +for Dom0, DomUs and Xen itself. Currently only ARM64 is supported. + +In order to enable and use it, few steps are needed. + +- Enable coloring in XEN configuration file. + + CONFIG_COLORING=y + +- Enable/disable debug information (optional). + + CONFIG_COLORING_DEBUG=y/n + +Before digging into configuration instructions, configurers should first +understand the basics of cache coloring. + +Background +********** + +Cache hierarchy of a modern multi-core CPU typically has first levels dedicated +to each core (hence using multiple cache units), while the last level is shared +among all of them. Such configuration implies that memory operations on one +core (e.g. running a DomU) are able to generate interference on another core +(e.g .hosting another DomU). Cache coloring allows eliminating this +mutual interference, and thus guaranteeing higher and more predictable +performances for memory accesses. +The key concept underlying cache coloring is a fragmentation of the memory +space into a set of sub-spaces called colors that are mapped to disjoint cache +partitions. Technically, the whole memory space is first divided into a number +of subsequent regions. Then each region is in turn divided into a number of +subsequent sub-colors. The generic i-th color is then obtained by all the +i-th sub-colors in each region. + +.. raw:: html + + <pre> + Region j Region j+1 + ..................... ............ + . . . + . . + _ _ _______________ _ _____________________ _ _ + | | | | | | | + | c_0 | c_1 | | c_n | c_0 | c_1 | + _ _ _|_____|_____|_ _ _|_____|_____|_____|_ _ _ + : : + : :... ... . + : color 0 + :........................... ... . + : + . . ..................................: + </pre> + +There are two pragmatic lesson to be learnt. + +1. If one wants to avoid cache interference between two domains, different + colors needs to be used for their memory. + +2. Color assignment must privilege contiguity in the partitioning. E.g., + assigning colors (0,1) to domain I and (2,3) to domain J is better than + assigning colors (0,2) to I and (1,3) to J. + + +Color(s) selection format +************************** + +Regardless of the domain that has to be colored (Dom0, DomUs and Xen), +the color selection can be expressed using the same syntax. In particular, +the latter is expressed as a comma-separated list of hyphen-separated intervals +of color numbers, as in `0-4,5-8,10-15`. Ranges are always represented using +strings. Note that no spaces are allowed. + +The number of available colors depends on the LLC layout of the specific +platform and determines the maximum allowed value. This number can be either +calculated [#f1]_ or read from the output given by the hypervisor during boot, +if DEBUG logging is enabled. + +Examples: + ++---------------------+-----------------------------------+ +|**Configuration** |**Actual selection** | ++---------------------+-----------------------------------+ +| 1-2,5-8 | [1, 2, 5, 6, 7, 8] | ++---------------------+-----------------------------------+ +| 0-8,3-8 | [0, 1, 2, 3, 4, 5, 6, 7, 8] | ++---------------------+-----------------------------------+ +| 0-0 | [0] | ++---------------------+-----------------------------------+ + +General coloring parameters +*************************** + +Four additional parameters in the Xen command line are used to define the +underlying coloring policy, which is not directly configurable otherwise. + +Please refer to the relative documentation in docs/man/xl.cfg.pod.5.in. + +Dom0less support +**************** +Support for the Dom0less experimental features is provided. Color selection for +a virtual machine is defined by the attribute `colors`, whose format is not a +string for ranges list, but a bitmask. It suffices to set all and only the bits +having a position equal to the chosen colors, leaving unset all the others. For +example, if we choose 8 colors out of 16, we can use a bitmask with 8 bits set +and 8 bit unset, like: + +- `0xff00` -> `1111 1111 0000 0000` +- `0x0ff0` -> `0000 1111 1111 0000` +- `0x3c3c` -> `0011 1100 0011 1100` + +Configuration example: + +.. raw:: html + + <pre> + xen,xen-bootargs = "console=dtuart dtuart=serial0 dom0_mem=1G dom0_max_vcpus=1 sched=null way_size=65536 xen_colors=0-1 dom0_colors=2-6"; + xen,dom0-bootargs "console=hvc0 earlycon=xen earlyprintk=xen root=/dev/ram0" + + dom0 { + compatible = "xen,linux-zimage" "xen,multiboot-module"; + reg = <0x0 0x1000000 0x0 15858176>; + }; + + dom0-ramdisk { + compatible = "xen,linux-initrd" "xen,multiboot-module"; + reg = <0x0 0x2000000 0x0 20638062>; + }; + + domU0 { + #address-cells = <0x1>; + #size-cells = <0x1>; + compatible = "xen,domain"; + memory = <0x0 0x40000>; + colors = <0x0 0x0f00>; + cpus = <0x1>; + vpl011 = <0x1>; + + module@2000000 { + compatible = "multiboot,kernel", "multiboot,module"; + reg = <0x2000000 0xffffff>; + bootargs = "console=ttyAMA0"; + }; + + module@30000000 { + compatible = "multiboot,ramdisk", "multiboot,module"; + reg = <0x3000000 0xffffff>; + }; + }; + </pre> + +Please refer to the relative documentation in +docs/misc/arm/device-tree/booting.txt. + + +Known issues +************ + +Explicitly define way_size in QEMU +################################## + +Currently, QEMU does not have a comprehensive cache model, so the cache coloring +support fails to detect a cache geometry where to operate. In this case, the +boot hangs as soon as the Xen image is loaded. To overcome this issue, it is +enough to specify the way_size parameter in the command line. Any multiple +greater than 1 of the page size allows the coloring mechanism to work, but the +precise behavior on the system that QEMU is emulating can be obtained with its +way_size. For instance, set way_size=65536. + + +Fail to boot colored DomUs with large memory size +################################################# + +If the kernel used for Dom0 does not contain the upstream commit +3941552aec1e04d63999988a057ae09a1c56ebeb and uses the hypercall buffer device, +colored DomUs with memory size larger then 127 MB cannot be created. This is +caused by the default limit of this buffer of 64 pages. The solution is to +manually apply the above patch, or to check if there is an updated version of +the kernel in use for Dom0 that contains this change. + +Notes: +****** + +.. [#f1] To compute the number of available colors on a platform, one can simply + divide `way_size` by `page_size`, where: `page_size` is the size of the page + used on the system (usually 4 KiB); `way_size` is size of each LLC way. For + example, an Arm Cortex-A53 with a 16-ways associative 1 MiB LLC enable 16 + colors, when pages are 4 KiB. + + -- 2.30.2 Shared caches in multi-core CPU architectures represent a problem for predictability of memory access latency. This jeopardizes applicability of many Arm platform in real-time critical and mixed-criticality scenarios. We introduce support for cache partitioning with page coloring, a transparent software technique that enables isolation between domains and Xen, and thus avoids cache interference. When creating a domain, a simple syntax (e.g. `0-3` or `4-11`) allows the user to define assignments of cache partitions ids, called colors, where assigning different colors guarantees no mutual eviction on cache will ever happen. This instructs the Xen memory allocator to provide the i-th color assignee only with pages that maps to color i, i.e. that are indexed in the i-th cache partition. The proposed implementation supports the dom0less feature. The proposed implementation doesn't support the static-mem feature. The solution has been tested in several scenarios, including Xilinx Zynq MPSoCs. Carlo Nonato (11): xen/common: add cache coloring common code xen/arm: add initial support for LLC coloring on arm64 xen/arm: permit non direct-mapped Dom0 construction xen/arm: add Dom0 cache coloring support xen: extend domctl interface for cache coloring tools: add support for cache coloring configuration xen/arm: add support for cache coloring configuration via device-tree xen/page_alloc: introduce preserved page flags macro xen: add cache coloring allocator for domains xen/arm: make consider_modules() available for xen relocation xen/arm: add cache coloring support for Xen Luca Miccio (1): xen/arm: add Xen cache colors command line parameter SUPPORT.md | 7 + docs/index.rst | 1 + docs/man/xl.cfg.5.pod.in | 6 + docs/misc/arm/device-tree/booting.txt | 4 + docs/misc/cache-coloring.rst | 246 +++++++++++++++ docs/misc/xen-command-line.pandoc | 70 +++++ tools/include/libxl.h | 5 + tools/include/xenctrl.h | 9 + tools/libs/ctrl/xc_domain.c | 35 +++ tools/libs/light/libxl_create.c | 18 ++ tools/libs/light/libxl_types.idl | 1 + tools/xl/xl_parse.c | 38 ++- xen/arch/arm/Kconfig | 1 + xen/arch/arm/Makefile | 1 + xen/arch/arm/alternative.c | 30 +- xen/arch/arm/arm32/mmu/mm.c | 95 +----- xen/arch/arm/arm64/mmu/head.S | 58 +++- xen/arch/arm/arm64/mmu/mm.c | 28 +- xen/arch/arm/dom0less-build.c | 60 +--- xen/arch/arm/domain_build.c | 106 ++++++- xen/arch/arm/include/asm/domain_build.h | 1 + xen/arch/arm/include/asm/mm.h | 5 + xen/arch/arm/include/asm/mmu/layout.h | 3 + xen/arch/arm/include/asm/processor.h | 16 + xen/arch/arm/include/asm/setup.h | 3 + xen/arch/arm/llc-coloring.c | 141 +++++++++ xen/arch/arm/mmu/setup.c | 195 +++++++++++- xen/arch/arm/setup.c | 13 +- xen/common/Kconfig | 29 ++ xen/common/Makefile | 1 + xen/common/domain.c | 3 + xen/common/domctl.c | 10 + xen/common/keyhandler.c | 3 + xen/common/llc-coloring.c | 382 ++++++++++++++++++++++++ xen/common/page_alloc.c | 208 ++++++++++++- xen/include/public/domctl.h | 9 + xen/include/xen/llc-coloring.h | 64 ++++ xen/include/xen/sched.h | 5 + xen/include/xen/xmalloc.h | 12 + 39 files changed, 1749 insertions(+), 173 deletions(-) create mode 100644 docs/misc/cache-coloring.rst create mode 100644 xen/arch/arm/llc-coloring.c create mode 100644 xen/common/llc-coloring.c create mode 100644 xen/include/xen/llc-coloring.h -- 2.43.0 Last Level Cache (LLC) coloring allows to partition the cache in smaller chunks called cache colors. Since not all architectures can actually implement it, add a HAS_LLC_COLORING Kconfig option. LLC_COLORS_ORDER Kconfig option has a range maximum of 10 (2^10 = 1024) because that's the number of colors that fit in a 4 KiB page when integers are 4 bytes long. LLC colors are a property of the domain, so struct domain has to be extended. Based on original work from: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Carlo Nonato <carlo.nonato@minervasys.tech> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- v10: - fixed commit message to use LLC_COLORS_ORDER - added documentation to index.rst - moved check on CONFIG_NUMA in arch/arm/Kconfig (next patch) - fixed copyright line - fixed array type for colors parameter in print_colors() - added check on (way_size & ~PAGE_MASK) v9: - dropped _MAX_ from CONFIG_MAX_LLC_COLORS_ORDER v8: - minor documentation fixes - "llc-coloring=on" is inferred from "llc-nr-ways" and "llc-size" usage - turned CONFIG_NR_LLC_COLORS to CONFIG_MAX_LLC_COLORS_ORDER, base-2 exponent - moved Kconfig options to common/Kconfig - don't crash if computed max_nr_colors is too large v7: - SUPPORT.md changes added to this patch - extended documentation to better address applicability of cache coloring - "llc-nr-ways" and "llc-size" params introduced in favor of "llc-way-size" - moved dump_llc_coloring_info() call in 'm' keyhandler (pagealloc_info()) v6: - moved almost all code in common - moved documentation in this patch - reintroduced range for CONFIG_NR_LLC_COLORS - reintroduced some stub functions to reduce the number of checks on llc_coloring_enabled - moved domain_llc_coloring_free() in same patch where allocation happens - turned "d->llc_colors" to pointer-to-const - llc_coloring_init() now returns void and panics if errors are found v5: - used - instead of _ for filenames - removed domain_create_llc_colored() - removed stub functions - coloring domain fields are now #ifdef protected v4: - Kconfig options moved to xen/arch - removed range for CONFIG_NR_LLC_COLORS - added "llc_coloring_enabled" global to later implement the boot-time switch - added domain_create_llc_colored() to be able to pass colors - added is_domain_llc_colored() macro --- SUPPORT.md | 7 ++ docs/index.rst | 1 + docs/misc/cache-coloring.rst | 116 ++++++++++++++++++++++++++++++ docs/misc/xen-command-line.pandoc | 37 ++++++++++ xen/common/Kconfig | 21 ++++++ xen/common/Makefile | 1 + xen/common/keyhandler.c | 3 + xen/common/llc-coloring.c | 115 +++++++++++++++++++++++++++++ xen/common/page_alloc.c | 3 + xen/include/xen/llc-coloring.h | 37 ++++++++++ xen/include/xen/sched.h | 5 ++ 11 files changed, 346 insertions(+) create mode 100644 docs/misc/cache-coloring.rst create mode 100644 xen/common/llc-coloring.c create mode 100644 xen/include/xen/llc-coloring.h diff --git a/SUPPORT.md b/SUPPORT.md index XXXXXXX..XXXXXXX 100644 --- a/SUPPORT.md +++ b/SUPPORT.md @@ -XXX,XX +XXX,XX @@ by maintaining multiple physical to machine (p2m) memory mappings. Status, x86 HVM: Tech Preview Status, ARM: Tech Preview +### Cache coloring + +Allows to reserve Last Level Cache (LLC) partitions for Dom0, DomUs and Xen +itself. + + Status, Arm64: Experimental + ## Resource Management ### CPU Pools diff --git a/docs/index.rst b/docs/index.rst index XXXXXXX..XXXXXXX 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -XXX,XX +XXX,XX @@ Documents in need of some rearranging. designs/launch/hyperlaunch-devicetree misc/xen-makefiles/makefiles misra/index + misc/cache-coloring Miscellanea diff --git a/docs/misc/cache-coloring.rst b/docs/misc/cache-coloring.rst new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/docs/misc/cache-coloring.rst @@ -XXX,XX +XXX,XX @@ +Xen cache coloring user guide +============================= + +The cache coloring support in Xen allows to reserve Last Level Cache (LLC) +partitions for Dom0, DomUs and Xen itself. Currently only ARM64 is supported. +Cache coloring realizes per-set cache partitioning in software and is applicable +to shared LLCs as implemented in Cortex-A53, Cortex-A72 and similar CPUs. + +To compile LLC coloring support set ``CONFIG_LLC_COLORING=y``. + +If needed, change the maximum number of colors with +``CONFIG_LLC_COLORS_ORDER=<n>``. + +Runtime configuration is done via `Command line parameters`_. + +Background +********** + +Cache hierarchy of a modern multi-core CPU typically has first levels dedicated +to each core (hence using multiple cache units), while the last level is shared +among all of them. Such configuration implies that memory operations on one +core (e.g. running a DomU) are able to generate interference on another core +(e.g. hosting another DomU). Cache coloring realizes per-set cache-partitioning +in software and mitigates this, guaranteeing more predictable performances for +memory accesses. +Software-based cache coloring is particularly useful in those situations where +no hardware mechanisms (e.g., DSU-based way partitioning) are available to +partition caches. This is the case for e.g., Cortex-A53, A57 and A72 CPUs that +feature a L2 LLC cache shared among all cores. + +The key concept underlying cache coloring is a fragmentation of the memory +space into a set of sub-spaces called colors that are mapped to disjoint cache +partitions. Technically, the whole memory space is first divided into a number +of subsequent regions. Then each region is in turn divided into a number of +subsequent sub-colors. The generic i-th color is then obtained by all the +i-th sub-colors in each region. + +:: + + Region j Region j+1 + ..................... ............ + . . . + . . + _ _ _______________ _ _____________________ _ _ + | | | | | | | + | c_0 | c_1 | | c_n | c_0 | c_1 | + _ _ _|_____|_____|_ _ _|_____|_____|_____|_ _ _ + : : + : :... ... . + : color 0 + :........................... ... . + : + . . ..................................: + +How colors are actually defined depends on the function that maps memory to +cache lines. In case of physically-indexed, physically-tagged caches with linear +mapping, the set index is found by extracting some contiguous bits from the +physical address. This allows colors to be defined as shown in figure: they +appear in memory as subsequent blocks of equal size and repeats themselves after +``n`` different colors, where ``n`` is the total number of colors. + +If some kind of bit shuffling appears in the mapping function, then colors +assume a different layout in memory. Those kind of caches aren't supported by +the current implementation. + +**Note**: Finding the exact cache mapping function can be a really difficult +task since it's not always documented in the CPU manual. As said Cortex-A53, A57 +and A72 are known to work with the current implementation. + +How to compute the number of colors +################################### + +Given the linear mapping from physical memory to cache lines for granted, the +number of available colors for a specific platform is computed using three +parameters: + +- the size of the LLC. +- the number of the LLC ways. +- the page size used by Xen. + +The first two parameters can be found in the processor manual, while the third +one is the minimum mapping granularity. Dividing the cache size by the number of +its ways we obtain the size of a way. Dividing this number by the page size, +the number of total cache colors is found. So for example an Arm Cortex-A53 +with a 16-ways associative 1 MiB LLC can isolate up to 16 colors when pages are +4 KiB in size. + +Effective colors assignment +########################### + +When assigning colors, if one wants to avoid cache interference between two +domains, different colors needs to be used for their memory. + +Command line parameters +*********************** + +Specific documentation is available at `docs/misc/xen-command-line.pandoc`. + ++----------------------+-------------------------------+ +| **Parameter** | **Description** | ++----------------------+-------------------------------+ +| ``llc-coloring`` | Enable coloring at runtime | ++----------------------+-------------------------------+ +| ``llc-size`` | Set the LLC size | ++----------------------+-------------------------------+ +| ``llc-nr-ways`` | Set the LLC number of ways | ++----------------------+-------------------------------+ + +Auto-probing of LLC specs +######################### + +LLC size and number of ways are probed automatically by default. + +LLC specs can be manually set via the above command line parameters. This +bypasses any auto-probing and it's used to overcome failing situations, such as +flawed probing logic, or for debugging/testing purposes. diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc index XXXXXXX..XXXXXXX 100644 --- a/docs/misc/xen-command-line.pandoc +++ b/docs/misc/xen-command-line.pandoc @@ -XXX,XX +XXX,XX @@ This option is intended for debugging purposes only. Enable MSR_DEBUGCTL.LBR in hypervisor context to be able to dump the Last Interrupt/Exception To/From record with other registers. +### llc-coloring (arm64) +> `= <boolean>` + +> Default: `false` + +Flag to enable or disable LLC coloring support at runtime. This option is +available only when `CONFIG_LLC_COLORING` is enabled. See the general +cache coloring documentation for more info. + +### llc-nr-ways (arm64) +> `= <integer>` + +> Default: `Obtained from hardware` + +Specify the number of ways of the Last Level Cache. This option is available +only when `CONFIG_LLC_COLORING` is enabled. LLC size and number of ways are used +to find the number of supported cache colors. By default the value is +automatically computed by probing the hardware, but in case of specific needs, +it can be manually set. Those include failing probing and debugging/testing +purposes so that it's possible to emulate platforms with different number of +supported colors. If set, also "llc-size" must be set, otherwise the default +will be used. Note that using both options implies "llc-coloring=on". + +### llc-size (arm64) +> `= <size>` + +> Default: `Obtained from hardware` + +Specify the size of the Last Level Cache. This option is available only when +`CONFIG_LLC_COLORING` is enabled. LLC size and number of ways are used to find +the number of supported cache colors. By default the value is automatically +computed by probing the hardware, but in case of specific needs, it can be +manually set. Those include failing probing and debugging/testing purposes so +that it's possible to emulate platforms with different number of supported +colors. If set, also "llc-nr-ways" must be set, otherwise the default will be +used. Note that using both options implies "llc-coloring=on". + ### lock-depth-size > `= <integer>` diff --git a/xen/common/Kconfig b/xen/common/Kconfig index XXXXXXX..XXXXXXX 100644 --- a/xen/common/Kconfig +++ b/xen/common/Kconfig @@ -XXX,XX +XXX,XX @@ config HAS_IOPORTS config HAS_KEXEC bool +config HAS_LLC_COLORING + bool + config HAS_PIRQ bool @@ -XXX,XX +XXX,XX @@ config TRACEBUFFER to be collected at run time for debugging or performance analysis. Memory and execution overhead when not active is minimal. +config LLC_COLORING + bool "Last Level Cache (LLC) coloring" if EXPERT + depends on HAS_LLC_COLORING + +config LLC_COLORS_ORDER + int "Maximum number of LLC colors (base-2 exponent)" + range 1 10 + default 7 + depends on LLC_COLORING + help + Controls the build-time size of various arrays associated with LLC + coloring. The value is a base-2 exponent. Refer to cache coloring + documentation for how to compute the number of colors supported by the + platform. This is only an upper bound. The runtime value is autocomputed + or manually set via cmdline parameters. + The default value corresponds to an 8 MiB 16-ways LLC, which should be + more than what's needed in the general case. + endmenu diff --git a/xen/common/Makefile b/xen/common/Makefile index XXXXXXX..XXXXXXX 100644 --- a/xen/common/Makefile +++ b/xen/common/Makefile @@ -XXX,XX +XXX,XX @@ obj-y += keyhandler.o obj-$(CONFIG_KEXEC) += kexec.o obj-$(CONFIG_KEXEC) += kimage.o obj-$(CONFIG_LIVEPATCH) += livepatch.o livepatch_elf.o +obj-$(CONFIG_LLC_COLORING) += llc-coloring.o obj-$(CONFIG_MEM_ACCESS) += mem_access.o obj-y += memory.o obj-y += multicall.o diff --git a/xen/common/keyhandler.c b/xen/common/keyhandler.c index XXXXXXX..XXXXXXX 100644 --- a/xen/common/keyhandler.c +++ b/xen/common/keyhandler.c @@ -XXX,XX +XXX,XX @@ #include <asm/regs.h> #include <xen/delay.h> #include <xen/keyhandler.h> +#include <xen/llc-coloring.h> #include <xen/param.h> #include <xen/sections.h> #include <xen/shutdown.h> @@ -XXX,XX +XXX,XX @@ static void cf_check dump_domains(unsigned char key) arch_dump_domain_info(d); + domain_dump_llc_colors(d); + rangeset_domain_printk(d); dump_pageframe_info(d); diff --git a/xen/common/llc-coloring.c b/xen/common/llc-coloring.c new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/xen/common/llc-coloring.c @@ -XXX,XX +XXX,XX @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Last Level Cache (LLC) coloring common code + * + * Copyright (C) 2024, Advanced Micro Devices, Inc. + * Copyright (C) 2024, Minerva Systems SRL + */ +#include <xen/keyhandler.h> +#include <xen/llc-coloring.h> +#include <xen/param.h> + +#define NR_LLC_COLORS (1U << CONFIG_LLC_COLORS_ORDER) + +static bool __ro_after_init llc_coloring_enabled; +boolean_param("llc-coloring", llc_coloring_enabled); + +static unsigned int __initdata llc_size; +size_param("llc-size", llc_size); +static unsigned int __initdata llc_nr_ways; +integer_param("llc-nr-ways", llc_nr_ways); +/* Number of colors available in the LLC */ +static unsigned int __ro_after_init max_nr_colors; + +static void print_colors(const unsigned int colors[], unsigned int num_colors) +{ + unsigned int i; + + printk("{ "); + for ( i = 0; i < num_colors; i++ ) + { + unsigned int start = colors[i], end = start; + + printk("%u", start); + + for ( ; i < num_colors - 1 && end + 1 == colors[i + 1]; i++, end++ ) + ; + + if ( start != end ) + printk("-%u", end); + + if ( i < num_colors - 1 ) + printk(", "); + } + printk(" }\n"); +} + +void __init llc_coloring_init(void) +{ + unsigned int way_size; + + if ( llc_size && llc_nr_ways ) + { + llc_coloring_enabled = true; + way_size = llc_size / llc_nr_ways; + } + else if ( !llc_coloring_enabled ) + return; + else + { + way_size = get_llc_way_size(); + if ( !way_size ) + panic("LLC probing failed and 'llc-size' or 'llc-nr-ways' missing\n"); + } + + if ( way_size & ~PAGE_MASK ) + panic("LLC way size must be a multiple of PAGE_SIZE\n"); + + /* + * The maximum number of colors must be a power of 2 in order to correctly + * map them to bits of an address. + */ + max_nr_colors = way_size >> PAGE_SHIFT; + + if ( max_nr_colors & (max_nr_colors - 1) ) + panic("Number of LLC colors (%u) isn't a power of 2\n", max_nr_colors); + + if ( max_nr_colors > NR_LLC_COLORS ) + { + printk(XENLOG_WARNING + "Number of LLC colors (%u) too big. Using configured max %u\n", + max_nr_colors, NR_LLC_COLORS); + max_nr_colors = NR_LLC_COLORS; + } else if ( max_nr_colors < 2 ) + panic("Number of LLC colors %u < 2\n", max_nr_colors); + + arch_llc_coloring_init(); +} + +void dump_llc_coloring_info(void) +{ + if ( !llc_coloring_enabled ) + return; + + printk("LLC coloring info:\n"); + printk(" Number of LLC colors supported: %u\n", max_nr_colors); +} + +void domain_dump_llc_colors(const struct domain *d) +{ + if ( !llc_coloring_enabled ) + return; + + printk("%u LLC colors: ", d->num_llc_colors); + print_colors(d->llc_colors, d->num_llc_colors); +} + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index XXXXXXX..XXXXXXX 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -XXX,XX +XXX,XX @@ #include <xen/irq.h> #include <xen/keyhandler.h> #include <xen/lib.h> +#include <xen/llc-coloring.h> #include <xen/mm.h> #include <xen/nodemask.h> #include <xen/numa.h> @@ -XXX,XX +XXX,XX @@ static void cf_check pagealloc_info(unsigned char key) } printk(" Dom heap: %lukB free\n", total << (PAGE_SHIFT-10)); + + dump_llc_coloring_info(); } static __init int cf_check pagealloc_keyhandler_init(void) diff --git a/xen/include/xen/llc-coloring.h b/xen/include/xen/llc-coloring.h new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/xen/include/xen/llc-coloring.h @@ -XXX,XX +XXX,XX @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Last Level Cache (LLC) coloring common header + * + * Copyright (C) 2024, Advanced Micro Devices, Inc. + * Copyright (C) 2024, Minerva Systems SRL + */ +#ifndef __COLORING_H__ +#define __COLORING_H__ + +#include <xen/sched.h> +#include <public/domctl.h> + +#ifdef CONFIG_LLC_COLORING +void llc_coloring_init(void); +void dump_llc_coloring_info(void); +void domain_dump_llc_colors(const struct domain *d); +#else +static inline void llc_coloring_init(void) {} +static inline void dump_llc_coloring_info(void) {} +static inline void domain_dump_llc_colors(const struct domain *d) {} +#endif + +unsigned int get_llc_way_size(void); +void arch_llc_coloring_init(void); + +#endif /* __COLORING_H__ */ + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index XXXXXXX..XXXXXXX 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -XXX,XX +XXX,XX @@ struct domain /* Holding CDF_* constant. Internal flags for domain creation. */ unsigned int cdf; + +#ifdef CONFIG_LLC_COLORING + unsigned int num_llc_colors; + const unsigned int *llc_colors; +#endif }; static inline struct page_list_head *page_to_list( -- 2.43.0 LLC coloring needs to know the last level cache layout in order to make the best use of it. This can be probed by inspecting the CLIDR_EL1 register, so the Last Level is defined as the last level visible by this register. Note that this excludes system caches in some platforms. Static memory allocation and cache coloring are incompatible because static memory can't be guaranteed to use only colors assigned to the domain. Panic during DomUs creation when both are enabled. Based on original work from: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Carlo Nonato <carlo.nonato@minervasys.tech> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- v10: - moved CONFIG_NUMA check in arch/arm/Kconfig v9: - no changes v8: - no changes v7: - only minor changes v6: - get_llc_way_size() now checks for at least separate I/D caches v5: - used - instead of _ for filenames - moved static-mem check in this patch - moved dom0 colors parsing in next patch - moved color allocation and configuration in next patch - moved check_colors() in next patch - colors are now printed in short form v4: - added "llc-coloring" cmdline option for the boot-time switch - dom0 colors are now checked during domain init as for any other domain - fixed processor.h masks bit width - check for overflow in parse_color_config() - check_colors() now checks also that colors are sorted and unique --- docs/misc/cache-coloring.rst | 14 +++++ xen/arch/arm/Kconfig | 1 + xen/arch/arm/Makefile | 1 + xen/arch/arm/dom0less-build.c | 6 +++ xen/arch/arm/include/asm/processor.h | 16 ++++++ xen/arch/arm/llc-coloring.c | 78 ++++++++++++++++++++++++++++ xen/arch/arm/setup.c | 3 ++ xen/common/llc-coloring.c | 2 +- xen/include/xen/llc-coloring.h | 4 ++ 9 files changed, 124 insertions(+), 1 deletion(-) create mode 100644 xen/arch/arm/llc-coloring.c diff --git a/docs/misc/cache-coloring.rst b/docs/misc/cache-coloring.rst index XXXXXXX..XXXXXXX 100644 --- a/docs/misc/cache-coloring.rst +++ b/docs/misc/cache-coloring.rst @@ -XXX,XX +XXX,XX @@ Auto-probing of LLC specs LLC size and number of ways are probed automatically by default. +In the Arm implementation, this is done by inspecting the CLIDR_EL1 register. +This means that other system caches that aren't visible there are ignored. + LLC specs can be manually set via the above command line parameters. This bypasses any auto-probing and it's used to overcome failing situations, such as flawed probing logic, or for debugging/testing purposes. + +Known issues and limitations +**************************** + +"xen,static-mem" isn't supported when coloring is enabled +######################################################### + +In the domain configuration, "xen,static-mem" allows memory to be statically +allocated to the domain. This isn't possible when LLC coloring is enabled, +because that memory can't be guaranteed to use only colors assigned to the +domain. diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/Kconfig +++ b/xen/arch/arm/Kconfig @@ -XXX,XX +XXX,XX @@ config ARM_64 depends on !ARM_32 select 64BIT select HAS_FAST_MULTIPLY + select HAS_LLC_COLORING if !NUMA config ARM def_bool y diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/Makefile +++ b/xen/arch/arm/Makefile @@ -XXX,XX +XXX,XX @@ obj-$(CONFIG_IOREQ_SERVER) += ioreq.o obj-y += irq.o obj-y += kernel.init.o obj-$(CONFIG_LIVEPATCH) += livepatch.o +obj-$(CONFIG_LLC_COLORING) += llc-coloring.o obj-$(CONFIG_MEM_ACCESS) += mem_access.o obj-y += mm.o obj-y += monitor.o diff --git a/xen/arch/arm/dom0less-build.c b/xen/arch/arm/dom0less-build.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/dom0less-build.c +++ b/xen/arch/arm/dom0less-build.c @@ -XXX,XX +XXX,XX @@ #include <xen/grant_table.h> #include <xen/iocap.h> #include <xen/libfdt/libfdt.h> +#include <xen/llc-coloring.h> #include <xen/sched.h> #include <xen/serial.h> #include <xen/sizes.h> @@ -XXX,XX +XXX,XX @@ void __init create_domUs(void) panic("No more domain IDs available\n"); if ( dt_find_property(node, "xen,static-mem", NULL) ) + { + if ( llc_coloring_enabled ) + panic("LLC coloring and static memory are incompatible\n"); + flags |= CDF_staticmem; + } if ( dt_property_read_bool(node, "direct-map") ) { diff --git a/xen/arch/arm/include/asm/processor.h b/xen/arch/arm/include/asm/processor.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/processor.h +++ b/xen/arch/arm/include/asm/processor.h @@ -XXX,XX +XXX,XX @@ #define CTR_IDC_SHIFT 28 #define CTR_DIC_SHIFT 29 +/* CCSIDR Current Cache Size ID Register */ +#define CCSIDR_LINESIZE_MASK _AC(0x7, UL) +#define CCSIDR_NUMSETS_SHIFT 13 +#define CCSIDR_NUMSETS_MASK _AC(0x3fff, UL) +#define CCSIDR_NUMSETS_SHIFT_FEAT_CCIDX 32 +#define CCSIDR_NUMSETS_MASK_FEAT_CCIDX _AC(0xffffff, UL) + +/* CSSELR Cache Size Selection Register */ +#define CSSELR_LEVEL_MASK _AC(0x7, UL) +#define CSSELR_LEVEL_SHIFT 1 + +/* CLIDR Cache Level ID Register */ +#define CLIDR_CTYPEn_SHIFT(n) (3 * ((n) - 1)) +#define CLIDR_CTYPEn_MASK _AC(0x7, UL) +#define CLIDR_CTYPEn_LEVELS 7 + #define ICACHE_POLICY_VPIPT 0 #define ICACHE_POLICY_AIVIVT 1 #define ICACHE_POLICY_VIPT 2 diff --git a/xen/arch/arm/llc-coloring.c b/xen/arch/arm/llc-coloring.c new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/xen/arch/arm/llc-coloring.c @@ -XXX,XX +XXX,XX @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Last Level Cache (LLC) coloring support for ARM + * + * Copyright (C) 2024, Advanced Micro Devices, Inc. + * Copyright (C) 2024, Minerva Systems SRL + */ +#include <xen/llc-coloring.h> +#include <xen/types.h> + +#include <asm/processor.h> +#include <asm/sysregs.h> + +/* Return the LLC way size by probing the hardware */ +unsigned int __init get_llc_way_size(void) +{ + register_t ccsidr_el1; + register_t clidr_el1 = READ_SYSREG(CLIDR_EL1); + register_t csselr_el1 = READ_SYSREG(CSSELR_EL1); + register_t id_aa64mmfr2_el1 = READ_SYSREG(ID_AA64MMFR2_EL1); + uint32_t ccsidr_numsets_shift = CCSIDR_NUMSETS_SHIFT; + uint32_t ccsidr_numsets_mask = CCSIDR_NUMSETS_MASK; + unsigned int n, line_size, num_sets; + + for ( n = CLIDR_CTYPEn_LEVELS; n != 0; n-- ) + { + uint8_t ctype_n = (clidr_el1 >> CLIDR_CTYPEn_SHIFT(n)) & + CLIDR_CTYPEn_MASK; + + /* Unified cache (see Arm ARM DDI 0487J.a D19.2.27) */ + if ( ctype_n == 0b100 ) + break; + } + + if ( n == 0 ) + return 0; + + WRITE_SYSREG((n - 1) << CSSELR_LEVEL_SHIFT, CSSELR_EL1); + isb(); + + ccsidr_el1 = READ_SYSREG(CCSIDR_EL1); + + /* Arm ARM: (Log2(Number of bytes in cache line)) - 4 */ + line_size = 1U << ((ccsidr_el1 & CCSIDR_LINESIZE_MASK) + 4); + + /* If FEAT_CCIDX is enabled, CCSIDR_EL1 has a different bit layout */ + if ( (id_aa64mmfr2_el1 >> ID_AA64MMFR2_CCIDX_SHIFT) & 0x7 ) + { + ccsidr_numsets_shift = CCSIDR_NUMSETS_SHIFT_FEAT_CCIDX; + ccsidr_numsets_mask = CCSIDR_NUMSETS_MASK_FEAT_CCIDX; + } + + /* Arm ARM: (Number of sets in cache) - 1 */ + num_sets = ((ccsidr_el1 >> ccsidr_numsets_shift) & ccsidr_numsets_mask) + 1; + + printk(XENLOG_INFO "LLC found: L%u (line size: %u bytes, sets num: %u)\n", + n, line_size, num_sets); + + /* Restore value in CSSELR_EL1 */ + WRITE_SYSREG(csselr_el1, CSSELR_EL1); + isb(); + + return line_size * num_sets; +} + +void __init arch_llc_coloring_init(void) +{ +} + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/setup.c +++ b/xen/arch/arm/setup.c @@ -XXX,XX +XXX,XX @@ #include <xen/device_tree.h> #include <xen/domain_page.h> #include <xen/grant_table.h> +#include <xen/llc-coloring.h> #include <xen/types.h> #include <xen/string.h> #include <xen/serial.h> @@ -XXX,XX +XXX,XX @@ void asmlinkage __init start_xen(unsigned long fdt_paddr) printk("Command line: %s\n", cmdline); cmdline_parse(cmdline); + llc_coloring_init(); + setup_mm(); vm_init(); diff --git a/xen/common/llc-coloring.c b/xen/common/llc-coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/common/llc-coloring.c +++ b/xen/common/llc-coloring.c @@ -XXX,XX +XXX,XX @@ #define NR_LLC_COLORS (1U << CONFIG_LLC_COLORS_ORDER) -static bool __ro_after_init llc_coloring_enabled; +bool __ro_after_init llc_coloring_enabled; boolean_param("llc-coloring", llc_coloring_enabled); static unsigned int __initdata llc_size; diff --git a/xen/include/xen/llc-coloring.h b/xen/include/xen/llc-coloring.h index XXXXXXX..XXXXXXX 100644 --- a/xen/include/xen/llc-coloring.h +++ b/xen/include/xen/llc-coloring.h @@ -XXX,XX +XXX,XX @@ #include <public/domctl.h> #ifdef CONFIG_LLC_COLORING +extern bool llc_coloring_enabled; + void llc_coloring_init(void); void dump_llc_coloring_info(void); void domain_dump_llc_colors(const struct domain *d); #else +#define llc_coloring_enabled false + static inline void llc_coloring_init(void) {} static inline void dump_llc_coloring_info(void) {} static inline void domain_dump_llc_colors(const struct domain *d) {} -- 2.43.0 Cache coloring requires Dom0 not to be direct-mapped because of its non contiguous mapping nature, so allocate_memory() is needed in this case. 8d2c3ab18cc1 ("arm/dom0less: put dom0less feature code in a separate module") moved allocate_memory() in dom0less_build.c. In order to use it in Dom0 construction bring it back to domain_build.c and declare it in domain_build.h. Take the opportunity to adapt the implementation of allocate_memory() so that it uses the host layout when called on the hwdom, via find_unallocated_memory(). Signed-off-by: Carlo Nonato <carlo.nonato@minervasys.tech> --- v10: - fixed a compilation bug that happened when dom0less support was disabled v9: - no changes v8: - patch adapted to new changes to allocate_memory() v7: - allocate_memory() now uses the host layout when called on the hwdom v6: - new patch --- xen/arch/arm/dom0less-build.c | 44 ------------ xen/arch/arm/domain_build.c | 96 +++++++++++++++++++++++-- xen/arch/arm/include/asm/domain_build.h | 1 + 3 files changed, 93 insertions(+), 48 deletions(-) diff --git a/xen/arch/arm/dom0less-build.c b/xen/arch/arm/dom0less-build.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/dom0less-build.c +++ b/xen/arch/arm/dom0less-build.c @@ -XXX,XX +XXX,XX @@ bool __init is_dom0less_mode(void) return ( !dom0found && domUfound ); } -static void __init allocate_memory(struct domain *d, struct kernel_info *kinfo) -{ - struct membanks *mem = kernel_info_get_mem(kinfo); - unsigned int i; - paddr_t bank_size; - - printk(XENLOG_INFO "Allocating mappings totalling %ldMB for %pd:\n", - /* Don't want format this as PRIpaddr (16 digit hex) */ - (unsigned long)(kinfo->unassigned_mem >> 20), d); - - mem->nr_banks = 0; - bank_size = MIN(GUEST_RAM0_SIZE, kinfo->unassigned_mem); - if ( !allocate_bank_memory(kinfo, gaddr_to_gfn(GUEST_RAM0_BASE), - bank_size) ) - goto fail; - - bank_size = MIN(GUEST_RAM1_SIZE, kinfo->unassigned_mem); - if ( !allocate_bank_memory(kinfo, gaddr_to_gfn(GUEST_RAM1_BASE), - bank_size) ) - goto fail; - - if ( kinfo->unassigned_mem ) - goto fail; - - for( i = 0; i < mem->nr_banks; i++ ) - { - printk(XENLOG_INFO "%pd BANK[%d] %#"PRIpaddr"-%#"PRIpaddr" (%ldMB)\n", - d, - i, - mem->bank[i].start, - mem->bank[i].start + mem->bank[i].size, - /* Don't want format this as PRIpaddr (16 digit hex) */ - (unsigned long)(mem->bank[i].size >> 20)); - } - - return; - -fail: - panic("Failed to allocate requested domain memory." - /* Don't want format this as PRIpaddr (16 digit hex) */ - " %ldKB unallocated. Fix the VMs configurations.\n", - (unsigned long)kinfo->unassigned_mem >> 10); -} - #ifdef CONFIG_VGICV2 static int __init make_gicv2_domU_node(struct kernel_info *kinfo) { diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/domain_build.c +++ b/xen/arch/arm/domain_build.c @@ -XXX,XX +XXX,XX @@ static void __init allocate_memory_11(struct domain *d, } } -#ifdef CONFIG_DOM0LESS_BOOT bool __init allocate_domheap_memory(struct domain *d, paddr_t tot_size, alloc_domheap_mem_cb cb, void *extra) { @@ -XXX,XX +XXX,XX @@ bool __init allocate_bank_memory(struct kernel_info *kinfo, gfn_t sgfn, return true; } -#endif /* * When PCI passthrough is available we want to keep the @@ -XXX,XX +XXX,XX @@ out: return res; } +void __init allocate_memory(struct domain *d, struct kernel_info *kinfo) +{ + struct membanks *mem = kernel_info_get_mem(kinfo); + unsigned int i, nr_banks = 2; + paddr_t bank_start, bank_size; + struct membanks *hwdom_ext_regions = NULL; + + printk(XENLOG_INFO "Allocating mappings totalling %ldMB for %pd:\n", + /* Don't want format this as PRIpaddr (16 digit hex) */ + (unsigned long)(kinfo->unassigned_mem >> 20), d); + + mem->nr_banks = 0; + /* + * Use host memory layout for hwdom. Only case for this is when LLC coloring + * is enabled. + */ + if ( is_hardware_domain(d) ) + { + ASSERT(llc_coloring_enabled); + + hwdom_ext_regions = xzalloc_flex_struct(struct membanks, bank, + NR_MEM_BANKS); + if ( !hwdom_ext_regions ) + goto fail; + hwdom_ext_regions->max_banks = NR_MEM_BANKS; + + if ( find_unallocated_memory(kinfo, hwdom_ext_regions) ) + goto fail; + + nr_banks = hwdom_ext_regions->nr_banks; + } + + for ( i = 0; kinfo->unassigned_mem > 0 && nr_banks > 0; i++, nr_banks-- ) + { + if ( is_hardware_domain(d) ) + { + bank_start = hwdom_ext_regions->bank[i].start; + bank_size = hwdom_ext_regions->bank[i].size; + + if ( bank_size < min_t(paddr_t, kinfo->unassigned_mem, MB(128)) ) + continue; + } + else + { + if ( i == 0 ) + { + bank_start = GUEST_RAM0_BASE; + bank_size = GUEST_RAM0_SIZE; + } + else if ( i == 1 ) + { + bank_start = GUEST_RAM1_BASE; + bank_size = GUEST_RAM1_SIZE; + } + else + goto fail; + } + + bank_size = MIN(bank_size, kinfo->unassigned_mem); + if ( !allocate_bank_memory(kinfo, gaddr_to_gfn(bank_start), bank_size) ) + goto fail; + } + + if ( kinfo->unassigned_mem ) + goto fail; + + for( i = 0; i < mem->nr_banks; i++ ) + { + printk(XENLOG_INFO "%pd BANK[%d] %#"PRIpaddr"-%#"PRIpaddr" (%ldMB)\n", + d, + i, + mem->bank[i].start, + mem->bank[i].start + mem->bank[i].size, + /* Don't want format this as PRIpaddr (16 digit hex) */ + (unsigned long)(mem->bank[i].size >> 20)); + } + + xfree(hwdom_ext_regions); + return; + +fail: + panic("Failed to allocate requested domain memory." + /* Don't want format this as PRIpaddr (16 digit hex) */ + " %ldKB unallocated. Fix the VMs configurations.\n", + (unsigned long)kinfo->unassigned_mem >> 10); +} + static int __init handle_pci_range(const struct dt_device_node *dev, uint64_t addr, uint64_t len, void *data) { @@ -XXX,XX +XXX,XX @@ int __init make_hypervisor_node(struct domain *d, ext_regions->max_banks = NR_MEM_BANKS; - if ( is_domain_direct_mapped(d) ) + if ( domain_use_host_layout(d) ) { if ( !is_iommu_enabled(d) ) res = find_unallocated_memory(kinfo, ext_regions); @@ -XXX,XX +XXX,XX @@ static int __init construct_dom0(struct domain *d) /* type must be set before allocate_memory */ d->arch.type = kinfo.type; #endif - allocate_memory_11(d, &kinfo); + if ( is_domain_direct_mapped(d) ) + allocate_memory_11(d, &kinfo); + else + allocate_memory(d, &kinfo); find_gnttab_region(d, &kinfo); rc = process_shm_chosen(d, &kinfo); diff --git a/xen/arch/arm/include/asm/domain_build.h b/xen/arch/arm/include/asm/domain_build.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/domain_build.h +++ b/xen/arch/arm/include/asm/domain_build.h @@ -XXX,XX +XXX,XX @@ bool allocate_domheap_memory(struct domain *d, paddr_t tot_size, alloc_domheap_mem_cb cb, void *extra); bool allocate_bank_memory(struct kernel_info *kinfo, gfn_t sgfn, paddr_t tot_size); +void allocate_memory(struct domain *d, struct kernel_info *kinfo); int construct_domain(struct domain *d, struct kernel_info *kinfo); int domain_fdt_begin_node(void *fdt, const char *name, uint64_t unit); int make_chosen_node(const struct kernel_info *kinfo); -- 2.43.0 Add a command line parameter to allow the user to set the coloring configuration for Dom0. A common configuration syntax for cache colors is introduced and documented. Take the opportunity to also add: - default configuration notion. - function to check well-formed configurations. Direct mapping Dom0 isn't possible when coloring is enabled, so CDF_directmap flag is removed when creating it. Based on original work from: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Carlo Nonato <carlo.nonato@minervasys.tech> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> Reviewed-by: Jan Beulich <jbeulich@suse.com> --- v10: - fixed array type for colors parameter in check_colors() v9: - moved domain_llc_coloring_free() in next patch cause it's never used for dom0 v8: - added bound check on dom0_num_colors - default colors array set just once v7: - parse_color_config() doesn't accept leading/trailing commas anymore - removed alloc_colors() helper v6: - moved domain_llc_coloring_free() in this patch - removed domain_alloc_colors() in favor of a more explicit allocation - parse_color_config() now accepts the size of the array to be filled - allocate_memory() moved in another patch v5: - Carlo Nonato as the new author - moved dom0 colors parsing (parse_colors()) in this patch - added dom0_set_llc_colors() to set dom0 colors after creation - moved color allocation and checking in this patch - error handling when allocating color arrays - FIXME: copy pasted allocate_memory() cause it got moved v4: - dom0 colors are dynamically allocated as for any other domain (colors are duplicated in dom0_colors and in the new array, but logic is simpler) --- docs/misc/cache-coloring.rst | 29 ++++++++ docs/misc/xen-command-line.pandoc | 9 +++ xen/arch/arm/domain_build.c | 10 ++- xen/common/llc-coloring.c | 119 +++++++++++++++++++++++++++++- xen/include/xen/llc-coloring.h | 1 + 5 files changed, 166 insertions(+), 2 deletions(-) diff --git a/docs/misc/cache-coloring.rst b/docs/misc/cache-coloring.rst index XXXXXXX..XXXXXXX 100644 --- a/docs/misc/cache-coloring.rst +++ b/docs/misc/cache-coloring.rst @@ -XXX,XX +XXX,XX @@ Specific documentation is available at `docs/misc/xen-command-line.pandoc`. +----------------------+-------------------------------+ | ``llc-nr-ways`` | Set the LLC number of ways | +----------------------+-------------------------------+ +| ``dom0-llc-colors`` | Dom0 color configuration | ++----------------------+-------------------------------+ + +Colors selection format +*********************** + +Regardless of the memory pool that has to be colored (Xen, Dom0/DomUs), +the color selection can be expressed using the same syntax. In particular a +comma-separated list of colors or ranges of colors is used. +Ranges are hyphen-separated intervals (such as `0-4`) and are inclusive on both +sides. + +Note that: + +- no spaces are allowed between values. +- no overlapping ranges or duplicated colors are allowed. +- values must be written in ascending order. + +Examples: + ++-------------------+-----------------------------+ +| **Configuration** | **Actual selection** | ++-------------------+-----------------------------+ +| 1-2,5-8 | [1, 2, 5, 6, 7, 8] | ++-------------------+-----------------------------+ +| 4-8,10,11,12 | [4, 5, 6, 7, 8, 10, 11, 12] | ++-------------------+-----------------------------+ +| 0 | [0] | ++-------------------+-----------------------------+ Auto-probing of LLC specs ######################### diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc index XXXXXXX..XXXXXXX 100644 --- a/docs/misc/xen-command-line.pandoc +++ b/docs/misc/xen-command-line.pandoc @@ -XXX,XX +XXX,XX @@ Controls for the dom0 IOMMU setup. Specify a list of IO ports to be excluded from dom0 access. +### dom0-llc-colors (arm64) +> `= List of [ <integer> | <integer>-<integer> ]` + +> Default: `All available LLC colors` + +Specify dom0 LLC color configuration. This option is available only when +`CONFIG_LLC_COLORING` is enabled. If the parameter is not set, all available +colors are used. + ### dom0_max_vcpus Either: diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/domain_build.c +++ b/xen/arch/arm/domain_build.c @@ -XXX,XX +XXX,XX @@ #include <xen/init.h> #include <xen/compile.h> #include <xen/lib.h> +#include <xen/llc-coloring.h> #include <xen/mm.h> #include <xen/param.h> #include <xen/domain_page.h> @@ -XXX,XX +XXX,XX @@ void __init create_dom0(void) .max_maptrack_frames = -1, .grant_opts = XEN_DOMCTL_GRANT_version(opt_gnttab_max_version), }; + unsigned int flags = CDF_privileged; int rc; /* The vGIC for DOM0 is exactly emulating the hardware GIC */ @@ -XXX,XX +XXX,XX @@ void __init create_dom0(void) panic("SVE vector length error\n"); } - dom0 = domain_create(0, &dom0_cfg, CDF_privileged | CDF_directmap); + if ( !llc_coloring_enabled ) + flags |= CDF_directmap; + + dom0 = domain_create(0, &dom0_cfg, flags); if ( IS_ERR(dom0) ) panic("Error creating domain 0 (rc = %ld)\n", PTR_ERR(dom0)); + if ( llc_coloring_enabled && (rc = dom0_set_llc_colors(dom0)) ) + panic("Error initializing LLC coloring for domain 0 (rc = %d)", rc); + if ( alloc_dom0_vcpu0(dom0) == NULL ) panic("Error creating domain 0 vcpu0\n"); diff --git a/xen/common/llc-coloring.c b/xen/common/llc-coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/common/llc-coloring.c +++ b/xen/common/llc-coloring.c @@ -XXX,XX +XXX,XX @@ static unsigned int __initdata llc_nr_ways; integer_param("llc-nr-ways", llc_nr_ways); /* Number of colors available in the LLC */ static unsigned int __ro_after_init max_nr_colors; +/* Default coloring configuration */ +static unsigned int __ro_after_init default_colors[NR_LLC_COLORS]; + +static unsigned int __initdata dom0_colors[NR_LLC_COLORS]; +static unsigned int __initdata dom0_num_colors; + +/* + * Parse the coloring configuration given in the buf string, following the + * syntax below. + * + * COLOR_CONFIGURATION ::= COLOR | RANGE,...,COLOR | RANGE + * RANGE ::= COLOR-COLOR + * + * Example: "0,2-6,15-16" represents the set of colors: 0,2,3,4,5,6,15,16. + */ +static int __init parse_color_config(const char *buf, unsigned int colors[], + unsigned int max_num_colors, + unsigned int *num_colors) +{ + const char *s = buf; + + *num_colors = 0; + + while ( *s != '\0' ) + { + unsigned int color, start, end; + + start = simple_strtoul(s, &s, 0); + + if ( *s == '-' ) /* Range */ + { + s++; + end = simple_strtoul(s, &s, 0); + } + else /* Single value */ + end = start; + + if ( start > end || (end - start) > (UINT_MAX - *num_colors) || + (*num_colors + (end - start)) >= max_num_colors ) + return -EINVAL; + + /* Colors are range checked in check_colors() */ + for ( color = start; color <= end; color++ ) + colors[(*num_colors)++] = color; + + if ( *s == ',' ) + s++; + else if ( *s != '\0' ) + break; + } + + return *s ? -EINVAL : 0; +} + +static int __init parse_dom0_colors(const char *s) +{ + return parse_color_config(s, dom0_colors, ARRAY_SIZE(dom0_colors), + &dom0_num_colors); +} +custom_param("dom0-llc-colors", parse_dom0_colors); static void print_colors(const unsigned int colors[], unsigned int num_colors) { @@ -XXX,XX +XXX,XX @@ static void print_colors(const unsigned int colors[], unsigned int num_colors) printk(" }\n"); } +static bool __init check_colors(const unsigned int colors[], + unsigned int num_colors) +{ + unsigned int i; + + for ( i = 0; i < num_colors; i++ ) + { + if ( colors[i] >= max_nr_colors ) + { + printk(XENLOG_ERR "LLC color %u >= %u\n", colors[i], max_nr_colors); + return false; + } + } + + return true; +} + void __init llc_coloring_init(void) { - unsigned int way_size; + unsigned int way_size, i; if ( llc_size && llc_nr_ways ) { @@ -XXX,XX +XXX,XX @@ void __init llc_coloring_init(void) } else if ( max_nr_colors < 2 ) panic("Number of LLC colors %u < 2\n", max_nr_colors); + for ( i = 0; i < max_nr_colors; i++ ) + default_colors[i] = i; + arch_llc_coloring_init(); } @@ -XXX,XX +XXX,XX @@ void domain_dump_llc_colors(const struct domain *d) print_colors(d->llc_colors, d->num_llc_colors); } +static void __init domain_set_default_colors(struct domain *d) +{ + printk(XENLOG_WARNING + "LLC color config not found for %pd, using all colors\n", d); + + d->llc_colors = default_colors; + d->num_llc_colors = max_nr_colors; +} + +int __init dom0_set_llc_colors(struct domain *d) +{ + typeof(*dom0_colors) *colors; + + if ( !dom0_num_colors ) + { + domain_set_default_colors(d); + return 0; + } + + if ( dom0_num_colors > max_nr_colors || + !check_colors(dom0_colors, dom0_num_colors) ) + { + printk(XENLOG_ERR "%pd: bad LLC color config\n", d); + return -EINVAL; + } + + colors = xmalloc_array(typeof(*dom0_colors), dom0_num_colors); + if ( !colors ) + return -ENOMEM; + + memcpy(colors, dom0_colors, sizeof(*colors) * dom0_num_colors); + d->llc_colors = colors; + d->num_llc_colors = dom0_num_colors; + + return 0; +} + /* * Local variables: * mode: C diff --git a/xen/include/xen/llc-coloring.h b/xen/include/xen/llc-coloring.h index XXXXXXX..XXXXXXX 100644 --- a/xen/include/xen/llc-coloring.h +++ b/xen/include/xen/llc-coloring.h @@ -XXX,XX +XXX,XX @@ static inline void domain_dump_llc_colors(const struct domain *d) {} unsigned int get_llc_way_size(void); void arch_llc_coloring_init(void); +int dom0_set_llc_colors(struct domain *d); #endif /* __COLORING_H__ */ -- 2.43.0 Add a new domctl hypercall to allow the user to set LLC coloring configurations. Colors can be set only once, just after domain creation, since recoloring isn't supported. Based on original work from: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Carlo Nonato <carlo.nonato@minervasys.tech> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> Reviewed-by: Jan Beulich <jbeulich@suse.com> --- v10: - no changes v9: - minor printk message changes - moved domain_llc_coloring_free() in this patch v8: - fixed memory leak on error path of domain_set_llc_colors() v7: - -EOPNOTSUPP returned in case of hypercall called without llc_coloring_enabled - domain_set_llc_colors_domctl() renamed to domain_set_llc_colors() - added padding and input bound checks to domain_set_llc_colors() - removed alloc_colors() helper usage from domain_set_llc_colors() v6: - reverted the XEN_DOMCTL_INTERFACE_VERSION bump - reverted to uint32 for the guest handle - explicit padding added to the domctl struct - rewrote domain_set_llc_colors_domctl() to be more explicit v5: - added a new hypercall to set colors - uint for the guest handle v4: - updated XEN_DOMCTL_INTERFACE_VERSION --- xen/common/domain.c | 3 ++ xen/common/domctl.c | 10 +++++++ xen/common/llc-coloring.c | 55 ++++++++++++++++++++++++++++++++-- xen/include/public/domctl.h | 9 ++++++ xen/include/xen/llc-coloring.h | 4 +++ 5 files changed, 78 insertions(+), 3 deletions(-) diff --git a/xen/common/domain.c b/xen/common/domain.c index XXXXXXX..XXXXXXX 100644 --- a/xen/common/domain.c +++ b/xen/common/domain.c @@ -XXX,XX +XXX,XX @@ #include <xen/xenoprof.h> #include <xen/irq.h> #include <xen/argo.h> +#include <xen/llc-coloring.h> #include <asm/p2m.h> #include <asm/processor.h> #include <public/sched.h> @@ -XXX,XX +XXX,XX @@ void domain_destroy(struct domain *d) { BUG_ON(!d->is_dying); + domain_llc_coloring_free(d); + /* May be already destroyed, or get_domain() can race us. */ if ( atomic_cmpxchg(&d->refcnt, 0, DOMAIN_DESTROYED) != 0 ) return; diff --git a/xen/common/domctl.c b/xen/common/domctl.c index XXXXXXX..XXXXXXX 100644 --- a/xen/common/domctl.c +++ b/xen/common/domctl.c @@ -XXX,XX +XXX,XX @@ #include <xen/types.h> #include <xen/lib.h> +#include <xen/llc-coloring.h> #include <xen/err.h> #include <xen/mm.h> #include <xen/sched.h> @@ -XXX,XX +XXX,XX @@ long do_domctl(XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl) __HYPERVISOR_domctl, "h", u_domctl); break; + case XEN_DOMCTL_set_llc_colors: + if ( op->u.set_llc_colors.pad ) + ret = -EINVAL; + else if ( llc_coloring_enabled ) + ret = domain_set_llc_colors(d, &op->u.set_llc_colors); + else + ret = -EOPNOTSUPP; + break; + default: ret = arch_do_domctl(op, d, u_domctl); break; diff --git a/xen/common/llc-coloring.c b/xen/common/llc-coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/common/llc-coloring.c +++ b/xen/common/llc-coloring.c @@ -XXX,XX +XXX,XX @@ * Copyright (C) 2024, Advanced Micro Devices, Inc. * Copyright (C) 2024, Minerva Systems SRL */ +#include <xen/guest_access.h> #include <xen/keyhandler.h> #include <xen/llc-coloring.h> #include <xen/param.h> @@ -XXX,XX +XXX,XX @@ static void print_colors(const unsigned int colors[], unsigned int num_colors) printk(" }\n"); } -static bool __init check_colors(const unsigned int colors[], - unsigned int num_colors) +static bool check_colors(const unsigned int colors[], unsigned int num_colors) { unsigned int i; @@ -XXX,XX +XXX,XX @@ void domain_dump_llc_colors(const struct domain *d) print_colors(d->llc_colors, d->num_llc_colors); } -static void __init domain_set_default_colors(struct domain *d) +static void domain_set_default_colors(struct domain *d) { printk(XENLOG_WARNING "LLC color config not found for %pd, using all colors\n", d); @@ -XXX,XX +XXX,XX @@ int __init dom0_set_llc_colors(struct domain *d) return 0; } +int domain_set_llc_colors(struct domain *d, + const struct xen_domctl_set_llc_colors *config) +{ + unsigned int *colors; + + if ( d->num_llc_colors ) + return -EEXIST; + + if ( !config->num_llc_colors ) + { + domain_set_default_colors(d); + return 0; + } + + if ( config->num_llc_colors > max_nr_colors ) + return -EINVAL; + + colors = xmalloc_array(unsigned int, config->num_llc_colors); + if ( !colors ) + return -ENOMEM; + + if ( copy_from_guest(colors, config->llc_colors, config->num_llc_colors) ) + { + xfree(colors); + return -EFAULT; + } + + if ( !check_colors(colors, config->num_llc_colors) ) + { + printk(XENLOG_ERR "%pd: bad LLC color config\n", d); + xfree(colors); + return -EINVAL; + } + + d->llc_colors = colors; + d->num_llc_colors = config->num_llc_colors; + + return 0; +} + +void domain_llc_coloring_free(struct domain *d) +{ + if ( !llc_coloring_enabled || d->llc_colors == default_colors ) + return; + + /* free pointer-to-const using __va(__pa()) */ + xfree(__va(__pa(d->llc_colors))); +} + /* * Local variables: * mode: C diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h index XXXXXXX..XXXXXXX 100644 --- a/xen/include/public/domctl.h +++ b/xen/include/public/domctl.h @@ -XXX,XX +XXX,XX @@ struct xen_domctl_dt_overlay { }; #endif +struct xen_domctl_set_llc_colors { + /* IN LLC coloring parameters */ + uint32_t num_llc_colors; + uint32_t pad; + XEN_GUEST_HANDLE_64(uint32) llc_colors; +}; + struct xen_domctl { uint32_t cmd; #define XEN_DOMCTL_createdomain 1 @@ -XXX,XX +XXX,XX @@ struct xen_domctl { #define XEN_DOMCTL_set_paging_mempool_size 86 #define XEN_DOMCTL_dt_overlay 87 #define XEN_DOMCTL_gsi_permission 88 +#define XEN_DOMCTL_set_llc_colors 89 #define XEN_DOMCTL_gdbsx_guestmemio 1000 #define XEN_DOMCTL_gdbsx_pausevcpu 1001 #define XEN_DOMCTL_gdbsx_unpausevcpu 1002 @@ -XXX,XX +XXX,XX @@ struct xen_domctl { #if defined(__arm__) || defined(__aarch64__) struct xen_domctl_dt_overlay dt_overlay; #endif + struct xen_domctl_set_llc_colors set_llc_colors; uint8_t pad[128]; } u; }; diff --git a/xen/include/xen/llc-coloring.h b/xen/include/xen/llc-coloring.h index XXXXXXX..XXXXXXX 100644 --- a/xen/include/xen/llc-coloring.h +++ b/xen/include/xen/llc-coloring.h @@ -XXX,XX +XXX,XX @@ extern bool llc_coloring_enabled; void llc_coloring_init(void); void dump_llc_coloring_info(void); void domain_dump_llc_colors(const struct domain *d); +void domain_llc_coloring_free(struct domain *d); #else #define llc_coloring_enabled false static inline void llc_coloring_init(void) {} static inline void dump_llc_coloring_info(void) {} static inline void domain_dump_llc_colors(const struct domain *d) {} +static inline void domain_llc_coloring_free(struct domain *d) {} #endif unsigned int get_llc_way_size(void); void arch_llc_coloring_init(void); int dom0_set_llc_colors(struct domain *d); +int domain_set_llc_colors(struct domain *d, + const struct xen_domctl_set_llc_colors *config); #endif /* __COLORING_H__ */ -- 2.43.0 Add a new "llc_colors" parameter that defines the LLC color assignment for a domain. The user can specify one or more color ranges using the same syntax used everywhere else for color config described in the documentation. The parameter is defined as a list of strings that represent the color ranges. Documentation is also added. Based on original work from: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Carlo Nonato <carlo.nonato@minervasys.tech> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- v10: - no changes v9: - turned warning into error in case of coloring not enabled v8: - warn the user in case of coloring not supported at hypervisor level v7: - removed unneeded NULL check before xc_hypercall_buffer_free() in xc_domain_set_llc_colors() v6: - no edits v5: - added LIBXL_HAVE_BUILDINFO_LLC_COLORS - moved color configuration in xc_domain_set_llc_colors() cause of the new hypercall v4: - removed overlapping color ranges checks during parsing - moved hypercall buffer initialization in libxenctrl --- docs/man/xl.cfg.5.pod.in | 6 +++++ tools/include/libxl.h | 5 +++++ tools/include/xenctrl.h | 9 ++++++++ tools/libs/ctrl/xc_domain.c | 35 +++++++++++++++++++++++++++++ tools/libs/light/libxl_create.c | 18 +++++++++++++++ tools/libs/light/libxl_types.idl | 1 + tools/xl/xl_parse.c | 38 +++++++++++++++++++++++++++++++- 7 files changed, 111 insertions(+), 1 deletion(-) diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in index XXXXXXX..XXXXXXX 100644 --- a/docs/man/xl.cfg.5.pod.in +++ b/docs/man/xl.cfg.5.pod.in @@ -XXX,XX +XXX,XX @@ raised. =over 4 +=item B<llc_colors=[ "RANGE", "RANGE", ...]> + +Specify the Last Level Cache (LLC) color configuration for the guest. +B<RANGE> can be either a single color value or a hypen-separated closed +interval of colors (such as "0-4"). + =item B<nr_spis="NR_SPIS"> An optional integer parameter specifying the number of SPIs (Shared diff --git a/tools/include/libxl.h b/tools/include/libxl.h index XXXXXXX..XXXXXXX 100644 --- a/tools/include/libxl.h +++ b/tools/include/libxl.h @@ -XXX,XX +XXX,XX @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, const libxl_mac *src); */ #define LIBXL_HAVE_BUILDINFO_HVM_SYSTEM_FIRMWARE +/* + * The libxl_domain_build_info has the llc_colors array. + */ +#define LIBXL_HAVE_BUILDINFO_LLC_COLORS 1 + /* * ERROR_REMUS_XXX error code only exists from Xen 4.5, Xen 4.6 and it * is changed to ERROR_CHECKPOINT_XXX in Xen 4.7 diff --git a/tools/include/xenctrl.h b/tools/include/xenctrl.h index XXXXXXX..XXXXXXX 100644 --- a/tools/include/xenctrl.h +++ b/tools/include/xenctrl.h @@ -XXX,XX +XXX,XX @@ int xc_livepatch_replace(xc_interface *xch, char *name, uint32_t timeout, uint32 int xc_domain_cacheflush(xc_interface *xch, uint32_t domid, xen_pfn_t start_pfn, xen_pfn_t nr_pfns); +/* + * Set LLC colors for a domain. + * It can only be used directly after domain creation. An attempt to use it + * afterwards will result in an error. + */ +int xc_domain_set_llc_colors(xc_interface *xch, uint32_t domid, + const unsigned int *llc_colors, + unsigned int num_llc_colors); + #if defined(__arm__) || defined(__aarch64__) int xc_dt_overlay(xc_interface *xch, void *overlay_fdt, uint32_t overlay_fdt_size, uint8_t overlay_op); diff --git a/tools/libs/ctrl/xc_domain.c b/tools/libs/ctrl/xc_domain.c index XXXXXXX..XXXXXXX 100644 --- a/tools/libs/ctrl/xc_domain.c +++ b/tools/libs/ctrl/xc_domain.c @@ -XXX,XX +XXX,XX @@ int xc_domain_soft_reset(xc_interface *xch, domctl.domain = domid; return do_domctl(xch, &domctl); } + +int xc_domain_set_llc_colors(xc_interface *xch, uint32_t domid, + const unsigned int *llc_colors, + unsigned int num_llc_colors) +{ + struct xen_domctl domctl = {}; + DECLARE_HYPERCALL_BUFFER(uint32_t, local); + int ret = -1; + + if ( num_llc_colors ) + { + size_t bytes = sizeof(uint32_t) * num_llc_colors; + + local = xc_hypercall_buffer_alloc(xch, local, bytes); + if ( local == NULL ) + { + PERROR("Could not allocate LLC colors for set_llc_colors"); + ret = -ENOMEM; + goto out; + } + memcpy(local, llc_colors, bytes); + set_xen_guest_handle(domctl.u.set_llc_colors.llc_colors, local); + } + + domctl.cmd = XEN_DOMCTL_set_llc_colors; + domctl.domain = domid; + domctl.u.set_llc_colors.num_llc_colors = num_llc_colors; + + ret = do_domctl(xch, &domctl); + +out: + xc_hypercall_buffer_free(xch, local); + + return ret; +} /* * Local variables: * mode: C diff --git a/tools/libs/light/libxl_create.c b/tools/libs/light/libxl_create.c index XXXXXXX..XXXXXXX 100644 --- a/tools/libs/light/libxl_create.c +++ b/tools/libs/light/libxl_create.c @@ -XXX,XX +XXX,XX @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config, /* A new domain now exists */ *domid = local_domid; + ret = xc_domain_set_llc_colors(ctx->xch, local_domid, + b_info->llc_colors, + b_info->num_llc_colors); + if (ret < 0) { + if (errno == EOPNOTSUPP) { + if (b_info->num_llc_colors > 0) { + LOGED(ERROR, local_domid, + "LLC coloring not enabled in the hypervisor"); + rc = ERROR_FAIL; + goto out; + } + } else { + LOGED(ERROR, local_domid, "LLC colors allocation failed"); + rc = ERROR_FAIL; + goto out; + } + } + rc = libxl__is_domid_recent(gc, local_domid, &recent); if (rc) goto out; diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl index XXXXXXX..XXXXXXX 100644 --- a/tools/libs/light/libxl_types.idl +++ b/tools/libs/light/libxl_types.idl @@ -XXX,XX +XXX,XX @@ libxl_domain_build_info = Struct("domain_build_info",[ ("ioports", Array(libxl_ioport_range, "num_ioports")), ("irqs", Array(uint32, "num_irqs")), ("iomem", Array(libxl_iomem_range, "num_iomem")), + ("llc_colors", Array(uint32, "num_llc_colors")), ("claim_mode", libxl_defbool), ("event_channels", uint32), ("kernel", string), diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c index XXXXXXX..XXXXXXX 100644 --- a/tools/xl/xl_parse.c +++ b/tools/xl/xl_parse.c @@ -XXX,XX +XXX,XX @@ void parse_config_data(const char *config_source, XLU_ConfigList *cpus, *vbds, *nics, *pcis, *cvfbs, *cpuids, *vtpms, *usbctrls, *usbdevs, *p9devs, *vdispls, *pvcallsifs_devs; XLU_ConfigList *channels, *ioports, *irqs, *iomem, *viridian, *dtdevs, - *mca_caps, *smbios; + *mca_caps, *smbios, *llc_colors; int num_ioports, num_irqs, num_iomem, num_cpus, num_viridian, num_mca_caps; int num_smbios; int pci_power_mgmt = 0; @@ -XXX,XX +XXX,XX @@ void parse_config_data(const char *config_source, int pci_permissive = 0; int pci_seize = 0; int i, e; + int num_llc_colors; char *kernel_basename; libxl_domain_create_info *c_info = &d_config->c_info; @@ -XXX,XX +XXX,XX @@ void parse_config_data(const char *config_source, if (!xlu_cfg_get_long (config, "maxmem", &l, 0)) b_info->max_memkb = l * 1024; + if (!xlu_cfg_get_list(config, "llc_colors", &llc_colors, &num_llc_colors, 0)) { + int cur_index = 0; + + b_info->num_llc_colors = 0; + for (i = 0; i < num_llc_colors; i++) { + uint32_t start = 0, end = 0, k; + + buf = xlu_cfg_get_listitem(llc_colors, i); + if (!buf) { + fprintf(stderr, + "xl: Can't get element %d in LLC color list\n", i); + exit(1); + } + + if (sscanf(buf, "%" SCNu32 "-%" SCNu32, &start, &end) != 2) { + if (sscanf(buf, "%" SCNu32, &start) != 1) { + fprintf(stderr, "xl: Invalid LLC color range: %s\n", buf); + exit(1); + } + end = start; + } else if (start > end) { + fprintf(stderr, + "xl: Start LLC color is greater than end: %s\n", buf); + exit(1); + } + + b_info->num_llc_colors += (end - start) + 1; + b_info->llc_colors = (uint32_t *)realloc(b_info->llc_colors, + sizeof(*b_info->llc_colors) * b_info->num_llc_colors); + + for (k = start; k <= end; k++) + b_info->llc_colors[cur_index++] = k; + } + } + if (!xlu_cfg_get_long (config, "vcpus", &l, 0)) { vcpus = l; if (libxl_cpu_bitmap_alloc(ctx, &b_info->avail_vcpus, l)) { -- 2.43.0 Add the "llc-colors" Device Tree attribute to express DomUs and Dom0less color configurations. Based on original work from: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Carlo Nonato <carlo.nonato@minervasys.tech> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> Reviewed-by: Jan Beulich <jbeulich@suse.com> # non-Arm --- v10: - no changes v9: - use best-effort allocation in domain_set_llc_colors_from_str() v8: - fixed memory leak on error path of domain_set_llc_colors_from_str() - realloc colors array after parsing from string to reduce memory usage v7: - removed alloc_colors() helper usage from domain_set_llc_colors_from_str() v6: - rewrote domain_set_llc_colors_from_str() to be more explicit v5: - static-mem check has been moved in a previous patch - added domain_set_llc_colors_from_str() to set colors after domain creation --- docs/misc/arm/device-tree/booting.txt | 4 +++ docs/misc/cache-coloring.rst | 48 +++++++++++++++++++++++++++ xen/arch/arm/dom0less-build.c | 10 ++++++ xen/common/llc-coloring.c | 41 +++++++++++++++++++++++ xen/include/xen/llc-coloring.h | 1 + xen/include/xen/xmalloc.h | 12 +++++++ 6 files changed, 116 insertions(+) diff --git a/docs/misc/arm/device-tree/booting.txt b/docs/misc/arm/device-tree/booting.txt index XXXXXXX..XXXXXXX 100644 --- a/docs/misc/arm/device-tree/booting.txt +++ b/docs/misc/arm/device-tree/booting.txt @@ -XXX,XX +XXX,XX @@ with the following properties: An integer specifying the number of vcpus to allocate to the guest. +- llc-colors + A string specifying the LLC color configuration for the guest. + Refer to docs/misc/cache_coloring.rst for syntax. + - vpl011 An empty property to enable/disable a virtual pl011 for the guest to diff --git a/docs/misc/cache-coloring.rst b/docs/misc/cache-coloring.rst index XXXXXXX..XXXXXXX 100644 --- a/docs/misc/cache-coloring.rst +++ b/docs/misc/cache-coloring.rst @@ -XXX,XX +XXX,XX @@ If needed, change the maximum number of colors with ``CONFIG_LLC_COLORS_ORDER=<n>``. Runtime configuration is done via `Command line parameters`_. +For DomUs follow `DomUs configuration`_. Background ********** @@ -XXX,XX +XXX,XX @@ LLC specs can be manually set via the above command line parameters. This bypasses any auto-probing and it's used to overcome failing situations, such as flawed probing logic, or for debugging/testing purposes. +DomUs configuration +******************* + +DomUs colors can be set either in the ``xl`` configuration file (documentation +at `docs/man/xl.cfg.pod.5.in`) or via Device Tree, also for Dom0less +configurations (documentation at `docs/misc/arm/device-tree/booting.txt`) using +the ``llc-colors`` option. For example: + +:: + + xen,xen-bootargs = "console=dtuart dtuart=serial0 dom0_mem=1G dom0_max_vcpus=1 sched=null llc-coloring=on dom0-llc-colors=2-6"; + xen,dom0-bootargs "console=hvc0 earlycon=xen earlyprintk=xen root=/dev/ram0" + + dom0 { + compatible = "xen,linux-zimage" "xen,multiboot-module"; + reg = <0x0 0x1000000 0x0 15858176>; + }; + + dom0-ramdisk { + compatible = "xen,linux-initrd" "xen,multiboot-module"; + reg = <0x0 0x2000000 0x0 20638062>; + }; + + domU0 { + #address-cells = <0x1>; + #size-cells = <0x1>; + compatible = "xen,domain"; + memory = <0x0 0x40000>; + llc-colors = "4-8,10,11,12"; + cpus = <0x1>; + vpl011 = <0x1>; + + module@2000000 { + compatible = "multiboot,kernel", "multiboot,module"; + reg = <0x2000000 0xffffff>; + bootargs = "console=ttyAMA0"; + }; + + module@30000000 { + compatible = "multiboot,ramdisk", "multiboot,module"; + reg = <0x3000000 0xffffff>; + }; + }; + +**Note:** If no color configuration is provided for a domain, the default one, +which corresponds to all available colors is used instead. + Known issues and limitations **************************** diff --git a/xen/arch/arm/dom0less-build.c b/xen/arch/arm/dom0less-build.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/dom0less-build.c +++ b/xen/arch/arm/dom0less-build.c @@ -XXX,XX +XXX,XX @@ void __init create_domUs(void) bool iommu = false; const struct dt_device_node *cpupool_node, *chosen = dt_find_node_by_path("/chosen"); + const char *llc_colors_str = NULL; BUG_ON(chosen == NULL); dt_for_each_child_node(chosen, node) @@ -XXX,XX +XXX,XX @@ void __init create_domUs(void) #endif } + dt_property_read_string(node, "llc-colors", &llc_colors_str); + if ( !llc_coloring_enabled && llc_colors_str ) + panic("'llc-colors' found, but LLC coloring is disabled\n"); + /* * The variable max_init_domid is initialized with zero, so here it's * very important to use the pre-increment operator to call @@ -XXX,XX +XXX,XX @@ void __init create_domUs(void) panic("Error creating domain %s (rc = %ld)\n", dt_node_name(node), PTR_ERR(d)); + if ( llc_coloring_enabled && + (rc = domain_set_llc_colors_from_str(d, llc_colors_str)) ) + panic("Error initializing LLC coloring for domain %s (rc = %d)\n", + dt_node_name(node), rc); + d->is_console = true; dt_device_set_used_by(node, d->domain_id); diff --git a/xen/common/llc-coloring.c b/xen/common/llc-coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/common/llc-coloring.c +++ b/xen/common/llc-coloring.c @@ -XXX,XX +XXX,XX @@ * Copyright (C) 2024, Advanced Micro Devices, Inc. * Copyright (C) 2024, Minerva Systems SRL */ +#include "xen/xmalloc.h" #include <xen/guest_access.h> #include <xen/keyhandler.h> #include <xen/llc-coloring.h> @@ -XXX,XX +XXX,XX @@ void domain_llc_coloring_free(struct domain *d) xfree(__va(__pa(d->llc_colors))); } +int __init domain_set_llc_colors_from_str(struct domain *d, const char *str) +{ + int err; + unsigned int *colors, num_colors; + + if ( !str ) + { + domain_set_default_colors(d); + return 0; + } + + colors = xmalloc_array(unsigned int, max_nr_colors); + if ( !colors ) + return -ENOMEM; + + err = parse_color_config(str, colors, max_nr_colors, &num_colors); + if ( err ) + { + printk(XENLOG_ERR "Error parsing LLC color configuration"); + xfree(colors); + return err; + } + + if ( !check_colors(colors, num_colors) ) + { + printk(XENLOG_ERR "%pd: bad LLC color config\n", d); + xfree(colors); + return -EINVAL; + } + + /* Adjust the size cause it was initially set to max_nr_colors */ + d->llc_colors = xrealloc_array(colors, num_colors); + if ( !d->llc_colors ) + d->llc_colors = colors; + + d->num_llc_colors = num_colors; + + return 0; +} + /* * Local variables: * mode: C diff --git a/xen/include/xen/llc-coloring.h b/xen/include/xen/llc-coloring.h index XXXXXXX..XXXXXXX 100644 --- a/xen/include/xen/llc-coloring.h +++ b/xen/include/xen/llc-coloring.h @@ -XXX,XX +XXX,XX @@ void arch_llc_coloring_init(void); int dom0_set_llc_colors(struct domain *d); int domain_set_llc_colors(struct domain *d, const struct xen_domctl_set_llc_colors *config); +int domain_set_llc_colors_from_str(struct domain *d, const char *str); #endif /* __COLORING_H__ */ diff --git a/xen/include/xen/xmalloc.h b/xen/include/xen/xmalloc.h index XXXXXXX..XXXXXXX 100644 --- a/xen/include/xen/xmalloc.h +++ b/xen/include/xen/xmalloc.h @@ -XXX,XX +XXX,XX @@ ((_type *)_xmalloc_array(sizeof(_type), __alignof__(_type), _num)) #define xzalloc_array(_type, _num) \ ((_type *)_xzalloc_array(sizeof(_type), __alignof__(_type), _num)) +#define xrealloc_array(_ptr, _num) \ + ((typeof(_ptr))_xrealloc_array(_ptr, sizeof(typeof(*(_ptr))), \ + __alignof__(typeof(*(_ptr))), _num)) /* Allocate space for a structure with a flexible array of typed objects. */ #define xzalloc_flex_struct(type, field, nr) \ @@ -XXX,XX +XXX,XX @@ static inline void *_xzalloc_array( return _xzalloc(size * num, align); } +static inline void *_xrealloc_array( + void *ptr, unsigned long size, unsigned long align, unsigned long num) +{ + /* Check for overflow. */ + if ( size && num > UINT_MAX / size ) + return NULL; + return _xrealloc(ptr, size * num, align); +} + /* * Pooled allocator interface. */ -- 2.43.0 PGC_static, PGC_extra and PGC_broken need to be preserved when assigning a page. Define a new macro that groups those flags and use it instead of or'ing every time. To make preserved flags even more meaningful, they are kept also when switching state in mark_page_free(). Enforce the removal of PGC_extra before freeing domain pages as this is considered an error and can cause ASSERT violations. Signed-off-by: Carlo Nonato <carlo.nonato@minervasys.tech> --- v10: - fixed commit message v9: - add PGC_broken to PGC_preserved - clear PGC_extra in alloc_domheap_pages() only if MEMF_no_refcount is set v8: - fixed PGC_extra ASSERT fail in alloc_domheap_pages() by removing PGC_extra before freeing v7: - PGC_preserved used also in mark_page_free() v6: - preserved_flags renamed to PGC_preserved - PGC_preserved is used only in assign_pages() v5: - new patch --- xen/common/page_alloc.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index XXXXXXX..XXXXXXX 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -XXX,XX +XXX,XX @@ #endif #define PGC_no_buddy_merge PGC_static +#define PGC_preserved (PGC_extra | PGC_static | PGC_broken) #ifndef PGT_TYPE_INFO_INITIALIZER #define PGT_TYPE_INFO_INITIALIZER 0 @@ -XXX,XX +XXX,XX @@ static bool mark_page_free(struct page_info *pg, mfn_t mfn) { case PGC_state_inuse: BUG_ON(pg->count_info & PGC_broken); - pg->count_info = PGC_state_free; + pg->count_info = PGC_state_free | (pg->count_info & PGC_preserved); break; case PGC_state_offlining: - pg->count_info = (pg->count_info & PGC_broken) | - PGC_state_offlined; + pg->count_info = (pg->count_info & PGC_preserved) | PGC_state_offlined; pg_offlined = true; break; @@ -XXX,XX +XXX,XX @@ int assign_pages( for ( i = 0; i < nr; i++ ) { - ASSERT(!(pg[i].count_info & ~(PGC_extra | PGC_static))); + ASSERT(!(pg[i].count_info & ~PGC_preserved)); if ( pg[i].count_info & PGC_extra ) extra_pages++; } @@ -XXX,XX +XXX,XX @@ int assign_pages( page_set_owner(&pg[i], d); smp_wmb(); /* Domain pointer must be visible before updating refcnt. */ pg[i].count_info = - (pg[i].count_info & (PGC_extra | PGC_static)) | PGC_allocated | 1; + (pg[i].count_info & PGC_preserved) | PGC_allocated | 1; page_list_add_tail(&pg[i], page_to_list(d, &pg[i])); } @@ -XXX,XX +XXX,XX @@ struct page_info *alloc_domheap_pages( } if ( assign_page(pg, order, d, memflags) ) { + if ( memflags & MEMF_no_refcount ) + { + unsigned long i; + + for ( i = 0; i < (1UL << order); i++ ) + pg[i].count_info &= ~PGC_extra; + } + free_heap_pages(pg, order, memflags & MEMF_no_scrub); return NULL; } @@ -XXX,XX +XXX,XX @@ void free_domheap_pages(struct page_info *pg, unsigned int order) { ASSERT(d->extra_pages); d->extra_pages--; + pg[i].count_info &= ~PGC_extra; } } -- 2.43.0 Add a new memory page allocator that implements the cache coloring mechanism. The allocation algorithm enforces equal frequency distribution of cache partitions, following the coloring configuration of a domain. This allows for an even utilization of cache sets for every domain. Pages are stored in a color-indexed array of lists. Those lists are filled by a simple init function which computes the color of each page. When a domain requests a page, the allocator extracts the page from the list with the maximum number of free pages among those that the domain can access, given its coloring configuration. The allocator can only handle requests of order-0 pages. This allows for easier implementation and since cache coloring targets only embedded systems, it's assumed not to be a major problem. The buddy allocator must coexist with the colored one because the Xen heap isn't colored. For this reason a new Kconfig option and a command line parameter are added to let the user set the amount of memory reserved for the buddy allocator. Even when cache coloring is enabled, this memory isn't managed by the colored allocator. Colored heap information is dumped in the dump_heap() debug-key function. Based on original work from: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> Signed-off-by: Carlo Nonato <carlo.nonato@minervasys.tech> --- v10: - stated explicit dependency on CONFIG_LLC_COLORING for buddy-alloc-size - fix for MISRA rule 20.7 parenthesis v9: - added ASSERT(order == 0) when freeing a colored page - moved buddy_alloc_size initialization logic in Kconfig v8: - requests that uses MEMF_* flags that can't be served are now going to fail - free_color_heap_page() is called directly from free_heap_pages() v7: - requests to alloc_color_heap_page() now fail if MEMF_bits is used v6: - colored allocator functions are now static v5: - Carlo Nonato as the new author - the colored allocator balances color usage for each domain and it searches linearly only in the number of colors (FIXME removed) - addedd scrub functionality - removed stub functions (still requires some macro definition) - addr_to_color turned to mfn_to_color for easier operations - removed BUG_ON in init_color_heap_pages() in favor of panic() - only non empty page lists are logged in dump_color_heap() v4: - moved colored allocator code after buddy allocator because it now has some dependencies on buddy functions - buddy_alloc_size is now used only by the colored allocator - fixed a bug that allowed the buddy to merge pages when they were colored - free_color_heap_page() now calls mark_page_free() - free_color_heap_page() uses of the frametable array for faster searches - added FIXME comment for the linear search in free_color_heap_page() - removed alloc_color_domheap_page() to let the colored allocator exploit some more buddy allocator code - alloc_color_heap_page() now allocs min address pages first - reduced the mess in end_boot_allocator(): use the first loop for init_color_heap_pages() - fixed page_list_add_prev() (list.h) since it was doing the opposite of what it was supposed to do - fixed page_list_add_prev() (non list.h) to check also for next existence - removed unused page_list_add_next() - moved p2m code in another patch --- docs/misc/cache-coloring.rst | 37 ++++++ docs/misc/xen-command-line.pandoc | 14 +++ xen/arch/arm/include/asm/mm.h | 5 + xen/common/Kconfig | 8 ++ xen/common/llc-coloring.c | 13 +++ xen/common/page_alloc.c | 188 +++++++++++++++++++++++++++++- xen/include/xen/llc-coloring.h | 4 + 7 files changed, 265 insertions(+), 4 deletions(-) diff --git a/docs/misc/cache-coloring.rst b/docs/misc/cache-coloring.rst index XXXXXXX..XXXXXXX 100644 --- a/docs/misc/cache-coloring.rst +++ b/docs/misc/cache-coloring.rst @@ -XXX,XX +XXX,XX @@ To compile LLC coloring support set ``CONFIG_LLC_COLORING=y``. If needed, change the maximum number of colors with ``CONFIG_LLC_COLORS_ORDER=<n>``. +If needed, change the buddy allocator reserved size with +``CONFIG_BUDDY_ALLOCATOR_SIZE=<n>``. + Runtime configuration is done via `Command line parameters`_. For DomUs follow `DomUs configuration`_. @@ -XXX,XX +XXX,XX @@ Specific documentation is available at `docs/misc/xen-command-line.pandoc`. +----------------------+-------------------------------+ | ``dom0-llc-colors`` | Dom0 color configuration | +----------------------+-------------------------------+ +| ``buddy-alloc-size`` | Buddy allocator reserved size | ++----------------------+-------------------------------+ Colors selection format *********************** @@ -XXX,XX +XXX,XX @@ the ``llc-colors`` option. For example: **Note:** If no color configuration is provided for a domain, the default one, which corresponds to all available colors is used instead. +Colored allocator and buddy allocator +************************************* + +The colored allocator distributes pages based on color configurations of +domains so that each domains only gets pages of its own colors. +The colored allocator is meant as an alternative to the buddy allocator because +its allocation policy is by definition incompatible with the generic one. Since +the Xen heap is not colored yet, we need to support the coexistence of the two +allocators and some memory must be left for the buddy one. Buddy memory +reservation is configured via Kconfig or via command-line. + Known issues and limitations **************************** @@ -XXX,XX +XXX,XX @@ In the domain configuration, "xen,static-mem" allows memory to be statically allocated to the domain. This isn't possible when LLC coloring is enabled, because that memory can't be guaranteed to use only colors assigned to the domain. + +Cache coloring is intended only for embedded systems +#################################################### + +The current implementation aims to satisfy the need of predictability in +embedded systems with small amount of memory to be managed in a colored way. +Given that, some shortcuts are taken in the development. Expect worse +performances on larger systems. + +Colored allocator can only make use of order-0 pages +#################################################### + +The cache coloring technique relies on memory mappings and on the smallest +mapping granularity to achieve the maximum number of colors (cache partitions) +possible. This granularity is what is normally called a page and, in Xen +terminology, the order-0 page is the smallest one. The fairly simple +colored allocator currently implemented, makes use only of such pages. +It must be said that a more complex one could, in theory, adopt higher order +pages if the colors selection contained adjacent colors. Two subsequent colors, +for example, can be represented by an order-1 page, four colors correspond to +an order-2 page, etc. diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc index XXXXXXX..XXXXXXX 100644 --- a/docs/misc/xen-command-line.pandoc +++ b/docs/misc/xen-command-line.pandoc @@ -XXX,XX +XXX,XX @@ and not running softirqs. Reduce this if softirqs are not being run frequently enough. Setting this to a high value may cause boot failure, particularly if the NMI watchdog is also enabled. +### buddy-alloc-size (arm64) +> `= <size>` + +> Default: `64M` + +Amount of memory reserved for the buddy allocator when colored allocator is +active. This option is available only when `CONFIG_LLC_COLORING` is enabled. +The colored allocator is meant as an alternative to the buddy allocator, +because its allocation policy is by definition incompatible with the generic +one. Since the Xen heap systems is not colored yet, we need to support the +coexistence of the two allocators for now. This parameter, which is optional +and for expert only, it's used to set the amount of memory reserved to the +buddy allocator. + ### cet = List of [ shstk=<bool>, ibt=<bool> ] diff --git a/xen/arch/arm/include/asm/mm.h b/xen/arch/arm/include/asm/mm.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/mm.h +++ b/xen/arch/arm/include/asm/mm.h @@ -XXX,XX +XXX,XX @@ struct page_info #else #define PGC_static 0 #endif +#ifdef CONFIG_LLC_COLORING +/* Page is cache colored */ +#define _PGC_colored PG_shift(4) +#define PGC_colored PG_mask(1, 4) +#endif /* ... */ /* Page is broken? */ #define _PGC_broken PG_shift(7) diff --git a/xen/common/Kconfig b/xen/common/Kconfig index XXXXXXX..XXXXXXX 100644 --- a/xen/common/Kconfig +++ b/xen/common/Kconfig @@ -XXX,XX +XXX,XX @@ config LLC_COLORS_ORDER The default value corresponds to an 8 MiB 16-ways LLC, which should be more than what's needed in the general case. +config BUDDY_ALLOCATOR_SIZE + int "Buddy allocator reserved memory size (MiB)" if LLC_COLORING + default "0" if !LLC_COLORING + default "64" + help + Amount of memory reserved for the buddy allocator to serve Xen heap, + working alongside the colored one. + endmenu diff --git a/xen/common/llc-coloring.c b/xen/common/llc-coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/common/llc-coloring.c +++ b/xen/common/llc-coloring.c @@ -XXX,XX +XXX,XX @@ static unsigned int __ro_after_init default_colors[NR_LLC_COLORS]; static unsigned int __initdata dom0_colors[NR_LLC_COLORS]; static unsigned int __initdata dom0_num_colors; +#define mfn_color_mask (max_nr_colors - 1) +#define mfn_to_color(mfn) (mfn_x(mfn) & mfn_color_mask) + /* * Parse the coloring configuration given in the buf string, following the * syntax below. @@ -XXX,XX +XXX,XX @@ int __init domain_set_llc_colors_from_str(struct domain *d, const char *str) return 0; } +unsigned int page_to_llc_color(const struct page_info *pg) +{ + return mfn_to_color(page_to_mfn(pg)); +} + +unsigned int get_max_nr_llc_colors(void) +{ + return max_nr_colors; +} + /* * Local variables: * mode: C diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index XXXXXXX..XXXXXXX 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -XXX,XX +XXX,XX @@ #define PGC_static 0 #endif -#define PGC_no_buddy_merge PGC_static -#define PGC_preserved (PGC_extra | PGC_static | PGC_broken) +#ifndef PGC_colored +#define PGC_colored 0 +#endif + +#define PGC_no_buddy_merge (PGC_static | PGC_colored) +#define PGC_preserved (PGC_extra | PGC_static | PGC_broken | PGC_colored) #ifndef PGT_TYPE_INFO_INITIALIZER #define PGT_TYPE_INFO_INITIALIZER 0 @@ -XXX,XX +XXX,XX @@ static bool mark_page_free(struct page_info *pg, mfn_t mfn) return pg_offlined; } +static void free_color_heap_page(struct page_info *pg, bool need_scrub); + /* Free 2^@order set of pages. */ static void free_heap_pages( struct page_info *pg, unsigned int order, bool need_scrub) @@ -XXX,XX +XXX,XX @@ static void free_heap_pages( pg[i].count_info |= PGC_need_scrub; poison_one_page(&pg[i]); } + + if ( pg->count_info & PGC_colored ) + { + ASSERT(order == 0); + + free_color_heap_page(pg, need_scrub); + spin_unlock(&heap_lock); + return; + } } avail[node][zone] += 1 << order; @@ -XXX,XX +XXX,XX @@ static unsigned long avail_heap_pages( return free_pages; } +/************************* + * COLORED SIDE-ALLOCATOR + * + * Pages are grouped by LLC color in lists which are globally referred to as the + * color heap. Lists are populated in end_boot_allocator(). + * After initialization there will be N lists where N is the number of + * available colors on the platform. + */ +static struct page_list_head *__ro_after_init _color_heap; +#define color_heap(color) (&_color_heap[color]) + +static unsigned long *__ro_after_init free_colored_pages; + +/* Memory required for buddy allocator to work with colored one */ +static unsigned long __initdata buddy_alloc_size = + MB(CONFIG_BUDDY_ALLOCATOR_SIZE); +size_param("buddy-alloc-size", buddy_alloc_size); + +#ifdef CONFIG_LLC_COLORING +#define domain_num_llc_colors(d) ((d)->num_llc_colors) +#define domain_llc_color(d, i) ((d)->llc_colors[i]) +#else +#define domain_num_llc_colors(d) 0 +#define domain_llc_color(d, i) 0 +#endif + +static void free_color_heap_page(struct page_info *pg, bool need_scrub) +{ + unsigned int color; + + color = page_to_llc_color(pg); + free_colored_pages[color]++; + /* + * Head insertion allows re-using cache-hot pages in configurations without + * sharing of colors. + */ + page_list_add(pg, color_heap(color)); +} + +static struct page_info *alloc_color_heap_page(unsigned int memflags, + const struct domain *d) +{ + struct page_info *pg = NULL; + unsigned int i, color = 0; + unsigned long max = 0; + bool need_tlbflush = false; + uint32_t tlbflush_timestamp = 0; + bool need_scrub; + + if ( memflags & ~(MEMF_no_refcount | MEMF_no_owner | MEMF_no_tlbflush | + MEMF_no_icache_flush | MEMF_no_scrub) ) + return NULL; + + spin_lock(&heap_lock); + + for ( i = 0; i < domain_num_llc_colors(d); i++ ) + { + unsigned long free = free_colored_pages[domain_llc_color(d, i)]; + + if ( free > max ) + { + color = domain_llc_color(d, i); + pg = page_list_first(color_heap(color)); + max = free; + } + } + + if ( !pg ) + { + spin_unlock(&heap_lock); + return NULL; + } + + need_scrub = pg->count_info & PGC_need_scrub; + pg->count_info = PGC_state_inuse | (pg->count_info & PGC_preserved); + free_colored_pages[color]--; + page_list_del(pg, color_heap(color)); + + if ( !(memflags & MEMF_no_tlbflush) ) + accumulate_tlbflush(&need_tlbflush, pg, &tlbflush_timestamp); + + init_free_page_fields(pg); + + spin_unlock(&heap_lock); + + if ( !(memflags & MEMF_no_scrub) ) + { + if ( need_scrub ) + scrub_one_page(pg); + else + check_one_page(pg); + } + + if ( need_tlbflush ) + filtered_flush_tlb_mask(tlbflush_timestamp); + + flush_page_to_ram(mfn_x(page_to_mfn(pg)), + !(memflags & MEMF_no_icache_flush)); + + return pg; +} + +static void __init init_color_heap_pages(struct page_info *pg, + unsigned long nr_pages) +{ + unsigned int i; + bool need_scrub = opt_bootscrub == BOOTSCRUB_IDLE; + + if ( buddy_alloc_size >= PAGE_SIZE ) + { + unsigned long buddy_pages = min(PFN_DOWN(buddy_alloc_size), nr_pages); + + init_heap_pages(pg, buddy_pages); + nr_pages -= buddy_pages; + buddy_alloc_size -= buddy_pages << PAGE_SHIFT; + pg += buddy_pages; + } + + if ( !_color_heap ) + { + unsigned int max_nr_colors = get_max_nr_llc_colors(); + + _color_heap = xmalloc_array(struct page_list_head, max_nr_colors); + free_colored_pages = xzalloc_array(unsigned long, max_nr_colors); + if ( !_color_heap || !free_colored_pages ) + panic("Can't allocate colored heap. Buddy reserved size is too low"); + + for ( i = 0; i < max_nr_colors; i++ ) + INIT_PAGE_LIST_HEAD(color_heap(i)); + } + + for ( i = 0; i < nr_pages; i++ ) + { + pg[i].count_info = PGC_colored; + free_color_heap_page(&pg[i], need_scrub); + } +} + +static void dump_color_heap(void) +{ + unsigned int color; + + printk("Dumping color heap info\n"); + for ( color = 0; color < get_max_nr_llc_colors(); color++ ) + if ( free_colored_pages[color] > 0 ) + printk("Color heap[%u]: %lu pages\n", + color, free_colored_pages[color]); +} + void __init end_boot_allocator(void) { unsigned int i; @@ -XXX,XX +XXX,XX @@ void __init end_boot_allocator(void) for ( i = nr_bootmem_regions; i-- > 0; ) { struct bootmem_region *r = &bootmem_region_list[i]; - if ( r->s < r->e ) + + if ( r->s >= r->e ) + continue; + + if ( llc_coloring_enabled ) + init_color_heap_pages(mfn_to_page(_mfn(r->s)), r->e - r->s); + else init_heap_pages(mfn_to_page(_mfn(r->s)), r->e - r->s); } nr_bootmem_regions = 0; @@ -XXX,XX +XXX,XX @@ struct page_info *alloc_domheap_pages( if ( memflags & MEMF_no_owner ) memflags |= MEMF_no_refcount; - if ( !dma_bitsize ) + /* Only domains are supported for coloring */ + if ( d && llc_coloring_enabled ) + { + /* Colored allocation must be done on 0 order */ + if ( order || (pg = alloc_color_heap_page(memflags, d)) == NULL ) + return NULL; + } + else if ( !dma_bitsize ) memflags &= ~MEMF_no_dma; else if ( (dma_zone = bits_to_zone(dma_bitsize)) < zone_hi ) pg = alloc_heap_pages(dma_zone + 1, zone_hi, order, memflags, d); @@ -XXX,XX +XXX,XX @@ static void cf_check dump_heap(unsigned char key) continue; printk("Node %d has %lu unscrubbed pages\n", i, node_need_scrub[i]); } + + if ( llc_coloring_enabled ) + dump_color_heap(); } static __init int cf_check register_heap_trigger(void) diff --git a/xen/include/xen/llc-coloring.h b/xen/include/xen/llc-coloring.h index XXXXXXX..XXXXXXX 100644 --- a/xen/include/xen/llc-coloring.h +++ b/xen/include/xen/llc-coloring.h @@ -XXX,XX +XXX,XX @@ int domain_set_llc_colors(struct domain *d, const struct xen_domctl_set_llc_colors *config); int domain_set_llc_colors_from_str(struct domain *d, const char *str); +struct page_info; +unsigned int page_to_llc_color(const struct page_info *pg); +unsigned int get_max_nr_llc_colors(void); + #endif /* __COLORING_H__ */ /* -- 2.43.0 From: Luca Miccio <lucmiccio@gmail.com> Add a new command line parameter to configure Xen cache colors. These colors are dumped together with other coloring info. Benchmarking the VM interrupt response time provides an estimation of LLC usage by Xen's most latency-critical runtime task. Results on Arm Cortex-A53 on Xilinx Zynq UltraScale+ XCZU9EG show that one color, which reserves 64 KiB of L2, is enough to attain best responsiveness: - Xen 1 color latency: 3.1 us - Xen 2 color latency: 3.1 us Since this is the most common target for Arm cache coloring, the default amount of Xen colors is set to one. More colors are instead very likely to be needed on processors whose L1 cache is physically-indexed and physically-tagged, such as Cortex-A57. In such cases, coloring applies to L1 also, and there typically are two distinct L1-colors. Therefore, reserving only one color for Xen would senselessly partitions a cache memory that is already private, i.e. underutilize it. Signed-off-by: Luca Miccio <lucmiccio@gmail.com> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> Signed-off-by: Carlo Nonato <carlo.nonato@minervasys.tech> Reviewed-by: Jan Beulich <jbeulich@suse.com> --- v10: - no changes v9: - no changes v8: - added bound check on xen_colors in llc_coloring_init() v7: - removed XEN_DEFAULT_COLOR - XEN_DEFAULT_NUM_COLORS is now used in a for loop to set xen default colors --- docs/misc/cache-coloring.rst | 2 ++ docs/misc/xen-command-line.pandoc | 10 ++++++++++ xen/common/llc-coloring.c | 31 ++++++++++++++++++++++++++++++- 3 files changed, 42 insertions(+), 1 deletion(-) diff --git a/docs/misc/cache-coloring.rst b/docs/misc/cache-coloring.rst index XXXXXXX..XXXXXXX 100644 --- a/docs/misc/cache-coloring.rst +++ b/docs/misc/cache-coloring.rst @@ -XXX,XX +XXX,XX @@ Specific documentation is available at `docs/misc/xen-command-line.pandoc`. +----------------------+-------------------------------+ | ``buddy-alloc-size`` | Buddy allocator reserved size | +----------------------+-------------------------------+ +| ``xen-llc-colors`` | Xen color configuration | ++----------------------+-------------------------------+ Colors selection format *********************** diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc index XXXXXXX..XXXXXXX 100644 --- a/docs/misc/xen-command-line.pandoc +++ b/docs/misc/xen-command-line.pandoc @@ -XXX,XX +XXX,XX @@ mode. **WARNING: `x2apic_phys` is deprecated and superseded by `x2apic-mode`. The latter takes precedence if both are set.** +### xen-llc-colors (arm64) +> `= List of [ <integer> | <integer>-<integer> ]` + +> Default: `0: the lowermost color` + +Specify Xen LLC color configuration. This options is available only when +`CONFIG_LLC_COLORING` is enabled. +Two colors are most likely needed on platforms where private caches are +physically indexed, e.g. the L1 instruction cache of the Arm Cortex-A57. + ### xenheap_megabytes (arm32) > `= <size>` diff --git a/xen/common/llc-coloring.c b/xen/common/llc-coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/common/llc-coloring.c +++ b/xen/common/llc-coloring.c @@ -XXX,XX +XXX,XX @@ #include <xen/param.h> #define NR_LLC_COLORS (1U << CONFIG_LLC_COLORS_ORDER) +#define XEN_DEFAULT_NUM_COLORS 1 bool __ro_after_init llc_coloring_enabled; boolean_param("llc-coloring", llc_coloring_enabled); @@ -XXX,XX +XXX,XX @@ static unsigned int __ro_after_init default_colors[NR_LLC_COLORS]; static unsigned int __initdata dom0_colors[NR_LLC_COLORS]; static unsigned int __initdata dom0_num_colors; +static unsigned int __ro_after_init xen_colors[NR_LLC_COLORS]; +static unsigned int __ro_after_init xen_num_colors; + #define mfn_color_mask (max_nr_colors - 1) #define mfn_to_color(mfn) (mfn_x(mfn) & mfn_color_mask) @@ -XXX,XX +XXX,XX @@ static int __init parse_dom0_colors(const char *s) } custom_param("dom0-llc-colors", parse_dom0_colors); -static void print_colors(const unsigned int colors[], unsigned int num_colors) +static int __init parse_xen_colors(const char *s) +{ + return parse_color_config(s, xen_colors, ARRAY_SIZE(xen_colors), + &xen_num_colors); +} +custom_param("xen-llc-colors", parse_xen_colors); + +static void print_colors(const unsigned int *colors, unsigned int num_colors) { unsigned int i; @@ -XXX,XX +XXX,XX @@ void __init llc_coloring_init(void) for ( i = 0; i < max_nr_colors; i++ ) default_colors[i] = i; + if ( !xen_num_colors ) + { + unsigned int i; + + xen_num_colors = MIN(XEN_DEFAULT_NUM_COLORS, max_nr_colors); + + printk(XENLOG_WARNING + "Xen LLC color config not found. Using first %u colors\n", + xen_num_colors); + for ( i = 0; i < xen_num_colors; i++ ) + xen_colors[i] = i; + } + else if ( xen_num_colors > max_nr_colors || + !check_colors(xen_colors, xen_num_colors) ) + panic("Bad LLC color config for Xen\n"); + arch_llc_coloring_init(); } @@ -XXX,XX +XXX,XX @@ void dump_llc_coloring_info(void) printk("LLC coloring info:\n"); printk(" Number of LLC colors supported: %u\n", max_nr_colors); + printk(" Xen LLC colors (%u): ", xen_num_colors); + print_colors(xen_colors, xen_num_colors); } void domain_dump_llc_colors(const struct domain *d) -- 2.43.0 Cache coloring must physically relocate Xen in order to color the hypervisor and consider_modules() is a key function that is needed to find a new available physical address. 672d67f339c0 ("xen/arm: Split MMU-specific setup_mm() and related code out") moved consider_modules() under arm32. Move it to mmu/setup.c and make it non-static so that it can be used outside. Signed-off-by: Carlo Nonato <carlo.nonato@minervasys.tech> --- v10: - no changes v9: - no changes v8: - patch adapted to new changes to consider_modules() v7: - moved consider_modules() to arm/mmu/setup.c v6: - new patch --- xen/arch/arm/arm32/mmu/mm.c | 95 +------------------------------ xen/arch/arm/include/asm/setup.h | 3 + xen/arch/arm/mmu/setup.c | 97 ++++++++++++++++++++++++++++++++ 3 files changed, 101 insertions(+), 94 deletions(-) diff --git a/xen/arch/arm/arm32/mmu/mm.c b/xen/arch/arm/arm32/mmu/mm.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/arm32/mmu/mm.c +++ b/xen/arch/arm/arm32/mmu/mm.c @@ -XXX,XX +XXX,XX @@ #include <asm/fixmap.h> #include <asm/static-memory.h> #include <asm/static-shmem.h> +#include <asm/setup.h> static unsigned long opt_xenheap_megabytes __initdata; integer_param("xenheap_megabytes", opt_xenheap_megabytes); @@ -XXX,XX +XXX,XX @@ static void __init setup_directmap_mappings(unsigned long base_mfn, directmap_virt_end = XENHEAP_VIRT_START + nr_mfns * PAGE_SIZE; } -/* - * Returns the end address of the highest region in the range s..e - * with required size and alignment that does not conflict with the - * modules from first_mod to nr_modules. - * - * For non-recursive callers first_mod should normally be 0 (all - * modules and Xen itself) or 1 (all modules but not Xen). - */ -static paddr_t __init consider_modules(paddr_t s, paddr_t e, - uint32_t size, paddr_t align, - int first_mod) -{ - const struct membanks *reserved_mem = bootinfo_get_reserved_mem(); -#ifdef CONFIG_STATIC_SHM - const struct membanks *shmem = bootinfo_get_shmem(); -#endif - const struct bootmodules *mi = &bootinfo.modules; - int i; - int nr; - - s = (s+align-1) & ~(align-1); - e = e & ~(align-1); - - if ( s > e || e - s < size ) - return 0; - - /* First check the boot modules */ - for ( i = first_mod; i < mi->nr_mods; i++ ) - { - paddr_t mod_s = mi->module[i].start; - paddr_t mod_e = mod_s + mi->module[i].size; - - if ( s < mod_e && mod_s < e ) - { - mod_e = consider_modules(mod_e, e, size, align, i+1); - if ( mod_e ) - return mod_e; - - return consider_modules(s, mod_s, size, align, i+1); - } - } - - /* - * i is the current bootmodule we are evaluating, across all - * possible kinds of bootmodules. - * - * When retrieving the corresponding reserved-memory addresses, we - * need to index the reserved_mem bank starting from 0, and only counting - * the reserved-memory modules. Hence, we need to use i - nr. - */ - nr = mi->nr_mods; - for ( ; i - nr < reserved_mem->nr_banks; i++ ) - { - paddr_t r_s = reserved_mem->bank[i - nr].start; - paddr_t r_e = r_s + reserved_mem->bank[i - nr].size; - - if ( s < r_e && r_s < e ) - { - r_e = consider_modules(r_e, e, size, align, i + 1); - if ( r_e ) - return r_e; - - return consider_modules(s, r_s, size, align, i + 1); - } - } - -#ifdef CONFIG_STATIC_SHM - nr += reserved_mem->nr_banks; - for ( ; i - nr < shmem->nr_banks; i++ ) - { - paddr_t r_s, r_e; - - r_s = shmem->bank[i - nr].start; - - /* Shared memory banks can contain INVALID_PADDR as start */ - if ( INVALID_PADDR == r_s ) - continue; - - r_e = r_s + shmem->bank[i - nr].size; - - if ( s < r_e && r_s < e ) - { - r_e = consider_modules(r_e, e, size, align, i + 1); - if ( r_e ) - return r_e; - - return consider_modules(s, r_s, size, align, i + 1); - } - } -#endif - - return e; -} - /* * Find a contiguous region that fits in the static heap region with * required size and alignment, and return the end address of the region diff --git a/xen/arch/arm/include/asm/setup.h b/xen/arch/arm/include/asm/setup.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/setup.h +++ b/xen/arch/arm/include/asm/setup.h @@ -XXX,XX +XXX,XX @@ struct init_info unsigned int cpuid; }; +paddr_t consider_modules(paddr_t s, paddr_t e, uint32_t size, paddr_t align, + int first_mod); + #endif /* * Local variables: diff --git a/xen/arch/arm/mmu/setup.c b/xen/arch/arm/mmu/setup.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/mmu/setup.c +++ b/xen/arch/arm/mmu/setup.c @@ -XXX,XX +XXX,XX @@ */ #include <xen/init.h> +#include <xen/lib.h> #include <xen/libfdt/libfdt.h> +#include <xen/libfdt/libfdt-xen.h> +#include <xen/llc-coloring.h> #include <xen/sections.h> #include <xen/sizes.h> #include <xen/vmap.h> @@ -XXX,XX +XXX,XX @@ static void xen_pt_enforce_wnx(void) flush_xen_tlb_local(); } +/* + * Returns the end address of the highest region in the range s..e + * with required size and alignment that does not conflict with the + * modules from first_mod to nr_modules. + * + * For non-recursive callers first_mod should normally be 0 (all + * modules and Xen itself) or 1 (all modules but not Xen). + */ +paddr_t __init consider_modules(paddr_t s, paddr_t e, + uint32_t size, paddr_t align, + int first_mod) +{ + const struct membanks *reserved_mem = bootinfo_get_reserved_mem(); +#ifdef CONFIG_STATIC_SHM + const struct membanks *shmem = bootinfo_get_shmem(); +#endif + const struct bootmodules *mi = &bootinfo.modules; + int i; + int nr; + + s = (s+align-1) & ~(align-1); + e = e & ~(align-1); + + if ( s > e || e - s < size ) + return 0; + + /* First check the boot modules */ + for ( i = first_mod; i < mi->nr_mods; i++ ) + { + paddr_t mod_s = mi->module[i].start; + paddr_t mod_e = mod_s + mi->module[i].size; + + if ( s < mod_e && mod_s < e ) + { + mod_e = consider_modules(mod_e, e, size, align, i+1); + if ( mod_e ) + return mod_e; + + return consider_modules(s, mod_s, size, align, i+1); + } + } + + /* + * i is the current bootmodule we are evaluating, across all + * possible kinds of bootmodules. + * + * When retrieving the corresponding reserved-memory addresses, we + * need to index the reserved_mem bank starting from 0, and only counting + * the reserved-memory modules. Hence, we need to use i - nr. + */ + nr = mi->nr_mods; + for ( ; i - nr < reserved_mem->nr_banks; i++ ) + { + paddr_t r_s = reserved_mem->bank[i - nr].start; + paddr_t r_e = r_s + reserved_mem->bank[i - nr].size; + + if ( s < r_e && r_s < e ) + { + r_e = consider_modules(r_e, e, size, align, i + 1); + if ( r_e ) + return r_e; + + return consider_modules(s, r_s, size, align, i + 1); + } + } + +#ifdef CONFIG_STATIC_SHM + nr += reserved_mem->nr_banks; + for ( ; i - nr < shmem->nr_banks; i++ ) + { + paddr_t r_s, r_e; + + r_s = shmem->bank[i - nr].start; + + /* Shared memory banks can contain INVALID_PADDR as start */ + if ( INVALID_PADDR == r_s ) + continue; + + r_e = r_s + shmem->bank[i - nr].size; + + if ( s < r_e && r_s < e ) + { + r_e = consider_modules(r_e, e, size, align, i + 1); + if ( r_e ) + return r_e; + + return consider_modules(s, r_s, size, align, i + 1); + } + } +#endif + + return e; +} + /* * Boot-time pagetable setup. * Changes here may need matching changes in head.S -- 2.43.0 Add the cache coloring support for Xen physical space. Since Xen must be relocated to a new physical space, some relocation functionalities must be brought back: - the virtual address of the new space is taken from 0c18fb76323b ("xen/arm: Remove unused BOOT_RELOC_VIRT_START"). - relocate_xen() and get_xen_paddr() are taken from f60658c6ae47 ("xen/arm: Stop relocating Xen"). setup_pagetables() must be adapted for coloring and for relocation. Runtime page tables are used to map the colored space, but they are also linked in boot tables so that the new space is temporarily available for relocation. This implies that Xen protection must happen after the copy. Finally, since the alternative framework needs to remap the Xen text and inittext sections, this operation must be done in a coloring-aware way. The function xen_remap_colored() is introduced for that. Signed-off-by: Carlo Nonato <carlo.nonato@minervasys.tech> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> --- v10: - no changes v9: - patch adapted to changes to setup_pagetables() v8: - moved xen_colored_map_size() to arm/llc-coloring.c v7: - added BUG_ON() checks to arch_llc_coloring_init() and create_llc_coloring_mappings() v6: - squashed with BOOT_RELOC_VIRT_START patch - consider_modules() moved in another patch - removed psci and smpboot code because of new idmap work already handles that - moved xen_remap_colored() in alternative.c since it's only used there - removed xen_colored_temp[] in favor of xen_xenmap[] usage for mapping - use of boot_module_find_by_kind() to remove the need of extra parameter in setup_pagetables() - moved get_xen_paddr() in arm/llc-coloring.c since it's only used there v5: - FIXME: consider_modules copy pasted since it got moved v4: - removed set_value_for_secondary() because it was wrongly cleaning cache - relocate_xen() now calls switch_ttbr_id() --- xen/arch/arm/alternative.c | 30 +++++++- xen/arch/arm/arm64/mmu/head.S | 58 +++++++++++++++- xen/arch/arm/arm64/mmu/mm.c | 28 ++++++-- xen/arch/arm/include/asm/mmu/layout.h | 3 + xen/arch/arm/llc-coloring.c | 63 +++++++++++++++++ xen/arch/arm/mmu/setup.c | 98 +++++++++++++++++++++++---- xen/arch/arm/setup.c | 10 ++- xen/common/llc-coloring.c | 18 +++++ xen/include/xen/llc-coloring.h | 13 ++++ 9 files changed, 300 insertions(+), 21 deletions(-) diff --git a/xen/arch/arm/alternative.c b/xen/arch/arm/alternative.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/alternative.c +++ b/xen/arch/arm/alternative.c @@ -XXX,XX +XXX,XX @@ #include <xen/init.h> #include <xen/types.h> #include <xen/kernel.h> +#include <xen/llc-coloring.h> #include <xen/mm.h> #include <xen/vmap.h> #include <xen/smp.h> @@ -XXX,XX +XXX,XX @@ static int __apply_alternatives_multi_stop(void *xenmap) return 0; } +static void __init *xen_remap_colored(mfn_t xen_mfn, paddr_t xen_size) +{ + unsigned int i; + void *xenmap; + mfn_t *xen_colored_mfns, mfn; + + xen_colored_mfns = xmalloc_array(mfn_t, xen_size >> PAGE_SHIFT); + if ( !xen_colored_mfns ) + panic("Can't allocate LLC colored MFNs\n"); + + for_each_xen_colored_mfn ( xen_mfn, mfn, i ) + { + xen_colored_mfns[i] = mfn; + } + + xenmap = vmap(xen_colored_mfns, xen_size >> PAGE_SHIFT); + xfree(xen_colored_mfns); + + return xenmap; +} + /* * This function should only be called during boot and before CPU0 jump * into the idle_loop. @@ -XXX,XX +XXX,XX @@ void __init apply_alternatives_all(void) * The text and inittext section are read-only. So re-map Xen to * be able to patch the code. */ - xenmap = __vmap(&xen_mfn, 1U << xen_order, 1, 1, PAGE_HYPERVISOR, - VMAP_DEFAULT); + if ( llc_coloring_enabled ) + xenmap = xen_remap_colored(xen_mfn, xen_size); + else + xenmap = __vmap(&xen_mfn, 1U << xen_order, 1, 1, PAGE_HYPERVISOR, + VMAP_DEFAULT); + /* Re-mapping Xen is not expected to fail during boot. */ BUG_ON(!xenmap); diff --git a/xen/arch/arm/arm64/mmu/head.S b/xen/arch/arm/arm64/mmu/head.S index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/arm64/mmu/head.S +++ b/xen/arch/arm/arm64/mmu/head.S @@ -XXX,XX +XXX,XX @@ FUNC_LOCAL(fail) b 1b END(fail) +/* + * Copy Xen to new location and switch TTBR + * x0 ttbr + * x1 source address + * x2 destination address + * x3 length + * + * Source and destination must be word aligned, length is rounded up + * to a 16 byte boundary. + * + * MUST BE VERY CAREFUL when saving things to RAM over the copy + */ +ENTRY(relocate_xen) + /* + * Copy 16 bytes at a time using: + * x9: counter + * x10: data + * x11: data + * x12: source + * x13: destination + */ + mov x9, x3 + mov x12, x1 + mov x13, x2 + +1: ldp x10, x11, [x12], #16 + stp x10, x11, [x13], #16 + + subs x9, x9, #16 + bgt 1b + + /* + * Flush destination from dcache using: + * x9: counter + * x10: step + * x11: vaddr + * + * This is to ensure data is visible to the instruction cache + */ + dsb sy + + mov x9, x3 + ldr x10, =dcache_line_bytes /* x10 := step */ + ldr x10, [x10] + mov x11, x2 + +1: dc cvac, x11 + + add x11, x11, x10 + subs x9, x9, x10 + bgt 1b + + /* No need for dsb/isb because they are alredy done in switch_ttbr_id */ + b switch_ttbr_id + /* * Switch TTBR * @@ -XXX,XX +XXX,XX @@ FUNC(switch_ttbr_id) /* * 5) Flush I-cache - * This should not be necessary but it is kept for safety. + * This should not be necessary in the general case, but it's needed + * for cache coloring because code is relocated in that case. */ ic iallu isb diff --git a/xen/arch/arm/arm64/mmu/mm.c b/xen/arch/arm/arm64/mmu/mm.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/arm64/mmu/mm.c +++ b/xen/arch/arm/arm64/mmu/mm.c @@ -XXX,XX +XXX,XX @@ /* SPDX-License-Identifier: GPL-2.0-only */ #include <xen/init.h> +#include <xen/llc-coloring.h> #include <xen/mm.h> #include <xen/pfn.h> @@ -XXX,XX +XXX,XX @@ void update_boot_mapping(bool enable) } extern void switch_ttbr_id(uint64_t ttbr); +extern void relocate_xen(uint64_t ttbr, void *src, void *dst, size_t len); typedef void (switch_ttbr_fn)(uint64_t ttbr); +typedef void (relocate_xen_fn)(uint64_t ttbr, void *src, void *dst, size_t len); void __init switch_ttbr(uint64_t ttbr) { - vaddr_t id_addr = virt_to_maddr(switch_ttbr_id); - switch_ttbr_fn *fn = (switch_ttbr_fn *)id_addr; + vaddr_t vaddr, id_addr; lpae_t pte; + if ( llc_coloring_enabled ) + vaddr = (vaddr_t)relocate_xen; + else + vaddr = (vaddr_t)switch_ttbr_id; + + id_addr = virt_to_maddr(vaddr); + /* Enable the identity mapping in the boot page tables */ update_identity_mapping(true); /* Enable the identity mapping in the runtime page tables */ - pte = pte_of_xenaddr((vaddr_t)switch_ttbr_id); + pte = pte_of_xenaddr(vaddr); pte.pt.table = 1; pte.pt.xn = 0; pte.pt.ro = 1; write_pte(&xen_third_id[third_table_offset(id_addr)], pte); /* Switch TTBR */ - fn(ttbr); + if ( llc_coloring_enabled ) + { + relocate_xen_fn *fn = (relocate_xen_fn *)id_addr; + + fn(ttbr, _start, (void *)BOOT_RELOC_VIRT_START, _end - _start); + } + else + { + switch_ttbr_fn *fn = (switch_ttbr_fn *)id_addr; + + fn(ttbr); + } /* * Disable the identity mapping in the runtime page tables. diff --git a/xen/arch/arm/include/asm/mmu/layout.h b/xen/arch/arm/include/asm/mmu/layout.h index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/include/asm/mmu/layout.h +++ b/xen/arch/arm/include/asm/mmu/layout.h @@ -XXX,XX +XXX,XX @@ * 10M - 12M Fixmap: special-purpose 4K mapping slots * 12M - 16M Early boot mapping of FDT * 16M - 18M Livepatch vmap (if compiled in) + * 16M - 24M Cache-colored Xen text, data, bss (temporary, if compiled in) * * 1G - 2G VMAP: ioremap and early_ioremap * @@ -XXX,XX +XXX,XX @@ #define BOOT_FDT_VIRT_START (FIXMAP_VIRT_START + FIXMAP_VIRT_SIZE) #define BOOT_FDT_VIRT_SIZE _AT(vaddr_t, MB(4)) +#define BOOT_RELOC_VIRT_START (BOOT_FDT_VIRT_START + BOOT_FDT_VIRT_SIZE) + #ifdef CONFIG_LIVEPATCH #define LIVEPATCH_VMAP_START (BOOT_FDT_VIRT_START + BOOT_FDT_VIRT_SIZE) #define LIVEPATCH_VMAP_SIZE _AT(vaddr_t, MB(2)) diff --git a/xen/arch/arm/llc-coloring.c b/xen/arch/arm/llc-coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/llc-coloring.c +++ b/xen/arch/arm/llc-coloring.c @@ -XXX,XX +XXX,XX @@ #include <asm/processor.h> #include <asm/sysregs.h> +#include <asm/setup.h> /* Return the LLC way size by probing the hardware */ unsigned int __init get_llc_way_size(void) @@ -XXX,XX +XXX,XX @@ unsigned int __init get_llc_way_size(void) return line_size * num_sets; } +/** + * get_xen_paddr - get physical address to relocate Xen to + * + * Xen is relocated to as near to the top of RAM as possible and + * aligned to a XEN_PADDR_ALIGN boundary. + */ +static paddr_t __init get_xen_paddr(paddr_t xen_size) +{ + const struct membanks *mem = bootinfo_get_mem(); + paddr_t min_size, paddr = 0; + unsigned int i; + + min_size = (xen_size + (XEN_PADDR_ALIGN-1)) & ~(XEN_PADDR_ALIGN-1); + + /* Find the highest bank with enough space. */ + for ( i = 0; i < mem->nr_banks; i++ ) + { + const struct membank *bank = &mem->bank[i]; + paddr_t s, e; + + if ( bank->size >= min_size ) + { + e = consider_modules(bank->start, bank->start + bank->size, + min_size, XEN_PADDR_ALIGN, 0); + if ( !e ) + continue; + +#ifdef CONFIG_ARM_32 + /* Xen must be under 4GB */ + if ( e > GB(4) ) + e = GB(4); + if ( e < bank->start ) + continue; +#endif + + s = e - min_size; + + if ( s > paddr ) + paddr = s; + } + } + + if ( !paddr ) + panic("Not enough memory to relocate Xen\n"); + + printk("Placing Xen at 0x%"PRIpaddr"-0x%"PRIpaddr"\n", + paddr, paddr + min_size); + + return paddr; +} + +static paddr_t __init xen_colored_map_size(void) +{ + return ROUNDUP((_end - _start) * get_max_nr_llc_colors(), XEN_PADDR_ALIGN); +} + void __init arch_llc_coloring_init(void) { + struct bootmodule *xen_bootmodule = boot_module_find_by_kind(BOOTMOD_XEN); + + BUG_ON(!xen_bootmodule); + + xen_bootmodule->size = xen_colored_map_size(); + xen_bootmodule->start = get_xen_paddr(xen_bootmodule->size); } /* diff --git a/xen/arch/arm/mmu/setup.c b/xen/arch/arm/mmu/setup.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/mmu/setup.c +++ b/xen/arch/arm/mmu/setup.c @@ -XXX,XX +XXX,XX @@ #undef virt_to_mfn #define virt_to_mfn(va) _mfn(__virt_to_mfn(va)) +#define virt_to_reloc_virt(virt) \ + (((vaddr_t)virt) - XEN_VIRT_START + BOOT_RELOC_VIRT_START) + /* Main runtime page tables */ /* @@ -XXX,XX +XXX,XX @@ static void __init __maybe_unused build_assertions(void) /* 2MB aligned regions */ BUILD_BUG_ON(XEN_VIRT_START & ~SECOND_MASK); BUILD_BUG_ON(FIXMAP_ADDR(0) & ~SECOND_MASK); + BUILD_BUG_ON(BOOT_RELOC_VIRT_START & ~SECOND_MASK); /* 1GB aligned regions */ #ifdef CONFIG_ARM_32 BUILD_BUG_ON(XENHEAP_VIRT_START & ~FIRST_MASK); @@ -XXX,XX +XXX,XX @@ static void __init __maybe_unused build_assertions(void) lpae_t __init pte_of_xenaddr(vaddr_t va) { + if ( llc_coloring_enabled ) + va = virt_to_reloc_virt(va); + return mfn_to_xen_entry(virt_to_mfn(va), MT_NORMAL); } @@ -XXX,XX +XXX,XX @@ paddr_t __init consider_modules(paddr_t s, paddr_t e, return e; } +static void __init create_llc_coloring_mappings(void) +{ + lpae_t pte; + unsigned int i; + struct bootmodule *xen_bootmodule = boot_module_find_by_kind(BOOTMOD_XEN); + mfn_t start_mfn = maddr_to_mfn(xen_bootmodule->start), mfn; + + for_each_xen_colored_mfn ( start_mfn, mfn, i ) + { + pte = mfn_to_xen_entry(mfn, MT_NORMAL); + pte.pt.table = 1; /* level 3 mappings always have this bit set */ + xen_xenmap[i] = pte; + } + + for ( i = 0; i < XEN_NR_ENTRIES(2); i++ ) + { + vaddr_t va = BOOT_RELOC_VIRT_START + (i << XEN_PT_LEVEL_SHIFT(2)); + + pte = mfn_to_xen_entry(virt_to_mfn(xen_xenmap + + i * XEN_PT_LPAE_ENTRIES), + MT_NORMAL); + pte.pt.table = 1; + write_pte(&boot_second[second_table_offset(va)], pte); + } +} + /* - * Boot-time pagetable setup. + * Boot-time pagetable setup with coloring support * Changes here may need matching changes in head.S + * + * The cache coloring support consists of: + * - Create colored mapping that conforms to Xen color selection in xen_xenmap[] + * - Link the mapping in boot page tables using BOOT_RELOC_VIRT_START as vaddr + * - pte_of_xenaddr() takes care of translating addresses to the new space + * during runtime page tables creation + * - Relocate xen and update TTBR with the new address in the colored space + * (see switch_ttbr()) + * - Protect the new space */ void __init setup_pagetables(void) { @@ -XXX,XX +XXX,XX @@ void __init setup_pagetables(void) lpae_t pte, *p; int i; + if ( llc_coloring_enabled ) + create_llc_coloring_mappings(); + arch_setup_page_tables(); #ifdef CONFIG_ARM_64 @@ -XXX,XX +XXX,XX @@ void __init setup_pagetables(void) break; pte = pte_of_xenaddr(va); pte.pt.table = 1; /* third level mappings always have this bit set */ - if ( is_kernel_text(va) || is_kernel_inittext(va) ) - { - pte.pt.xn = 0; - pte.pt.ro = 1; - } - if ( is_kernel_rodata(va) ) - pte.pt.ro = 1; + pte.pt.xn = 0; /* Permissions will be enforced later. Allow execution */ xen_xenmap[i] = pte; } @@ -XXX,XX +XXX,XX @@ void __init setup_pagetables(void) ttbr = virt_to_maddr(cpu0_pgtable); #endif - switch_ttbr(ttbr); - - xen_pt_enforce_wnx(); - #ifdef CONFIG_ARM_32 per_cpu(xen_pgtable, 0) = cpu0_pgtable; #endif + + if ( llc_coloring_enabled ) + ttbr = virt_to_maddr(virt_to_reloc_virt(THIS_CPU_PGTABLE)); + + switch_ttbr(ttbr); + + /* Protect Xen */ + for ( i = 0; i < XEN_NR_ENTRIES(3); i++ ) + { + vaddr_t va = XEN_VIRT_START + (i << PAGE_SHIFT); + lpae_t *entry = xen_xenmap + i; + + if ( !is_kernel(va) ) + break; + + pte = read_atomic(entry); + + if ( is_kernel_text(va) || is_kernel_inittext(va) ) + { + pte.pt.xn = 0; + pte.pt.ro = 1; + } else if ( is_kernel_rodata(va) ) { + pte.pt.ro = 1; + pte.pt.xn = 1; + } else { + pte.pt.xn = 1; + pte.pt.ro = 0; + } + + write_pte(entry, pte); + } + + /* + * We modified live page-tables. Ensure the TLBs are invalidated + * before setting enforcing the WnX permissions. + */ + flush_xen_tlb_local(); + + xen_pt_enforce_wnx(); } void *__init arch_vmap_virt_end(void) diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c index XXXXXXX..XXXXXXX 100644 --- a/xen/arch/arm/setup.c +++ b/xen/arch/arm/setup.c @@ -XXX,XX +XXX,XX @@ void asmlinkage __init start_xen(unsigned long fdt_paddr) /* Initialize traps early allow us to get backtrace when an error occurred */ init_traps(); - setup_pagetables(); - smp_clear_cpu_maps(); device_tree_flattened = early_fdt_map(fdt_paddr); @@ -XXX,XX +XXX,XX @@ void asmlinkage __init start_xen(unsigned long fdt_paddr) llc_coloring_init(); + /* + * Page tables must be setup after LLC coloring initialization because + * coloring info are required in order to create colored mappings + */ + setup_pagetables(); + /* Device-tree was mapped in boot page tables, remap it in the new tables */ + device_tree_flattened = early_fdt_map(fdt_paddr); + setup_mm(); vm_init(); diff --git a/xen/common/llc-coloring.c b/xen/common/llc-coloring.c index XXXXXXX..XXXXXXX 100644 --- a/xen/common/llc-coloring.c +++ b/xen/common/llc-coloring.c @@ -XXX,XX +XXX,XX @@ static unsigned int __ro_after_init xen_num_colors; #define mfn_color_mask (max_nr_colors - 1) #define mfn_to_color(mfn) (mfn_x(mfn) & mfn_color_mask) +#define get_mfn_with_color(mfn, color) \ + (_mfn((mfn_x(mfn) & ~mfn_color_mask) | (color))) /* * Parse the coloring configuration given in the buf string, following the @@ -XXX,XX +XXX,XX @@ unsigned int get_max_nr_llc_colors(void) return max_nr_colors; } +mfn_t __init xen_colored_mfn(mfn_t mfn) +{ + unsigned int i, color = mfn_to_color(mfn); + + for ( i = 0; i < xen_num_colors; i++ ) + { + if ( color == xen_colors[i] ) + return mfn; + else if ( color < xen_colors[i] ) + return get_mfn_with_color(mfn, xen_colors[i]); + } + + /* Jump to next color space (max_nr_colors mfns) and use the first color */ + return get_mfn_with_color(mfn_add(mfn, max_nr_colors), xen_colors[0]); +} + /* * Local variables: * mode: C diff --git a/xen/include/xen/llc-coloring.h b/xen/include/xen/llc-coloring.h index XXXXXXX..XXXXXXX 100644 --- a/xen/include/xen/llc-coloring.h +++ b/xen/include/xen/llc-coloring.h @@ -XXX,XX +XXX,XX @@ static inline void domain_dump_llc_colors(const struct domain *d) {} static inline void domain_llc_coloring_free(struct domain *d) {} #endif +/** + * Iterate over each Xen mfn in the colored space. + * @start_mfn: the first mfn that needs to be colored. + * @mfn: the current mfn. + * @i: loop index. + */ +#define for_each_xen_colored_mfn(start_mfn, mfn, i) \ + for ( i = 0, mfn = xen_colored_mfn(start_mfn); \ + i < (_end - _start) >> PAGE_SHIFT; \ + i++, mfn = xen_colored_mfn(mfn_add(mfn, 1)) ) + unsigned int get_llc_way_size(void); void arch_llc_coloring_init(void); int dom0_set_llc_colors(struct domain *d); @@ -XXX,XX +XXX,XX @@ struct page_info; unsigned int page_to_llc_color(const struct page_info *pg); unsigned int get_max_nr_llc_colors(void); +mfn_t xen_colored_mfn(mfn_t mfn); + #endif /* __COLORING_H__ */ /* -- 2.43.0