Hi Stefano,
> -----Original Message-----
> From: Stefano Stabellini <sstabellini@kernel.org>
> Sent: 2021年8月26日 8:09
> To: Wei Chen <Wei.Chen@arm.com>
> Cc: xen-devel@lists.xenproject.org; sstabellini@kernel.org; julien@xen.org;
> jbeulich@suse.com; Bertrand Marquis <Bertrand.Marquis@arm.com>;
> andrew.cooper3@citrix.com
> Subject: Re: [XEN RFC PATCH 00/40] Add device tree based NUMA support to
> Arm64
>
> Thanks for the big contribution!
>
> I just wanted to let you know that the series passed all the gitlab-ci
> build tests without issues.
>
> The runtime tests originally failed due to unrelated problems (there was
> a Debian testing upgrade that broke Gitlab-CI.) I fix the underlying
> issue and restarted the failed tests and now they passed.
>
> This is the pipeline:
> https://gitlab.com/xen-project/patchew/xen/-/pipelines/351484940
>
> There are still two runtime x86 tests that fail but I don't think the
> failures are related to your series.
>
>
Thanks for testing this series : )
> On Wed, 11 Aug 2021, Wei Chen wrote:
> > Xen memory allocation and scheduler modules are NUMA aware.
> > But actually, on x86 has implemented the architecture APIs
> > to support NUMA. Arm was providing a set of fake architecture
> > APIs to make it compatible with NUMA awared memory allocation
> > and scheduler.
> >
> > Arm system was working well as a single node NUMA system with
> > these fake APIs, because we didn't have multiple nodes NUMA
> > system on Arm. But in recent years, more and more Arm devices
> > support multiple nodes NUMA system. Like TX2, some Hisilicon
> > chips and the Ampere Altra.
> >
> > So now we have a new problem. When Xen is running on these Arm
> > devices, Xen still treat them as single node SMP systems. The
> > NUMA affinity capability of Xen memory allocation and scheduler
> > becomes meaningless. Because they rely on input data that does
> > not reflect real NUMA layout.
> >
> > Xen still think the access time for all of the memory is the
> > same for all CPUs. However, Xen may allocate memory to a VM
> > from different NUMA nodes with different access speeds. This
> > difference can be amplified in workloads inside VM, causing
> > performance instability and timeouts.
> >
> > So in this patch series, we implement a set of NUMA API to use
> > device tree to describe the NUMA layout. We reuse most of the
> > code of x86 NUMA to create and maintain the mapping between
> > memory and CPU, create the matrix between any two NUMA nodes.
> > Except ACPI and some x86 specified code, we have moved other
> > code to common. In next stage, when we implement ACPI based
> > NUMA for Arm64, we may move the ACPI NUMA code to common too,
> > but in current stage, we keep it as x86 only.
> >
> > This patch serires has been tested and booted well on one
> > Arm64 NUMA machine and one HPE x86 NUMA machine.
> >
> > Hongda Deng (2):
> > xen/arm: return default DMA bit width when platform is not set
> > xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0
> >
> > Wei Chen (38):
> > tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf
> > xen/arm: Print a 64-bit number in hex from early uart
> > xen/x86: Initialize memnodemapsize while faking NUMA node
> > xen: decouple NUMA from ACPI in Kconfig
> > xen/arm: use !CONFIG_NUMA to keep fake NUMA API
> > xen/x86: Move NUMA memory node map functions to common
> > xen/x86: Move numa_add_cpu_node to common
> > xen/x86: Move NR_NODE_MEMBLKS macro to common
> > xen/x86: Move NUMA nodes and memory block ranges to common
> > xen/x86: Move numa_initmem_init to common
> > xen/arm: introduce numa_set_node for Arm
> > xen/arm: set NUMA nodes max number to 64 by default
> > xen/x86: move NUMA API from x86 header to common header
> > xen/arm: Create a fake NUMA node to use common code
> > xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
> > xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
> > xen: fdt: Introduce a helper to check fdt node type
> > xen/arm: implement node distance helpers for Arm64
> > xen/arm: introduce device_tree_numa as a switch for device tree NUMA
> > xen/arm: introduce a helper to parse device tree processor node
> > xen/arm: introduce a helper to parse device tree memory node
> > xen/arm: introduce a helper to parse device tree NUMA distance map
> > xen/arm: unified entry to parse all NUMA data from device tree
> > xen/arm: Add boot and secondary CPU to NUMA system
> > xen/arm: build CPU NUMA node map while creating cpu_logical_map
> > xen/x86: decouple nodes_cover_memory with E820 map
> > xen/arm: implement Arm arch helpers Arm to get memory map info
> > xen: move NUMA memory and CPU parsed nodemasks to common
> > xen/x86: move nodes_cover_memory to common
> > xen/x86: make acpi_scan_nodes to be neutral
> > xen: export bad_srat and srat_disabled to extern
> > xen: move numa_scan_nodes from x86 to common
> > xen: enable numa_scan_nodes for device tree based NUMA
> > xen/arm: keep guest still be NUMA unware
> > xen: introduce an arch helper to do NUMA init failed fallback
> > xen/arm: enable device tree based NUMA in system init
> > xen/x86: move numa_setup to common to support NUMA switch in command
> > line
> > xen/x86: move dump_numa info hotkey to common
> >
> > tools/libs/util/libxlu_pci.c | 3 +-
> > xen/arch/arm/Kconfig | 10 +
> > xen/arch/arm/Makefile | 2 +
> > xen/arch/arm/arm64/head.S | 9 +-
> > xen/arch/arm/bootfdt.c | 8 +-
> > xen/arch/arm/domain_build.c | 17 +-
> > xen/arch/arm/efi/efi-boot.h | 25 --
> > xen/arch/arm/numa.c | 162 +++++++++
> > xen/arch/arm/numa_device_tree.c | 292 ++++++++++++++++
> > xen/arch/arm/platform.c | 4 +-
> > xen/arch/arm/setup.c | 14 +
> > xen/arch/arm/smpboot.c | 37 +-
> > xen/arch/x86/Kconfig | 2 +-
> > xen/arch/x86/numa.c | 421 +----------------------
> > xen/arch/x86/srat.c | 147 +-------
> > xen/common/Kconfig | 3 +
> > xen/common/Makefile | 1 +
> > xen/common/libfdt/fdt_ro.c | 15 +
> > xen/common/numa.c | 588 ++++++++++++++++++++++++++++++++
> > xen/common/page_alloc.c | 2 +-
> > xen/drivers/acpi/Kconfig | 3 +-
> > xen/drivers/acpi/Makefile | 2 +-
> > xen/include/asm-arm/numa.h | 33 ++
> > xen/include/asm-arm/setup.h | 6 +
> > xen/include/asm-x86/acpi.h | 4 -
> > xen/include/asm-x86/config.h | 1 -
> > xen/include/asm-x86/numa.h | 65 +---
> > xen/include/asm-x86/setup.h | 1 -
> > xen/include/xen/libfdt/libfdt.h | 25 ++
> > xen/include/xen/nodemask.h | 2 +
> > xen/include/xen/numa.h | 80 +++++
> > 31 files changed, 1325 insertions(+), 659 deletions(-)
> > create mode 100644 xen/arch/arm/numa.c
> > create mode 100644 xen/arch/arm/numa_device_tree.c
> > create mode 100644 xen/common/numa.c
> >
> > --
> > 2.25.1
> >