support NUMA emulation for genertic arch

[PATCH v1 0/2] support NUMA emulation for genertic arch

Posted by Rongwei Wang 1 year, 11 months ago

A brief introduction
====================

The NUMA emulation can fake more node base on a single
node system, e.g.

one node system:

[root@localhost ~]# numactl -H
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 31788 MB
node 0 free: 31446 MB
node distances:
node   0
  0:  10

add numa=fake=2 (fake 2 node on each origin node):

[root@localhost ~]# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 15806 MB
node 0 free: 15451 MB
node 1 cpus: 0 1 2 3 4 5 6 7
node 1 size: 16029 MB
node 1 free: 15989 MB
node distances:
node   0   1
  0:  10  10
  1:  10  10

As above shown, a new node has been faked. As cpus, the realization
of x86 NUMA emulation is kept. Maybe each node should has 4 cores is
better (not sure, next to do if so).

Why do this
===========

It seems has following reasons:
  (1) In x86 host, apply NUMA emulation can fake more nodes environment
      to test or verify some performance stuff, but arm64 only has
      one method that modify ACPI table to do this. It's troublesome
      more or less.
  (2) Reduce competition for some locks. Here an example we found:
      will-it-scale/tlb_flush1_processes -t 96 -s 10, it shows obvious
      hotspot on lruvec->lock when test in single environment. What's
      more, The performance improved greatly if test in two more nodes
      system. The data shows below (more is better):

      ---------------------------------------------------------------------
      threads/process |   1     |     12   |     24   |   48     |   96
      ---------------------------------------------------------------------
      one node        | 14 1122 | 110 5372 | 111 2615 | 79 7084  | 72 4516
      ---------------------------------------------------------------------
      numa=fake=2     | 14 1168 | 144 4848 | 215 9070 | 157 0412 | 142 3968
      ---------------------------------------------------------------------
                      | For concurrency 12, no lruvec->lock hotspot. For 24,
      hotspot         | one node has 24% hotspot on lruvec->lock, but
                      | two nodes env hasn't.
      ---------------------------------------------------------------------

As for risks (e.g. numa balance...), they need to be discussed here.

Lastly, it seems not a good choice to realize x86 and other genertic
archs separately. But it can indeed avoid some architecture related
APIs adjustments and alleviate future maintenance. The previous RFC
link see [1].

Any advice are welcome, Thanks!

Change log
==========

RFC v1 -> v1
* add new CONFIG_NUMA_FAKE for genertic archs.
* keep x86 implementation, realize numa emulation in driver/base/ for
  genertic arch, e.g, arm64.

[1] RFC v1: https://patchwork.kernel.org/project/linux-arm-kernel/cover/20231012024842.99703-1-rongwei.wang@linux.alibaba.com/

Rongwei Wang (2):
  arch_numa: remove __init for early_cpu_to_node
  numa: introduce numa emulation for genertic arch

 drivers/base/Kconfig          |   9 +
 drivers/base/Makefile         |   1 +
 drivers/base/arch_numa.c      |  32 +-
 drivers/base/numa_emulation.c | 909 ++++++++++++++++++++++++++++++++++
 drivers/base/numa_emulation.h |  41 ++
 include/asm-generic/numa.h    |   2 +-
 6 files changed, 992 insertions(+), 2 deletions(-)
 create mode 100644 drivers/base/numa_emulation.c
 create mode 100644 drivers/base/numa_emulation.h

-- 
2.32.0.3.gf3a3e56d6

Re: [PATCH v1 0/2] support NUMA emulation for genertic arch

Posted by Mike Rapoport 1 year, 11 months ago

On Tue, Feb 20, 2024 at 07:36:00PM +0800, Rongwei Wang wrote:
> A brief introduction
> ====================
> 
> The NUMA emulation can fake more node base on a single
> node system, e.g.

... 
 
> Lastly, it seems not a good choice to realize x86 and other genertic
> archs separately. But it can indeed avoid some architecture related
> APIs adjustments and alleviate future maintenance.

Why is it a good choice? Copying 1k lines from x86 to a new place and
having to maintain two copies does not sound like a good choice to me.

> The previous RFC link see [1].
> 
> Any advice are welcome, Thanks!
> 
> Change log
> ==========
> 
> RFC v1 -> v1
> * add new CONFIG_NUMA_FAKE for genertic archs.
> * keep x86 implementation, realize numa emulation in driver/base/ for
>   genertic arch, e.g, arm64.
> 
> [1] RFC v1: https://patchwork.kernel.org/project/linux-arm-kernel/cover/20231012024842.99703-1-rongwei.wang@linux.alibaba.com/
> 
> Rongwei Wang (2):
>   arch_numa: remove __init for early_cpu_to_node
>   numa: introduce numa emulation for genertic arch
> 
>  drivers/base/Kconfig          |   9 +
>  drivers/base/Makefile         |   1 +
>  drivers/base/arch_numa.c      |  32 +-
>  drivers/base/numa_emulation.c | 909 ++++++++++++++++++++++++++++++++++
>  drivers/base/numa_emulation.h |  41 ++
>  include/asm-generic/numa.h    |   2 +-
>  6 files changed, 992 insertions(+), 2 deletions(-)
>  create mode 100644 drivers/base/numa_emulation.c
>  create mode 100644 drivers/base/numa_emulation.h
> 
> -- 
> 2.32.0.3.gf3a3e56d6
> 
> 

-- 
Sincerely yours,
Mike.

Re: [PATCH v1 0/2] support NUMA emulation for genertic arch

Posted by Pierre Gondois 1 year, 11 months ago


On 2/21/24 07:12, Mike Rapoport wrote:
> On Tue, Feb 20, 2024 at 07:36:00PM +0800, Rongwei Wang wrote:
>> A brief introduction
>> ====================
>>
>> The NUMA emulation can fake more node base on a single
>> node system, e.g.
> 
> ...
>   
>> Lastly, it seems not a good choice to realize x86 and other genertic
>> archs separately. But it can indeed avoid some architecture related
>> APIs adjustments and alleviate future maintenance.
> 
> Why is it a good choice? Copying 1k lines from x86 to a new place and
> having to maintain two copies does not sound like a good choice to me.

I agree it would be better to avoid duplication and extract the common
code from the original x86 implementation. The RFC seemed to go more
in this direction.
Also NITs:
- genertic -> generic
- there is a 'ifdef CONFIG_X86' in drivers/base/numa_emulation.c,
   but the file should not be used by x86 as the arch doesn't set
   CONFIG_GENERIC_ARCH_NUMA

Regards,
Pierre

> 
>> The previous RFC link see [1].
>>
>> Any advice are welcome, Thanks!
>>
>> Change log
>> ==========
>>
>> RFC v1 -> v1
>> * add new CONFIG_NUMA_FAKE for genertic archs.
>> * keep x86 implementation, realize numa emulation in driver/base/ for
>>    genertic arch, e.g, arm64.
>>
>> [1] RFC v1: https://patchwork.kernel.org/project/linux-arm-kernel/cover/20231012024842.99703-1-rongwei.wang@linux.alibaba.com/
>>
>> Rongwei Wang (2):
>>    arch_numa: remove __init for early_cpu_to_node
>>    numa: introduce numa emulation for genertic arch
>>
>>   drivers/base/Kconfig          |   9 +
>>   drivers/base/Makefile         |   1 +
>>   drivers/base/arch_numa.c      |  32 +-
>>   drivers/base/numa_emulation.c | 909 ++++++++++++++++++++++++++++++++++
>>   drivers/base/numa_emulation.h |  41 ++
>>   include/asm-generic/numa.h    |   2 +-
>>   6 files changed, 992 insertions(+), 2 deletions(-)
>>   create mode 100644 drivers/base/numa_emulation.c
>>   create mode 100644 drivers/base/numa_emulation.h
>>
>> -- 
>> 2.32.0.3.gf3a3e56d6
>>
>>
>

Re: [PATCH v1 0/2] support NUMA emulation for genertic arch

Posted by Rongwei Wang 1 year, 11 months ago


On 2/21/24 11:51 PM, Pierre Gondois wrote:
>
>
> On 2/21/24 07:12, Mike Rapoport wrote:
>> On Tue, Feb 20, 2024 at 07:36:00PM +0800, Rongwei Wang wrote:
>>> A brief introduction
>>> ====================
>>>
>>> The NUMA emulation can fake more node base on a single
>>> node system, e.g.
>>
>> ...
>>> Lastly, it seems not a good choice to realize x86 and other genertic
>>> archs separately. But it can indeed avoid some architecture related
>>> APIs adjustments and alleviate future maintenance.
>>
>> Why is it a good choice? Copying 1k lines from x86 to a new place and
>> having to maintain two copies does not sound like a good choice to me.
Hi Pierre
> I agree it would be better to avoid duplication and extract the common
> code from the original x86 implementation. The RFC seemed to go more
> in this direction.
> Also NITs:
> - genertic -> generic
Thanks, my fault, zhaoyu also found this (thanks).
> - there is a 'ifdef CONFIG_X86' in drivers/base/numa_emulation.c,
>   but the file should not be used by x86 as the arch doesn't set
>   CONFIG_GENERIC_ARCH_NUMA
>
Actually, I have not think about how to ask the question. I'm also try 
to original direction like RFC version, but found much APIs need to be 
updated, and there are many APIs are similar but a little difference. 
That seems much modification needed in more than one arch if go in 
original direction.

But if all think original method is right, I will continue it in RFC 
version.

Thanks for your time to review.
> Regards,
> Pierre
>
>>
>>> The previous RFC link see [1].
>>>
>>> Any advice are welcome, Thanks!
>>>
>>> Change log
>>> ==========
>>>
>>> RFC v1 -> v1
>>> * add new CONFIG_NUMA_FAKE for genertic archs.
>>> * keep x86 implementation, realize numa emulation in driver/base/ for
>>>    genertic arch, e.g, arm64.
>>>
>>> [1] RFC v1: 
>>> https://patchwork.kernel.org/project/linux-arm-kernel/cover/20231012024842.99703-1-rongwei.wang@linux.alibaba.com/
>>>
>>> Rongwei Wang (2):
>>>    arch_numa: remove __init for early_cpu_to_node
>>>    numa: introduce numa emulation for genertic arch
>>>
>>>   drivers/base/Kconfig          |   9 +
>>>   drivers/base/Makefile         |   1 +
>>>   drivers/base/arch_numa.c      |  32 +-
>>>   drivers/base/numa_emulation.c | 909 
>>> ++++++++++++++++++++++++++++++++++
>>>   drivers/base/numa_emulation.h |  41 ++
>>>   include/asm-generic/numa.h    |   2 +-
>>>   6 files changed, 992 insertions(+), 2 deletions(-)
>>>   create mode 100644 drivers/base/numa_emulation.c
>>>   create mode 100644 drivers/base/numa_emulation.h
>>>
>>> -- 
>>> 2.32.0.3.gf3a3e56d6
>>>
>>>
>>

-- 
Thanks,
-wrw