include/linux/hugetlb.h | 2 +- include/linux/padata.h | 2 + kernel/padata.c | 8 +- mm/hugetlb.c | 201 +++++++++++++++++++++++++++------------- mm/mm_init.c | 1 + 5 files changed, 148 insertions(+), 66 deletions(-)
Hi all, hugetlb init parallelization has now been updated to v2.
To David Hildenbrand: padata multithread utilities has been used to reduce
code complexity.
To David Rientjes: The patch for measuring time will be separately included
in the reply. Please test during your free time, thanks.
# Introduction
Hugetlb initialization during boot takes up a considerable amount of time.
For instance, on a 2TB system, initializing 1,800 1GB huge pages takes 1-2
seconds out of 10 seconds. Initializing 11,776 1GB pages on a 12TB Intel
host takes 65.2 seconds [1], which is 17.4% of the total 373.78 seconds boot
time. This is a noteworthy figure.
Inspired by [2] and [3], hugetlb initialization can also be accelerated
through parallelization. Kernel already has infrastructure like
padata_do_multithreaded, this patch uses it to achieve effective results
by minimal modifications.
[1] https://lore.kernel.org/all/783f8bac-55b8-5b95-eb6a-11a583675000@google.com/
[2] https://lore.kernel.org/all/20200527173608.2885243-1-daniel.m.jordan@oracle.com/
[3] https://lore.kernel.org/all/20230906112605.2286994-1-usama.arif@bytedance.com/
# Test result
test no patch(ms) patched(ms) saved
------------------- -------------- ------------- --------
256c2t(4 node) 2M 2624 956 63.57%
256c2t(4 node) 1G 2679 1582 40.95%
128c1t(2 node) 2M 1788 684 61.74%
128c1t(2 node) 1G 3160 1618 48.80%
# Change log
Changes in v2:
- Reduce complexity with `padata_do_multithreaded`
- Support 1G hugetlb
v1:
- https://lore.kernel.org/all/20231123133036.68540-1-gang.li@linux.dev/
- parallelize 2M hugetlb initialization with workqueue
Gang Li (5):
hugetlb: code clean for hugetlb_hstate_alloc_pages
hugetlb: split hugetlb_hstate_alloc_pages
padata: dispatch works on different nodes
hugetlb: parallelize 2M hugetlb allocation and initialization
hugetlb: parallelize 1G hugetlb initialization
include/linux/hugetlb.h | 2 +-
include/linux/padata.h | 2 +
kernel/padata.c | 8 +-
mm/hugetlb.c | 201 +++++++++++++++++++++++++++-------------
mm/mm_init.c | 1 +
5 files changed, 148 insertions(+), 66 deletions(-)
--
2.30.2
On Fri, 8 Dec 2023, Gang Li wrote: > Hi all, hugetlb init parallelization has now been updated to v2. > > To David Hildenbrand: padata multithread utilities has been used to reduce > code complexity. > > To David Rientjes: The patch for measuring time will be separately included > in the reply. Please test during your free time, thanks. > I'd love to, but what kernel is this based on? :) I can't get this to apply to any kernels that I have recently benchmarked with. > # Introduction > Hugetlb initialization during boot takes up a considerable amount of time. > For instance, on a 2TB system, initializing 1,800 1GB huge pages takes 1-2 > seconds out of 10 seconds. Initializing 11,776 1GB pages on a 12TB Intel > host takes 65.2 seconds [1], which is 17.4% of the total 373.78 seconds boot > time. This is a noteworthy figure. > > Inspired by [2] and [3], hugetlb initialization can also be accelerated > through parallelization. Kernel already has infrastructure like > padata_do_multithreaded, this patch uses it to achieve effective results > by minimal modifications. > > [1] https://lore.kernel.org/all/783f8bac-55b8-5b95-eb6a-11a583675000@google.com/ > [2] https://lore.kernel.org/all/20200527173608.2885243-1-daniel.m.jordan@oracle.com/ > [3] https://lore.kernel.org/all/20230906112605.2286994-1-usama.arif@bytedance.com/ > > # Test result > test no patch(ms) patched(ms) saved > ------------------- -------------- ------------- -------- > 256c2t(4 node) 2M 2624 956 63.57% > 256c2t(4 node) 1G 2679 1582 40.95% > 128c1t(2 node) 2M 1788 684 61.74% > 128c1t(2 node) 1G 3160 1618 48.80% > > # Change log > Changes in v2: > - Reduce complexity with `padata_do_multithreaded` > - Support 1G hugetlb > > v1: > - https://lore.kernel.org/all/20231123133036.68540-1-gang.li@linux.dev/ > - parallelize 2M hugetlb initialization with workqueue > > Gang Li (5): > hugetlb: code clean for hugetlb_hstate_alloc_pages > hugetlb: split hugetlb_hstate_alloc_pages > padata: dispatch works on different nodes > hugetlb: parallelize 2M hugetlb allocation and initialization > hugetlb: parallelize 1G hugetlb initialization > > include/linux/hugetlb.h | 2 +- > include/linux/padata.h | 2 + > kernel/padata.c | 8 +- > mm/hugetlb.c | 201 +++++++++++++++++++++++++++------------- > mm/mm_init.c | 1 + > 5 files changed, 148 insertions(+), 66 deletions(-) > > -- > 2.30.2 > >
On 12/12/23 14:14, David Rientjes wrote: > On Fri, 8 Dec 2023, Gang Li wrote: > > > Hi all, hugetlb init parallelization has now been updated to v2. > > > > To David Hildenbrand: padata multithread utilities has been used to reduce > > code complexity. > > > > To David Rientjes: The patch for measuring time will be separately included > > in the reply. Please test during your free time, thanks. > > > > I'd love to, but what kernel is this based on? :) I can't get this to > apply to any kernels that I have recently benchmarked with. I was able to apply and build on top of v6.7-rc5. Gang Li, Since hugetlb now depends on CONFIG_PADATA, the Kconfig file should be updated to reflect this. -- Mike Kravetz
On Tue, 12 Dec 2023, Mike Kravetz wrote:
> On 12/12/23 14:14, David Rientjes wrote:
> > On Fri, 8 Dec 2023, Gang Li wrote:
> >
> > > Hi all, hugetlb init parallelization has now been updated to v2.
> > >
> > > To David Hildenbrand: padata multithread utilities has been used to reduce
> > > code complexity.
> > >
> > > To David Rientjes: The patch for measuring time will be separately included
> > > in the reply. Please test during your free time, thanks.
> > >
> >
> > I'd love to, but what kernel is this based on? :) I can't get this to
> > apply to any kernels that I have recently benchmarked with.
>
> I was able to apply and build on top of v6.7-rc5.
>
> Gang Li,
> Since hugetlb now depends on CONFIG_PADATA, the Kconfig file should be
> updated to reflect this.
Gotcha, thanks.
I got this:
ld: error: undefined symbol: padata_do_multithreaded
referenced by hugetlb.c:3470 (./mm/hugetlb.c:3470)
vmlinux.o:(gather_bootmem_prealloc)
referenced by hugetlb.c:3592 (./mm/hugetlb.c:3592)
vmlinux.o:(hugetlb_hstate_alloc_pages_non_gigantic)
referenced by hugetlb.c:3599 (./mm/hugetlb.c:3599)
vmlinux.o:(hugetlb_hstate_alloc_pages_non_gigantic)
So, yeah we need to enable DEFERRED_STRUCT_PAGE_INIT for this to build.
On 6.6 I measured "hugepagesz=1G hugepages=11776" on as 12TB host to be
77s this time around.
A latest Linus build with this patch set does not boot successfully, so
I'll need to look into that and try to capture the failure. Not sure if
it's related to this patch or the latest Linus build in general.
Hi, On 2023/12/13 08:10, David Rientjes wrote: > On 6.6 I measured "hugepagesz=1G hugepages=11776" on as 12TB host to be > 77s this time around. Thanks for your test! Is this the total kernel boot time, or just the hugetlb initialization time? > > A latest Linus build with this patch set does not boot successfully, so Which branch/tag is it compiled on? I test this patch on v6.7-rc4 and next-20231130. > I'll need to look into that and try to capture the failure. Not sure if > it's related to this patch or the latest Linus build in general. >
On Mon, 18 Dec 2023, Gang Li wrote: > Hi, > > On 2023/12/13 08:10, David Rientjes wrote: > > On 6.6 I measured "hugepagesz=1G hugepages=11776" on as 12TB host to be > > 77s this time around. > > Thanks for your test! Is this the total kernel boot time, or just the > hugetlb initialization time? > Ah, sorry for not being specific. It's just the hugetlb preallocation of 11776 1GB hugetlb pages, total boot takes a few more minutes. > > A latest Linus build with this patch set does not boot successfully, so > > Which branch/tag is it compiled on? > I test this patch on v6.7-rc4 and next-20231130. > It was the latest Linus tip of tree. I'll continue to try again until I get a successful boot and report back, serial console won't be possible for unrelated reasons.
On Thu, 21 Dec 2023, David Rientjes wrote:
> > Hi,
> >
> > On 2023/12/13 08:10, David Rientjes wrote:
> > > On 6.6 I measured "hugepagesz=1G hugepages=11776" on as 12TB host to be
> > > 77s this time around.
> >
> > Thanks for your test! Is this the total kernel boot time, or just the
> > hugetlb initialization time?
> >
>
> Ah, sorry for not being specific. It's just the hugetlb preallocation of
> 11776 1GB hugetlb pages, total boot takes a few more minutes.
>
I had to apply this to get the patch series to compile on 6.7-rc7:
diff --git a/kernel/padata.c b/kernel/padata.c
--- a/kernel/padata.c
+++ b/kernel/padata.c
@@ -485,7 +485,7 @@ void __init padata_do_multithreaded(struct padata_mt_job *job)
struct padata_work my_work, *pw;
struct padata_mt_job_state ps;
LIST_HEAD(works);
- int nworks, nid;
+ int nworks, nid = 0;
if (job->size == 0)
return;
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3300,7 +3300,7 @@ int alloc_bootmem_huge_page(struct hstate *h, int nid)
int __alloc_bootmem_huge_page(struct hstate *h, int nid)
{
struct huge_bootmem_page *m = NULL; /* initialize for clang */
- int nr_nodes, node;
+ int nr_nodes, node = NUMA_NO_NODE;
/* do node specific alloc */
if (nid != NUMA_NO_NODE) {
With that, I compared "hugepagesz=1G hugepages=11776" before and after on
a 12TB host with eight NUMA nodes.
Compared to 77s of total initialization time before, with this series I
measured 18.3s.
Feel free to add this into the changelog once the initialization issues
are fixed up and I'm happy to ack it.
Thanks!
On 2023/12/25 13:21, David Rientjes wrote: > With that, I compared "hugepagesz=1G hugepages=11776" before and after on > a 12TB host with eight NUMA nodes. > > Compared to 77s of total initialization time before, with this series I > measured 18.3s. > > Feel free to add this into the changelog once the initialization issues > are fixed up and I'm happy to ack it. > > Thanks! Cool! Thank you ;)
On 12/08/23 10:52, Gang Li wrote:
> Hi all, hugetlb init parallelization has now been updated to v2.
Thanks for your efforts, and sorry for my late comments.
> To David Hildenbrand: padata multithread utilities has been used to reduce
> code complexity.
>
> To David Rientjes: The patch for measuring time will be separately included
> in the reply. Please test during your free time, thanks.
>
> # Introduction
> Hugetlb initialization during boot takes up a considerable amount of time.
> For instance, on a 2TB system, initializing 1,800 1GB huge pages takes 1-2
> seconds out of 10 seconds. Initializing 11,776 1GB pages on a 12TB Intel
> host takes 65.2 seconds [1], which is 17.4% of the total 373.78 seconds boot
> time. This is a noteworthy figure.
One issue to be concerned with is hugetlb page allocation on systems with
unbalanced numa node memory. Commit f60858f9d327 ("hugetlbfs: don't retry
when pool page allocations start to fail") was added to deal with issues
reported on such systems. So, users are certainly using hugetlb pages
on systems with imbalances.
If performing allocations in parallel, I believe we would want the total
number of hugetlb pages allocated to be the same as today. For example,
consider a simple 2 node system with 16GB total memory:
node 0: 2GB
node 1: 14GB
With today's code, allocating 6656 2MB pages via the kernel command line
results in:
node 0: 924 pages
node 1: 5732 pages
total: 6656 pages
With code to parallel allocations in this series:
node 0: 924 pages
node 1: 1547 pages
total: 2471 pages
--
Mike Kravetz
On 2023/12/13 04:06, Mike Kravetz wrote: > With today's code, allocating 6656 2MB pages via the kernel command line > results in: > node 0: 924 pages > node 1: 5732 pages > total: 6656 pages > > With code to parallel allocations in this series: > node 0: 924 pages > node 1: 1547 pages > total: 2471 pages Hi Mike, Disable numa_aware for hugetlb_alloc_node should solve this problem. I will fix it in v3.
© 2016 - 2025 Red Hat, Inc.