From nobody Tue Dec 16 16:37:59 2025 Received: from out-177.mta0.migadu.com (out-177.mta0.migadu.com [91.218.175.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 782EC1474A4 for ; Thu, 22 Feb 2024 14:04:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610690; cv=none; b=sGUFOgsC4egy+wuHmRHyKGNgmxB2uj7P1iSuywJSHRxCkexdtNWlfDd6YjbGsseFfEv+SEsaWiMwkKCtphkX64yPL70zK+rVj6PYCDOjMjrzztO1FPKV8Ho68/c9U1MlM9dL2MsiiCEO+F3Itp15rdLsvletDf8ps2Crl1FIlBE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610690; c=relaxed/simple; bh=9Nlv5yU9lraOMOB9dfIUMrxAw7ypqQN0Obqwia5pjOo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Tv7tR7D1/QoVuyuZxrH9a9W651ymOGuK9NDh7wCJ+z/oF/dUw8TExGAvetfsvexl7IBu0KNBQmvGtLSl2bDhOhUgAlaOSiR+5NklFEDGUKZF524oNw8KnnOBp9+/CXybxoz1Zpz2QcB4qKFSd9jCOoL+FlYGYNW/ny7+jOgAIBk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=i3uRvd9J; arc=none smtp.client-ip=91.218.175.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="i3uRvd9J" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1708610686; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qLDJ0oXYyRPh9zz0wE8IXI1bSRdqq7PDnnR6TGU9VgU=; b=i3uRvd9JstCtGlLa8/tyWtExImE0JtGxaojcIRs7n7nhFDNAP1r0mFxWuFGnV0DVJck360 +ybW3ROURTCTJyB8p7i3fvB985dnVQ3LNtm4RXv8GdUo6fpAsYQ3cBv0H6OF+4mHm7dqT7 SiYS5mOQGsqbMvVVng+9nU3mjbwRkQs= From: Gang Li To: Andrew Morton Cc: David Hildenbrand , David Rientjes , Muchun Song , Tim Chen , Steffen Klassert , Daniel Jordan , Jane Chu , "Paul E . McKenney" , Randy Dunlap , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v6 1/8] hugetlb: code clean for hugetlb_hstate_alloc_pages Date: Thu, 22 Feb 2024 22:04:14 +0800 Message-Id: <20240222140422.393911-2-gang.li@linux.dev> In-Reply-To: <20240222140422.393911-1-gang.li@linux.dev> References: <20240222140422.393911-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" The readability of `hugetlb_hstate_alloc_pages` is poor. By cleaning the code, its readability can be improved, facilitating future modifications. This patch extracts two functions to reduce the complexity of `hugetlb_hstate_alloc_pages` and has no functional changes. - hugetlb_hstate_alloc_pages_node_specific() to handle iterates through each online node and performs allocation if necessary. - hugetlb_hstate_alloc_pages_report() report error during allocation. And the value of h->max_huge_pages is updated accordingly. Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Muchun Song Reviewed-by: Tim Chen --- mm/hugetlb.c | 46 +++++++++++++++++++++++++++++----------------- 1 file changed, 29 insertions(+), 17 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c53a41d07cd3a..794f3e6a19bb6 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3482,6 +3482,33 @@ static void __init hugetlb_hstate_alloc_pages_onenod= e(struct hstate *h, int nid) h->max_huge_pages_node[nid] =3D i; } =20 +static bool __init hugetlb_hstate_alloc_pages_specific_nodes(struct hstate= *h) +{ + int i; + bool node_specific_alloc =3D false; + + for_each_online_node(i) { + if (h->max_huge_pages_node[i] > 0) { + hugetlb_hstate_alloc_pages_onenode(h, i); + node_specific_alloc =3D true; + } + } + + return node_specific_alloc; +} + +static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long alloc= ated, struct hstate *h) +{ + if (allocated < h->max_huge_pages) { + char buf[32]; + + string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); + pr_warn("HugeTLB: allocating %lu of page size %s failed. Only allocated= %lu hugepages.\n", + h->max_huge_pages, buf, allocated); + h->max_huge_pages =3D allocated; + } +} + /* * NOTE: this routine is called in different contexts for gigantic and * non-gigantic pages. @@ -3499,7 +3526,6 @@ static void __init hugetlb_hstate_alloc_pages(struct = hstate *h) struct folio *folio; LIST_HEAD(folio_list); nodemask_t *node_alloc_noretry; - bool node_specific_alloc =3D false; =20 /* skip gigantic hugepages allocation if hugetlb_cma enabled */ if (hstate_is_gigantic(h) && hugetlb_cma_size) { @@ -3508,14 +3534,7 @@ static void __init hugetlb_hstate_alloc_pages(struct= hstate *h) } =20 /* do node specific alloc */ - for_each_online_node(i) { - if (h->max_huge_pages_node[i] > 0) { - hugetlb_hstate_alloc_pages_onenode(h, i); - node_specific_alloc =3D true; - } - } - - if (node_specific_alloc) + if (hugetlb_hstate_alloc_pages_specific_nodes(h)) return; =20 /* below will do all node balanced alloc */ @@ -3558,14 +3577,7 @@ static void __init hugetlb_hstate_alloc_pages(struct= hstate *h) /* list will be empty if hstate_is_gigantic */ prep_and_add_allocated_folios(h, &folio_list); =20 - if (i < h->max_huge_pages) { - char buf[32]; - - string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); - pr_warn("HugeTLB: allocating %lu of page size %s failed. Only allocated= %lu hugepages.\n", - h->max_huge_pages, buf, i); - h->max_huge_pages =3D i; - } + hugetlb_hstate_alloc_pages_errcheck(i, h); kfree(node_alloc_noretry); } =20 --=20 2.20.1 From nobody Tue Dec 16 16:37:59 2025 Received: from out-187.mta0.migadu.com (out-187.mta0.migadu.com [91.218.175.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E1981474B3 for ; Thu, 22 Feb 2024 14:04:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.187 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610693; cv=none; b=fnp6E9R21YzuV4+vMmmy6x/eUqtbtgoXr1iM4ERzKPKh2Z/zhxdAjkOP3n9ompLIhAzF4WZMgvn0yiQKL1e45sCp455/xvNpGF3nqbQPFCsP/J+8SR/HBdBisTwsYy+I4TR2ItugOyp8+h4mIfLAaDTb8t/EIquk+ZfqoMWPh84= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610693; c=relaxed/simple; bh=vRuxUrdqrEjW/kmQKHVnf2KKQjRlpz3t45eHDJuC1Z8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Sf+Fkr8HhW+ZK2vz8JSRm2tGDHbmGizSuWT8yDISkkNjH11VAiKYmWA+5u4yLqe0BUh0Ow+hPPstokk5SyTNPNVytqBgBmYhpjAH9kJ8Jsh2VWgzl2TPtSYkHVKA/UqcJpnATX9i02Zk0GmRa2i7QqvDxBMXEncB9sIThA/81p4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=ZZ38NdKG; arc=none smtp.client-ip=91.218.175.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="ZZ38NdKG" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1708610690; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GFm+BsVb+EJLE4T/S60F9nhII3K5Okpm0T3KX+Ypj6g=; b=ZZ38NdKGJjXCAsgG44/Qo//cf/3cc1EknNNpxFahUoIgcacNCf55oBecrLxSCFmnIx/6Yc zc0joimqN7Kl9zD1PYmr5FlyOXa9qomyj3kBYRaaA7C375f+IuMQnFTecYf2S6/jKeHeH3 JIJykJuzg6dwPf8qafigyzC4+gneD6Q= From: Gang Li To: Andrew Morton Cc: David Hildenbrand , David Rientjes , Muchun Song , Tim Chen , Steffen Klassert , Daniel Jordan , Jane Chu , "Paul E . McKenney" , Randy Dunlap , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v6 2/8] hugetlb: split hugetlb_hstate_alloc_pages Date: Thu, 22 Feb 2024 22:04:15 +0800 Message-Id: <20240222140422.393911-3-gang.li@linux.dev> In-Reply-To: <20240222140422.393911-1-gang.li@linux.dev> References: <20240222140422.393911-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" 1G and 2M huge pages have different allocation and initialization logic, which leads to subtle differences in parallelization. Therefore, it is appropriate to split hugetlb_hstate_alloc_pages into gigantic and non-gigantic. This patch has no functional changes. Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Tim Chen Reviewed-by: Muchun Song --- mm/hugetlb.c | 87 ++++++++++++++++++++++++++-------------------------- 1 file changed, 43 insertions(+), 44 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 794f3e6a19bb6..647e52596e2da 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3509,6 +3509,43 @@ static void __init hugetlb_hstate_alloc_pages_errche= ck(unsigned long allocated, } } =20 +static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstat= e *h) +{ + unsigned long i; + + for (i =3D 0; i < h->max_huge_pages; ++i) { + if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE)) + break; + cond_resched(); + } + + return i; +} + +static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h) +{ + unsigned long i; + struct folio *folio; + LIST_HEAD(folio_list); + nodemask_t node_alloc_noretry; + + /* Bit mask controlling how hard we retry per-node allocations.*/ + nodes_clear(node_alloc_noretry); + + for (i =3D 0; i < h->max_huge_pages; ++i) { + folio =3D alloc_pool_huge_folio(h, &node_states[N_MEMORY], + &node_alloc_noretry); + if (!folio) + break; + list_add(&folio->lru, &folio_list); + cond_resched(); + } + + prep_and_add_allocated_folios(h, &folio_list); + + return i; +} + /* * NOTE: this routine is called in different contexts for gigantic and * non-gigantic pages. @@ -3522,10 +3559,7 @@ static void __init hugetlb_hstate_alloc_pages_errche= ck(unsigned long allocated, */ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { - unsigned long i; - struct folio *folio; - LIST_HEAD(folio_list); - nodemask_t *node_alloc_noretry; + unsigned long allocated; =20 /* skip gigantic hugepages allocation if hugetlb_cma enabled */ if (hstate_is_gigantic(h) && hugetlb_cma_size) { @@ -3538,47 +3572,12 @@ static void __init hugetlb_hstate_alloc_pages(struc= t hstate *h) return; =20 /* below will do all node balanced alloc */ - if (!hstate_is_gigantic(h)) { - /* - * Bit mask controlling how hard we retry per-node allocations. - * Ignore errors as lower level routines can deal with - * node_alloc_noretry =3D=3D NULL. If this kmalloc fails at boot - * time, we are likely in bigger trouble. - */ - node_alloc_noretry =3D kmalloc(sizeof(*node_alloc_noretry), - GFP_KERNEL); - } else { - /* allocations done at boot time */ - node_alloc_noretry =3D NULL; - } - - /* bit mask controlling how hard we retry per-node allocations */ - if (node_alloc_noretry) - nodes_clear(*node_alloc_noretry); - - for (i =3D 0; i < h->max_huge_pages; ++i) { - if (hstate_is_gigantic(h)) { - /* - * gigantic pages not added to list as they are not - * added to pools now. - */ - if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE)) - break; - } else { - folio =3D alloc_pool_huge_folio(h, &node_states[N_MEMORY], - node_alloc_noretry); - if (!folio) - break; - list_add(&folio->lru, &folio_list); - } - cond_resched(); - } - - /* list will be empty if hstate_is_gigantic */ - prep_and_add_allocated_folios(h, &folio_list); + if (hstate_is_gigantic(h)) + allocated =3D hugetlb_gigantic_pages_alloc_boot(h); + else + allocated =3D hugetlb_pages_alloc_boot(h); =20 - hugetlb_hstate_alloc_pages_errcheck(i, h); - kfree(node_alloc_noretry); + hugetlb_hstate_alloc_pages_errcheck(allocated, h); } =20 static void __init hugetlb_init_hstates(void) --=20 2.20.1 From nobody Tue Dec 16 16:37:59 2025 Received: from out-170.mta0.migadu.com (out-170.mta0.migadu.com [91.218.175.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 91E60148FFE for ; Thu, 22 Feb 2024 14:04:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610697; cv=none; b=q3WlXVQ9i5sGxKTaFuWmWDF7ELpaTPfWelgQe00yMCjyT76JFCvvdGqhFciq7INihQajgVJat+s7tz6nQ3FP7FNGuNaGxEQHPMGrtPUmZ3UMYFcioIpOjI7TTKrufSfd2Kc4EpzjBE2zsdKCbYnlZ6/rPiGUQxVgyHhhUQ8dVe0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610697; c=relaxed/simple; bh=GJdtD9vyuX0d/G7xkiT5uC/YyxLu8BmsN8PyxCBFDIM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=XwvJ8TUoV7BAnu+mRfT/UpfiXTNxSq/Xons6FK9TZpgWTD+1ovJ8ER5cM96gBmZL9SPMqlK3j51Mxgk1Dp7HpKlx0U081r+GDeUEMYwEgqVrp5zYFSt0kQvd7HyemXQbyZFImfgeYazW58N8uOxRdXcdBAcBd1xar+7Rq6COCtI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=GMb+VdCD; arc=none smtp.client-ip=91.218.175.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="GMb+VdCD" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1708610693; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=o+8qigIYBf+CF6nmzaQESLXIfknRBaMPQJkianD0US0=; b=GMb+VdCDD/mQRiGk4B7cYFJq6Uvre0TApLBB9BKJ+sRd1zHzqt/ccv8JB7NwYBykKXKcVg yGsJtPCbMDS2g3UmDUDyGkzjAGtoVsFp1hOGY9p3OVE2ZXd6DuJMgkbwGJhkPF9OGmvazP YrigqBnLGrT2H9Joq7NpfDGHlVTmvWM= From: Gang Li To: Andrew Morton Cc: David Hildenbrand , David Rientjes , Muchun Song , Tim Chen , Steffen Klassert , Daniel Jordan , Jane Chu , "Paul E . McKenney" , Randy Dunlap , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v6 3/8] hugetlb: pass *next_nid_to_alloc directly to for_each_node_mask_to_alloc Date: Thu, 22 Feb 2024 22:04:16 +0800 Message-Id: <20240222140422.393911-4-gang.li@linux.dev> In-Reply-To: <20240222140422.393911-1-gang.li@linux.dev> References: <20240222140422.393911-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" With parallelization of hugetlb allocation across different threads, each thread works on a differnet node to allocate pages from, instead of all allocating from a common node h->next_nid_to_alloc. To address this, it's necessary to assign a separate next_nid_to_alloc for each thread. Consequently, the hstate_next_node_to_alloc and for_each_node_mask_to_alloc have been modified to directly accept a *next_nid_to_alloc parameter, ensuring thread-specific allocation and avoiding concurrent access issues. Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Tim Chen Reviewed-by: Muchun Song --- mm/hugetlb.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 647e52596e2da..d1ce1a52ad504 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1464,15 +1464,15 @@ static int get_valid_node_allowed(int nid, nodemask= _t *nodes_allowed) * next node from which to allocate, handling wrap at end of node * mask. */ -static int hstate_next_node_to_alloc(struct hstate *h, +static int hstate_next_node_to_alloc(int *next_node, nodemask_t *nodes_allowed) { int nid; =20 VM_BUG_ON(!nodes_allowed); =20 - nid =3D get_valid_node_allowed(h->next_nid_to_alloc, nodes_allowed); - h->next_nid_to_alloc =3D next_node_allowed(nid, nodes_allowed); + nid =3D get_valid_node_allowed(*next_node, nodes_allowed); + *next_node =3D next_node_allowed(nid, nodes_allowed); =20 return nid; } @@ -1495,10 +1495,10 @@ static int hstate_next_node_to_free(struct hstate *= h, nodemask_t *nodes_allowed) return nid; } =20 -#define for_each_node_mask_to_alloc(hs, nr_nodes, node, mask) \ +#define for_each_node_mask_to_alloc(next_node, nr_nodes, node, mask) \ for (nr_nodes =3D nodes_weight(*mask); \ nr_nodes > 0 && \ - ((node =3D hstate_next_node_to_alloc(hs, mask)) || 1); \ + ((node =3D hstate_next_node_to_alloc(next_node, mask)) || 1); \ nr_nodes--) =20 #define for_each_node_mask_to_free(hs, nr_nodes, node, mask) \ @@ -2350,12 +2350,13 @@ static void prep_and_add_allocated_folios(struct hs= tate *h, */ static struct folio *alloc_pool_huge_folio(struct hstate *h, nodemask_t *nodes_allowed, - nodemask_t *node_alloc_noretry) + nodemask_t *node_alloc_noretry, + int *next_node) { gfp_t gfp_mask =3D htlb_alloc_mask(h) | __GFP_THISNODE; int nr_nodes, node; =20 - for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) { + for_each_node_mask_to_alloc(next_node, nr_nodes, node, nodes_allowed) { struct folio *folio; =20 folio =3D only_alloc_fresh_hugetlb_folio(h, gfp_mask, node, @@ -3310,7 +3311,7 @@ int __alloc_bootmem_huge_page(struct hstate *h, int n= id) goto found; } /* allocate from next node when distributing huge pages */ - for_each_node_mask_to_alloc(h, nr_nodes, node, &node_states[N_MEMORY]) { + for_each_node_mask_to_alloc(&h->next_nid_to_alloc, nr_nodes, node, &node_= states[N_MEMORY]) { m =3D memblock_alloc_try_nid_raw( huge_page_size(h), huge_page_size(h), 0, MEMBLOCK_ALLOC_ACCESSIBLE, node); @@ -3679,7 +3680,7 @@ static int adjust_pool_surplus(struct hstate *h, node= mask_t *nodes_allowed, VM_BUG_ON(delta !=3D -1 && delta !=3D 1); =20 if (delta < 0) { - for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) { + for_each_node_mask_to_alloc(&h->next_nid_to_alloc, nr_nodes, node, nodes= _allowed) { if (h->surplus_huge_pages_node[node]) goto found; } @@ -3794,7 +3795,8 @@ static int set_max_huge_pages(struct hstate *h, unsig= ned long count, int nid, cond_resched(); =20 folio =3D alloc_pool_huge_folio(h, nodes_allowed, - node_alloc_noretry); + node_alloc_noretry, + &h->next_nid_to_alloc); if (!folio) { prep_and_add_allocated_folios(h, &page_list); spin_lock_irq(&hugetlb_lock); --=20 2.20.1 From nobody Tue Dec 16 16:37:59 2025 Received: from out-186.mta0.migadu.com (out-186.mta0.migadu.com [91.218.175.186]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3C32D14A08D for ; Thu, 22 Feb 2024 14:04:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.186 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610701; cv=none; b=nYubhOsTKrxfl+hltQo8yVCbbdjUlYecC5CW7odlg5neyCt86uf942BSKgSAB8NoZq+ht5YlMEUweb8sEQX46Mc/qaXrAl5s+raQRX99JYdO3RUFDvL8Wf//f9JkV1juvEJ90a/K7t17XnrdefbB+68NRFRqA3gF6KNBSFtjjbU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610701; c=relaxed/simple; bh=pKjNCBtJRKlytDtOYpIKfNyflGK1XMd1FZLmFDtWJWQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QPoo8aZI4PLaCnGPrm1+GsPIsvZUI6wj3jVVdKRIKFGwJ8FBaqf9oPDrw2hYzW7SjVKVtUAvA5s2UsES3a5xbehosQc6r145hCtvCROlswi24SLDEUtEzYahTgytNXqRkBExbIYipmd6xliRs2C924wvkE405/NnE8mOiCBIviw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=p1Ko7q4H; arc=none smtp.client-ip=91.218.175.186 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="p1Ko7q4H" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1708610697; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MgzBTPzPKbW03C1p+bf5b2VOJPKCTieqosPALSU6d3A=; b=p1Ko7q4HwLZ8IRyvsRmFPEObb08w0kLTfT8Eo/yUShmw9X8SanRExrBoYN4E8QizQ7JtJr y6CDhqCmusgjbGbPQnWZj7zwS/EdtoiwysOQK8U4EQ2jhWPEDI0N15nmbWXhOwCYYBftee 20rXO2q25jdViRTYxjq4nVcOPEYzQBk= From: Gang Li To: Andrew Morton Cc: David Hildenbrand , David Rientjes , Muchun Song , Tim Chen , Steffen Klassert , Daniel Jordan , Jane Chu , "Paul E . McKenney" , Randy Dunlap , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v6 4/8] padata: dispatch works on different nodes Date: Thu, 22 Feb 2024 22:04:17 +0800 Message-Id: <20240222140422.393911-5-gang.li@linux.dev> In-Reply-To: <20240222140422.393911-1-gang.li@linux.dev> References: <20240222140422.393911-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" When a group of tasks that access different nodes are scheduled on the same node, they may encounter bandwidth bottlenecks and access latency. Thus, numa_aware flag is introduced here, allowing tasks to be distributed across different nodes to fully utilize the advantage of multi-node systems. Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Muchun Song Reviewed-by: Tim Chen Acked-by: Daniel Jordan --- include/linux/padata.h | 2 ++ kernel/padata.c | 14 ++++++++++++-- mm/mm_init.c | 1 + 3 files changed, 15 insertions(+), 2 deletions(-) diff --git a/include/linux/padata.h b/include/linux/padata.h index 495b16b6b4d72..8f418711351bc 100644 --- a/include/linux/padata.h +++ b/include/linux/padata.h @@ -137,6 +137,7 @@ struct padata_shell { * appropriate for one worker thread to do at once. * @max_threads: Max threads to use for the job, actual number may be less * depending on task size and minimum chunk size. + * @numa_aware: Distribute jobs to different nodes with CPU in a round rob= in fashion. */ struct padata_mt_job { void (*thread_fn)(unsigned long start, unsigned long end, void *arg); @@ -146,6 +147,7 @@ struct padata_mt_job { unsigned long align; unsigned long min_chunk; int max_threads; + bool numa_aware; }; =20 /** diff --git a/kernel/padata.c b/kernel/padata.c index 179fb1518070c..e3f639ff16707 100644 --- a/kernel/padata.c +++ b/kernel/padata.c @@ -485,7 +485,8 @@ void __init padata_do_multithreaded(struct padata_mt_jo= b *job) struct padata_work my_work, *pw; struct padata_mt_job_state ps; LIST_HEAD(works); - int nworks; + int nworks, nid; + static atomic_t last_used_nid __initdata; =20 if (job->size =3D=3D 0) return; @@ -517,7 +518,16 @@ void __init padata_do_multithreaded(struct padata_mt_j= ob *job) ps.chunk_size =3D roundup(ps.chunk_size, job->align); =20 list_for_each_entry(pw, &works, pw_list) - queue_work(system_unbound_wq, &pw->pw_work); + if (job->numa_aware) { + int old_node =3D atomic_read(&last_used_nid); + + do { + nid =3D next_node_in(old_node, node_states[N_CPU]); + } while (!atomic_try_cmpxchg(&last_used_nid, &old_node, nid)); + queue_work_node(nid, system_unbound_wq, &pw->pw_work); + } else { + queue_work(system_unbound_wq, &pw->pw_work); + } =20 /* Use the current thread, which saves starting a workqueue worker. */ padata_work_init(&my_work, padata_mt_helper, &ps, PADATA_WORK_ONSTACK); diff --git a/mm/mm_init.c b/mm/mm_init.c index 2c19f5515e36c..549e76af8f82a 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2231,6 +2231,7 @@ static int __init deferred_init_memmap(void *data) .align =3D PAGES_PER_SECTION, .min_chunk =3D PAGES_PER_SECTION, .max_threads =3D max_threads, + .numa_aware =3D false, }; =20 padata_do_multithreaded(&job); --=20 2.20.1 From nobody Tue Dec 16 16:37:59 2025 Received: from out-170.mta0.migadu.com (out-170.mta0.migadu.com [91.218.175.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A1D9914691D for ; Thu, 22 Feb 2024 14:05:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610704; cv=none; b=MUmDAnjJIzS9gCr+IdbGtoE2mu1pcxAxlvnPgaRdkxfatUrfIR0ya3/n2AXwDnz9ojDgeY/crV3BKuH4sXHgJc0snouJSU/mpAUlp6lLeu3lslH5sfHj4fJ/VNEAwPoZ9Cf+8HMj1N2fqx+Cx9ZYWCblJDqSdYKmZY4MbdDfa4g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610704; c=relaxed/simple; bh=tQai8VYTD5u9kmna2i0v3NDlggEePdlk6wRlzITeLz4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QU38ab7YDD8SNlGQvhTp+iagjGHdS1muUUjfHjXxfIHPvemrFFJKk4VQLAR3ENbrsKjrM2hHs6hXace8b0eB8XUjwFUjTEAxBspVm0ZTzMf8uT3FufhanKsbn9WYF63VXtxo/4vNO2z3GSWKPOWL70oObWnWPOwhOpYiAbdJPvQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=EWql7fnc; arc=none smtp.client-ip=91.218.175.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="EWql7fnc" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1708610701; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P5dkM7DuGNd3n4uTkQacavwzunNC39kM4edC7ugWujE=; b=EWql7fncR1Nj1MIpD5GRXCTp3WENf4dEY69tv+jJU3JilH4N5PjL/V0Pe5FGwckjEP/Uyc TMSelN5Z8oMXIJ4mqBgNDiz4LmUYpAOxzwz8N/sBA/f6cjS1ksHyDB50z50WXk6Wjh9xGP vTIbBboABNWUcFmQwXUyWrABeLtlfvs= From: Gang Li To: Andrew Morton Cc: David Hildenbrand , David Rientjes , Muchun Song , Tim Chen , Steffen Klassert , Daniel Jordan , Jane Chu , "Paul E . McKenney" , Randy Dunlap , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v6 5/8] padata: downgrade padata_do_multithreaded to serial execution for non-SMP Date: Thu, 22 Feb 2024 22:04:18 +0800 Message-Id: <20240222140422.393911-6-gang.li@linux.dev> In-Reply-To: <20240222140422.393911-1-gang.li@linux.dev> References: <20240222140422.393911-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" hugetlb parallelization depends on PADATA, and PADATA depends on SMP. PADATA consists of two distinct functionality: One part is padata_do_multithreaded which disregards order and simply divides tasks into several groups for parallel execution. Hugetlb init parallelization depends on padata_do_multithreaded. The other part is composed of a set of APIs that, while handling data in an out-of-order parallel manner, can eventually return the data with ordered sequence. Currently Only `crypto/pcrypt.c` use them. All users of PADATA of non-SMP case currently only use padata_do_multithreaded. It is easy to implement a serial one in include/linux/padata.h. And it is not necessary to implement another functionality unless the only user of crypto/pcrypt.c does not depend on SMP in the future. Signed-off-by: Gang Li Tested-by: Paul E. McKenney Acked-by: Daniel Jordan --- include/linux/padata.h | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/include/linux/padata.h b/include/linux/padata.h index 8f418711351bc..0146daf344306 100644 --- a/include/linux/padata.h +++ b/include/linux/padata.h @@ -180,10 +180,6 @@ struct padata_instance { =20 #ifdef CONFIG_PADATA extern void __init padata_init(void); -#else -static inline void __init padata_init(void) {} -#endif - extern struct padata_instance *padata_alloc(const char *name); extern void padata_free(struct padata_instance *pinst); extern struct padata_shell *padata_alloc_shell(struct padata_instance *pin= st); @@ -194,4 +190,12 @@ extern void padata_do_serial(struct padata_priv *padat= a); extern void __init padata_do_multithreaded(struct padata_mt_job *job); extern int padata_set_cpumask(struct padata_instance *pinst, int cpumask_t= ype, cpumask_var_t cpumask); +#else +static inline void __init padata_init(void) {} +static inline void __init padata_do_multithreaded(struct padata_mt_job *jo= b) +{ + job->thread_fn(job->start, job->start + job->size, job->fn_arg); +} +#endif + #endif --=20 2.20.1 From nobody Tue Dec 16 16:37:59 2025 Received: from out-171.mta0.migadu.com (out-171.mta0.migadu.com [91.218.175.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AF5B914C5AF for ; Thu, 22 Feb 2024 14:05:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610708; cv=none; b=XnVte5m/zxhnlxgpp8OI0FFo9rE4ZD951vtdNoW5VBHIbhJ6ivm//9hnzUWNZl0o1cpZO38ZYXvz5zYGETaj/cWPdR6K6cREc0i4k/iDZjorVKr0WAhlsd/iSfZ9zlzI7gS47eumDeXj07R/0agZaTxmr/Ww0gH2jtHyjxK1MoM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610708; c=relaxed/simple; bh=R5MnQwc9t4v2E0fp8qGfSsi085XwCBGRqhTGk/VczIo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ESO/7zax262KNBGzZ7VtwWk1XICtKULFYHW2J5b3y0KdmNJwPZA28vPM9Yy+iMjVXXzGu7mEp5SkGbCLBrgOh2BBI0cSEmPat4X24upnFDPlnd8/JWdKPDBhQqoL3Z1+PJlk+yinJVimOPBgoKANMhRw04x9OLbkeGs3bBkrDCI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=qg/JjqKp; arc=none smtp.client-ip=91.218.175.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="qg/JjqKp" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1708610704; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gKGGJ4keOOfrl00iluvEDQ3Ty93zFU24V1EYV5CxVyE=; b=qg/JjqKph94W1a8idl0Ardw/d1tXM/odKpkHBNYeJX6L3C8F1p+YAZQ9skiGNuOx7apSHg P50H5nQcpGnKjLptZs0X/vUJxmxyGb2wQzDUgVbKsnbaaN5OpGF0RgSBPk7VaTYu7ry88z y+/WrwgLKTRUrw1khfweAOXlhe62gqo= From: Gang Li To: Andrew Morton Cc: David Hildenbrand , David Rientjes , Muchun Song , Tim Chen , Steffen Klassert , Daniel Jordan , Jane Chu , "Paul E . McKenney" , Randy Dunlap , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v6 6/8] hugetlb: have CONFIG_HUGETLBFS select CONFIG_PADATA Date: Thu, 22 Feb 2024 22:04:19 +0800 Message-Id: <20240222140422.393911-7-gang.li@linux.dev> In-Reply-To: <20240222140422.393911-1-gang.li@linux.dev> References: <20240222140422.393911-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Allow hugetlb use padata_do_multithreaded for parallel initialization. Select CONFIG_PADATA in this case. Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Muchun Song Tested-by: Paul E. McKenney Acked-by: Daniel Jordan --- fs/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/Kconfig b/fs/Kconfig index 89fdbefd1075f..c0b9599bf8e3a 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -262,6 +262,7 @@ menuconfig HUGETLBFS depends on X86 || SPARC64 || ARCH_SUPPORTS_HUGETLBFS || BROKEN depends on (SYSFS || SYSCTL) select MEMFD_CREATE + select PADATA if SMP help hugetlbfs is a filesystem backing for HugeTLB pages, based on ramfs. For architectures that support it, say Y here and read --=20 2.20.1 From nobody Tue Dec 16 16:37:59 2025 Received: from out-180.mta0.migadu.com (out-180.mta0.migadu.com [91.218.175.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F5C614C5BD for ; Thu, 22 Feb 2024 14:05:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610713; cv=none; b=C3lvmeJxYR3yqdC3RPYyAKuWfKduMI/uhBRJISoVcN4fUqwDBV4hb9Y4WCSLUOCZ0izX3mFMSfTAyJTvthRF+yoUuyBGEUntVSggAKcSjBcS3WJt9EPkgNQ0zgtVDgrnUySJYJ6rojr6nGLjG+tTGBbx+eC7xzBiNvZ+LQwNudg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610713; c=relaxed/simple; bh=wDUtn4D4+R/+Ucrafy1XUZ39rLpK7nGNdp7cuRDIc2A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=S7HVLBr/QaO0zD9nTSHFob5hB0j8VUHtMuKYKxHyXTvOPWzt5efStkV85a685qpOJL/NBDq4+d/vr0/aEv0XhWlqZYtCRYdWnvEo6KSYcO5MAmLxdt0e8e/s1Ip6I7QBMtRZi2lvjLMfBxLmwqaAivvhRpqnU/ilHo3HY5HYiI4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=X1qraX2V; arc=none smtp.client-ip=91.218.175.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="X1qraX2V" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1708610708; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=N6t1qyq4TDUtkxpVn8LUmZjkdAyqh+tvT81M41wbLKc=; b=X1qraX2Vvj0/KBfu6EnJq/+fNR67xug/4C3rpiILP1VuAv3Lrl9M7fpLL3xNyIAibRgjh2 t6IVN2idjt+n0TQ0E+IbU5JFcgU226TBbWzoJOagUVks3iKjgV3ZA0lDtrM/fhY1oX0xtj SGDAg/B2R/m6fm9JvkcoCOxIIyohtc8= From: Gang Li To: Andrew Morton Cc: David Hildenbrand , David Rientjes , Muchun Song , Tim Chen , Steffen Klassert , Daniel Jordan , Jane Chu , "Paul E . McKenney" , Randy Dunlap , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v6 7/8] hugetlb: parallelize 2M hugetlb allocation and initialization Date: Thu, 22 Feb 2024 22:04:20 +0800 Message-Id: <20240222140422.393911-8-gang.li@linux.dev> In-Reply-To: <20240222140422.393911-1-gang.li@linux.dev> References: <20240222140422.393911-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" By distributing both the allocation and the initialization tasks across multiple threads, the initialization of 2M hugetlb will be faster, thereby improving the boot speed. Here are some test results: test case no patch(ms) patched(ms) saved ------------------- -------------- ------------- -------- 256c2T(4 node) 2M 3336 1051 68.52% 128c1T(2 node) 2M 1943 716 63.15% Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Muchun Song Acked-by: Daniel Jordan # padata --- mm/hugetlb.c | 73 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 56 insertions(+), 17 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d1ce1a52ad504..3ce957b3e350b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -35,6 +35,7 @@ #include #include #include +#include =20 #include #include @@ -3510,6 +3511,30 @@ static void __init hugetlb_hstate_alloc_pages_errche= ck(unsigned long allocated, } } =20 +static void __init hugetlb_pages_alloc_boot_node(unsigned long start, unsi= gned long end, void *arg) +{ + struct hstate *h =3D (struct hstate *)arg; + int i, num =3D end - start; + nodemask_t node_alloc_noretry; + LIST_HEAD(folio_list); + int next_node =3D first_online_node; + + /* Bit mask controlling how hard we retry per-node allocations.*/ + nodes_clear(node_alloc_noretry); + + for (i =3D 0; i < num; ++i) { + struct folio *folio =3D alloc_pool_huge_folio(h, &node_states[N_MEMORY], + &node_alloc_noretry, &next_node); + if (!folio) + break; + + list_move(&folio->lru, &folio_list); + cond_resched(); + } + + prep_and_add_allocated_folios(h, &folio_list); +} + static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstat= e *h) { unsigned long i; @@ -3525,26 +3550,40 @@ static unsigned long __init hugetlb_gigantic_pages_= alloc_boot(struct hstate *h) =20 static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h) { - unsigned long i; - struct folio *folio; - LIST_HEAD(folio_list); - nodemask_t node_alloc_noretry; - - /* Bit mask controlling how hard we retry per-node allocations.*/ - nodes_clear(node_alloc_noretry); + struct padata_mt_job job =3D { + .fn_arg =3D h, + .align =3D 1, + .numa_aware =3D true + }; =20 - for (i =3D 0; i < h->max_huge_pages; ++i) { - folio =3D alloc_pool_huge_folio(h, &node_states[N_MEMORY], - &node_alloc_noretry); - if (!folio) - break; - list_add(&folio->lru, &folio_list); - cond_resched(); - } + job.thread_fn =3D hugetlb_pages_alloc_boot_node; + job.start =3D 0; + job.size =3D h->max_huge_pages; =20 - prep_and_add_allocated_folios(h, &folio_list); + /* + * job.max_threads is twice the num_node_state(N_MEMORY), + * + * Tests below indicate that a multiplier of 2 significantly improves + * performance, and although larger values also provide improvements, + * the gains are marginal. + * + * Therefore, choosing 2 as the multiplier strikes a good balance between + * enhancing parallel processing capabilities and maintaining efficient + * resource management. + * + * +------------+-------+-------+-------+-------+-------+ + * | multiplier | 1 | 2 | 3 | 4 | 5 | + * +------------+-------+-------+-------+-------+-------+ + * | 256G 2node | 358ms | 215ms | 157ms | 134ms | 126ms | + * | 2T 4node | 979ms | 679ms | 543ms | 489ms | 481ms | + * | 50G 2node | 71ms | 44ms | 37ms | 30ms | 31ms | + * +------------+-------+-------+-------+-------+-------+ + */ + job.max_threads =3D num_node_state(N_MEMORY) * 2; + job.min_chunk =3D h->max_huge_pages / num_node_state(N_MEMORY) / 2; + padata_do_multithreaded(&job); =20 - return i; + return h->nr_huge_pages; } =20 /* --=20 2.20.1 From nobody Tue Dec 16 16:37:59 2025 Received: from out-174.mta0.migadu.com (out-174.mta0.migadu.com [91.218.175.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7FDE11474BD for ; Thu, 22 Feb 2024 14:05:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610716; cv=none; b=BvMd/kzRAZNqrimzKMENJ3S61j5nORVK6jVcY+/IZQ+3tY1CPCnWwqHE9hrciaomPHrDHBcNqYuSav9WHthUUVFDhuLy8Z4LuUGIe2FmwypPBqn4UK0Nt/IUXM77MJyn2gfgKzTy4N3IInkIepgw+hZ1/CYMxk/pW3PdjB7Cbts= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610716; c=relaxed/simple; bh=RPjBKA6DR5wkHsrMFWa2oUnkIDrSZ9zmAWeJ9svygWA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=q+1gxaKeeV7aFDXH2mwRgmkLzuqSnkOR0hiOKP0tehAeMAWX7qHu4645Xi+32Z7AWP576hxgpDfzns1XN876ksiJXNNGUhd1nIZrtX4zGac451j8lO1KUFhcEQn/ph6MZJLYL4hmLfoId7dzk9fkjV+AqN2ZWYbDdpxTzKDwKKs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=t41r5HaT; arc=none smtp.client-ip=91.218.175.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="t41r5HaT" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1708610711; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PkfbG9EDgmYztgYWXLtb18KzwOAsDRoGbeegEzJG3FQ=; b=t41r5HaT0mYHvm84//2lfA9azmzwtZPkLtm5E0DMzyiun/u/Gc1oQ9waBm96kAePKKkDSe Hi+OLO9VM8zbSrf+T3I+INs/MUO9PYJdI1PkVQUVo0g6y2VYj2OdAF9TygrPCwHNCzMTVQ ooYuq9JMPkcTZoUsPL2yAYMTAI4NREU= From: Gang Li To: Andrew Morton Cc: David Hildenbrand , David Rientjes , Muchun Song , Tim Chen , Steffen Klassert , Daniel Jordan , Jane Chu , "Paul E . McKenney" , Randy Dunlap , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v6 8/8] hugetlb: parallelize 1G hugetlb initialization Date: Thu, 22 Feb 2024 22:04:21 +0800 Message-Id: <20240222140422.393911-9-gang.li@linux.dev> In-Reply-To: <20240222140422.393911-1-gang.li@linux.dev> References: <20240222140422.393911-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Optimizing the initialization speed of 1G huge pages through parallelization. 1G hugetlbs are allocated from bootmem, a process that is already very fast and does not currently require optimization. Therefore, we focus on parallelizing only the initialization phase in `gather_bootmem_prealloc`. Here are some test results: test case no patch(ms) patched(ms) saved ------------------- -------------- ------------- -------- 256c2T(4 node) 1G 4745 2024 57.34% 128c1T(2 node) 1G 3358 1712 49.02% 12T 1G 77000 18300 76.23% Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Muchun Song Acked-by: Daniel Jordan # padata --- arch/powerpc/mm/hugetlbpage.c | 2 +- include/linux/hugetlb.h | 2 +- mm/hugetlb.c | 51 +++++++++++++++++++++++++++++------ 3 files changed, 45 insertions(+), 10 deletions(-) diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c index 16557d008eef5..594a4b7b2ca24 100644 --- a/arch/powerpc/mm/hugetlbpage.c +++ b/arch/powerpc/mm/hugetlbpage.c @@ -226,7 +226,7 @@ static int __init pseries_alloc_bootmem_huge_page(struc= t hstate *hstate) return 0; m =3D phys_to_virt(gpage_freearray[--nr_gpages]); gpage_freearray[nr_gpages] =3D 0; - list_add(&m->list, &huge_boot_pages); + list_add(&m->list, &huge_boot_pages[0]); m->hstate =3D hstate; return 1; } diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index c1ee640d87b11..77b30a8c6076b 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -178,7 +178,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_a= rea_struct *vma, struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage); =20 extern int sysctl_hugetlb_shm_group; -extern struct list_head huge_boot_pages; +extern struct list_head huge_boot_pages[MAX_NUMNODES]; =20 /* arch callbacks */ =20 diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 3ce957b3e350b..1ac38e7606461 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -69,7 +69,7 @@ static bool hugetlb_cma_folio(struct folio *folio, unsign= ed int order) #endif static unsigned long hugetlb_cma_size __initdata; =20 -__initdata LIST_HEAD(huge_boot_pages); +__initdata struct list_head huge_boot_pages[MAX_NUMNODES]; =20 /* for command line parsing */ static struct hstate * __initdata parsed_hstate; @@ -3301,7 +3301,7 @@ int alloc_bootmem_huge_page(struct hstate *h, int nid) int __alloc_bootmem_huge_page(struct hstate *h, int nid) { struct huge_bootmem_page *m =3D NULL; /* initialize for clang */ - int nr_nodes, node; + int nr_nodes, node =3D nid; =20 /* do node specific alloc */ if (nid !=3D NUMA_NO_NODE) { @@ -3339,7 +3339,7 @@ int __alloc_bootmem_huge_page(struct hstate *h, int n= id) huge_page_size(h) - PAGE_SIZE); /* Put them into a private list first because mem_map is not up yet */ INIT_LIST_HEAD(&m->list); - list_add(&m->list, &huge_boot_pages); + list_add(&m->list, &huge_boot_pages[node]); m->hstate =3D h; return 1; } @@ -3390,8 +3390,6 @@ static void __init prep_and_add_bootmem_folios(struct= hstate *h, /* Send list for bulk vmemmap optimization processing */ hugetlb_vmemmap_optimize_folios(h, folio_list); =20 - /* Add all new pool pages to free lists in one lock cycle */ - spin_lock_irqsave(&hugetlb_lock, flags); list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { if (!folio_test_hugetlb_vmemmap_optimized(folio)) { /* @@ -3404,23 +3402,25 @@ static void __init prep_and_add_bootmem_folios(stru= ct hstate *h, HUGETLB_VMEMMAP_RESERVE_PAGES, pages_per_huge_page(h)); } + /* Subdivide locks to achieve better parallel performance */ + spin_lock_irqsave(&hugetlb_lock, flags); __prep_account_new_huge_page(h, folio_nid(folio)); enqueue_hugetlb_folio(h, folio); + spin_unlock_irqrestore(&hugetlb_lock, flags); } - spin_unlock_irqrestore(&hugetlb_lock, flags); } =20 /* * Put bootmem huge pages into the standard lists after mem_map is up. * Note: This only applies to gigantic (order > MAX_PAGE_ORDER) pages. */ -static void __init gather_bootmem_prealloc(void) +static void __init gather_bootmem_prealloc_node(unsigned long nid) { LIST_HEAD(folio_list); struct huge_bootmem_page *m; struct hstate *h =3D NULL, *prev_h =3D NULL; =20 - list_for_each_entry(m, &huge_boot_pages, list) { + list_for_each_entry(m, &huge_boot_pages[nid], list) { struct page *page =3D virt_to_page(m); struct folio *folio =3D (void *)page; =20 @@ -3453,6 +3453,31 @@ static void __init gather_bootmem_prealloc(void) prep_and_add_bootmem_folios(h, &folio_list); } =20 +static void __init gather_bootmem_prealloc_parallel(unsigned long start, + unsigned long end, void *arg) +{ + int nid; + + for (nid =3D start; nid < end; nid++) + gather_bootmem_prealloc_node(nid); +} + +static void __init gather_bootmem_prealloc(void) +{ + struct padata_mt_job job =3D { + .thread_fn =3D gather_bootmem_prealloc_parallel, + .fn_arg =3D NULL, + .start =3D 0, + .size =3D num_node_state(N_MEMORY), + .align =3D 1, + .min_chunk =3D 1, + .max_threads =3D num_node_state(N_MEMORY), + .numa_aware =3D true, + }; + + padata_do_multithreaded(&job); +} + static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, in= t nid) { unsigned long i; @@ -3600,6 +3625,7 @@ static unsigned long __init hugetlb_pages_alloc_boot(= struct hstate *h) static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { unsigned long allocated; + static bool initialied __initdata; =20 /* skip gigantic hugepages allocation if hugetlb_cma enabled */ if (hstate_is_gigantic(h) && hugetlb_cma_size) { @@ -3607,6 +3633,15 @@ static void __init hugetlb_hstate_alloc_pages(struct= hstate *h) return; } =20 + /* hugetlb_hstate_alloc_pages will be called many times, initialize huge_= boot_pages once */ + if (!initialied) { + int i =3D 0; + + for (i =3D 0; i < MAX_NUMNODES; i++) + INIT_LIST_HEAD(&huge_boot_pages[i]); + initialied =3D true; + } + /* do node specific alloc */ if (hugetlb_hstate_alloc_pages_specific_nodes(h)) return; --=20 2.20.1