From nobody Tue Sep 9 22:47:39 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1215BEB64DD for ; Thu, 27 Jul 2023 20:47:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232388AbjG0UrF (ORCPT ); Thu, 27 Jul 2023 16:47:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34388 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231570AbjG0Uq4 (ORCPT ); Thu, 27 Jul 2023 16:46:56 -0400 Received: from mail-lj1-x232.google.com (mail-lj1-x232.google.com [IPv6:2a00:1450:4864:20::232]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6597271D for ; Thu, 27 Jul 2023 13:46:33 -0700 (PDT) Received: by mail-lj1-x232.google.com with SMTP id 38308e7fff4ca-2b701e1ca63so21959291fa.1 for ; Thu, 27 Jul 2023 13:46:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490792; x=1691095592; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oaqNw2Hm1guEkvjK0qvKkO1TrGxzTe44nwo0h0OS4xY=; b=Qh4G5pkJ2CklC1dn5cBz4RJXwctIF25YGEWE2oQ3ZrOzxnKalFKae/8pznmb31lV7C WTzPyronTTWL8g0r5I8lSbz/AdU7gvJCGq8K2Qd4vWzRvFiQ/FfCwXl6dsQ9CmNizTh+ m0AH/lVmLiMfUecg6qsdvv9XMrT/D7wOGOTWcG9Zt50oRSe1Bavo9Ko7yaB4wjJ+Qnnr BNWwV0vvSuj9U4WuwZrl1x2sIhyQwELI33CtL1QqIzp+Rtzz205zsveFrvcH9XBhLeZX G7lPHvXhJp/f1c2Jmw7kl1BPg6BGcbtzSnM+LE3IJIaKxCmCB8OfQyprqC6Rh3SHBpsg Wpxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490792; x=1691095592; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oaqNw2Hm1guEkvjK0qvKkO1TrGxzTe44nwo0h0OS4xY=; b=cIWRrNJAxXNGf/cCZhI/WhleJ19H1lVTspF1KEz1+LkNa+BpyAJ+m7YXZSFRX8KH8d pneT9TvEiEmnUN16gKM9HWiLhanRdxidkvdlBx83NWaPgWc7MHxjiNinmoXPLKkoH6fB vzWqrEoM4yFn/SvLONJh7akI9MvWdrNPRS1PNOrGiimjXAJ2F/sWASw2/7LTPI7dnEwF ucIZ/UfvzsJQzMCnPHlc9B4/CNQJA19qkYKkZgHwELSTZ0F8vBGofPZxVfybIitiO6SH IfmZh99jDUMozxHvz5OpiwSAuTAW6R2XGEZ2GUspB2NvXC2BhFijze1zaKQarx1M7mNO VwSA== X-Gm-Message-State: ABy/qLZgHUZWAldkarwnuXffsAWSX4T5nA+KGPwAlHXFCiUCjrncRqVT zta/RDF8HM2Gb6VMjtJo1WXDyg== X-Google-Smtp-Source: APBJJlEpvm0P4NAPU6V060zctU8cT38iuWapH30/cDkVR/gUSQA0rrC4oV9wP9Dr8ocbwFl306QznA== X-Received: by 2002:a05:651c:1042:b0:2b6:da64:321 with SMTP id x2-20020a05651c104200b002b6da640321mr95486ljm.45.1690490792019; Thu, 27 Jul 2023 13:46:32 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:31 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 1/6] mm: hugetlb: Skip prep of tail pages when HVO is enabled Date: Thu, 27 Jul 2023 21:46:19 +0100 Message-Id: <20230727204624.1942372-2-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When vmemmap is optimizable, it will free all the duplicated tail pages in hugetlb_vmemmap_optimize while preparing the new hugepage. Hence, there is no need to prepare them. For 1G x86 hugepages, it avoids preparing 262144 - 64 =3D 262080 struct pages per hugepage. Signed-off-by: Usama Arif --- mm/hugetlb.c | 32 +++++++++++++++++++++++--------- mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 7 +++++++ 3 files changed, 31 insertions(+), 10 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 64a3239b6407..58cf5978bee1 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1943,13 +1943,24 @@ static void prep_new_hugetlb_folio(struct hstate *h= , struct folio *folio, int ni } =20 static bool __prep_compound_gigantic_folio(struct folio *folio, - unsigned int order, bool demote) + unsigned int order, bool demote, + bool hvo) { int i, j; int nr_pages =3D 1 << order; struct page *p; =20 __folio_clear_reserved(folio); + +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + /* + * No need to prep pages that will be freed later by hugetlb_vmemmap_opti= mize + * in prep_new_huge_page. Hence, reduce nr_pages to the pages that will b= e kept. + */ + if (hvo) + nr_pages =3D HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page); +#endif + for (i =3D 0; i < nr_pages; i++) { p =3D folio_page(folio, i); =20 @@ -2020,15 +2031,15 @@ static bool __prep_compound_gigantic_folio(struct f= olio *folio, } =20 static bool prep_compound_gigantic_folio(struct folio *folio, - unsigned int order) + unsigned int order, bool hvo) { - return __prep_compound_gigantic_folio(folio, order, false); + return __prep_compound_gigantic_folio(folio, order, false, hvo); } =20 static bool prep_compound_gigantic_folio_for_demote(struct folio *folio, - unsigned int order) + unsigned int order, bool hvo) { - return __prep_compound_gigantic_folio(folio, order, true); + return __prep_compound_gigantic_folio(folio, order, true, hvo); } =20 /* @@ -2185,7 +2196,8 @@ static struct folio *alloc_fresh_hugetlb_folio(struct= hstate *h, if (!folio) return NULL; if (hstate_is_gigantic(h)) { - if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { + if (!prep_compound_gigantic_folio(folio, huge_page_order(h), + vmemmap_should_optimize(h, &folio->page))) { /* * Rare failure to convert pages to compound page. * Free pages and try again - ONCE! @@ -3201,7 +3213,8 @@ static void __init gather_bootmem_prealloc(void) =20 VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(folio_ref_count(folio) !=3D 1); - if (prep_compound_gigantic_folio(folio, huge_page_order(h))) { + if (prep_compound_gigantic_folio(folio, huge_page_order(h), + vmemmap_should_optimize(h, page))) { WARN_ON(folio_test_reserved(folio)); prep_new_hugetlb_folio(h, folio, folio_nid(folio)); free_huge_page(page); /* add to the hugepage allocator */ @@ -3624,8 +3637,9 @@ static int demote_free_hugetlb_folio(struct hstate *h= , struct folio *folio) subpage =3D folio_page(folio, i); inner_folio =3D page_folio(subpage); if (hstate_is_gigantic(target_hstate)) - prep_compound_gigantic_folio_for_demote(inner_folio, - target_hstate->order); + prep_compound_gigantic_folio_for_demote(folio, + target_hstate->order, + vmemmap_should_optimize(target_hstate, subpage)); else prep_compound_page(subpage, target_hstate->order); folio_change_private(inner_folio, NULL); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c2007ef5e9b0..b721e87de2b3 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -486,7 +486,7 @@ int hugetlb_vmemmap_restore(const struct hstate *h, str= uct page *head) } =20 /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ -static bool vmemmap_should_optimize(const struct hstate *h, const struct p= age *head) +bool vmemmap_should_optimize(const struct hstate *h, const struct page *he= ad) { if (!READ_ONCE(vmemmap_optimize_enabled)) return false; diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 25bd0e002431..07555d2dc0cb 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -13,6 +13,7 @@ #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); +bool vmemmap_should_optimize(const struct hstate *h, const struct page *he= ad); =20 /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See @@ -51,6 +52,12 @@ static inline unsigned int hugetlb_vmemmap_optimizable_s= ize(const struct hstate { return 0; } + +bool vmemmap_should_optimize(const struct hstate *h, const struct page *he= ad) +{ + return false; +} + #endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ =20 static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h) --=20 2.25.1 From nobody Tue Sep 9 22:47:39 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52D22C001DC for ; Thu, 27 Jul 2023 20:46:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231721AbjG0Uqi (ORCPT ); Thu, 27 Jul 2023 16:46:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229522AbjG0Uqf (ORCPT ); Thu, 27 Jul 2023 16:46:35 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 923272D45 for ; Thu, 27 Jul 2023 13:46:34 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id ffacd0b85a97d-3176a439606so1400731f8f.3 for ; Thu, 27 Jul 2023 13:46:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490793; x=1691095593; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; b=ZZUlyW4YzysoCHAYMp2/sA6YrKmTeac3t3azsTg4jDgno5140SsxzboZ2DX+sYyTar QBt4NCcbIwVqU6ECOkAxzfNVYTJSA4UoAO9vNAzRgOzSciNduCZzg6hsTy30yZygE/FX Y3IhdC9ei8B4xOpa33YxJ+rAYbRMQmfBhHWbkpR2MzbCVvhFxC5cl2XdP9qY+d1bHhgd lqfn5I8dfHTphHuAk/OGTexXituKKGFsXZtEGV8Pe+xS2H+P9mt/hFkl6RI2Q/3BRFmT 8QbfWy8NIL5JwByXBCIWXDjISwNspvj4RcypyTUIR9HRGShbKOG5H+xfqxUKSelybJ2e i7JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490793; x=1691095593; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; b=XGLUlnn1iD+NYgCwrKBTGtwffgNr6WkRbv0VflyZIkqC9iSxHp+mrMKa+L1mFJketw mVA5rEGN6XBAUqxuHxI4b5AjZcVXWXP33T9FmONwd10gEkGiZw+iSfyEq52HZUgvAkx6 1O7hJGnbD90U4ybeRe5UTHPNzBmtYZoW/++QWNLKsYq3bkJOEXQA+4ZyLFi0YNpbweXR xalkJP/GY7OyWKxIKb49CzfvMMuV6xMV61lCVTOfDiP/dS6r6nw4ldDZb73l9sgEupck RfGE36Q5qRBSYLcvdmzbKyfkwhwPCHDpM0N3PUy5ov4O39Y7QBVx2mc6cD+5YJ9HVG1e 7WdA== X-Gm-Message-State: ABy/qLbliNXnrAtc00L9SaqcPaiRNlWyHXfTxAtIxjsiyMVGRo2blJ1M jQJMhG7V1eLcN6MH2gwPWYlEdA== X-Google-Smtp-Source: APBJJlG66jp7ldTsiVfnEBvuqVAQDlHt8IOfvNpmcOCKQllr9otAKlvj+wF8PQJddxyevra73/5ciQ== X-Received: by 2002:adf:ef4f:0:b0:314:32b6:af3 with SMTP id c15-20020adfef4f000000b0031432b60af3mr204498wrp.5.1690490792997; Thu, 27 Jul 2023 13:46:32 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:32 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 2/6] mm: hugetlb_vmemmap: Use nid of the head page to reallocate it Date: Thu, 27 Jul 2023 21:46:20 +0100 Message-Id: <20230727204624.1942372-3-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" If tail page prep and initialization is skipped, then the "start" page will not contain the correct nid. Use the nid from first vmemap page. Signed-off-by: Usama Arif --- mm/hugetlb_vmemmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index b721e87de2b3..bdf750a4786b 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -324,7 +324,7 @@ static int vmemmap_remap_free(unsigned long start, unsi= gned long end, .reuse_addr =3D reuse, .vmemmap_pages =3D &vmemmap_pages, }; - int nid =3D page_to_nid((struct page *)start); + int nid =3D page_to_nid((struct page *)reuse); gfp_t gfp_mask =3D GFP_KERNEL | __GFP_THISNODE | __GFP_NORETRY | __GFP_NOWARN; =20 --=20 2.25.1 From nobody Tue Sep 9 22:47:39 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1AFDEB64DD for ; Thu, 27 Jul 2023 20:46:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231862AbjG0Uq5 (ORCPT ); Thu, 27 Jul 2023 16:46:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34396 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231773AbjG0Uqy (ORCPT ); Thu, 27 Jul 2023 16:46:54 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A99882D47 for ; Thu, 27 Jul 2023 13:46:35 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-3fde57684d7so14158905e9.2 for ; Thu, 27 Jul 2023 13:46:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490794; x=1691095594; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=R/2kTGP5bmExpEckq+8Rz1vqcaG4uRUICK5uorcV/EQ=; b=BuN5xyW/6weGNXRlIjeHAIrAgftiJaDdwcAFdinE25QOj+gk0KxhfbPi6NDrWScFMY VMJS1fp/fwlHj4io7fNeedGnL3Vq6LPtuewysLC8Sjo6xuSSwb5C7A3QVA6pUp3BuGal peUgBBXdkLuyAG0r0MtOAu8bJxWihdBSiUkx7+JUoxmU92FDCSc3+a203D99Te3djvrE GFjNEAyNZJnsGcpwIVLN7dd8/TClS2qfeDsn0scV9uZaGKj9iuXLvZP5q+GVR9jZNOl4 uoPcfh3+yJsGI/CKwLu45iuV0qqJWtVosbUaOpv+ylvrytQ2cUfc7YMZnclJk3oiesVt HmMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490794; x=1691095594; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=R/2kTGP5bmExpEckq+8Rz1vqcaG4uRUICK5uorcV/EQ=; b=frvOzM0GhjlH7wz+Y2s9u7rO5iDv1IBP7Y94bM3QteHwrnIsXQ3p6kq83SxTpfhtWF DEhetDq/R3iPTwmupCO5f6HuxsUf91bs6qtb9VXP5d+7nsCjW9810GM/Ci0zf26t8CrS hSvFdZ0tTKysOaR41PcaMocLtbSlSFyYsvSCzlxCGm7mJMN8MervEdztWpjN3TrPzfEu nVCDdAuOUYrJ1IBI9bNzxaeNZZKN2l+Q2cfmg46NylWd3Q+TQ6Z09DFQTDM4xErgXCWx eJCj+4tAdwZope0VQOBcL/+zHCzEAr2LRrA5yIWn1v7Hm60VhD4SPlVTTR5HxbHO2tWM jW6Q== X-Gm-Message-State: ABy/qLarESVAETB2HGzxazjrEN7FD3r3QL7i1pd9WNo/Vm7xael1K7/6 plbcrkQwKm5ZLpTElt6/qdNwjw== X-Google-Smtp-Source: APBJJlFsQ9jv1pf3Fn+r5CYo5PNZoF1WnIrPiX/ItiAceHqurCAQzDyFXe3uUVsJ5LM8vIz8zQ9d8g== X-Received: by 2002:a05:600c:21cf:b0:3fd:ee50:d6a4 with SMTP id x15-20020a05600c21cf00b003fdee50d6a4mr141098wmj.17.1690490794119; Thu, 27 Jul 2023 13:46:34 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:33 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 3/6] memblock: add parameter to memblock_setclr_flag for selecting memblock_type Date: Thu, 27 Jul 2023 21:46:21 +0100 Message-Id: <20230727204624.1942372-4-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This is in preparation for setting flags (for e.g. to not initialize struct pages) on reserved memory region. Signed-off-by: Usama Arif --- mm/memblock.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/mm/memblock.c b/mm/memblock.c index f9e61e565a53..4fd431d16ef2 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -897,11 +897,16 @@ int __init_memblock memblock_physmem_add(phys_addr_t = base, phys_addr_t size) * Return: 0 on success, -errno on failure. */ static int __init_memblock memblock_setclr_flag(phys_addr_t base, - phys_addr_t size, int set, int flag) + phys_addr_t size, int set, int flag, bool reserved) { - struct memblock_type *type =3D &memblock.memory; + struct memblock_type *type; int i, ret, start_rgn, end_rgn; =20 + if (reserved) + type =3D &memblock.reserved; + else + type =3D &memblock.memory; + ret =3D memblock_isolate_range(type, base, size, &start_rgn, &end_rgn); if (ret) return ret; @@ -928,7 +933,7 @@ static int __init_memblock memblock_setclr_flag(phys_ad= dr_t base, */ int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t si= ze) { - return memblock_setclr_flag(base, size, 1, MEMBLOCK_HOTPLUG); + return memblock_setclr_flag(base, size, 1, MEMBLOCK_HOTPLUG, 0); } =20 /** @@ -940,7 +945,7 @@ int __init_memblock memblock_mark_hotplug(phys_addr_t b= ase, phys_addr_t size) */ int __init_memblock memblock_clear_hotplug(phys_addr_t base, phys_addr_t s= ize) { - return memblock_setclr_flag(base, size, 0, MEMBLOCK_HOTPLUG); + return memblock_setclr_flag(base, size, 0, MEMBLOCK_HOTPLUG, 0); } =20 /** @@ -957,7 +962,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t ba= se, phys_addr_t size) =20 system_has_some_mirror =3D true; =20 - return memblock_setclr_flag(base, size, 1, MEMBLOCK_MIRROR); + return memblock_setclr_flag(base, size, 1, MEMBLOCK_MIRROR, 0); } =20 /** @@ -977,7 +982,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t ba= se, phys_addr_t size) */ int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 1, MEMBLOCK_NOMAP); + return memblock_setclr_flag(base, size, 1, MEMBLOCK_NOMAP, 0); } =20 /** @@ -989,7 +994,7 @@ int __init_memblock memblock_mark_nomap(phys_addr_t bas= e, phys_addr_t size) */ int __init_memblock memblock_clear_nomap(phys_addr_t base, phys_addr_t siz= e) { - return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP); + return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP, 0); } =20 static bool should_skip_region(struct memblock_type *type, --=20 2.25.1 From nobody Tue Sep 9 22:47:39 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6076FC001DC for ; Thu, 27 Jul 2023 20:47:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231933AbjG0UrO (ORCPT ); Thu, 27 Jul 2023 16:47:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232213AbjG0UrB (ORCPT ); Thu, 27 Jul 2023 16:47:01 -0400 Received: from mail-lj1-x234.google.com (mail-lj1-x234.google.com [IPv6:2a00:1450:4864:20::234]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B87852D45 for ; Thu, 27 Jul 2023 13:46:36 -0700 (PDT) Received: by mail-lj1-x234.google.com with SMTP id 38308e7fff4ca-2b701e1ca63so21959931fa.1 for ; Thu, 27 Jul 2023 13:46:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490795; x=1691095595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JQe+LnlORYwelNDP2OagFjwF2/T/1IMvjhocyHCh0HM=; b=gVT8uA85Q/2RybadFL1WMGqwI8huoiPfSEh3Ga80yhJTkx2wqYddCczcvHucvrLvi3 8LyY62f1ojKZNCIJ1RJJ6AJm/H/D5kfULuGeQgjkEhp/sdQz5z18I7dL/szLmlIfJpYD u61EVQdgSt8wbKxI6DB5hsAcwBuLB2hgvn0jrCvRww9Z0lGEXVNLysQBc2Fe36p46uCT jl2OXbNIY2R4lV5NY2Ex7WgCxOm/ALG64TE59hb/8mRc6i3LDTDk1ah1xfi+hOTG8MYd 4LLgZYdnHkKk+A17Ls6+GXL7nzPHfRkZeRmwu0Uv3hqpWoCBuyANT80yrLWtGgfm1gge gtcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490795; x=1691095595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JQe+LnlORYwelNDP2OagFjwF2/T/1IMvjhocyHCh0HM=; b=fIwHg8/52HS2L+/h6TlO/UClI7myuMGNqms80m595MZbW/ka/rULtjlJm4N8KnXJ6m MhRGlKPptJGbH58GvKiXwHeggjCEFm95YAfcP9grwnmuc+c5qI6sYJgJhHqpa2cmRV0x l/5ZsxvA5tnDIuClAJOW54O1Ol2/Ry8ozTb+V5+J/SB+cJavogtQUv+vY0IQDCzcZZA/ 4+xfIMcAg7ABkXtU5i5bHG5psl5Ua+7d5syCXILErkGSPWnsPzfNDFIWPLJaWYQgsP1A KjdR1fr/KgwqdX6DnS0my2vIQDb6OFUCQ9hOhXxwFEXikMyPez0jcxDQZGqJ780RhOIB V28A== X-Gm-Message-State: ABy/qLZe596ymFhynY4Aqp2Gw3heu6nQkznfmI9tmOT7B04RD0ji9BKv d49NuY0uYgUsPPhUCy1XZ6V6fw== X-Google-Smtp-Source: APBJJlFfgB46RLeDSeeOkYepjrGae0ZQ1eLESbLa8jxhbH1/PX3Edir8uGhF64v2uGzIFXEGRina+Q== X-Received: by 2002:a2e:3003:0:b0:2b5:974f:385 with SMTP id w3-20020a2e3003000000b002b5974f0385mr129660ljw.9.1690490795035; Thu, 27 Jul 2023 13:46:35 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:34 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 4/6] memblock: introduce MEMBLOCK_RSRV_NOINIT flag Date: Thu, 27 Jul 2023 21:46:22 +0100 Message-Id: <20230727204624.1942372-5-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" For reserved memory regions marked with this flag, reserve_bootmem_region is not called during memmap_init_reserved_pages. This can be used to avoid struct page initialization for regions which won't need them, for e.g. hugepages with HVO enabled. Signed-off-by: Usama Arif --- include/linux/memblock.h | 7 +++++++ mm/memblock.c | 32 ++++++++++++++++++++++++++------ 2 files changed, 33 insertions(+), 6 deletions(-) diff --git a/include/linux/memblock.h b/include/linux/memblock.h index f71ff9f0ec81..7f9d06c08592 100644 --- a/include/linux/memblock.h +++ b/include/linux/memblock.h @@ -47,6 +47,7 @@ enum memblock_flags { MEMBLOCK_MIRROR =3D 0x2, /* mirrored region */ MEMBLOCK_NOMAP =3D 0x4, /* don't add to kernel direct mapping */ MEMBLOCK_DRIVER_MANAGED =3D 0x8, /* always detected via a driver */ + MEMBLOCK_RSRV_NOINIT =3D 0x10, /* don't call reserve_bootmem_region for t= his region */ }; =20 /** @@ -125,6 +126,7 @@ int memblock_clear_hotplug(phys_addr_t base, phys_addr_= t size); int memblock_mark_mirror(phys_addr_t base, phys_addr_t size); int memblock_mark_nomap(phys_addr_t base, phys_addr_t size); int memblock_clear_nomap(phys_addr_t base, phys_addr_t size); +int memblock_rsrv_mark_noinit(phys_addr_t base, phys_addr_t size); =20 void memblock_free_all(void); void memblock_free(void *ptr, size_t size); @@ -259,6 +261,11 @@ static inline bool memblock_is_nomap(struct memblock_r= egion *m) return m->flags & MEMBLOCK_NOMAP; } =20 +static inline bool memblock_is_noinit(struct memblock_region *m) +{ + return m->flags & MEMBLOCK_RSRV_NOINIT; +} + static inline bool memblock_is_driver_managed(struct memblock_region *m) { return m->flags & MEMBLOCK_DRIVER_MANAGED; diff --git a/mm/memblock.c b/mm/memblock.c index 4fd431d16ef2..3a15708af3b6 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -997,6 +997,22 @@ int __init_memblock memblock_clear_nomap(phys_addr_t b= ase, phys_addr_t size) return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP, 0); } =20 +/** + * memblock_rsrv_mark_noinit - Mark a reserved memory region with flag MEM= BLOCK_RSRV_NOINIT. + * @base: the base phys addr of the region + * @size: the size of the region + * + * For memory regions marked with %MEMBLOCK_RSRV_NOINIT, reserve_bootmem_r= egion + * is not called during memmap_init_reserved_pages, hence struct pages are= not + * initialized for this region. + * + * Return: 0 on success, -errno on failure. + */ +int __init_memblock memblock_rsrv_mark_noinit(phys_addr_t base, phys_addr_= t size) +{ + return memblock_setclr_flag(base, size, 1, MEMBLOCK_RSRV_NOINIT, 1); +} + static bool should_skip_region(struct memblock_type *type, struct memblock_region *m, int nid, int flags) @@ -2113,13 +2129,17 @@ static void __init memmap_init_reserved_pages(void) memblock_set_node(start, end, &memblock.reserved, nid); } =20 - /* initialize struct pages for the reserved regions */ + /* + * initialize struct pages for reserved regions that don't have + * the MEMBLOCK_RSRV_NOINIT flag set + */ for_each_reserved_mem_region(region) { - nid =3D memblock_get_region_node(region); - start =3D region->base; - end =3D start + region->size; - - reserve_bootmem_region(start, end, nid); + if (!memblock_is_noinit(region)) { + nid =3D memblock_get_region_node(region); + start =3D region->base; + end =3D start + region->size; + reserve_bootmem_region(start, end, nid); + } } } =20 --=20 2.25.1 From nobody Tue Sep 9 22:47:39 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E5E4EB64DD for ; Thu, 27 Jul 2023 20:46:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231858AbjG0Uqk (ORCPT ); Thu, 27 Jul 2023 16:46:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34412 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231773AbjG0Uqi (ORCPT ); Thu, 27 Jul 2023 16:46:38 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 492FC2D5B for ; Thu, 27 Jul 2023 13:46:37 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id 5b1f17b1804b1-3fbc5d5746cso15093815e9.2 for ; Thu, 27 Jul 2023 13:46:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490796; x=1691095596; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cuTqSOqGGusC4+STWjmE7AEvtPXrFRozZqvR7YJO9tM=; b=MYvinfuvPzSR6QZNB+kO061wB6tpem3LnUdQ68duY2x9Lf58oM1yxzxSQNGFKohFk/ EI1cawtc+AbWjfvWpBvppmYxUriHX3hcCjpQz4l+rQHliweF9u1TU856lSFvZen+b0ac /Iqd2lMerRuef36wCLT08cM3aet4zD4TwG6rd3HjnQ3e8jECc7SC98cUtcttUQ97dfkH 2TCsQq6Rp2S9nECR41e32NdAk0O1RVp9RR8BBusJoiRVB5fSCtegpCAa22SLN6tNpX0M IR0L8UXTdcMg4LsESCJXNbMtMfHWQw91m2aMdBO9Amm4VRFmS4e2CIzdeIHkjpK5IJLK fqUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490796; x=1691095596; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cuTqSOqGGusC4+STWjmE7AEvtPXrFRozZqvR7YJO9tM=; b=C3UXuym9yAG/L2o+sGCamZRSgWaxLiW2A6fEkpaULf3yf4lgJ63cmZken26jliHGAn 54PBPRKt3Oyyt2LaaFPhuEnqzvCxkXKmsTwOY0oGmga9mspWDXZx7vmXxuO4Y55/A89o VdXuknTwmrEsS8SfMq1WnMQDv2YiKPsF56BapUZ4C0SVQ5kxcDGWfpBpPbb49mfkb49k X59gYpBlnb2m4Yji1S2ffCljCdoCRIkBau9pqpEwXfTvllsHD0C77yghxofaUgr7/DCb kK3GbTDJcBQp8fnJp0Qm6PKclVrgpRj1AFALVLF//Q8EYez5I6784AidfgUyes1hPk18 L21A== X-Gm-Message-State: ABy/qLa0uNh9y8xXTl6nhrrKdvqtAsCvDHWEKAp1aAkXE/0YuphmgpjZ kSz/0WZJ7UXPN0g8oJ9gOpOynA== X-Google-Smtp-Source: APBJJlEOKaj+kNcenpgjxlaFx5BKYk7IpRTqAJr1UfKl04jzi5rThxvhDJdy8KNbbbphmnOSEh7YMw== X-Received: by 2002:a7b:ce16:0:b0:3f9:b748:ff3f with SMTP id m22-20020a7bce16000000b003f9b748ff3fmr151792wmc.1.1690490795841; Thu, 27 Jul 2023 13:46:35 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:35 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 5/6] mm: move allocation of gigantic hstates to the start of mm_core_init Date: Thu, 27 Jul 2023 21:46:23 +0100 Message-Id: <20230727204624.1942372-6-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Whether the initialization of tail struct pages of a hugepage happens or not will become dependent on the commandline parameter hugetlb_free_vmemmap in the future. Hence, hugetlb_hstate_alloc_pages needs to be after command line parameters are parsed and the start of mm_core_init is a good point. Signed-off-by: Usama Arif --- include/linux/hugetlb.h | 1 + mm/hugetlb.c | 18 ++++++++++-------- mm/mm_init.c | 4 ++++ 3 files changed, 15 insertions(+), 8 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ca3c8e10f24a..2b20553deef3 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1271,4 +1271,5 @@ hugetlb_walk(struct vm_area_struct *vma, unsigned lon= g addr, unsigned long sz) return huge_pte_offset(vma->vm_mm, addr, sz); } =20 +void __init hugetlb_hstate_alloc_gigantic_pages(void); #endif /* _LINUX_HUGETLB_H */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 58cf5978bee1..c1fcf2af591a 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4418,14 +4418,6 @@ static int __init hugepages_setup(char *s) } } =20 - /* - * Global state is always initialized later in hugetlb_init. - * But we need to allocate gigantic hstates here early to still - * use the bootmem allocator. - */ - if (hugetlb_max_hstate && hstate_is_gigantic(parsed_hstate)) - hugetlb_hstate_alloc_pages(parsed_hstate); - last_mhp =3D mhp; =20 return 1; @@ -4437,6 +4429,16 @@ static int __init hugepages_setup(char *s) } __setup("hugepages=3D", hugepages_setup); =20 +void __init hugetlb_hstate_alloc_gigantic_pages(void) +{ + int i; + + for (i =3D 0; i < HUGE_MAX_HSTATE; i++) { + if (hstate_is_gigantic(&hstates[i])) + hugetlb_hstate_alloc_pages(&hstates[i]); + } +} + /* * hugepagesz command line processing * A specific huge page size can only be specified once with hugepagesz. diff --git a/mm/mm_init.c b/mm/mm_init.c index a1963c3322af..5585c66c3c42 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -26,6 +26,7 @@ #include #include #include +#include #include "internal.h" #include "slab.h" #include "shuffle.h" @@ -2768,6 +2769,9 @@ static void __init mem_init_print_info(void) */ void __init mm_core_init(void) { +#ifdef CONFIG_HUGETLBFS + hugetlb_hstate_alloc_gigantic_pages(); +#endif /* Initializations relying on SMP setup */ build_all_zonelists(NULL); page_alloc_init_cpuhp(); --=20 2.25.1 From nobody Tue Sep 9 22:47:39 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41127EB64DD for ; Thu, 27 Jul 2023 20:47:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232359AbjG0UrM (ORCPT ); Thu, 27 Jul 2023 16:47:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34414 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232217AbjG0UrB (ORCPT ); Thu, 27 Jul 2023 16:47:01 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 386382D60 for ; Thu, 27 Jul 2023 13:46:38 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id ffacd0b85a97d-3174aac120aso1423313f8f.2 for ; Thu, 27 Jul 2023 13:46:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490797; x=1691095597; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ckgW/P+ejqydrp6JQqQrrU5HjkGQNsndgwMWGYDB/pc=; b=MHW+96etwxTrHcysNxNv4Jt5Keu/21p5ZeNuDIbQx9GnIkEA1kq7sTTqw6Q/8BayM4 1Acht/ZPiVv8tOChy+mgBTfHG6ixCBKorlj2dJl7YTk0/sTWgbMWE9WJCFXy+OQ9Iijl F797zrYkH1WHN8+kZpWGtO4V6mpRCt3ijeyEEL/x0uHMVkx2JVs2E0JF144vy0BqQwCm brRLxremJQxfBWLsYq+wA+OhATOq4v3hK/HHE03k794S7TLjgAEkUPQNfZuGNCFBakRv 3NI8Jepy+Nx2mW0XqXn6WjNkB/tP3+1AiL7EGltSxqblDgE7Db3FfhwIHlnqEs7zbPpH c/9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490797; x=1691095597; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ckgW/P+ejqydrp6JQqQrrU5HjkGQNsndgwMWGYDB/pc=; b=HcWvJSbMTkhtzPPMgy1c8PJUZ1VeaGPrrkHY2pYet46UdygVtss9oCzPVKcvTe54Of GFptEERuCyulR0+3BQnd+Xi5c4qTwRUi6vZa9VSV7GSrOhjrkUDsiy8m+l+TLKcMlw/9 04VlTdOcJlrKfOSspUJbWsVWzC084xNSbAhqxYIanqCr3UhYlISQrMwPStMhqRh7NX/8 CiSSsc/NV4fZxhr1qNJYtzYQi8BHZHDqd6TTvHteeJJsXP5CrCiGEHTzUIaTtsOdzXl5 0F3GHHss1U4MpAzRPWUrpSePHQCqUtwcRQYg/xedRpbnbqeiCKsqES12+0aQ/twvGsVw nkjQ== X-Gm-Message-State: ABy/qLbewnaQyyodr5P7r8U4wej9aBuz3Z54VvVNXpAzqORGF8rV2yee tUPAXQWL5CiH5K8vo9HC0Y1u0A== X-Google-Smtp-Source: APBJJlEO7+YO7D/m+lNz0Wmsb1Fis9T5rZ6fiE8hYl22nZAClbV/pTs8BoCD41J0AwcBpujlXdeyTA== X-Received: by 2002:adf:db44:0:b0:314:34dd:aaec with SMTP id f4-20020adfdb44000000b0031434ddaaecmr229166wrj.8.1690490796779; Thu, 27 Jul 2023 13:46:36 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:36 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 6/6] mm: hugetlb: Skip initialization of struct pages freed later by HVO Date: Thu, 27 Jul 2023 21:46:24 +0100 Message-Id: <20230727204624.1942372-7-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This is done by marking the region for which to skip initialization with the MEMBLOCK_RSRV_NOINIT flag. If the region is for hugepages and if HVO is enabled, then those struct pages which will be freed later don't need to be initialized. This can save significant time when a large number of hugepages are allocated at boot time. HUGETLB_VMEMMAP_RESERVE_SIZE struct pages at the start of hugepage still need to be initialized. Signed-off-by: Usama Arif --- mm/hugetlb.c | 21 +++++++++++++++++++++ mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 3 +++ 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c1fcf2af591a..bb2b12f41026 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3166,6 +3166,9 @@ int __alloc_bootmem_huge_page(struct hstate *h, int n= id) { struct huge_bootmem_page *m =3D NULL; /* initialize for clang */ int nr_nodes, node; + phys_addr_t hugetlb_vmemmap_reserve_size =3D + HUGETLB_VMEMMAP_RESERVE_SIZE * sizeof(struct page); + phys_addr_t noinit_base; =20 /* do node specific alloc */ if (nid !=3D NUMA_NO_NODE) { @@ -3173,6 +3176,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int = nid) 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid); if (!m) return 0; + + if (vmemmap_optimize_enabled && hugetlb_vmemmap_optimizable(h)) { + noinit_base =3D virt_to_phys( + (void *)((phys_addr_t) m + hugetlb_vmemmap_reserve_size)); + memblock_rsrv_mark_noinit( + noinit_base, + huge_page_size(h) - hugetlb_vmemmap_reserve_size); + } + goto found; } /* allocate from next node when distributing huge pages */ @@ -3187,6 +3199,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int = nid) */ if (!m) return 0; + + if (vmemmap_optimize_enabled && hugetlb_vmemmap_optimizable(h)) { + noinit_base =3D virt_to_phys( + (void *)((phys_addr_t) m + hugetlb_vmemmap_reserve_size)); + memblock_rsrv_mark_noinit( + noinit_base, + huge_page_size(h) - hugetlb_vmemmap_reserve_size); + } + goto found; } =20 diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index bdf750a4786b..b5b7834e0f42 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -443,7 +443,7 @@ static int vmemmap_remap_alloc(unsigned long start, uns= igned long end, DEFINE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); =20 -static bool vmemmap_optimize_enabled =3D IS_ENABLED(CONFIG_HUGETLB_PAGE_OP= TIMIZE_VMEMMAP_DEFAULT_ON); +bool vmemmap_optimize_enabled =3D IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_= VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); =20 /** diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 07555d2dc0cb..cb5171abe683 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -64,4 +64,7 @@ static inline bool hugetlb_vmemmap_optimizable(const stru= ct hstate *h) { return hugetlb_vmemmap_optimizable_size(h) !=3D 0; } + +extern bool vmemmap_optimize_enabled; + #endif /* _LINUX_HUGETLB_VMEMMAP_H */ --=20 2.25.1