From nobody Tue Sep 9 22:30:30 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 006ECC001DF for ; Sun, 30 Jul 2023 15:16:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229807AbjG3PQl (ORCPT ); Sun, 30 Jul 2023 11:16:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50614 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230024AbjG3PQi (ORCPT ); Sun, 30 Jul 2023 11:16:38 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A5AA1713 for ; Sun, 30 Jul 2023 08:16:12 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-3fbc59de0e2so34772095e9.3 for ; Sun, 30 Jul 2023 08:16:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730170; x=1691334970; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=P5VE1b09XGrn3hjFzsE3qaVui4TN6DlsvAzQ6XWShV0=; b=LhfXuIzeZE5fNFxToBAlvDQA5EoZbK+3WEM+AjQQxsQ0JxOdwHbtoU7CSVU9Y3ua/I YUpJ5p9ziGSrXpohlrNQkGF+742Gm7k1GzsA5f33b42pcc1fc/iAwrWonfQ7MYbv1q7h YVGa2uWWvFfRDEkTBXC8BuX87YIa3rtNxtLUF7dmSJ7zSlON6lGYhX3vs8+xJgQ7inEi v2WUTfER2YUR5IubnbmBUcgeUw1db17u0y+MPFkcMPvVpO4iH/iea96utluPPBbXOwxl JtX5YfmoepWtGVwANwedcUUAaZIqjKeijQsq21YfiLsa2xpb4FFh20dY0yu/BPDgxfy1 opbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730170; x=1691334970; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=P5VE1b09XGrn3hjFzsE3qaVui4TN6DlsvAzQ6XWShV0=; b=eBSOQhRwN3F3x+//9i/Nz3GBC5czXB1M4mGu8iHcUhxx+eIcIIs5XP12E7ScoVc1HQ UIYqYvJD8kRyjfwxOrmeFO68IRMesRqCFreSh6JixMmRPV3ajP1L7LopY/gmme5c3u9c WJzQ2Rd1FOLh5RuWBhRMEvC+Pxf2FcPVZd+ij//+2m+y9+Q3FA6aRB7iCfcqWQBU+E7A 26RV5XsNC8HC+Gktqd3w8dicV+JeaYfGl4P8o4QH+ceLDv5WmFO9Yu6XBA9LiWK5rk4B FAP/hUyMhWpKKo3+N3YgWb4PDvpl5hvUUgnKx/DCGlxXjqR0H7s8D6wXmituWBpYUy5q f7CA== X-Gm-Message-State: ABy/qLYoAHD4N70d/lLBTBbpfhcNZGD/rcKzZQb6P9N5RWnkbVtb94iX R+NrQM8RNExTlMdI+aOqZUdXpA== X-Google-Smtp-Source: APBJJlGPOGrrpNvuwalf5EOZlDbgmbBWoE71UsgklAK9FFQ8DCMd6wl0g32vEm3QO8b2B4WvXFrH5A== X-Received: by 2002:a7b:cc8c:0:b0:3fc:175:ade7 with SMTP id p12-20020a7bcc8c000000b003fc0175ade7mr5638740wma.38.1690730170432; Sun, 30 Jul 2023 08:16:10 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:09 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 1/6] mm: hugetlb: Skip prep of tail pages when HVO is enabled Date: Sun, 30 Jul 2023 16:16:01 +0100 Message-Id: <20230730151606.2871391-2-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When vmemmap is optimizable, it will free all the duplicated tail pages in hugetlb_vmemmap_optimize while preparing the new hugepage. Hence, there is no need to prepare them. For 1G x86 hugepages, it avoids preparing 262144 - 64 =3D 262080 struct pages per hugepage. The indirection of using __prep_compound_gigantic_folio is also removed, as it just creates extra functions to indicate demote which can be done with the argument. Signed-off-by: Usama Arif --- mm/hugetlb.c | 32 ++++++++++++++------------------ mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 15 +++++++++++---- 3 files changed, 26 insertions(+), 23 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 64a3239b6407..541c07b6d60f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1942,14 +1942,23 @@ static void prep_new_hugetlb_folio(struct hstate *h= , struct folio *folio, int ni spin_unlock_irq(&hugetlb_lock); } =20 -static bool __prep_compound_gigantic_folio(struct folio *folio, - unsigned int order, bool demote) +static bool prep_compound_gigantic_folio(struct folio *folio, struct hstat= e *h, bool demote) { int i, j; + int order =3D huge_page_order(h); int nr_pages =3D 1 << order; struct page *p; =20 __folio_clear_reserved(folio); + + /* + * No need to prep pages that will be freed later by hugetlb_vmemmap_opti= mize. + * Hence, reduce nr_pages to the pages that will be kept. + */ + if (IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP) && + vmemmap_should_optimize(h, &folio->page)) + nr_pages =3D HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page); + for (i =3D 0; i < nr_pages; i++) { p =3D folio_page(folio, i); =20 @@ -2019,18 +2028,6 @@ static bool __prep_compound_gigantic_folio(struct fo= lio *folio, return false; } =20 -static bool prep_compound_gigantic_folio(struct folio *folio, - unsigned int order) -{ - return __prep_compound_gigantic_folio(folio, order, false); -} - -static bool prep_compound_gigantic_folio_for_demote(struct folio *folio, - unsigned int order) -{ - return __prep_compound_gigantic_folio(folio, order, true); -} - /* * PageHuge() only returns true for hugetlbfs pages, but not for normal or * transparent huge pages. See the PageTransHuge() documentation for more @@ -2185,7 +2182,7 @@ static struct folio *alloc_fresh_hugetlb_folio(struct= hstate *h, if (!folio) return NULL; if (hstate_is_gigantic(h)) { - if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { + if (!prep_compound_gigantic_folio(folio, h, false)) { /* * Rare failure to convert pages to compound page. * Free pages and try again - ONCE! @@ -3201,7 +3198,7 @@ static void __init gather_bootmem_prealloc(void) =20 VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(folio_ref_count(folio) !=3D 1); - if (prep_compound_gigantic_folio(folio, huge_page_order(h))) { + if (prep_compound_gigantic_folio(folio, h, false)) { WARN_ON(folio_test_reserved(folio)); prep_new_hugetlb_folio(h, folio, folio_nid(folio)); free_huge_page(page); /* add to the hugepage allocator */ @@ -3624,8 +3621,7 @@ static int demote_free_hugetlb_folio(struct hstate *h= , struct folio *folio) subpage =3D folio_page(folio, i); inner_folio =3D page_folio(subpage); if (hstate_is_gigantic(target_hstate)) - prep_compound_gigantic_folio_for_demote(inner_folio, - target_hstate->order); + prep_compound_gigantic_folio(inner_folio, target_hstate, true); else prep_compound_page(subpage, target_hstate->order); folio_change_private(inner_folio, NULL); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c2007ef5e9b0..b721e87de2b3 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -486,7 +486,7 @@ int hugetlb_vmemmap_restore(const struct hstate *h, str= uct page *head) } =20 /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ -static bool vmemmap_should_optimize(const struct hstate *h, const struct p= age *head) +bool vmemmap_should_optimize(const struct hstate *h, const struct page *he= ad) { if (!READ_ONCE(vmemmap_optimize_enabled)) return false; diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 25bd0e002431..3e7978a9af73 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -10,16 +10,17 @@ #define _LINUX_HUGETLB_VMEMMAP_H #include =20 -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); -void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); - /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See * Documentation/vm/vmemmap_dedup.rst. */ #define HUGETLB_VMEMMAP_RESERVE_SIZE PAGE_SIZE =20 +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); +void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); +bool vmemmap_should_optimize(const struct hstate *h, const struct page *he= ad); + static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) { return pages_per_huge_page(h) * sizeof(struct page); @@ -51,6 +52,12 @@ static inline unsigned int hugetlb_vmemmap_optimizable_s= ize(const struct hstate { return 0; } + +static inline bool vmemmap_should_optimize(const struct hstate *h, const s= truct page *head) +{ + return false; +} + #endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ =20 static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h) --=20 2.25.1 From nobody Tue Sep 9 22:30:30 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DE57C001DF for ; Sun, 30 Jul 2023 15:16:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230040AbjG3PQn (ORCPT ); Sun, 30 Jul 2023 11:16:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230027AbjG3PQi (ORCPT ); Sun, 30 Jul 2023 11:16:38 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E56421716 for ; Sun, 30 Jul 2023 08:16:12 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-3fbc5d5742bso41888745e9.2 for ; Sun, 30 Jul 2023 08:16:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730171; x=1691334971; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; b=JoxH10l13AhAr8ISmwB2i1gOf6hMfDgdag1NYZGf8eqVGmEdh4R3IE2D1prI5+wzm7 vu1QD7xvEBstHjEIbxAMlgCCO00ONBH3G5zPFMoDk4Drvk6SzML5WNLP4tfo6lnQh865 TX13dZrkIeWswvkw6xN/xmei3GAwgiUombSwNiUwcW22Vp57PvkxefJyTKdVVA+u+gzP VbGxb9p/bzQfa6R6GWeFqN5krmkkMZbJxyXakbPqdmWiMydGR5JelRYvQMAxOrQf62pU o4lWtSXohDF8KsxmkSED8zQCamat1slkfvH7Xda+Eyuy3MLasON3KHOua9umK5WY8Nob wW5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730171; x=1691334971; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; b=RMg2GfdFISe3acNSNOvZ9cm0P9VWd0Zb2Ad1F5XQz4wsNUj+BoWzPGtB9Ei/1o86vC kFPBKuzBs0SIm1Lp1iyQHe0A3triHfOn6CMmdFMKqgWTayOz7W6yWqX+xQV+Fk61M/rJ R4NQ85/KvwOYHs4ilJsMpDJlCU2TL9R6ZycgsE2TiVwixWcHPlUIvxN+jdwNhHQ4Lz3B m/p7OErswrzoSyKku5JU+LUc33wgieqFX3X3P+ySmqB10+VhXwL6Pfsv5uWloqO6v37F LMFT6m8udhud/JaFiGbdnomVGAAmlARW+UznJ2wGHy5DUzC5Mgj0ZkB7MvU/4x7Df3gg x6LA== X-Gm-Message-State: ABy/qLbd6bAzhkiuORD4pY0uxbXQTzFydHg2tc7dA08gWsTyJAH1npUV KwNb+faOysaCcPHTeFG5dQepOQ== X-Google-Smtp-Source: APBJJlFtcqfY902vGXRLqPZchnNVVitcnCdU3eEgHgB4d1amxrHvfEkxn/i1iktNVcoW6QdeU+DsCA== X-Received: by 2002:a7b:cbd5:0:b0:3fe:1db2:5168 with SMTP id n21-20020a7bcbd5000000b003fe1db25168mr1370994wmi.11.1690730171283; Sun, 30 Jul 2023 08:16:11 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:10 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 2/6] mm: hugetlb_vmemmap: Use nid of the head page to reallocate it Date: Sun, 30 Jul 2023 16:16:02 +0100 Message-Id: <20230730151606.2871391-3-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" If tail page prep and initialization is skipped, then the "start" page will not contain the correct nid. Use the nid from first vmemap page. Signed-off-by: Usama Arif --- mm/hugetlb_vmemmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index b721e87de2b3..bdf750a4786b 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -324,7 +324,7 @@ static int vmemmap_remap_free(unsigned long start, unsi= gned long end, .reuse_addr =3D reuse, .vmemmap_pages =3D &vmemmap_pages, }; - int nid =3D page_to_nid((struct page *)start); + int nid =3D page_to_nid((struct page *)reuse); gfp_t gfp_mask =3D GFP_KERNEL | __GFP_THISNODE | __GFP_NORETRY | __GFP_NOWARN; =20 --=20 2.25.1 From nobody Tue Sep 9 22:30:30 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E31DC04A94 for ; Sun, 30 Jul 2023 15:16:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230016AbjG3PQR (ORCPT ); Sun, 30 Jul 2023 11:16:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50622 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229918AbjG3PQO (ORCPT ); Sun, 30 Jul 2023 11:16:14 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9C5F3138 for ; Sun, 30 Jul 2023 08:16:13 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-3fba8e2aa52so42188475e9.1 for ; Sun, 30 Jul 2023 08:16:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730172; x=1691334972; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LjF+zN6TJ1BvEvRrI5C4CD8rIkkkthqR+fjsUt4Q8CQ=; b=Vg/2oAVG+oc700uPemJAECQqoZhvkHC4+v5G5umPPiGRagPclA/x9n9QgA5SDk8rdg xAP9ReHyrkmHzKD+QCBzbqzaesJvP/4YCVXhkUBDq7oFIh6fXXlQ/W6sesudYok3JgtG i8bNtt0JeBjQ7J+lYkbC40TeksYu8cuU6k1rZccXxVLpTekSilTdDcASxORLzoJggl+u dRvUz2KpmJ9ZuYWecFzFZAf1jF7ZS6Fe1gbbhfyhcMe2hud5pyNq2/bry4VpryWWF7x9 5XjSiGdDOVXVOjoB9TsdMcNi/ewnPLEIYUwh7l9m4DJjOtr0WmlRT+SiTcjbgC7BRS1N Zt9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730172; x=1691334972; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LjF+zN6TJ1BvEvRrI5C4CD8rIkkkthqR+fjsUt4Q8CQ=; b=a1u51rrFCKwhbysKj25kUQzqMgF1p+lTpRpo/E+2EreLrDNo4by/xnIEI71kLJDM3Y ekrAw3Y4BhzlFyyO2/YUfqmAO7PvAaPBGDOND+z51YTcYjV3S4VOvZhbBMd4Hfi7Mmi8 pwKbeC83i6YGYdDi8a+YcBeMib+H+K74fSlhm49KUQP3qSKDDdvMAhD7B5JMpnWHRz11 U5XPINj2A6CYAKtH9w1Bl4wubqJkwkByee6aHMJB4iXV3hwymBnDKUahszwKcodfj6pJ VIRcNPdHm3X/UTYLZr/BiuT36shaDodImA03BjZQ7LkGFvKhbKhoPCPT3NNCkoDQDFq5 ad0Q== X-Gm-Message-State: ABy/qLbOPXSMQ2MAtfcqdkcpBphnzHeod0VAc3dEUiKiLtLkiZ3IdohO WzQD8C88niBY4LOJAImguKcpGg== X-Google-Smtp-Source: APBJJlEBTcG9vQb7sO+eVhsdznDus6PkQqwHLouS3PQ0kKj5rr+MyIEAIBdbHeu9lJsdSY/4wUqiMA== X-Received: by 2002:a7b:c450:0:b0:3fb:c990:3b2 with SMTP id l16-20020a7bc450000000b003fbc99003b2mr5960153wmi.34.1690730172214; Sun, 30 Jul 2023 08:16:12 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:11 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 3/6] memblock: pass memblock_type to memblock_setclr_flag Date: Sun, 30 Jul 2023 16:16:03 +0100 Message-Id: <20230730151606.2871391-4-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This allows setting flags to both memblock types and is in preparation for setting flags (for e.g. to not initialize struct pages) on reserved memory region. Signed-off-by: Usama Arif --- mm/memblock.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/mm/memblock.c b/mm/memblock.c index f9e61e565a53..43cb4404d94c 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -896,10 +896,9 @@ int __init_memblock memblock_physmem_add(phys_addr_t b= ase, phys_addr_t size) * * Return: 0 on success, -errno on failure. */ -static int __init_memblock memblock_setclr_flag(phys_addr_t base, - phys_addr_t size, int set, int flag) +static int __init_memblock memblock_setclr_flag(struct memblock_type *type, + phys_addr_t base, phys_addr_t size, int set, int flag) { - struct memblock_type *type =3D &memblock.memory; int i, ret, start_rgn, end_rgn; =20 ret =3D memblock_isolate_range(type, base, size, &start_rgn, &end_rgn); @@ -928,7 +927,7 @@ static int __init_memblock memblock_setclr_flag(phys_ad= dr_t base, */ int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t si= ze) { - return memblock_setclr_flag(base, size, 1, MEMBLOCK_HOTPLUG); + return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_HOT= PLUG); } =20 /** @@ -940,7 +939,7 @@ int __init_memblock memblock_mark_hotplug(phys_addr_t b= ase, phys_addr_t size) */ int __init_memblock memblock_clear_hotplug(phys_addr_t base, phys_addr_t s= ize) { - return memblock_setclr_flag(base, size, 0, MEMBLOCK_HOTPLUG); + return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_HOT= PLUG); } =20 /** @@ -957,7 +956,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t ba= se, phys_addr_t size) =20 system_has_some_mirror =3D true; =20 - return memblock_setclr_flag(base, size, 1, MEMBLOCK_MIRROR); + return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_MIR= ROR); } =20 /** @@ -977,7 +976,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t ba= se, phys_addr_t size) */ int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 1, MEMBLOCK_NOMAP); + return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_NOM= AP); } =20 /** @@ -989,7 +988,7 @@ int __init_memblock memblock_mark_nomap(phys_addr_t bas= e, phys_addr_t size) */ int __init_memblock memblock_clear_nomap(phys_addr_t base, phys_addr_t siz= e) { - return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP); + return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_NOM= AP); } =20 static bool should_skip_region(struct memblock_type *type, --=20 2.25.1 From nobody Tue Sep 9 22:30:30 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2802FC001DF for ; Sun, 30 Jul 2023 15:16:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230075AbjG3PQs (ORCPT ); Sun, 30 Jul 2023 11:16:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50622 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230035AbjG3PQk (ORCPT ); Sun, 30 Jul 2023 11:16:40 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 89A54171A for ; Sun, 30 Jul 2023 08:16:14 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-3fe2048c910so2025485e9.1 for ; Sun, 30 Jul 2023 08:16:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730173; x=1691334973; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sRlgK3gsc81CRpAMRRPsywIlI5j9OUiW8IxZjNa20o0=; b=W/lVFu8oBYOHaQON2+IUAE4OeTID9l1Wz5BLxJdEu8LJBB9z/KErJPNGKn4gPKcqET TOz183J9eC9oj+adoUW1wOIA2x17GqKNs+kATE99/ZRY8Fjp8O5VirlHjPixUUm43ku4 z+wT3z1JbZFobFUIlYBs/gN3XFoPKT2xOzlfLL6MoMgEtn/SiRo+2YI4xg86K/JhVl3i MuhecVQM9cEuPwUzqT4PGXvn5nnp3Ror/whrcq0G86C37cVp/BcEGPKbi/b5/niap3Zq DtL+erRe6APzX6GQNORLFAEQj9gidto4zepL9icunXIYosg8Xv4r0ivr6Y0+U81B8I1M t8+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730173; x=1691334973; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sRlgK3gsc81CRpAMRRPsywIlI5j9OUiW8IxZjNa20o0=; b=YZ3KGxN0H8XlPhQWdPsNzjiPZEJQntgotTMelPIicrugNOD/xy5FzjAwbMCMohbOnT MU4GhI+WjCvJJrn5Ig3sTLnoylcdGB6inONai6y+uHAYOOxXs9gjCfN/0lT5sMcZPbTm A0jnaOrm248kwuMPthVEK5DRwfWqKNS6WJdPtv1Lf5WiS5NdAzWlE0yZ3NFkEG1HeFnt ML7GRs19ORDhLmK22TSigItHLPiV/SmXGHLxl07AjieVhRxLomR7vB5kl1DG2DPpvi9G 7yqEPYcrnpOro+w5KmTWRju+1dQUmZdlt1z5UcEkYDzl+uwanupD3Wrn+Icg0IFqfC0G Ku3g== X-Gm-Message-State: ABy/qLbvk6sCB0Rof9wXWm+4Yk7oNos8J6amGuYvI1idkCPNB+n9xPbk RhPscleu2TXuHL1HeuT6NXZe35/+l3W6viI/nNw= X-Google-Smtp-Source: APBJJlFzVJI2UxHIjpRrxyZbsv8rAC3x9G9eCF0qIIb2iLIkSv48sgus3BhU4mIBADGNAt8oxdN7wQ== X-Received: by 2002:a05:600c:2809:b0:3fa:9767:c816 with SMTP id m9-20020a05600c280900b003fa9767c816mr5021376wmb.39.1690730173023; Sun, 30 Jul 2023 08:16:13 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:12 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 4/6] memblock: introduce MEMBLOCK_RSRV_NOINIT flag Date: Sun, 30 Jul 2023 16:16:04 +0100 Message-Id: <20230730151606.2871391-5-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" For reserved memory regions marked with this flag, reserve_bootmem_region is not called during memmap_init_reserved_pages. This can be used to avoid struct page initialization for regions which won't need them, for e.g. hugepages with HVO enabled. Signed-off-by: Usama Arif --- include/linux/memblock.h | 9 +++++++++ mm/memblock.c | 30 +++++++++++++++++++++++++----- 2 files changed, 34 insertions(+), 5 deletions(-) diff --git a/include/linux/memblock.h b/include/linux/memblock.h index f71ff9f0ec81..de7760aeb870 100644 --- a/include/linux/memblock.h +++ b/include/linux/memblock.h @@ -40,6 +40,8 @@ extern unsigned long long max_possible_pfn; * via a driver, and never indicated in the firmware-provided memory map as * system RAM. This corresponds to IORESOURCE_SYSRAM_DRIVER_MANAGED in the * kernel resource tree. + * @MEMBLOCK_RSRV_NOINIT: memory region for which struct pages are not ini= tialized + * (only for reserved regions). */ enum memblock_flags { MEMBLOCK_NONE =3D 0x0, /* No special request */ @@ -47,6 +49,7 @@ enum memblock_flags { MEMBLOCK_MIRROR =3D 0x2, /* mirrored region */ MEMBLOCK_NOMAP =3D 0x4, /* don't add to kernel direct mapping */ MEMBLOCK_DRIVER_MANAGED =3D 0x8, /* always detected via a driver */ + MEMBLOCK_RSRV_NOINIT =3D 0x10, /* don't initialize struct pages */ }; =20 /** @@ -125,6 +128,7 @@ int memblock_clear_hotplug(phys_addr_t base, phys_addr_= t size); int memblock_mark_mirror(phys_addr_t base, phys_addr_t size); int memblock_mark_nomap(phys_addr_t base, phys_addr_t size); int memblock_clear_nomap(phys_addr_t base, phys_addr_t size); +int memblock_reserved_mark_noinit(phys_addr_t base, phys_addr_t size); =20 void memblock_free_all(void); void memblock_free(void *ptr, size_t size); @@ -259,6 +263,11 @@ static inline bool memblock_is_nomap(struct memblock_r= egion *m) return m->flags & MEMBLOCK_NOMAP; } =20 +static inline bool memblock_is_noinit(struct memblock_region *m) +{ + return m->flags & MEMBLOCK_RSRV_NOINIT; +} + static inline bool memblock_is_driver_managed(struct memblock_region *m) { return m->flags & MEMBLOCK_DRIVER_MANAGED; diff --git a/mm/memblock.c b/mm/memblock.c index 43cb4404d94c..40690ce0b06e 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -991,6 +991,21 @@ int __init_memblock memblock_clear_nomap(phys_addr_t b= ase, phys_addr_t size) return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_NOM= AP); } =20 +/** + * memblock_reserved_mark_noinit - Mark a reserved memory region with flag= MEMBLOCK_RSRV_NOINIT. + * @base: the base phys addr of the region + * @size: the size of the region + * + * struct pages will not be initialized for reserved memory regions marked= with + * %MEMBLOCK_RSRV_NOINIT. + * + * Return: 0 on success, -errno on failure. + */ +int __init_memblock memblock_reserved_mark_noinit(phys_addr_t base, phys_a= ddr_t size) +{ + return memblock_setclr_flag(&memblock.reserved, base, size, 1, MEMBLOCK_R= SRV_NOINIT); +} + static bool should_skip_region(struct memblock_type *type, struct memblock_region *m, int nid, int flags) @@ -2107,13 +2122,18 @@ static void __init memmap_init_reserved_pages(void) memblock_set_node(start, end, &memblock.reserved, nid); } =20 - /* initialize struct pages for the reserved regions */ + /* + * initialize struct pages for reserved regions that don't have + * the MEMBLOCK_RSRV_NOINIT flag set + */ for_each_reserved_mem_region(region) { - nid =3D memblock_get_region_node(region); - start =3D region->base; - end =3D start + region->size; + if (!memblock_is_noinit(region)) { + nid =3D memblock_get_region_node(region); + start =3D region->base; + end =3D start + region->size; =20 - reserve_bootmem_region(start, end, nid); + reserve_bootmem_region(start, end, nid); + } } } =20 --=20 2.25.1 From nobody Tue Sep 9 22:30:30 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DD7EC001E0 for ; Sun, 30 Jul 2023 15:16:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230028AbjG3PQV (ORCPT ); Sun, 30 Jul 2023 11:16:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50632 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229996AbjG3PQQ (ORCPT ); Sun, 30 Jul 2023 11:16:16 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45F69138 for ; Sun, 30 Jul 2023 08:16:15 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id 5b1f17b1804b1-3fbc12181b6so42071685e9.2 for ; Sun, 30 Jul 2023 08:16:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730174; x=1691334974; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SFM4pR1QacGlm/8xlIUmCJcy4VSy/s+0OM0YFxjuA8k=; b=TkyhfbVGe1Y7YHfHksu1wqG6hgQhYRmidCb5+Y61F1QRRs/h47YJFt0AIJh54Q0FKL gByq+10/h43JlVcCEVZjC6aUzw/Omy08uMjBB7CgI/FZQXN3z5WuJzdScj4/HnraGiyu il7QDEWZYGSpogyg92dYwrJa+Eqeb9M65dGnmL7nnCP2F14JH377zSxW/hWjC6XaZnyk O4M74hvreh6XC7+2F9vY3dYi8dCjt4haamygyIK9l6ZMqTa22DtdrGIfwCNlLo2xUbbo yHp+23v36B0XyqElADWdUdmxDLaLZYjFnCSrKJy9muMAQ+U2qOTBbbF4yWjE1y0Ngwfe EPHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730174; x=1691334974; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SFM4pR1QacGlm/8xlIUmCJcy4VSy/s+0OM0YFxjuA8k=; b=YW4EEIX4ylfJSBse2jsh5Dwd577GK00szMeZY1xSl5zbyNMxecH2sf27ZHgbnuUd3S g9sG+OpoEzfCScytXOPQLSbq0s8n8uASoiwpocWAmFuQKfsu7A2Jy3yofy33877XeIYC DS8shqJVhPQKDToft7zzOyEBie0jcJZDkAPHNQW8IE3G5AaD2F0UEhHjmizplXAvtuc0 tOdT7Nd7tmHPQ21BNpTjz89chSNz+w6Qr5ImFLy8KpY7vkvXAMHLAYmh39+DQhJMCz+x 6/s5n0RDRj0Z3EFyG2lS1FtHfalzEx5Jpo6AjOaML+MDV/AXYWQxVsacSvGvdpIA6TEY bxsw== X-Gm-Message-State: ABy/qLb3o9Dyp3eeJGMUfS/gYDusB9ow2MAgbPmr4WlLVedRFa2D8h/3 myGTE08cEfpcYTXgnYWakTpu2w== X-Google-Smtp-Source: APBJJlGH28j3h1b9uEfHTlqYomCBicErb0vATQvL1NGT50ZdEoNVQZEd6E1WZNtnMG3Mxzbn8725YA== X-Received: by 2002:a05:600c:d5:b0:3fb:e206:ca5f with SMTP id u21-20020a05600c00d500b003fbe206ca5fmr5709508wmm.31.1690730173889; Sun, 30 Jul 2023 08:16:13 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:13 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 5/6] mm: move allocation of gigantic hstates to the start of mm_core_init Date: Sun, 30 Jul 2023 16:16:05 +0100 Message-Id: <20230730151606.2871391-6-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Whether the initialization of tail struct pages of a hugepage happens or not will become dependent on the commandline parameter hugetlb_free_vmemmap in the future. Hence, hugetlb_hstate_alloc_pages needs to be after command line parameters are parsed and the start of mm_core_init is a good point. Signed-off-by: Usama Arif --- mm/hugetlb.c | 18 ++++++++++-------- mm/internal.h | 9 +++++++++ mm/mm_init.c | 6 ++++++ 3 files changed, 25 insertions(+), 8 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 541c07b6d60f..bf60545496d7 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4400,14 +4400,6 @@ static int __init hugepages_setup(char *s) } } =20 - /* - * Global state is always initialized later in hugetlb_init. - * But we need to allocate gigantic hstates here early to still - * use the bootmem allocator. - */ - if (hugetlb_max_hstate && hstate_is_gigantic(parsed_hstate)) - hugetlb_hstate_alloc_pages(parsed_hstate); - last_mhp =3D mhp; =20 return 1; @@ -4419,6 +4411,16 @@ static int __init hugepages_setup(char *s) } __setup("hugepages=3D", hugepages_setup); =20 +void __init hugetlb_hstate_alloc_gigantic_pages(void) +{ + int i; + + for (i =3D 0; i < HUGE_MAX_HSTATE; i++) { + if (hstate_is_gigantic(&hstates[i])) + hugetlb_hstate_alloc_pages(&hstates[i]); + } +} + /* * hugepagesz command line processing * A specific huge page size can only be specified once with hugepagesz. diff --git a/mm/internal.h b/mm/internal.h index a7d9e980429a..692bb1136a39 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1102,4 +1102,13 @@ struct vma_prepare { struct vm_area_struct *remove; struct vm_area_struct *remove2; }; + +#ifdef CONFIG_HUGETLBFS +void __init hugetlb_hstate_alloc_gigantic_pages(void); +#else +static inline void __init hugetlb_hstate_alloc_gigantic_pages(void); +{ +} +#endif /* CONFIG_HUGETLBFS */ + #endif /* __MM_INTERNAL_H */ diff --git a/mm/mm_init.c b/mm/mm_init.c index a1963c3322af..f2751ccd7d99 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -26,6 +26,7 @@ #include #include #include +#include #include "internal.h" #include "slab.h" #include "shuffle.h" @@ -2768,6 +2769,11 @@ static void __init mem_init_print_info(void) */ void __init mm_core_init(void) { + /* + * We need to allocate gigantic hstates here early to still use the bootm= em + * allocator. Non gigantic hstates are initialized later in hugetlb_init. + */ + hugetlb_hstate_alloc_gigantic_pages(); /* Initializations relying on SMP setup */ build_all_zonelists(NULL); page_alloc_init_cpuhp(); --=20 2.25.1 From nobody Tue Sep 9 22:30:30 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6592C001DF for ; Sun, 30 Jul 2023 15:16:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229521AbjG3PQv (ORCPT ); Sun, 30 Jul 2023 11:16:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230039AbjG3PQl (ORCPT ); Sun, 30 Jul 2023 11:16:41 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B3D2171B for ; Sun, 30 Jul 2023 08:16:16 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-3fbc5d5742eso42076825e9.3 for ; Sun, 30 Jul 2023 08:16:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730174; x=1691334974; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ed3bTY/DQrtVo38S3b99vtsCN5ydWYuckmFt+jfpXnQ=; b=CsLvLnj1VBExXRAAglTYPWH2EpYCorZggOAC3yIHs0vybciyxsExVcLP/39obiFOYD FverDpvbx+BWRlvxddS1Ucch0Lo8ySBj5n4n9qsls9ncd4/U0ukC8FCsg5LU0Ps7yXpS 0x73YYFy0A18lccSHORRLeDqKaS8nvvq1tbkN/1SKvxZESadhIFOz/0oIn1npIu6dGrI 2CSL9zF3tMODmGVmKAr7ZywN5DBo03IuiGI33kAtutuURXCt6kQH4LYr2FKRdNpyhVsi OuuSaUD++enIZiLNISEKbcIDH+KKX7DGzU9igsnu/+61mXgL/I4WQRrk8Tlu5bD6f8xb lAyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730174; x=1691334974; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ed3bTY/DQrtVo38S3b99vtsCN5ydWYuckmFt+jfpXnQ=; b=l5DFJAE0OqA9UJDZYpLQO/ng+i2JvHTjYiEzqZaaun9rcgHVcl1Uxb91s8q+qW0kuJ smr+zYXKp5p15eiiywO+ff9grWdzmeZ4FOjVEZYeguNG301skm9lSRGezrVHYnQPC73/ pZiNDPQ+5RIe+hPA16eZaQcZBEt5fe6K75RL2I4t3weCApIocySPqTje/eR6QyQJIldv la9NwSG9rDRlw7hJaMSd2gOCyoReqC8lnel64Ex+9NDyA1yphvGPQ33BJnbs+AYnNzKb J50d7GRRHTk7kB2WH5I9u+0P0xg9dctsWw9uIR6UnbeGeckIVR/YxJ99coCG+W57Na+7 jDzA== X-Gm-Message-State: ABy/qLZmMb2vUXBzzez8p08zQhbpIfAUk1AsJc7C/LX3sz0BEMu4URxX zDBnaWP7ieN7iy+swL2F3r10/Q== X-Google-Smtp-Source: APBJJlEvwIYdcin+SgJIaKf3f0NWjv2vOyb8TeOb/EE1tTkylJ9cPozIeSxvQxhFH8cAtZQs6UhkkA== X-Received: by 2002:a05:600c:3787:b0:3fa:97b3:7ce0 with SMTP id o7-20020a05600c378700b003fa97b37ce0mr3917250wmr.26.1690730174681; Sun, 30 Jul 2023 08:16:14 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:14 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 6/6] mm: hugetlb: Skip initialization of struct pages freed later by HVO Date: Sun, 30 Jul 2023 16:16:06 +0100 Message-Id: <20230730151606.2871391-7-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This is done by marking the region for which to skip initialization with the MEMBLOCK_RSRV_NOINIT flag. If the region is for hugepages and if HVO is enabled, then those struct pages which will be freed later don't need to be initialized. This can save significant time when a large number of hugepages are allocated at boot time. HUGETLB_VMEMMAP_RESERVE_SIZE struct pages at the start of hugepage still need to be initialized. Signed-off-by: Usama Arif --- mm/hugetlb.c | 21 +++++++++++++++++++++ mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 3 +++ 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bf60545496d7..8434100f60ae 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3151,6 +3151,9 @@ int __alloc_bootmem_huge_page(struct hstate *h, int n= id) { struct huge_bootmem_page *m =3D NULL; /* initialize for clang */ int nr_nodes, node; + phys_addr_t hugetlb_vmemmap_reserve_size =3D + HUGETLB_VMEMMAP_RESERVE_SIZE * sizeof(struct page); + phys_addr_t noinit_base; =20 /* do node specific alloc */ if (nid !=3D NUMA_NO_NODE) { @@ -3158,6 +3161,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int = nid) 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid); if (!m) return 0; + + if (vmemmap_optimize_enabled && hugetlb_vmemmap_optimizable(h)) { + noinit_base =3D virt_to_phys( + (void *)((phys_addr_t) m + hugetlb_vmemmap_reserve_size)); + memblock_reserved_mark_noinit( + noinit_base, + huge_page_size(h) - hugetlb_vmemmap_reserve_size); + } + goto found; } /* allocate from next node when distributing huge pages */ @@ -3172,6 +3184,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int = nid) */ if (!m) return 0; + + if (vmemmap_optimize_enabled && hugetlb_vmemmap_optimizable(h)) { + noinit_base =3D virt_to_phys( + (void *)((phys_addr_t) m + hugetlb_vmemmap_reserve_size)); + memblock_reserved_mark_noinit( + noinit_base, + huge_page_size(h) - hugetlb_vmemmap_reserve_size); + } + goto found; } =20 diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index bdf750a4786b..b5b7834e0f42 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -443,7 +443,7 @@ static int vmemmap_remap_alloc(unsigned long start, uns= igned long end, DEFINE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); =20 -static bool vmemmap_optimize_enabled =3D IS_ENABLED(CONFIG_HUGETLB_PAGE_OP= TIMIZE_VMEMMAP_DEFAULT_ON); +bool vmemmap_optimize_enabled =3D IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_= VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); =20 /** diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 3e7978a9af73..3fff6f611c19 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -64,4 +64,7 @@ static inline bool hugetlb_vmemmap_optimizable(const stru= ct hstate *h) { return hugetlb_vmemmap_optimizable_size(h) !=3D 0; } + +extern bool vmemmap_optimize_enabled; + #endif /* _LINUX_HUGETLB_VMEMMAP_H */ --=20 2.25.1