From nobody Wed Apr 15 19:23:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DF9FC433FE for ; Thu, 17 Nov 2022 21:16:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240855AbiKQVQK (ORCPT ); Thu, 17 Nov 2022 16:16:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240776AbiKQVP1 (ORCPT ); Thu, 17 Nov 2022 16:15:27 -0500 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 206A684334 for ; Thu, 17 Nov 2022 13:15:14 -0800 (PST) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AHLEGdP000786; Thu, 17 Nov 2022 21:15:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=VHJL7jnc3iptg9cxJxIdTtxv2yR1/WZxi2cQQ0+3Ggg=; b=nYOz+FOcckJb+g5YccNYBVmX9N6q7hV4Y7/wDIasmofgV8Q3uvfgkisWHn6DfhNU9003 im7abbzHcKtZazd6Ri72LzvpubAypV0Ktoonxokgd4tx2Otxd+ttavH82pSj2jwku98c 1st0p8RYhwYkEOSYRJ3YXH8LO/+wHzHLvgRfN0ELJn8o5ott7A1NkEgy+xYpFxgLRimh wRgPomYBroV86Ts+3T4BfIjlFyUGB0LlV7Y0zcsp/eHYhn4CSHl8NdO1c34SL/V9UnuD rABhm0oZeMGM5n3Dnleix5MHUHExPkviZiB6nuSiB1rEunsidIsjOmign1YzbqEqlkmb gg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3kv8ykthfs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:01 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 2AHKdRa6010850; Thu, 17 Nov 2022 21:15:00 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3kt1x9nqmp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:00 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2AHLEx7S024557; Thu, 17 Nov 2022 21:15:00 GMT Received: from sid-dell.us.oracle.com (dhcp-10-132-95-73.usdhcp.oraclecorp.com [10.132.95.73]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3kt1x9nqku-2; Thu, 17 Nov 2022 21:14:59 +0000 From: Sidhartha Kumar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: akpm@linux-foundation.org, songmuchun@bytedance.com, mike.kravetz@oracle.com, willy@infradead.org, almasrymina@google.com, linmiaohe@huawei.com, hughd@google.com, Sidhartha Kumar Subject: [PATCH mm-unstable v3 01/10] mm: add folio dtor and order setter functions Date: Thu, 17 Nov 2022 13:14:52 -0800 Message-Id: <20221117211501.17150-2-sidhartha.kumar@oracle.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221117211501.17150-1-sidhartha.kumar@oracle.com> References: <20221117211501.17150-1-sidhartha.kumar@oracle.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-17_06,2022-11-17_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2211170152 X-Proofpoint-GUID: m8uAdXpCu6nNc0-6aFGF4EOwAkQRCXfw X-Proofpoint-ORIG-GUID: m8uAdXpCu6nNc0-6aFGF4EOwAkQRCXfw Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Add folio equivalents for set_compound_order() and set_compound_page_dtor(). Also remove extra new-lines introduced by mm/hugetlb: convert move_hugetlb_state() to folios and mm/hugetlb_cgroup: convert hugetlb_cgroup_uncharge_page() to folios. Suggested-by: Mike Kravetz Suggested-by: Muchun Song Signed-off-by: Sidhartha Kumar --- include/linux/mm.h | 16 ++++++++++++++++ mm/hugetlb.c | 4 +--- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a48c5ad16a5e..2bdef8a5298a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -972,6 +972,13 @@ static inline void set_compound_page_dtor(struct page = *page, page[1].compound_dtor =3D compound_dtor; } =20 +static inline void folio_set_compound_dtor(struct folio *folio, + enum compound_dtor_id compound_dtor) +{ + VM_BUG_ON_FOLIO(compound_dtor >=3D NR_COMPOUND_DTORS, folio); + folio->_folio_dtor =3D compound_dtor; +} + void destroy_large_folio(struct folio *folio); =20 static inline int head_compound_pincount(struct page *head) @@ -987,6 +994,15 @@ static inline void set_compound_order(struct page *pag= e, unsigned int order) #endif } =20 +static inline void folio_set_compound_order(struct folio *folio, + unsigned int order) +{ + folio->_folio_order =3D order; +#ifdef CONFIG_64BIT + folio->_folio_nr_pages =3D order ? 1U << order : 0; +#endif +} + /* Returns the number of pages in this potentially compound page. */ static inline unsigned long compound_nr(struct page *page) { diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 002337cc27b5..157f2392c64f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1780,7 +1780,7 @@ static void __prep_new_hugetlb_folio(struct hstate *h= , struct folio *folio) { hugetlb_vmemmap_optimize(h, &folio->page); INIT_LIST_HEAD(&folio->lru); - folio->_folio_dtor =3D HUGETLB_PAGE_DTOR; + folio_set_compound_dtor(folio, HUGETLB_PAGE_DTOR); hugetlb_set_folio_subpool(folio, NULL); set_hugetlb_cgroup(folio, NULL); set_hugetlb_cgroup_rsvd(folio, NULL); @@ -2936,7 +2936,6 @@ struct page *alloc_huge_page(struct vm_area_struct *v= ma, * a reservation exists for the allocation. */ page =3D dequeue_huge_page_vma(h, vma, addr, avoid_reserve, gbl_chg); - if (!page) { spin_unlock_irq(&hugetlb_lock); page =3D alloc_buddy_huge_page_with_mpol(h, vma, addr); @@ -7349,7 +7348,6 @@ void move_hugetlb_state(struct folio *old_folio, stru= ct folio *new_folio, int re int old_nid =3D folio_nid(old_folio); int new_nid =3D folio_nid(new_folio); =20 - folio_set_hugetlb_temporary(old_folio); folio_clear_hugetlb_temporary(new_folio); =20 --=20 2.38.1 From nobody Wed Apr 15 19:23:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A280C43217 for ; Thu, 17 Nov 2022 21:15:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240624AbiKQVPz (ORCPT ); Thu, 17 Nov 2022 16:15:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240749AbiKQVP0 (ORCPT ); Thu, 17 Nov 2022 16:15:26 -0500 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5D20A65869 for ; Thu, 17 Nov 2022 13:15:13 -0800 (PST) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AHLE1Bx027418; Thu, 17 Nov 2022 21:15:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=Mwgxn09UOrTmFlxHKtjfX4954mioyEjow3Fh9kxIaBI=; b=2KWYgdPbiF5MtR/yKAQ3vqhVXP/ve/siJog2tUxdrICWKN0IQbG5YK9Wp9NgpVF/aPBp wdw7LnI7DY58EPUrFl0bmNF+P8FLYkGM64wPkGeRN7pphUBiBdiYeEVAv5D3QjS24QhA CBICDGh4XwcG3v3ufdv5iJcJYgN55XWdgwqmxNk88aZoM8sDFdjYvR7uRhhvj3iHerqX CqnK2z5P3Zz118UGeiWBnXNaVcVQxWcUkfoCBF7XUV26v/qVstCulXT3jBbNG65UsZ0J 1HQl0dbLkcR1gMrlSVrU9QVB2Jc7hexSs0XjISFE4f0vffO7CiSrA0cN0Yf7dw1i2eK2 aw== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3kv3jstfae-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:02 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 2AHKjtZp010826; Thu, 17 Nov 2022 21:15:01 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3kt1x9nqn4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:01 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2AHLEx7U024557; Thu, 17 Nov 2022 21:15:00 GMT Received: from sid-dell.us.oracle.com (dhcp-10-132-95-73.usdhcp.oraclecorp.com [10.132.95.73]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3kt1x9nqku-3; Thu, 17 Nov 2022 21:15:00 +0000 From: Sidhartha Kumar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: akpm@linux-foundation.org, songmuchun@bytedance.com, mike.kravetz@oracle.com, willy@infradead.org, almasrymina@google.com, linmiaohe@huawei.com, hughd@google.com, Sidhartha Kumar Subject: [PATCH mm-unstable v3 02/10] mm/hugetlb: convert destroy_compound_gigantic_page() to folios Date: Thu, 17 Nov 2022 13:14:53 -0800 Message-Id: <20221117211501.17150-3-sidhartha.kumar@oracle.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221117211501.17150-1-sidhartha.kumar@oracle.com> References: <20221117211501.17150-1-sidhartha.kumar@oracle.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-17_06,2022-11-17_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2211170152 X-Proofpoint-ORIG-GUID: k4U_EhAB5bo-KgdNZ2fRnJ-D-P-dCLLL X-Proofpoint-GUID: k4U_EhAB5bo-KgdNZ2fRnJ-D-P-dCLLL Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Convert page operations within __destroy_compound_gigantic_page() to the corresponding folio operations. Signed-off-by: Sidhartha Kumar --- mm/hugetlb.c | 43 +++++++++++++++++++++---------------------- 1 file changed, 21 insertions(+), 22 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 157f2392c64f..5edb81541ede 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1325,43 +1325,40 @@ static int hstate_next_node_to_free(struct hstate *= h, nodemask_t *nodes_allowed) nr_nodes--) =20 /* used to demote non-gigantic_huge pages as well */ -static void __destroy_compound_gigantic_page(struct page *page, +static void __destroy_compound_gigantic_folio(struct folio *folio, unsigned int order, bool demote) { int i; int nr_pages =3D 1 << order; struct page *p; =20 - atomic_set(compound_mapcount_ptr(page), 0); - atomic_set(subpages_mapcount_ptr(page), 0); - atomic_set(compound_pincount_ptr(page), 0); + atomic_set(folio_mapcount_ptr(folio), 0); + atomic_set(folio_subpages_mapcount_ptr(folio), 0); + atomic_set(folio_pincount_ptr(folio), 0); =20 for (i =3D 1; i < nr_pages; i++) { - p =3D nth_page(page, i); + p =3D folio_page(folio, i); p->mapping =3D NULL; clear_compound_head(p); if (!demote) set_page_refcounted(p); } =20 - set_compound_order(page, 0); -#ifdef CONFIG_64BIT - page[1].compound_nr =3D 0; -#endif - __ClearPageHead(page); + folio_set_compound_order(folio, 0); + folio_clear_head(folio); } =20 -static void destroy_compound_hugetlb_page_for_demote(struct page *page, +static void destroy_compound_hugetlb_folio_for_demote(struct folio *folio, unsigned int order) { - __destroy_compound_gigantic_page(page, order, true); + __destroy_compound_gigantic_folio(folio, order, true); } =20 #ifdef CONFIG_ARCH_HAS_GIGANTIC_PAGE -static void destroy_compound_gigantic_page(struct page *page, +static void destroy_compound_gigantic_folio(struct folio *folio, unsigned int order) { - __destroy_compound_gigantic_page(page, order, false); + __destroy_compound_gigantic_folio(folio, order, false); } =20 static void free_gigantic_page(struct page *page, unsigned int order) @@ -1430,7 +1427,7 @@ static struct page *alloc_gigantic_page(struct hstate= *h, gfp_t gfp_mask, return NULL; } static inline void free_gigantic_page(struct page *page, unsigned int orde= r) { } -static inline void destroy_compound_gigantic_page(struct page *page, +static inline void destroy_compound_gigantic_folio(struct folio *folio, unsigned int order) { } #endif =20 @@ -1477,8 +1474,8 @@ static void __remove_hugetlb_page(struct hstate *h, s= truct page *page, * * For gigantic pages set the destructor to the null dtor. This * destructor will never be called. Before freeing the gigantic - * page destroy_compound_gigantic_page will turn the compound page - * into a simple group of pages. After this the destructor does not + * page destroy_compound_gigantic_folio will turn the folio into a + * simple group of pages. After this the destructor does not * apply. * * This handles the case where more than one ref is held when and @@ -1559,6 +1556,7 @@ static void add_hugetlb_page(struct hstate *h, struct= page *page, static void __update_and_free_page(struct hstate *h, struct page *page) { int i; + struct folio *folio =3D page_folio(page); struct page *subpage; =20 if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) @@ -1587,8 +1585,8 @@ static void __update_and_free_page(struct hstate *h, = struct page *page) * Move PageHWPoison flag from head page to the raw error pages, * which makes any healthy subpages reusable. */ - if (unlikely(PageHWPoison(page))) - hugetlb_clear_page_hwpoison(page); + if (unlikely(folio_test_hwpoison(folio))) + hugetlb_clear_page_hwpoison(&folio->page); =20 for (i =3D 0; i < pages_per_huge_page(h); i++) { subpage =3D nth_page(page, i); @@ -1604,7 +1602,7 @@ static void __update_and_free_page(struct hstate *h, = struct page *page) */ if (hstate_is_gigantic(h) || hugetlb_cma_page(page, huge_page_order(h))) { - destroy_compound_gigantic_page(page, huge_page_order(h)); + destroy_compound_gigantic_folio(folio, huge_page_order(h)); free_gigantic_page(page, huge_page_order(h)); } else { __free_pages(page, huge_page_order(h)); @@ -3435,6 +3433,7 @@ static int demote_free_huge_page(struct hstate *h, st= ruct page *page) { int i, nid =3D page_to_nid(page); struct hstate *target_hstate; + struct folio *folio =3D page_folio(page); struct page *subpage; int rc =3D 0; =20 @@ -3453,10 +3452,10 @@ static int demote_free_huge_page(struct hstate *h, = struct page *page) } =20 /* - * Use destroy_compound_hugetlb_page_for_demote for all huge page + * Use destroy_compound_hugetlb_folio_for_demote for all huge page * sizes as it will not ref count pages. */ - destroy_compound_hugetlb_page_for_demote(page, huge_page_order(h)); + destroy_compound_hugetlb_folio_for_demote(folio, huge_page_order(h)); =20 /* * Taking target hstate mutex synchronizes with set_max_huge_pages. --=20 2.38.1 From nobody Wed Apr 15 19:23:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ABD44C433FE for ; Thu, 17 Nov 2022 21:16:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240841AbiKQVP5 (ORCPT ); Thu, 17 Nov 2022 16:15:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240754AbiKQVP1 (ORCPT ); Thu, 17 Nov 2022 16:15:27 -0500 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AFEFA8432B for ; Thu, 17 Nov 2022 13:15:14 -0800 (PST) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AHLEGi2000828; Thu, 17 Nov 2022 21:15:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=THT0/VFA6cUN/tQQrWJEQ7HX1532qlc3yOTzGPja16Y=; b=nYejCUa2hGvXjWw7PuLQKlQ+J+btYMrMHrzN6pLpEYuqK3gOpLDt8Z3af4VNc0psDfbd HHKKlnHmt64vymMOcRP95zUJMY+jK86ZUwz9/+Gvx+ugFUmqoFt7rFVYtdnYNoHsa/Db j0D5VY0vxwUdSCh5PIZAE+8AvHhg0DgmQ1y1jUWv1rBND4Ln9i8IaKiptLwnkz3LUn+r 2SvPUfh+qY64Zi/gRj4mwcxlgyI6j3oGvqB3HCigB1OSpYh6h89vKFNnUQV0iXb3o+JN YcJJ3u2hDhOADkWwizkK2w1jNjUHA9LyBDK3PytJk5aNP4BJllN1OJG99zBL+wAQQ+ns /Q== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3kv8ykthfw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:02 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 2AHKaQqD010773; Thu, 17 Nov 2022 21:15:02 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3kt1x9nqnq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:01 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2AHLEx7W024557; Thu, 17 Nov 2022 21:15:01 GMT Received: from sid-dell.us.oracle.com (dhcp-10-132-95-73.usdhcp.oraclecorp.com [10.132.95.73]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3kt1x9nqku-4; Thu, 17 Nov 2022 21:15:01 +0000 From: Sidhartha Kumar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: akpm@linux-foundation.org, songmuchun@bytedance.com, mike.kravetz@oracle.com, willy@infradead.org, almasrymina@google.com, linmiaohe@huawei.com, hughd@google.com, Sidhartha Kumar Subject: [PATCH mm-unstable v3 03/10] mm/hugetlb: convert dissolve_free_huge_page() to folios Date: Thu, 17 Nov 2022 13:14:54 -0800 Message-Id: <20221117211501.17150-4-sidhartha.kumar@oracle.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221117211501.17150-1-sidhartha.kumar@oracle.com> References: <20221117211501.17150-1-sidhartha.kumar@oracle.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-17_06,2022-11-17_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2211170152 X-Proofpoint-GUID: -PMEYwNm4fFfVTV2DX1WZVa4xzQORiXu X-Proofpoint-ORIG-GUID: -PMEYwNm4fFfVTV2DX1WZVa4xzQORiXu Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Removes compound_head() call by using a folio rather than a head page. Signed-off-by: Sidhartha Kumar --- mm/hugetlb.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 5edb81541ede..ff609efe5497 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2126,21 +2126,21 @@ static struct page *remove_pool_huge_page(struct hs= tate *h, int dissolve_free_huge_page(struct page *page) { int rc =3D -EBUSY; + struct folio *folio =3D page_folio(page); =20 retry: /* Not to disrupt normal path by vainly holding hugetlb_lock */ - if (!PageHuge(page)) + if (!folio_test_hugetlb(folio)) return 0; =20 spin_lock_irq(&hugetlb_lock); - if (!PageHuge(page)) { + if (!folio_test_hugetlb(folio)) { rc =3D 0; goto out; } =20 - if (!page_count(page)) { - struct page *head =3D compound_head(page); - struct hstate *h =3D page_hstate(head); + if (!folio_ref_count(folio)) { + struct hstate *h =3D folio_hstate(folio); if (!available_huge_pages(h)) goto out; =20 @@ -2148,7 +2148,7 @@ int dissolve_free_huge_page(struct page *page) * We should make sure that the page is already on the free list * when it is dissolved. */ - if (unlikely(!HPageFreed(head))) { + if (unlikely(!folio_test_hugetlb_freed(folio))) { spin_unlock_irq(&hugetlb_lock); cond_resched(); =20 @@ -2163,7 +2163,7 @@ int dissolve_free_huge_page(struct page *page) goto retry; } =20 - remove_hugetlb_page(h, head, false); + remove_hugetlb_page(h, &folio->page, false); h->max_huge_pages--; spin_unlock_irq(&hugetlb_lock); =20 @@ -2175,12 +2175,12 @@ int dissolve_free_huge_page(struct page *page) * Attempt to allocate vmemmmap here so that we can take * appropriate action on failure. */ - rc =3D hugetlb_vmemmap_restore(h, head); + rc =3D hugetlb_vmemmap_restore(h, &folio->page); if (!rc) { - update_and_free_page(h, head, false); + update_and_free_page(h, &folio->page, false); } else { spin_lock_irq(&hugetlb_lock); - add_hugetlb_page(h, head, false); + add_hugetlb_page(h, &folio->page, false); h->max_huge_pages++; spin_unlock_irq(&hugetlb_lock); } --=20 2.38.1 From nobody Wed Apr 15 19:23:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDF3DC433FE for ; Thu, 17 Nov 2022 21:16:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240847AbiKQVQB (ORCPT ); Thu, 17 Nov 2022 16:16:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240769AbiKQVP1 (ORCPT ); Thu, 17 Nov 2022 16:15:27 -0500 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B008C8432E for ; Thu, 17 Nov 2022 13:15:14 -0800 (PST) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AHLEGi4000828; Thu, 17 Nov 2022 21:15:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=mZHrU3BK7yqZ4Vo8Pufvz4V3D37bYSkveHg4q16NVVk=; b=2q6Lf1wtSEmE0fzM33CiX1J04qs6PeVqDv65/kU5yruoTVKxBxZs18NiukcrbI1tv6mr ravhqkmzcxbr/35IuXafDLBzVpxL0WV7LxoZLUuzy4LZMXpaJdz4v89PKOl5GyURCu5Q Q+a0OLotICOVvG+QK9mOLWb2E3Eim+EFq3Cx+Qi//T9dS+C9s2mswyXO5iLYv8bu6kUH iSPgDqjNa1qJqciGGETw62Jgoy4IJk9Xw4IB4NxnqKZAbJvz1z0JVpIj1eH6aDdFa/nk XbjOnTZTIGG1cZf9tqNlEUzdDyt4oWHzgqG12F3qRHU1beRzzGwtVzLgoQrbrJSOZomI 8Q== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3kv8ykthfy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:03 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 2AHKfT3T010805; Thu, 17 Nov 2022 21:15:02 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3kt1x9nqpd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:02 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2AHLEx7Y024557; Thu, 17 Nov 2022 21:15:02 GMT Received: from sid-dell.us.oracle.com (dhcp-10-132-95-73.usdhcp.oraclecorp.com [10.132.95.73]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3kt1x9nqku-5; Thu, 17 Nov 2022 21:15:02 +0000 From: Sidhartha Kumar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: akpm@linux-foundation.org, songmuchun@bytedance.com, mike.kravetz@oracle.com, willy@infradead.org, almasrymina@google.com, linmiaohe@huawei.com, hughd@google.com, Sidhartha Kumar Subject: [PATCH mm-unstable v3 04/10] mm/hugetlb: convert remove_hugetlb_page() to folios Date: Thu, 17 Nov 2022 13:14:55 -0800 Message-Id: <20221117211501.17150-5-sidhartha.kumar@oracle.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221117211501.17150-1-sidhartha.kumar@oracle.com> References: <20221117211501.17150-1-sidhartha.kumar@oracle.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-17_06,2022-11-17_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2211170152 X-Proofpoint-GUID: ixSxojTkZqKmQYaFlaJxm05HmpVNY5po X-Proofpoint-ORIG-GUID: ixSxojTkZqKmQYaFlaJxm05HmpVNY5po Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Removes page_folio() call by converting callers to directly pass a folio into __remove_hugetlb_page(). Signed-off-by: Sidhartha Kumar --- mm/hugetlb.c | 48 +++++++++++++++++++++++++----------------------- 1 file changed, 25 insertions(+), 23 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index ff609efe5497..d9604c0dac54 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1432,19 +1432,18 @@ static inline void destroy_compound_gigantic_folio(= struct folio *folio, #endif =20 /* - * Remove hugetlb page from lists, and update dtor so that page appears + * Remove hugetlb folio from lists, and update dtor so that the folio appe= ars * as just a compound page. * - * A reference is held on the page, except in the case of demote. + * A reference is held on the folio, except in the case of demote. * * Must be called with hugetlb lock held. */ -static void __remove_hugetlb_page(struct hstate *h, struct page *page, +static void __remove_hugetlb_folio(struct hstate *h, struct folio *folio, bool adjust_surplus, bool demote) { - int nid =3D page_to_nid(page); - struct folio *folio =3D page_folio(page); + int nid =3D folio_nid(folio); =20 VM_BUG_ON_FOLIO(hugetlb_cgroup_from_folio(folio), folio); VM_BUG_ON_FOLIO(hugetlb_cgroup_from_folio_rsvd(folio), folio); @@ -1453,9 +1452,9 @@ static void __remove_hugetlb_page(struct hstate *h, s= truct page *page, if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) return; =20 - list_del(&page->lru); + list_del(&folio->lru); =20 - if (HPageFreed(page)) { + if (folio_test_hugetlb_freed(folio)) { h->free_huge_pages--; h->free_huge_pages_node[nid]--; } @@ -1485,26 +1484,26 @@ static void __remove_hugetlb_page(struct hstate *h,= struct page *page, * be turned into a page of smaller size. */ if (!demote) - set_page_refcounted(page); + folio_ref_unfreeze(folio, 1); if (hstate_is_gigantic(h)) - set_compound_page_dtor(page, NULL_COMPOUND_DTOR); + folio_set_compound_dtor(folio, NULL_COMPOUND_DTOR); else - set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); + folio_set_compound_dtor(folio, COMPOUND_PAGE_DTOR); =20 h->nr_huge_pages--; h->nr_huge_pages_node[nid]--; } =20 -static void remove_hugetlb_page(struct hstate *h, struct page *page, +static void remove_hugetlb_folio(struct hstate *h, struct folio *folio, bool adjust_surplus) { - __remove_hugetlb_page(h, page, adjust_surplus, false); + __remove_hugetlb_folio(h, folio, adjust_surplus, false); } =20 -static void remove_hugetlb_page_for_demote(struct hstate *h, struct page *= page, +static void remove_hugetlb_folio_for_demote(struct hstate *h, struct folio= *folio, bool adjust_surplus) { - __remove_hugetlb_page(h, page, adjust_surplus, true); + __remove_hugetlb_folio(h, folio, adjust_surplus, true); } =20 static void add_hugetlb_page(struct hstate *h, struct page *page, @@ -1639,8 +1638,9 @@ static void free_hpage_workfn(struct work_struct *wor= k) /* * The VM_BUG_ON_PAGE(!PageHuge(page), page) in page_hstate() * is going to trigger because a previous call to - * remove_hugetlb_page() will set_compound_page_dtor(page, - * NULL_COMPOUND_DTOR), so do not use page_hstate() directly. + * remove_hugetlb_folio() will call folio_set_compound_dtor + * (folio, NULL_COMPOUND_DTOR), so do not use page_hstate() + * directly. */ h =3D size_to_hstate(page_size(page)); =20 @@ -1749,12 +1749,12 @@ void free_huge_page(struct page *page) h->resv_huge_pages++; =20 if (folio_test_hugetlb_temporary(folio)) { - remove_hugetlb_page(h, page, false); + remove_hugetlb_folio(h, folio, false); spin_unlock_irqrestore(&hugetlb_lock, flags); update_and_free_page(h, page, true); } else if (h->surplus_huge_pages_node[nid]) { /* remove the page from active list */ - remove_hugetlb_page(h, page, true); + remove_hugetlb_folio(h, folio, true); spin_unlock_irqrestore(&hugetlb_lock, flags); update_and_free_page(h, page, true); } else { @@ -2090,6 +2090,7 @@ static struct page *remove_pool_huge_page(struct hsta= te *h, { int nr_nodes, node; struct page *page =3D NULL; + struct folio *folio; =20 lockdep_assert_held(&hugetlb_lock); for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) { @@ -2101,7 +2102,8 @@ static struct page *remove_pool_huge_page(struct hsta= te *h, !list_empty(&h->hugepage_freelists[node])) { page =3D list_entry(h->hugepage_freelists[node].next, struct page, lru); - remove_hugetlb_page(h, page, acct_surplus); + folio =3D page_folio(page); + remove_hugetlb_folio(h, folio, acct_surplus); break; } } @@ -2163,7 +2165,7 @@ int dissolve_free_huge_page(struct page *page) goto retry; } =20 - remove_hugetlb_page(h, &folio->page, false); + remove_hugetlb_folio(h, folio, false); h->max_huge_pages--; spin_unlock_irq(&hugetlb_lock); =20 @@ -2801,7 +2803,7 @@ static int alloc_and_dissolve_huge_page(struct hstate= *h, struct page *old_page, * and enqueue_huge_page() for new_page. The counters will remain * stable since this happens under the lock. */ - remove_hugetlb_page(h, old_page, false); + remove_hugetlb_folio(h, old_folio, false); =20 /* * Ref count on new page is already zero as it was dropped @@ -3228,7 +3230,7 @@ static void try_to_free_low(struct hstate *h, unsigne= d long count, goto out; if (PageHighMem(page)) continue; - remove_hugetlb_page(h, page, false); + remove_hugetlb_folio(h, page_folio(page), false); list_add(&page->lru, &page_list); } } @@ -3439,7 +3441,7 @@ static int demote_free_huge_page(struct hstate *h, st= ruct page *page) =20 target_hstate =3D size_to_hstate(PAGE_SIZE << h->demote_order); =20 - remove_hugetlb_page_for_demote(h, page, false); + remove_hugetlb_folio_for_demote(h, folio, false); spin_unlock_irq(&hugetlb_lock); =20 rc =3D hugetlb_vmemmap_restore(h, page); --=20 2.38.1 From nobody Wed Apr 15 19:23:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EEA1C433FE for ; Thu, 17 Nov 2022 21:16:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240612AbiKQVQO (ORCPT ); Thu, 17 Nov 2022 16:16:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32982 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240785AbiKQVP2 (ORCPT ); Thu, 17 Nov 2022 16:15:28 -0500 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 402B8725D7 for ; Thu, 17 Nov 2022 13:15:15 -0800 (PST) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AHLEGdV000786; Thu, 17 Nov 2022 21:15:05 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=cqECmlw3uurUHKox1I6ZdisuXQoCt6s1cy/6t7wynMs=; b=qL30yY0jr6ljOLZCv8DWjFmNovpZ9cosgqnYBZK/BLygReSyOM7YNkJlVC9YScfaPZz7 2KcowcjTaereDgpfgTLlt8EmNh7okZh6igBMftOOVkszADxTTSCpzduB4CrCxwyh8jZ4 6gwXI9qvX/6zmKn3AbZ22z11txssGfrVudvVweFGGWuUePpc3Z30tf9t11VQly916CwR ul/sifON/HulEnUpJmitg4PiJR32sJHpGSVgVKL4xIG5LLW5mmc9olCteuYzTlm0B1X5 NbDAuVs7PmqydgoiLtwokE38K21vSqyOsnHC/OXjEVDNNdj3dAT8aSNkQygNz1fwhaeK qQ== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3kv8ykthg3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:05 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 2AHKrqUS010778; Thu, 17 Nov 2022 21:15:03 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3kt1x9nqpx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:03 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2AHLEx7a024557; Thu, 17 Nov 2022 21:15:02 GMT Received: from sid-dell.us.oracle.com (dhcp-10-132-95-73.usdhcp.oraclecorp.com [10.132.95.73]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3kt1x9nqku-6; Thu, 17 Nov 2022 21:15:02 +0000 From: Sidhartha Kumar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: akpm@linux-foundation.org, songmuchun@bytedance.com, mike.kravetz@oracle.com, willy@infradead.org, almasrymina@google.com, linmiaohe@huawei.com, hughd@google.com, Sidhartha Kumar Subject: [PATCH mm-unstable v3 05/10] mm/hugetlb: convert update_and_free_page() to folios Date: Thu, 17 Nov 2022 13:14:56 -0800 Message-Id: <20221117211501.17150-6-sidhartha.kumar@oracle.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221117211501.17150-1-sidhartha.kumar@oracle.com> References: <20221117211501.17150-1-sidhartha.kumar@oracle.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-17_06,2022-11-17_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2211170152 X-Proofpoint-GUID: c7lu4abh0h8pd6OSNEfIR-TS5DzbqwA6 X-Proofpoint-ORIG-GUID: c7lu4abh0h8pd6OSNEfIR-TS5DzbqwA6 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Make more progress on converting the free_huge_page() destructor to operate on folios by converting update_and_free_page() to folios. Signed-off-by: Sidhartha Kumar --- mm/hugetlb.c | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d9604c0dac54..80301fab56d8 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1478,7 +1478,7 @@ static void __remove_hugetlb_folio(struct hstate *h, = struct folio *folio, * apply. * * This handles the case where more than one ref is held when and - * after update_and_free_page is called. + * after update_and_free_hugetlb_folio is called. * * In the case of demote we do not ref count the page as it will soon * be turned into a page of smaller size. @@ -1609,7 +1609,7 @@ static void __update_and_free_page(struct hstate *h, = struct page *page) } =20 /* - * As update_and_free_page() can be called under any context, so we cannot + * As update_and_free_hugetlb_folio() can be called under any context, so = we cannot * use GFP_KERNEL to allocate vmemmap pages. However, we can defer the * actual freeing in a workqueue to prevent from using GFP_ATOMIC to alloc= ate * the vmemmap pages. @@ -1657,11 +1657,11 @@ static inline void flush_free_hpage_work(struct hst= ate *h) flush_work(&free_hpage_work); } =20 -static void update_and_free_page(struct hstate *h, struct page *page, +static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *= folio, bool atomic) { - if (!HPageVmemmapOptimized(page) || !atomic) { - __update_and_free_page(h, page); + if (!folio_test_hugetlb_vmemmap_optimized(folio) || !atomic) { + __update_and_free_page(h, &folio->page); return; } =20 @@ -1672,16 +1672,18 @@ static void update_and_free_page(struct hstate *h, = struct page *page, * empty. Otherwise, schedule_work() had been called but the workfn * hasn't retrieved the list yet. */ - if (llist_add((struct llist_node *)&page->mapping, &hpage_freelist)) + if (llist_add((struct llist_node *)&folio->mapping, &hpage_freelist)) schedule_work(&free_hpage_work); } =20 static void update_and_free_pages_bulk(struct hstate *h, struct list_head = *list) { struct page *page, *t_page; + struct folio *folio; =20 list_for_each_entry_safe(page, t_page, list, lru) { - update_and_free_page(h, page, false); + folio =3D page_folio(page); + update_and_free_hugetlb_folio(h, folio, false); cond_resched(); } } @@ -1751,12 +1753,12 @@ void free_huge_page(struct page *page) if (folio_test_hugetlb_temporary(folio)) { remove_hugetlb_folio(h, folio, false); spin_unlock_irqrestore(&hugetlb_lock, flags); - update_and_free_page(h, page, true); + update_and_free_hugetlb_folio(h, folio, true); } else if (h->surplus_huge_pages_node[nid]) { /* remove the page from active list */ remove_hugetlb_folio(h, folio, true); spin_unlock_irqrestore(&hugetlb_lock, flags); - update_and_free_page(h, page, true); + update_and_free_hugetlb_folio(h, folio, true); } else { arch_clear_hugepage_flags(page); enqueue_huge_page(h, page); @@ -2170,8 +2172,8 @@ int dissolve_free_huge_page(struct page *page) spin_unlock_irq(&hugetlb_lock); =20 /* - * Normally update_and_free_page will allocate required vmemmmap - * before freeing the page. update_and_free_page will fail to + * Normally update_and_free_hugtlb_folio will allocate required vmemmmap + * before freeing the page. update_and_free_hugtlb_folio will fail to * free the page if it can not allocate required vmemmap. We * need to adjust max_huge_pages if the page is not freed. * Attempt to allocate vmemmmap here so that we can take @@ -2179,7 +2181,7 @@ int dissolve_free_huge_page(struct page *page) */ rc =3D hugetlb_vmemmap_restore(h, &folio->page); if (!rc) { - update_and_free_page(h, &folio->page, false); + update_and_free_hugetlb_folio(h, folio, false); } else { spin_lock_irq(&hugetlb_lock); add_hugetlb_page(h, &folio->page, false); @@ -2816,7 +2818,7 @@ static int alloc_and_dissolve_huge_page(struct hstate= *h, struct page *old_page, * Pages have been replaced, we can safely free the old one. */ spin_unlock_irq(&hugetlb_lock); - update_and_free_page(h, old_page, false); + update_and_free_hugetlb_folio(h, old_folio, false); } =20 return ret; @@ -2825,7 +2827,7 @@ static int alloc_and_dissolve_huge_page(struct hstate= *h, struct page *old_page, spin_unlock_irq(&hugetlb_lock); /* Page has a zero ref count, but needs a ref to be freed */ folio_ref_unfreeze(new_folio, 1); - update_and_free_page(h, new_page, false); + update_and_free_hugetlb_folio(h, new_folio, false); =20 return ret; } --=20 2.38.1 From nobody Wed Apr 15 19:23:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58601C4332F for ; Thu, 17 Nov 2022 21:16:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240849AbiKQVQH (ORCPT ); Thu, 17 Nov 2022 16:16:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32948 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240779AbiKQVP1 (ORCPT ); Thu, 17 Nov 2022 16:15:27 -0500 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2078884335 for ; Thu, 17 Nov 2022 13:15:14 -0800 (PST) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AHLE1C0027418; Thu, 17 Nov 2022 21:15:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=0qWyxReQlxSizjSVKyD4aM3NoT7R0refv45YW4nsk6s=; b=N2QYyLOpbtBsCVAKGcAkRb2UOdDcw8rMtQOScRJnPLB/lMfveZgtrNoMpLYVHHwoGv7k NXJHtzAYZyNluJp3hnAq/yy8voikYsGf0BKPTciFLJuabeMyFzG48kL15cL6V/Gs8JfS Pw4zCuHb9WBNxhAr8YgJ8ZaXJ/IRPAxeRpe2RFAHMJ6ZM6Rw1iASvnaDPBmo58o60HEd vgJx2DmrVrO5a7RPB+l/+OTVgKzk6jywobciKEhHzDU9zHzEvZkKGpVHNNYdkKHdqL4m RcSZwOpVOh4rkB9K3VoMYsBJiXQjNWDqeaRQ7c9Mk89zFaIMymlMnMJT/DzftE0iuPpo MQ== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3kv3jstfam-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:04 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 2AHKdRa8010850; Thu, 17 Nov 2022 21:15:04 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3kt1x9nqqg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:03 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2AHLEx7c024557; Thu, 17 Nov 2022 21:15:03 GMT Received: from sid-dell.us.oracle.com (dhcp-10-132-95-73.usdhcp.oraclecorp.com [10.132.95.73]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3kt1x9nqku-7; Thu, 17 Nov 2022 21:15:03 +0000 From: Sidhartha Kumar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: akpm@linux-foundation.org, songmuchun@bytedance.com, mike.kravetz@oracle.com, willy@infradead.org, almasrymina@google.com, linmiaohe@huawei.com, hughd@google.com, Sidhartha Kumar Subject: [PATCH mm-unstable v3 06/10] mm/hugetlb: convert add_hugetlb_page() to folios and add hugetlb_cma_folio() Date: Thu, 17 Nov 2022 13:14:57 -0800 Message-Id: <20221117211501.17150-7-sidhartha.kumar@oracle.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221117211501.17150-1-sidhartha.kumar@oracle.com> References: <20221117211501.17150-1-sidhartha.kumar@oracle.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-17_06,2022-11-17_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2211170152 X-Proofpoint-ORIG-GUID: 9Z2UtI9eYvBe4UgxK_xjpndkp5YKbKbz X-Proofpoint-GUID: 9Z2UtI9eYvBe4UgxK_xjpndkp5YKbKbz Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Convert add_hugetlb_page() to take in a folio, also convert hugetlb_cma_page() to take in a folio. Signed-off-by: Sidhartha Kumar --- mm/hugetlb.c | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 80301fab56d8..bf36aa8e6072 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -54,13 +54,13 @@ struct hstate hstates[HUGE_MAX_HSTATE]; #ifdef CONFIG_CMA static struct cma *hugetlb_cma[MAX_NUMNODES]; static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata; -static bool hugetlb_cma_page(struct page *page, unsigned int order) +static bool hugetlb_cma_folio(struct folio *folio, unsigned int order) { - return cma_pages_valid(hugetlb_cma[page_to_nid(page)], page, + return cma_pages_valid(hugetlb_cma[folio_nid(folio)], &folio->page, 1 << order); } #else -static bool hugetlb_cma_page(struct page *page, unsigned int order) +static bool hugetlb_cma_folio(struct folio *folio, unsigned int order) { return false; } @@ -1506,17 +1506,17 @@ static void remove_hugetlb_folio_for_demote(struct = hstate *h, struct folio *foli __remove_hugetlb_folio(h, folio, adjust_surplus, true); } =20 -static void add_hugetlb_page(struct hstate *h, struct page *page, +static void add_hugetlb_folio(struct hstate *h, struct folio *folio, bool adjust_surplus) { int zeroed; - int nid =3D page_to_nid(page); + int nid =3D folio_nid(folio); =20 - VM_BUG_ON_PAGE(!HPageVmemmapOptimized(page), page); + VM_BUG_ON_FOLIO(!folio_test_hugetlb_vmemmap_optimized(folio), folio); =20 lockdep_assert_held(&hugetlb_lock); =20 - INIT_LIST_HEAD(&page->lru); + INIT_LIST_HEAD(&folio->lru); h->nr_huge_pages++; h->nr_huge_pages_node[nid]++; =20 @@ -1525,21 +1525,21 @@ static void add_hugetlb_page(struct hstate *h, stru= ct page *page, h->surplus_huge_pages_node[nid]++; } =20 - set_compound_page_dtor(page, HUGETLB_PAGE_DTOR); - set_page_private(page, 0); + folio_set_compound_dtor(folio, HUGETLB_PAGE_DTOR); + folio_change_private(folio, 0); /* * We have to set HPageVmemmapOptimized again as above - * set_page_private(page, 0) cleared it. + * folio_change_private(folio, 0) cleared it. */ - SetHPageVmemmapOptimized(page); + folio_set_hugetlb_vmemmap_optimized(folio); =20 /* - * This page is about to be managed by the hugetlb allocator and + * This folio is about to be managed by the hugetlb allocator and * should have no users. Drop our reference, and check for others * just in case. */ - zeroed =3D put_page_testzero(page); - if (!zeroed) + zeroed =3D folio_put_testzero(folio); + if (unlikely(!zeroed)) /* * It is VERY unlikely soneone else has taken a ref on * the page. In this case, we simply return as the @@ -1548,8 +1548,8 @@ static void add_hugetlb_page(struct hstate *h, struct= page *page, */ return; =20 - arch_clear_hugepage_flags(page); - enqueue_huge_page(h, page); + arch_clear_hugepage_flags(&folio->page); + enqueue_huge_page(h, &folio->page); } =20 static void __update_and_free_page(struct hstate *h, struct page *page) @@ -1575,7 +1575,7 @@ static void __update_and_free_page(struct hstate *h, = struct page *page) * page and put the page back on the hugetlb free list and treat * as a surplus page. */ - add_hugetlb_page(h, page, true); + add_hugetlb_folio(h, page_folio(page), true); spin_unlock_irq(&hugetlb_lock); return; } @@ -1600,7 +1600,7 @@ static void __update_and_free_page(struct hstate *h, = struct page *page) * need to be given back to CMA in free_gigantic_page. */ if (hstate_is_gigantic(h) || - hugetlb_cma_page(page, huge_page_order(h))) { + hugetlb_cma_folio(folio, huge_page_order(h))) { destroy_compound_gigantic_folio(folio, huge_page_order(h)); free_gigantic_page(page, huge_page_order(h)); } else { @@ -2184,7 +2184,7 @@ int dissolve_free_huge_page(struct page *page) update_and_free_hugetlb_folio(h, folio, false); } else { spin_lock_irq(&hugetlb_lock); - add_hugetlb_page(h, &folio->page, false); + add_hugetlb_folio(h, folio, false); h->max_huge_pages++; spin_unlock_irq(&hugetlb_lock); } @@ -3451,7 +3451,7 @@ static int demote_free_huge_page(struct hstate *h, st= ruct page *page) /* Allocation of vmemmmap failed, we can not demote page */ spin_lock_irq(&hugetlb_lock); set_page_refcounted(page); - add_hugetlb_page(h, page, false); + add_hugetlb_folio(h, page_folio(page), false); return rc; } =20 --=20 2.38.1 From nobody Wed Apr 15 19:23:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2EC14C43217 for ; Thu, 17 Nov 2022 21:16:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239449AbiKQVQZ (ORCPT ); Thu, 17 Nov 2022 16:16:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240787AbiKQVPm (ORCPT ); Thu, 17 Nov 2022 16:15:42 -0500 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF54D65845 for ; Thu, 17 Nov 2022 13:15:16 -0800 (PST) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AHLEGi5000828; Thu, 17 Nov 2022 21:15:05 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=DrYH6pRWKnBgFEybzVkbkIij9mLZr9efXDcEx32Vsb4=; b=Leq38lRgCQU5qt7/Ysz+q9zI6Qm3VvJkpXz/Mb5i+JrtVd4ny7qEIcgrC9gOg7ssEp0I wr0/0sh0IJp7rYduA8cxFMZCqK/vtfmBuPveWzNVLyhUuRam7gmFjxcAiyEIMNSL4GX5 ySQrUx/9ldTdx4Zi0wj78sMhe50cJwImpv8kc9w0VpM03BwLhTtHaYPTJI+wODNtMEIN uUWmFXEG126tA+OSAC3wmocUq4/+YTLvuwwjciSpY4v3MdSNNABtAfEUCXiCmMBisTQA /MdsK6BBCAHeNW8OZJhN0Jf/NNybgjFMBd4WbAtecTIBxDWzWg7mgq3TWsKKIlCjOyxl ew== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3kv8ykthg4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:05 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 2AHKjtZu010826; Thu, 17 Nov 2022 21:15:04 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3kt1x9nqr3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:04 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2AHLEx7e024557; Thu, 17 Nov 2022 21:15:04 GMT Received: from sid-dell.us.oracle.com (dhcp-10-132-95-73.usdhcp.oraclecorp.com [10.132.95.73]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3kt1x9nqku-8; Thu, 17 Nov 2022 21:15:04 +0000 From: Sidhartha Kumar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: akpm@linux-foundation.org, songmuchun@bytedance.com, mike.kravetz@oracle.com, willy@infradead.org, almasrymina@google.com, linmiaohe@huawei.com, hughd@google.com, Sidhartha Kumar Subject: [PATCH mm-unstable v3 07/10] mm/hugetlb: convert enqueue_huge_page() to folios Date: Thu, 17 Nov 2022 13:14:58 -0800 Message-Id: <20221117211501.17150-8-sidhartha.kumar@oracle.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221117211501.17150-1-sidhartha.kumar@oracle.com> References: <20221117211501.17150-1-sidhartha.kumar@oracle.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-17_06,2022-11-17_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2211170152 X-Proofpoint-GUID: E5SbJIJezI92l3ebRPRmBKsVEGxPXbVl X-Proofpoint-ORIG-GUID: E5SbJIJezI92l3ebRPRmBKsVEGxPXbVl Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Convert callers of enqueue_huge_page() to pass in a folio, function is renamed to enqueue_hugetlb_folio(). Signed-off-by: Sidhartha Kumar --- mm/hugetlb.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bf36aa8e6072..4eb6f3d6f46e 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1127,17 +1127,17 @@ static bool vma_has_reserves(struct vm_area_struct = *vma, long chg) return false; } =20 -static void enqueue_huge_page(struct hstate *h, struct page *page) +static void enqueue_hugetlb_folio(struct hstate *h, struct folio *folio) { - int nid =3D page_to_nid(page); + int nid =3D folio_nid(folio); =20 lockdep_assert_held(&hugetlb_lock); - VM_BUG_ON_PAGE(page_count(page), page); + VM_BUG_ON_FOLIO(folio_ref_count(folio), folio); =20 - list_move(&page->lru, &h->hugepage_freelists[nid]); + list_move(&folio->lru, &h->hugepage_freelists[nid]); h->free_huge_pages++; h->free_huge_pages_node[nid]++; - SetHPageFreed(page); + folio_set_hugetlb_freed(folio); } =20 static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) @@ -1549,7 +1549,7 @@ static void add_hugetlb_folio(struct hstate *h, struc= t folio *folio, return; =20 arch_clear_hugepage_flags(&folio->page); - enqueue_huge_page(h, &folio->page); + enqueue_hugetlb_folio(h, folio); } =20 static void __update_and_free_page(struct hstate *h, struct page *page) @@ -1761,7 +1761,7 @@ void free_huge_page(struct page *page) update_and_free_hugetlb_folio(h, folio, true); } else { arch_clear_hugepage_flags(page); - enqueue_huge_page(h, page); + enqueue_hugetlb_folio(h, folio); spin_unlock_irqrestore(&hugetlb_lock, flags); } } @@ -2436,7 +2436,7 @@ static int gather_surplus_pages(struct hstate *h, lon= g delta) if ((--needed) < 0) break; /* Add the page to the hugetlb allocator */ - enqueue_huge_page(h, page); + enqueue_hugetlb_folio(h, page_folio(page)); } free: spin_unlock_irq(&hugetlb_lock); @@ -2802,8 +2802,8 @@ static int alloc_and_dissolve_huge_page(struct hstate= *h, struct page *old_page, * Ok, old_page is still a genuine free hugepage. Remove it from * the freelist and decrease the counters. These will be * incremented again when calling __prep_account_new_huge_page() - * and enqueue_huge_page() for new_page. The counters will remain - * stable since this happens under the lock. + * and enqueue_hugetlb_folio() for new_folio. The counters will + * remain stable since this happens under the lock. */ remove_hugetlb_folio(h, old_folio, false); =20 @@ -2812,7 +2812,7 @@ static int alloc_and_dissolve_huge_page(struct hstate= *h, struct page *old_page, * earlier. It can be directly added to the pool free list. */ __prep_account_new_huge_page(h, nid); - enqueue_huge_page(h, new_page); + enqueue_hugetlb_folio(h, new_folio); =20 /* * Pages have been replaced, we can safely free the old one. --=20 2.38.1 From nobody Wed Apr 15 19:23:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D811C4332F for ; Thu, 17 Nov 2022 21:16:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240743AbiKQVQ2 (ORCPT ); Thu, 17 Nov 2022 16:16:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33416 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240788AbiKQVPm (ORCPT ); Thu, 17 Nov 2022 16:15:42 -0500 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 36B8685EC1 for ; Thu, 17 Nov 2022 13:15:17 -0800 (PST) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AHLE06O027408; Thu, 17 Nov 2022 21:15:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=cwkUHEJT5bDkOzVSGa3m03Vwd59Go/cano1leZnsjBU=; b=LOixh6FFTWbbWfiHL+JpRbJCgELtzq5o0DinqfQsTuTAwzKOzmFgfTOCf/VBOd05x25q v5dRM7M0lSSJZ6ob/thv9rHMmwfg7pR3qP3sUxL6mi9E4Zbjuy0Z92OPnduNFdPxnrsG G5SBoK4fXdyBGHtwVKA/ex+XJmSRcmRxGF1C59y87jrIpS7pyk2KIvGJRVVHIEliKsM4 v8WuSAQWUi94bH827o280py0bQoRNyH3JTnzKaGuVIjxIqa1ZFiClQtJcDB1c9+u/177 7BZgCP9bbVWhRwAgB8KzgChH73iI8gF8ibGu/0aKKfEgZytBiWPEMfBuAHwAK/1sFekn rw== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3kv3jstfap-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:06 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 2AHKZ2h8010810; Thu, 17 Nov 2022 21:15:05 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3kt1x9nqrt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:05 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2AHLEx7g024557; Thu, 17 Nov 2022 21:15:04 GMT Received: from sid-dell.us.oracle.com (dhcp-10-132-95-73.usdhcp.oraclecorp.com [10.132.95.73]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3kt1x9nqku-9; Thu, 17 Nov 2022 21:15:04 +0000 From: Sidhartha Kumar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: akpm@linux-foundation.org, songmuchun@bytedance.com, mike.kravetz@oracle.com, willy@infradead.org, almasrymina@google.com, linmiaohe@huawei.com, hughd@google.com, Sidhartha Kumar Subject: [PATCH mm-unstable v3 08/10] mm/hugetlb: convert free_gigantic_page() to folios Date: Thu, 17 Nov 2022 13:14:59 -0800 Message-Id: <20221117211501.17150-9-sidhartha.kumar@oracle.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221117211501.17150-1-sidhartha.kumar@oracle.com> References: <20221117211501.17150-1-sidhartha.kumar@oracle.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-17_06,2022-11-17_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2211170152 X-Proofpoint-ORIG-GUID: OIZ9hkeoC2gpbmdTxB42yBjbDmUCh8Fj X-Proofpoint-GUID: OIZ9hkeoC2gpbmdTxB42yBjbDmUCh8Fj Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Convert callers of free_gigantic_page() to use folios, function is then renamed to free_gigantic_folio(). Signed-off-by: Sidhartha Kumar --- mm/hugetlb.c | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4eb6f3d6f46e..38c5ca015363 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1361,18 +1361,20 @@ static void destroy_compound_gigantic_folio(struct = folio *folio, __destroy_compound_gigantic_folio(folio, order, false); } =20 -static void free_gigantic_page(struct page *page, unsigned int order) +static void free_gigantic_folio(struct folio *folio, unsigned int order) { /* * If the page isn't allocated using the cma allocator, * cma_release() returns false. */ #ifdef CONFIG_CMA - if (cma_release(hugetlb_cma[page_to_nid(page)], page, 1 << order)) + int nid =3D folio_nid(folio); + + if (cma_release(hugetlb_cma[nid], &folio->page, 1 << order)) return; #endif =20 - free_contig_range(page_to_pfn(page), 1 << order); + free_contig_range(folio_pfn(folio), 1 << order); } =20 #ifdef CONFIG_CONTIG_ALLOC @@ -1426,7 +1428,8 @@ static struct page *alloc_gigantic_page(struct hstate= *h, gfp_t gfp_mask, { return NULL; } -static inline void free_gigantic_page(struct page *page, unsigned int orde= r) { } +static inline void free_gigantic_folio(struct folio *folio, + unsigned int order) { } static inline void destroy_compound_gigantic_folio(struct folio *folio, unsigned int order) { } #endif @@ -1565,7 +1568,7 @@ static void __update_and_free_page(struct hstate *h, = struct page *page) * If we don't know which subpages are hwpoisoned, we can't free * the hugepage, so it's leaked intentionally. */ - if (HPageRawHwpUnreliable(page)) + if (folio_test_hugetlb_raw_hwp_unreliable(folio)) return; =20 if (hugetlb_vmemmap_restore(h, page)) { @@ -1575,7 +1578,7 @@ static void __update_and_free_page(struct hstate *h, = struct page *page) * page and put the page back on the hugetlb free list and treat * as a surplus page. */ - add_hugetlb_folio(h, page_folio(page), true); + add_hugetlb_folio(h, folio, true); spin_unlock_irq(&hugetlb_lock); return; } @@ -1588,7 +1591,7 @@ static void __update_and_free_page(struct hstate *h, = struct page *page) hugetlb_clear_page_hwpoison(&folio->page); =20 for (i =3D 0; i < pages_per_huge_page(h); i++) { - subpage =3D nth_page(page, i); + subpage =3D folio_page(folio, i); subpage->flags &=3D ~(1 << PG_locked | 1 << PG_error | 1 << PG_referenced | 1 << PG_dirty | 1 << PG_active | 1 << PG_private | @@ -1597,12 +1600,12 @@ static void __update_and_free_page(struct hstate *h= , struct page *page) =20 /* * Non-gigantic pages demoted from CMA allocated gigantic pages - * need to be given back to CMA in free_gigantic_page. + * need to be given back to CMA in free_gigantic_folio. */ if (hstate_is_gigantic(h) || hugetlb_cma_folio(folio, huge_page_order(h))) { destroy_compound_gigantic_folio(folio, huge_page_order(h)); - free_gigantic_page(page, huge_page_order(h)); + free_gigantic_folio(folio, huge_page_order(h)); } else { __free_pages(page, huge_page_order(h)); } @@ -2023,6 +2026,7 @@ static struct page *alloc_fresh_huge_page(struct hsta= te *h, nodemask_t *node_alloc_noretry) { struct page *page; + struct folio *folio; bool retry =3D false; =20 retry: @@ -2033,14 +2037,14 @@ static struct page *alloc_fresh_huge_page(struct hs= tate *h, nid, nmask, node_alloc_noretry); if (!page) return NULL; - + folio =3D page_folio(page); if (hstate_is_gigantic(h)) { if (!prep_compound_gigantic_page(page, huge_page_order(h))) { /* * Rare failure to convert pages to compound page. * Free pages and try again - ONCE! */ - free_gigantic_page(page, huge_page_order(h)); + free_gigantic_folio(folio, huge_page_order(h)); if (!retry) { retry =3D true; goto retry; @@ -3048,6 +3052,7 @@ static void __init gather_bootmem_prealloc(void) =20 list_for_each_entry(m, &huge_boot_pages, list) { struct page *page =3D virt_to_page(m); + struct folio *folio =3D page_folio(page); struct hstate *h =3D m->hstate; =20 VM_BUG_ON(!hstate_is_gigantic(h)); @@ -3058,7 +3063,7 @@ static void __init gather_bootmem_prealloc(void) free_huge_page(page); /* add to the hugepage allocator */ } else { /* VERY unlikely inflated ref count on a tail page */ - free_gigantic_page(page, huge_page_order(h)); + free_gigantic_folio(folio, huge_page_order(h)); } =20 /* --=20 2.38.1 From nobody Wed Apr 15 19:23:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C1CAC433FE for ; Thu, 17 Nov 2022 21:16:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240870AbiKQVQW (ORCPT ); Thu, 17 Nov 2022 16:16:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240786AbiKQVPm (ORCPT ); Thu, 17 Nov 2022 16:15:42 -0500 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 276CA85EC0 for ; Thu, 17 Nov 2022 13:15:17 -0800 (PST) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AHLDxoX027384; Thu, 17 Nov 2022 21:15:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=MCK/rXGUTZNQyhTZ+P51qmllsIFD0ccOv0nuQ2VWPKM=; b=lcm/LQgS2wFF2NPu1SgV0jnXhWYVgz5gbkvTwGuLN+FnqwghcTCNyuAJbYUCsVk8J8Jd fqYiK9XaWhhTpYcvIFRX2Hx8Z+EbEQJPFj5VANDJv2R7m6XwudYdcm1W4XNiaOKAoFJR N/2LM2Z8Otow4Ni/5fH0cN7u3eDoQi3wSZgMdYPz3hplWfRZNsDo4ziCSqhgCLxAwm2l 60e/iTx4AGARF1hZcbk36G59QrFEbUZHHbS9NsEcAZRDXXgc/SQMOYKkc5638d1+T4l0 +JOvkwW6TsfYLcgtDN4PFnmBTviRNlCTyThuuaurKsITC9behivCTpDY5TguB50p569X Cw== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3kv3jstfar-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:06 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 2AHKbHtW010865; Thu, 17 Nov 2022 21:15:06 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3kt1x9nqsh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:05 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2AHLEx7i024557; Thu, 17 Nov 2022 21:15:05 GMT Received: from sid-dell.us.oracle.com (dhcp-10-132-95-73.usdhcp.oraclecorp.com [10.132.95.73]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3kt1x9nqku-10; Thu, 17 Nov 2022 21:15:05 +0000 From: Sidhartha Kumar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: akpm@linux-foundation.org, songmuchun@bytedance.com, mike.kravetz@oracle.com, willy@infradead.org, almasrymina@google.com, linmiaohe@huawei.com, hughd@google.com, Sidhartha Kumar Subject: [PATCH mm-unstable v3 09/10] mm/hugetlb: convert hugetlb prep functions to folios Date: Thu, 17 Nov 2022 13:15:00 -0800 Message-Id: <20221117211501.17150-10-sidhartha.kumar@oracle.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221117211501.17150-1-sidhartha.kumar@oracle.com> References: <20221117211501.17150-1-sidhartha.kumar@oracle.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-17_06,2022-11-17_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2211170152 X-Proofpoint-ORIG-GUID: 8XBZ1VtzycdgxzpFFmiyrFJ9rwKXRbxK X-Proofpoint-GUID: 8XBZ1VtzycdgxzpFFmiyrFJ9rwKXRbxK Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Convert prep_new_huge_page() and __prep_compound_gigantic_page() to folios. Signed-off-by: Sidhartha Kumar --- mm/hugetlb.c | 61 +++++++++++++++++++++++++--------------------------- 1 file changed, 29 insertions(+), 32 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 38c5ca015363..6ecf9874c521 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1789,28 +1789,26 @@ static void __prep_new_hugetlb_folio(struct hstate = *h, struct folio *folio) set_hugetlb_cgroup_rsvd(folio, NULL); } =20 -static void prep_new_huge_page(struct hstate *h, struct page *page, int ni= d) +static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, = int nid) { - struct folio *folio =3D page_folio(page); - __prep_new_hugetlb_folio(h, folio); spin_lock_irq(&hugetlb_lock); __prep_account_new_huge_page(h, nid); spin_unlock_irq(&hugetlb_lock); } =20 -static bool __prep_compound_gigantic_page(struct page *page, unsigned int = order, - bool demote) +static bool __prep_compound_gigantic_folio(struct folio *folio, + unsigned int order, bool demote) { int i, j; int nr_pages =3D 1 << order; struct page *p; =20 - /* we rely on prep_new_huge_page to set the destructor */ - set_compound_order(page, order); - __SetPageHead(page); + /* we rely on prep_new_hugetlb_folio to set the destructor */ + folio_set_compound_order(folio, order); + __SetPageHead(&folio->page); for (i =3D 0; i < nr_pages; i++) { - p =3D nth_page(page, i); + p =3D folio_page(folio, i); =20 /* * For gigantic hugepages allocated through bootmem at @@ -1851,43 +1849,41 @@ static bool __prep_compound_gigantic_page(struct pa= ge *page, unsigned int order, VM_BUG_ON_PAGE(page_count(p), p); } if (i !=3D 0) - set_compound_head(p, page); + set_compound_head(p, &folio->page); } - atomic_set(compound_mapcount_ptr(page), -1); - atomic_set(subpages_mapcount_ptr(page), 0); - atomic_set(compound_pincount_ptr(page), 0); + atomic_set(folio_mapcount_ptr(folio), -1); + atomic_set(folio_subpages_mapcount_ptr(folio), 0); + atomic_set(folio_pincount_ptr(folio), 0); return true; =20 out_error: /* undo page modifications made above */ for (j =3D 0; j < i; j++) { - p =3D nth_page(page, j); + p =3D folio_page(folio, j); if (j !=3D 0) clear_compound_head(p); set_page_refcounted(p); } /* need to clear PG_reserved on remaining tail pages */ for (; j < nr_pages; j++) { - p =3D nth_page(page, j); + p =3D folio_page(folio, j); __ClearPageReserved(p); } - set_compound_order(page, 0); -#ifdef CONFIG_64BIT - page[1].compound_nr =3D 0; -#endif - __ClearPageHead(page); + folio_set_compound_order(folio, 0); + folio_clear_head(folio); return false; } =20 -static bool prep_compound_gigantic_page(struct page *page, unsigned int or= der) +static bool prep_compound_gigantic_folio(struct folio *folio, + unsigned int order) { - return __prep_compound_gigantic_page(page, order, false); + return __prep_compound_gigantic_folio(folio, order, false); } =20 -static bool prep_compound_gigantic_page_for_demote(struct page *page, +static bool prep_compound_gigantic_folio_for_demote(struct folio *folio, unsigned int order) { - return __prep_compound_gigantic_page(page, order, true); + return __prep_compound_gigantic_folio(folio, order, true); } =20 /* @@ -2039,7 +2035,7 @@ static struct page *alloc_fresh_huge_page(struct hsta= te *h, return NULL; folio =3D page_folio(page); if (hstate_is_gigantic(h)) { - if (!prep_compound_gigantic_page(page, huge_page_order(h))) { + if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { /* * Rare failure to convert pages to compound page. * Free pages and try again - ONCE! @@ -2052,7 +2048,7 @@ static struct page *alloc_fresh_huge_page(struct hsta= te *h, return NULL; } } - prep_new_huge_page(h, page, page_to_nid(page)); + prep_new_hugetlb_folio(h, folio, folio_nid(folio)); =20 return page; } @@ -3056,10 +3052,10 @@ static void __init gather_bootmem_prealloc(void) struct hstate *h =3D m->hstate; =20 VM_BUG_ON(!hstate_is_gigantic(h)); - WARN_ON(page_count(page) !=3D 1); - if (prep_compound_gigantic_page(page, huge_page_order(h))) { - WARN_ON(PageReserved(page)); - prep_new_huge_page(h, page, page_to_nid(page)); + WARN_ON(folio_ref_count(folio) !=3D 1); + if (prep_compound_gigantic_folio(folio, huge_page_order(h))) { + WARN_ON(folio_test_reserved(folio)); + prep_new_hugetlb_folio(h, folio, folio_nid(folio)); free_huge_page(page); /* add to the hugepage allocator */ } else { /* VERY unlikely inflated ref count on a tail page */ @@ -3478,13 +3474,14 @@ static int demote_free_huge_page(struct hstate *h, = struct page *page) for (i =3D 0; i < pages_per_huge_page(h); i +=3D pages_per_huge_page(target_hstate)) { subpage =3D nth_page(page, i); + folio =3D page_folio(subpage); if (hstate_is_gigantic(target_hstate)) - prep_compound_gigantic_page_for_demote(subpage, + prep_compound_gigantic_folio_for_demote(folio, target_hstate->order); else prep_compound_page(subpage, target_hstate->order); set_page_private(subpage, 0); - prep_new_huge_page(target_hstate, subpage, nid); + prep_new_hugetlb_folio(target_hstate, folio, nid); free_huge_page(subpage); } mutex_unlock(&target_hstate->resize_lock); --=20 2.38.1 From nobody Wed Apr 15 19:23:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2921C433FE for ; Thu, 17 Nov 2022 21:16:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234898AbiKQVQn (ORCPT ); Thu, 17 Nov 2022 16:16:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60854 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240586AbiKQVPn (ORCPT ); Thu, 17 Nov 2022 16:15:43 -0500 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C9E2D85EC6 for ; Thu, 17 Nov 2022 13:15:17 -0800 (PST) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AHLEGdW000786; Thu, 17 Nov 2022 21:15:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2022-7-12; bh=oiF9VDlWy9y8mIee/ulZOw0MitvLoK+ljs9NDxhkktU=; b=1b7T0wgGOZEvzRW4OxPtZxtc7HeAn4ePlc3cEQsC2+gC6OWyI5iuuEWJjyE2MKc191b3 BvnZefg+nNRj7XdqMkzIXhWh4YJ6rz8kuIBxHwo3Jg3RTtGIoHAEtpr8gapcTi0KfumP O2AAKZtmAGoFT2CPXlnczygj7posFpaHL0UPloZD1gEuHJpgSOg4uCs3HkI8xOFdxmn3 nWSMyVM++SvB7fUxCKwoN0DUz3/PUCJs+ROuIncAEbgin+Z9dMBX/1y/9paoVGICnZu6 49EQHAhxGjqxziFy94YrnhwZLoSSWdc9kcruEE+mXzs4rAP3h+UxpAbv+f4ZzpKCB4QE 2g== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3kv8ykthg8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:07 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 2AHKri2c011542; Thu, 17 Nov 2022 21:15:06 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3kt1x9nqt0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Nov 2022 21:15:06 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2AHLEx7k024557; Thu, 17 Nov 2022 21:15:06 GMT Received: from sid-dell.us.oracle.com (dhcp-10-132-95-73.usdhcp.oraclecorp.com [10.132.95.73]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3kt1x9nqku-11; Thu, 17 Nov 2022 21:15:06 +0000 From: Sidhartha Kumar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: akpm@linux-foundation.org, songmuchun@bytedance.com, mike.kravetz@oracle.com, willy@infradead.org, almasrymina@google.com, linmiaohe@huawei.com, hughd@google.com, Sidhartha Kumar Subject: [PATCH mm-unstable v3 10/10] mm/hugetlb: change hugetlb allocation functions to return a folio Date: Thu, 17 Nov 2022 13:15:01 -0800 Message-Id: <20221117211501.17150-11-sidhartha.kumar@oracle.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221117211501.17150-1-sidhartha.kumar@oracle.com> References: <20221117211501.17150-1-sidhartha.kumar@oracle.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-17_06,2022-11-17_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2211170152 X-Proofpoint-GUID: ndawDH4INcKZlpvlT2gHcL1g4jkY5Jbf X-Proofpoint-ORIG-GUID: ndawDH4INcKZlpvlT2gHcL1g4jkY5Jbf Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Many hugetlb allocation helper functions have now been converting to folios, update their higher level callers to be compatible with folios. Signed-off-by: Sidhartha Kumar --- mm/hugetlb.c | 98 ++++++++++++++++++++++++---------------------------- 1 file changed, 46 insertions(+), 52 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 6ecf9874c521..aa0a388bded4 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1378,7 +1378,7 @@ static void free_gigantic_folio(struct folio *folio, = unsigned int order) } =20 #ifdef CONFIG_CONTIG_ALLOC -static struct page *alloc_gigantic_page(struct hstate *h, gfp_t gfp_mask, +static struct folio *alloc_gigantic_folio(struct hstate *h, gfp_t gfp_mask, int nid, nodemask_t *nodemask) { unsigned long nr_pages =3D pages_per_huge_page(h); @@ -1394,7 +1394,7 @@ static struct page *alloc_gigantic_page(struct hstate= *h, gfp_t gfp_mask, page =3D cma_alloc(hugetlb_cma[nid], nr_pages, huge_page_order(h), true); if (page) - return page; + return page_folio(page); } =20 if (!(gfp_mask & __GFP_THISNODE)) { @@ -1405,17 +1405,16 @@ static struct page *alloc_gigantic_page(struct hsta= te *h, gfp_t gfp_mask, page =3D cma_alloc(hugetlb_cma[node], nr_pages, huge_page_order(h), true); if (page) - return page; + return page_folio(page); } } } #endif - - return alloc_contig_pages(nr_pages, gfp_mask, nid, nodemask); + return page_folio(alloc_contig_pages(nr_pages, gfp_mask, nid, nodemask)); } =20 #else /* !CONFIG_CONTIG_ALLOC */ -static struct page *alloc_gigantic_page(struct hstate *h, gfp_t gfp_mask, +static struct folio *alloc_gigantic_folio(struct hstate *h, gfp_t gfp_mask, int nid, nodemask_t *nodemask) { return NULL; @@ -1423,7 +1422,7 @@ static struct page *alloc_gigantic_page(struct hstate= *h, gfp_t gfp_mask, #endif /* CONFIG_CONTIG_ALLOC */ =20 #else /* !CONFIG_ARCH_HAS_GIGANTIC_PAGE */ -static struct page *alloc_gigantic_page(struct hstate *h, gfp_t gfp_mask, +static struct folio *alloc_gigantic_folio(struct hstate *h, gfp_t gfp_mask, int nid, nodemask_t *nodemask) { return NULL; @@ -1948,7 +1947,7 @@ pgoff_t hugetlb_basepage_index(struct page *page) return (index << compound_order(page_head)) + compound_idx; } =20 -static struct page *alloc_buddy_huge_page(struct hstate *h, +static struct folio *alloc_buddy_hugetlb_folio(struct hstate *h, gfp_t gfp_mask, int nid, nodemask_t *nmask, nodemask_t *node_alloc_noretry) { @@ -2007,7 +2006,7 @@ static struct page *alloc_buddy_huge_page(struct hsta= te *h, if (node_alloc_noretry && !page && alloc_try_hard) node_set(nid, *node_alloc_noretry); =20 - return page; + return page_folio(page); } =20 /* @@ -2017,23 +2016,21 @@ static struct page *alloc_buddy_huge_page(struct hs= tate *h, * Note that returned page is 'frozen': ref count of head page and all ta= il * pages is zero. */ -static struct page *alloc_fresh_huge_page(struct hstate *h, +static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, gfp_t gfp_mask, int nid, nodemask_t *nmask, nodemask_t *node_alloc_noretry) { - struct page *page; struct folio *folio; bool retry =3D false; =20 retry: if (hstate_is_gigantic(h)) - page =3D alloc_gigantic_page(h, gfp_mask, nid, nmask); + folio =3D alloc_gigantic_folio(h, gfp_mask, nid, nmask); else - page =3D alloc_buddy_huge_page(h, gfp_mask, + folio =3D alloc_buddy_hugetlb_folio(h, gfp_mask, nid, nmask, node_alloc_noretry); - if (!page) + if (!folio) return NULL; - folio =3D page_folio(page); if (hstate_is_gigantic(h)) { if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { /* @@ -2050,7 +2047,7 @@ static struct page *alloc_fresh_huge_page(struct hsta= te *h, } prep_new_hugetlb_folio(h, folio, folio_nid(folio)); =20 - return page; + return folio; } =20 /* @@ -2060,21 +2057,21 @@ static struct page *alloc_fresh_huge_page(struct hs= tate *h, static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowe= d, nodemask_t *node_alloc_noretry) { - struct page *page; + struct folio *folio; int nr_nodes, node; gfp_t gfp_mask =3D htlb_alloc_mask(h) | __GFP_THISNODE; =20 for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) { - page =3D alloc_fresh_huge_page(h, gfp_mask, node, nodes_allowed, - node_alloc_noretry); - if (page) + folio =3D alloc_fresh_hugetlb_folio(h, gfp_mask, node, + nodes_allowed, node_alloc_noretry); + if (folio) break; } =20 - if (!page) + if (!folio) return 0; =20 - free_huge_page(page); /* free it into the hugepage allocator */ + free_huge_page(&folio->page); /* free it into the hugepage allocator */ =20 return 1; } @@ -2235,7 +2232,7 @@ int dissolve_free_huge_pages(unsigned long start_pfn,= unsigned long end_pfn) static struct page *alloc_surplus_huge_page(struct hstate *h, gfp_t gfp_ma= sk, int nid, nodemask_t *nmask) { - struct page *page =3D NULL; + struct folio *folio =3D NULL; =20 if (hstate_is_gigantic(h)) return NULL; @@ -2245,8 +2242,8 @@ static struct page *alloc_surplus_huge_page(struct hs= tate *h, gfp_t gfp_mask, goto out_unlock; spin_unlock_irq(&hugetlb_lock); =20 - page =3D alloc_fresh_huge_page(h, gfp_mask, nid, nmask, NULL); - if (!page) + folio =3D alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, NULL); + if (!folio) return NULL; =20 spin_lock_irq(&hugetlb_lock); @@ -2258,43 +2255,42 @@ static struct page *alloc_surplus_huge_page(struct = hstate *h, gfp_t gfp_mask, * codeflow */ if (h->surplus_huge_pages >=3D h->nr_overcommit_huge_pages) { - SetHPageTemporary(page); + folio_set_hugetlb_temporary(folio); spin_unlock_irq(&hugetlb_lock); - free_huge_page(page); + free_huge_page(&folio->page); return NULL; } =20 h->surplus_huge_pages++; - h->surplus_huge_pages_node[page_to_nid(page)]++; + h->surplus_huge_pages_node[folio_nid(folio)]++; =20 out_unlock: spin_unlock_irq(&hugetlb_lock); =20 - return page; + return &folio->page; } =20 static struct page *alloc_migrate_huge_page(struct hstate *h, gfp_t gfp_ma= sk, int nid, nodemask_t *nmask) { - struct page *page; + struct folio *folio; =20 if (hstate_is_gigantic(h)) return NULL; =20 - page =3D alloc_fresh_huge_page(h, gfp_mask, nid, nmask, NULL); - if (!page) + folio =3D alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, NULL); + if (!folio) return NULL; =20 /* fresh huge pages are frozen */ - set_page_refcounted(page); - + folio_ref_unfreeze(folio, 1); /* * We do not account these pages as surplus because they are only * temporary and will be released properly on the last reference */ - SetHPageTemporary(page); + folio_set_hugetlb_temporary(folio); =20 - return page; + return &folio->page; } =20 /* @@ -2743,19 +2739,18 @@ void restore_reserve_on_error(struct hstate *h, str= uct vm_area_struct *vma, } =20 /* - * alloc_and_dissolve_huge_page - Allocate a new page and dissolve the old= one + * alloc_and_dissolve_hugetlb_folio - Allocate a new folio and dissolve + * the old one * @h: struct hstate old page belongs to * @old_page: Old page to dissolve * @list: List to isolate the page in case we need to * Returns 0 on success, otherwise negated error. */ -static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old= _page, - struct list_head *list) +static int alloc_and_dissolve_hugetlb_folio(struct hstate *h, + struct folio *old_folio, struct list_head *list) { gfp_t gfp_mask =3D htlb_alloc_mask(h) | __GFP_THISNODE; - struct folio *old_folio =3D page_folio(old_page); int nid =3D folio_nid(old_folio); - struct page *new_page; struct folio *new_folio; int ret =3D 0; =20 @@ -2766,26 +2761,25 @@ static int alloc_and_dissolve_huge_page(struct hsta= te *h, struct page *old_page, * the pool. This simplifies and let us do most of the processing * under the lock. */ - new_page =3D alloc_buddy_huge_page(h, gfp_mask, nid, NULL, NULL); - if (!new_page) + new_folio =3D alloc_buddy_hugetlb_folio(h, gfp_mask, nid, NULL, NULL); + if (!new_folio) return -ENOMEM; - new_folio =3D page_folio(new_page); __prep_new_hugetlb_folio(h, new_folio); =20 retry: spin_lock_irq(&hugetlb_lock); if (!folio_test_hugetlb(old_folio)) { /* - * Freed from under us. Drop new_page too. + * Freed from under us. Drop new_folio too. */ goto free_new; } else if (folio_ref_count(old_folio)) { /* - * Someone has grabbed the page, try to isolate it here. + * Someone has grabbed the folio, try to isolate it here. * Fail with -EBUSY if not possible. */ spin_unlock_irq(&hugetlb_lock); - ret =3D isolate_hugetlb(old_page, list); + ret =3D isolate_hugetlb(&old_folio->page, list); spin_lock_irq(&hugetlb_lock); goto free_new; } else if (!folio_test_hugetlb_freed(old_folio)) { @@ -2863,7 +2857,7 @@ int isolate_or_dissolve_huge_page(struct page *page, = struct list_head *list) if (folio_ref_count(folio) && !isolate_hugetlb(&folio->page, list)) ret =3D 0; else if (!folio_ref_count(folio)) - ret =3D alloc_and_dissolve_huge_page(h, &folio->page, list); + ret =3D alloc_and_dissolve_hugetlb_folio(h, folio, list); =20 return ret; } @@ -3081,14 +3075,14 @@ static void __init hugetlb_hstate_alloc_pages_oneno= de(struct hstate *h, int nid) if (!alloc_bootmem_huge_page(h, nid)) break; } else { - struct page *page; + struct folio *folio; gfp_t gfp_mask =3D htlb_alloc_mask(h) | __GFP_THISNODE; =20 - page =3D alloc_fresh_huge_page(h, gfp_mask, nid, + folio =3D alloc_fresh_hugetlb_folio(h, gfp_mask, nid, &node_states[N_MEMORY], NULL); - if (!page) + if (!folio) break; - free_huge_page(page); /* free it into the hugepage allocator */ + free_huge_page(&folio->page); /* free it into the hugepage allocator */ } cond_resched(); } --=20 2.38.1