From nobody Tue Dec 16 23:40:37 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 959A8C004C0 for ; Fri, 20 Oct 2023 18:33:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229738AbjJTSdv (ORCPT ); Fri, 20 Oct 2023 14:33:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37722 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229473AbjJTSdr (ORCPT ); Fri, 20 Oct 2023 14:33:47 -0400 Received: from mail-pf1-x429.google.com (mail-pf1-x429.google.com [IPv6:2607:f8b0:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A493D5B for ; Fri, 20 Oct 2023 11:33:44 -0700 (PDT) Received: by mail-pf1-x429.google.com with SMTP id d2e1a72fcca58-6b3c2607d9bso1064047b3a.1 for ; Fri, 20 Oct 2023 11:33:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697826824; x=1698431624; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rr6V2LCgJ9TVJLB96vsUnm/xqKZcto4Za9cX8Go8cec=; b=HYX1L559DtEq154vwCpkFBPlvKAHxN1+kc/Em7pUEga2+In6+pmqasjNOF3gLEFzN4 l+JOpLsoDjH/b0O6djwmNtee7I877AGNMNZ7sr63arV9huIvQQHj34LYVRkIDTMxQQdD JQszknP2THonG6VWkSZO0mzcPM0cs/i4HOz69FoAvnmLlcp7GDdIdpZc5S2aQ9K3uBUa GgavpWbXi1CHVA5GIV+0vmxj0hBDeiPebqmP7zs8IKBNzLYdNMOa/i5sAyJAzEyD3iUx 4muJBKUwPCMW9Rbe0f1YW2gvSmeITDfcP+CN8dtI3TzR92USHnUHwYDsixvpfa1gyF4p k9Fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697826824; x=1698431624; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rr6V2LCgJ9TVJLB96vsUnm/xqKZcto4Za9cX8Go8cec=; b=Gh8Iisf1nYWWVYJ51ujq7RqIajWROQRuSw0BKJIydygsj40xw5QaweHPYa29ZoxlQw v0PtUvdhww7/sn9gXUbW74cREMusXrkWywXvCB2ZOOu0/PtotLH9QRjMLygOdybASbBP GFv3+8wtYpQ4043dufwYnCqM9Tc5OoVfT1b5iFTXJL5RPr4x8RqNAdKm6eXA50C9P6Uz lFJMgSf3OXHKDJ5KkbFjppK/JhPgaa7Coz3OusX5D8UKj5JXnYLwssK1tiQRxxUt/LH1 dsxpkKYdCJe9utxWJmeL+5ryV38bZqETYEpU+OHEtabd6AIowpLDpi6ZIH56u040Kkg0 2xKw== X-Gm-Message-State: AOJu0YxiAVYPqJdSCyZZ5ZRACdrm7ADgPywVaHYJQ0TdqZXaKwjYe0+q 9Sm8CyDp6NK8UlWTT5g2ay5zKloEMkU= X-Google-Smtp-Source: AGHT+IGHwdGrDYHJU3ar4omw5xYzuwotpwlRbNDAoj/IKDWChBLhpJBOvVjPjaCZMcDR1LHeQ2sY9A== X-Received: by 2002:a05:6a20:158c:b0:17c:cd46:73d with SMTP id h12-20020a056a20158c00b0017ccd46073dmr386382pzj.20.1697826823892; Fri, 20 Oct 2023 11:33:43 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id z14-20020aa7990e000000b006b4ac8885b4sm1911345pff.14.2023.10.20.11.33.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 11:33:43 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, "Vishal Moola (Oracle)" , Matthew Wilcox Subject: [PATCH v3 1/5] mm/khugepaged: Convert __collapse_huge_page_isolate() to use folios Date: Fri, 20 Oct 2023 11:33:27 -0700 Message-Id: <20231020183331.10770-2-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231020183331.10770-1-vishal.moola@gmail.com> References: <20231020183331.10770-1-vishal.moola@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Replaces 11 calls to compound_head() with 1, and removes 1348 bytes of kernel text. Reviewed-by: Matthew Wilcox (Oracle) Signed-off-by: Vishal Moola (Oracle) Reviewed-by: David Hildenbrand Reviewed-by: Yang Shi --- mm/khugepaged.c | 45 +++++++++++++++++++++++---------------------- 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 88433cc25d8a..500756604488 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -542,6 +542,7 @@ static int __collapse_huge_page_isolate(struct vm_area_= struct *vma, struct list_head *compound_pagelist) { struct page *page =3D NULL; + struct folio *folio =3D NULL; pte_t *_pte; int none_or_zero =3D 0, shared =3D 0, result =3D SCAN_FAIL, referenced = =3D 0; bool writable =3D false; @@ -576,7 +577,8 @@ static int __collapse_huge_page_isolate(struct vm_area_= struct *vma, goto out; } =20 - VM_BUG_ON_PAGE(!PageAnon(page), page); + folio =3D page_folio(page); + VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio); =20 if (page_mapcount(page) > 1) { ++shared; @@ -588,16 +590,15 @@ static int __collapse_huge_page_isolate(struct vm_are= a_struct *vma, } } =20 - if (PageCompound(page)) { - struct page *p; - page =3D compound_head(page); + if (folio_test_large(folio)) { + struct folio *f; =20 /* * Check if we have dealt with the compound page * already */ - list_for_each_entry(p, compound_pagelist, lru) { - if (page =3D=3D p) + list_for_each_entry(f, compound_pagelist, lru) { + if (folio =3D=3D f) goto next; } } @@ -608,7 +609,7 @@ static int __collapse_huge_page_isolate(struct vm_area_= struct *vma, * is needed to serialize against split_huge_page * when invoked from the VM. */ - if (!trylock_page(page)) { + if (!folio_trylock(folio)) { result =3D SCAN_PAGE_LOCK; goto out; } @@ -624,8 +625,8 @@ static int __collapse_huge_page_isolate(struct vm_area_= struct *vma, * but not from this process. The other process cannot write to * the page, only trigger CoW. */ - if (!is_refcount_suitable(page)) { - unlock_page(page); + if (!is_refcount_suitable(&folio->page)) { + folio_unlock(folio); result =3D SCAN_PAGE_COUNT; goto out; } @@ -634,27 +635,27 @@ static int __collapse_huge_page_isolate(struct vm_are= a_struct *vma, * Isolate the page to avoid collapsing an hugepage * currently in use by the VM. */ - if (!isolate_lru_page(page)) { - unlock_page(page); + if (!folio_isolate_lru(folio)) { + folio_unlock(folio); result =3D SCAN_DEL_PAGE_LRU; goto out; } - mod_node_page_state(page_pgdat(page), - NR_ISOLATED_ANON + page_is_file_lru(page), - compound_nr(page)); - VM_BUG_ON_PAGE(!PageLocked(page), page); - VM_BUG_ON_PAGE(PageLRU(page), page); + node_stat_mod_folio(folio, + NR_ISOLATED_ANON + folio_is_file_lru(folio), + folio_nr_pages(folio)); + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); =20 - if (PageCompound(page)) - list_add_tail(&page->lru, compound_pagelist); + if (folio_test_large(folio)) + list_add_tail(&folio->lru, compound_pagelist); next: /* * If collapse was initiated by khugepaged, check that there is * enough young pte to justify collapsing the page */ if (cc->is_khugepaged && - (pte_young(pteval) || page_is_young(page) || - PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, + (pte_young(pteval) || folio_test_young(folio) || + folio_test_referenced(folio) || mmu_notifier_test_young(vma->vm_mm, address))) referenced++; =20 @@ -668,13 +669,13 @@ static int __collapse_huge_page_isolate(struct vm_are= a_struct *vma, result =3D SCAN_LACK_REFERENCED_PAGE; } else { result =3D SCAN_SUCCEED; - trace_mm_collapse_huge_page_isolate(page, none_or_zero, + trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero, referenced, writable, result); return result; } out: release_pte_pages(pte, _pte, compound_pagelist); - trace_mm_collapse_huge_page_isolate(page, none_or_zero, + trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero, referenced, writable, result); return result; } --=20 2.40.1 From nobody Tue Dec 16 23:40:37 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A09E2C001DF for ; Fri, 20 Oct 2023 18:33:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229596AbjJTSd4 (ORCPT ); Fri, 20 Oct 2023 14:33:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37736 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229723AbjJTSds (ORCPT ); Fri, 20 Oct 2023 14:33:48 -0400 Received: from mail-pf1-x435.google.com (mail-pf1-x435.google.com [IPv6:2607:f8b0:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 81F57112 for ; Fri, 20 Oct 2023 11:33:45 -0700 (PDT) Received: by mail-pf1-x435.google.com with SMTP id d2e1a72fcca58-6ba172c5f3dso1070394b3a.0 for ; Fri, 20 Oct 2023 11:33:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697826825; x=1698431625; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kOxvYaksrDFINgvTHvg7i3BLH6nikNi8tBwFFhT4+xI=; b=IiZSR70ybdNI3AWQ9+UGfUte2FcmKSqDQ31Ilhg4Z/kpgKRYeh3Geibq7t//IRFKyP 4jGhFcLhrfz0e4CnsrX4jM5Yhg1gohG0E0XQH4r1uqOcWrajeDMW93kQZBCkARxVcAjz tHrvw0MADRUkU/x1oe3NaVTTUIkGvqfEpVg1CXf+dH7GR2kquYD2lVj45r9djTdV0gT+ CoBcIm3l3n9z9bygs9VLZiaUHXA/fGRHK1fTjrhroUEhUQVrGmcZOzYb9IclutCGBmVx X0ugtJZM5V2y/Qq1PPI1Jm7Cpnxs5sejIjjGVHzBNDybDpu9IZvV49mlmG3U18fvxOAO Gj3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697826825; x=1698431625; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kOxvYaksrDFINgvTHvg7i3BLH6nikNi8tBwFFhT4+xI=; b=nIXgwvgv2MMUee001NekUieiDMzkwFOKzgduuNUaz9XFDw4XO2dQbhCZ/XUN7Ugg7G Fsrz5Pb9iijAQBSuHgzu+jxEGVV5zVCLvYNfnIS9RQ9PYuTPEAsll1LXgRg5e3scxfQz WiN9007jb5agOBLIWRng3u0GIXrN1DiophtzvS1/xGgEu55lSNMlqWC59/a57nEmibU4 clq2zheuVh7yTBz/7uMPXoQnjXrIXSl5J2lcYl84+DRWcrDCXSrwOVJOMNpIAykkN06s I+I7I1SC9/d9r4/rkD7AR98tr+1w6htk4HXDC4gKzIBXd9R2cfTIpoJSUQOOfR5sFxLP WU6w== X-Gm-Message-State: AOJu0YxU7mbuRtGkv7xFgUXJqtmQeFzpGr20Gq+zO1poKer3UWhEq9dc /TSf7fZ8NeJeStqCsxmsncg= X-Google-Smtp-Source: AGHT+IGjvWBVIPvS8d/Tp9SPsRWhqao+5hgFvjwUvgPczwPUlTiLiRzRdGbqQM/sCIzdkV5OmPNgVQ== X-Received: by 2002:a05:6a00:1393:b0:6bc:62d0:fffd with SMTP id t19-20020a056a00139300b006bc62d0fffdmr2943268pfg.15.1697826824857; Fri, 20 Oct 2023 11:33:44 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id z14-20020aa7990e000000b006b4ac8885b4sm1911345pff.14.2023.10.20.11.33.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 11:33:44 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, "Vishal Moola (Oracle)" Subject: [PATCH v3 2/5] mm/khugepaged: Convert hpage_collapse_scan_pmd() to use folios Date: Fri, 20 Oct 2023 11:33:28 -0700 Message-Id: <20231020183331.10770-3-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231020183331.10770-1-vishal.moola@gmail.com> References: <20231020183331.10770-1-vishal.moola@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Replaces 5 calls to compound_head(), and removes 1385 bytes of kernel text. Signed-off-by: Vishal Moola (Oracle) Reviewed-by: Rik van Riel Reviewed-by: Yang Shi --- mm/khugepaged.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 500756604488..6c4b5af43371 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1248,6 +1248,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *= mm, int result =3D SCAN_FAIL, referenced =3D 0; int none_or_zero =3D 0, shared =3D 0; struct page *page =3D NULL; + struct folio *folio =3D NULL; unsigned long _address; spinlock_t *ptl; int node =3D NUMA_NO_NODE, unmapped =3D 0; @@ -1334,29 +1335,28 @@ static int hpage_collapse_scan_pmd(struct mm_struct= *mm, } } =20 - page =3D compound_head(page); - + folio =3D page_folio(page); /* * Record which node the original page is from and save this * information to cc->node_load[]. * Khugepaged will allocate hugepage from the node has the max * hit record. */ - node =3D page_to_nid(page); + node =3D folio_nid(folio); if (hpage_collapse_scan_abort(node, cc)) { result =3D SCAN_SCAN_ABORT; goto out_unmap; } cc->node_load[node]++; - if (!PageLRU(page)) { + if (!folio_test_lru(folio)) { result =3D SCAN_PAGE_LRU; goto out_unmap; } - if (PageLocked(page)) { + if (folio_test_locked(folio)) { result =3D SCAN_PAGE_LOCK; goto out_unmap; } - if (!PageAnon(page)) { + if (!folio_test_anon(folio)) { result =3D SCAN_PAGE_ANON; goto out_unmap; } @@ -1371,7 +1371,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *= mm, * has excessive GUP pins (i.e. 512). Anyway the same check * will be done again later the risk seems low. */ - if (!is_refcount_suitable(page)) { + if (!is_refcount_suitable(&folio->page)) { result =3D SCAN_PAGE_COUNT; goto out_unmap; } @@ -1381,8 +1381,8 @@ static int hpage_collapse_scan_pmd(struct mm_struct *= mm, * enough young pte to justify collapsing the page */ if (cc->is_khugepaged && - (pte_young(pteval) || page_is_young(page) || - PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, + (pte_young(pteval) || folio_test_young(folio) || + folio_test_referenced(folio) || mmu_notifier_test_young(vma->vm_mm, address))) referenced++; } @@ -1404,7 +1404,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *= mm, *mmap_locked =3D false; } out: - trace_mm_khugepaged_scan_pmd(mm, page, writable, referenced, + trace_mm_khugepaged_scan_pmd(mm, &folio->page, writable, referenced, none_or_zero, result, unmapped); return result; } --=20 2.40.1 From nobody Tue Dec 16 23:40:37 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4406BC25B41 for ; Fri, 20 Oct 2023 18:33:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229800AbjJTSdx (ORCPT ); Fri, 20 Oct 2023 14:33:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37724 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229695AbjJTSds (ORCPT ); Fri, 20 Oct 2023 14:33:48 -0400 Received: from mail-pf1-x42b.google.com (mail-pf1-x42b.google.com [IPv6:2607:f8b0:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6CEFA1A8 for ; Fri, 20 Oct 2023 11:33:46 -0700 (PDT) Received: by mail-pf1-x42b.google.com with SMTP id d2e1a72fcca58-6b5cac99cfdso1054410b3a.2 for ; Fri, 20 Oct 2023 11:33:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697826826; x=1698431626; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Q/rAUjlYbudj5wnw8lHkufWutUwYDRgsClPJWjawwgo=; b=VH+0CnjMk9UQe/XogDr4e3WlaCk4NdyH46es8n3wMSx1EKeVd1oygRkYQV1ou956FI PSjOBcNMMYmIIIJIg1gNFUIlUxK1Mr5/drJnX2c+he51Fy0nZXajkRwpKSYuO6kIAWdG o3lihQaC7FRsnPtmCMf0sXUyxn1mfPbByh450gEqi7SuAXsRt1OxFEpZAfh6ZbYbNldS O//lADm9/p2bAtqRo/0J5eZTyaIT1asGObyvR7Wi/U4tvFP6poDs3PePyrE0qXagmPtm LS2VJRqa5k7S/udB61F/KY2ljLhHubwzDXh/W0BG0JszY+xDp0yPM3fc1WDlbj4e8dgk WnRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697826826; x=1698431626; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Q/rAUjlYbudj5wnw8lHkufWutUwYDRgsClPJWjawwgo=; b=M+wN21SnBMo2Wi//noFdAa6m4uRC0lkbVwVIkQRQ2zlekN8JQKPeBLqSgBFyQ1ecev VxtfJVjz+kc729qFnuSlDl3fHIKh3RZMZLPRBu2u1Zml14pCYU3uP7sSE6m5WReayWKw hp+bNzeSzegmkeOP/XuqCOr+wS0Ex2V2VLYx9ankpAj7s/she1toVN3lLSarHuLaaFaM aTU9neCa5h1tdF5KobUYCDFJAwEyO8cWn+guK3oWtJLSecTyfX3/y+9m1npxVJuc/c/T y5Ug4Z0f6jiWOmbwuoYFDcng85RUjEW9L4OJEPHtEtLmsX08HHnhlLAjMcO7bALn2pk8 HW6Q== X-Gm-Message-State: AOJu0YyqAGBOAmWT5fYiKtp2KZ5pLghXiwSjIn9jwXO0iavHiPg2GlIb 0hTW1WHviSixs9HJ6XVg6WcxJvlaqpA= X-Google-Smtp-Source: AGHT+IHyGeFVYpw0wd/VaSvhNfJGFkjxPCXQo0amteeNn2H2xRJuWdIEmOzTNv8bZATyXXNSf6XFFg== X-Received: by 2002:a05:6a00:2341:b0:6b3:2106:94b0 with SMTP id j1-20020a056a00234100b006b3210694b0mr2956504pfj.23.1697826825800; Fri, 20 Oct 2023 11:33:45 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id z14-20020aa7990e000000b006b4ac8885b4sm1911345pff.14.2023.10.20.11.33.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 11:33:45 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, "Vishal Moola (Oracle)" Subject: [PATCH v3 3/5] mm/khugepaged: Convert is_refcount_suitable() to use folios Date: Fri, 20 Oct 2023 11:33:29 -0700 Message-Id: <20231020183331.10770-4-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231020183331.10770-1-vishal.moola@gmail.com> References: <20231020183331.10770-1-vishal.moola@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Both callers of is_refcount_suitable() have been converted to use folios, so convert it to take in a folio. Both callers only operate on head pages of folios so mapcount/refcount conversions here are trivial. Removes 3 calls to compound head, and removes 315 bytes of kernel text. Signed-off-by: Vishal Moola (Oracle) Reviewed-by: David Hildenbrand Reviewed-by: Yang Shi --- mm/khugepaged.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6c4b5af43371..9efd8ff68f06 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -524,15 +524,15 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte, } } =20 -static bool is_refcount_suitable(struct page *page) +static bool is_refcount_suitable(struct folio *folio) { int expected_refcount; =20 - expected_refcount =3D total_mapcount(page); - if (PageSwapCache(page)) - expected_refcount +=3D compound_nr(page); + expected_refcount =3D folio_mapcount(folio); + if (folio_test_swapcache(folio)) + expected_refcount +=3D folio_nr_pages(folio); =20 - return page_count(page) =3D=3D expected_refcount; + return folio_ref_count(folio) =3D=3D expected_refcount; } =20 static int __collapse_huge_page_isolate(struct vm_area_struct *vma, @@ -625,7 +625,7 @@ static int __collapse_huge_page_isolate(struct vm_area_= struct *vma, * but not from this process. The other process cannot write to * the page, only trigger CoW. */ - if (!is_refcount_suitable(&folio->page)) { + if (!is_refcount_suitable(folio)) { folio_unlock(folio); result =3D SCAN_PAGE_COUNT; goto out; @@ -1371,7 +1371,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *= mm, * has excessive GUP pins (i.e. 512). Anyway the same check * will be done again later the risk seems low. */ - if (!is_refcount_suitable(&folio->page)) { + if (!is_refcount_suitable(folio)) { result =3D SCAN_PAGE_COUNT; goto out_unmap; } --=20 2.40.1 From nobody Tue Dec 16 23:40:37 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D911C00A8F for ; Fri, 20 Oct 2023 18:33:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229893AbjJTSd6 (ORCPT ); Fri, 20 Oct 2023 14:33:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37740 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229726AbjJTSds (ORCPT ); Fri, 20 Oct 2023 14:33:48 -0400 Received: from mail-pg1-x52a.google.com (mail-pg1-x52a.google.com [IPv6:2607:f8b0:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 62806D63 for ; Fri, 20 Oct 2023 11:33:47 -0700 (PDT) Received: by mail-pg1-x52a.google.com with SMTP id 41be03b00d2f7-5859a7d6556so882222a12.0 for ; Fri, 20 Oct 2023 11:33:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697826827; x=1698431627; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xk5ybvp+SNZIaZ3pEA3Be7vi8BfZn75lCz2iohEvZyM=; b=l9B4cEmI+le35OCk8zv59oEfyblRpzTvTu3zdY9zOhtmFxbCbrDz8gO/iH5Ehn/Q1R bEjFLT2zrM+hHEsYApgYvqMXneg+GS56muHYvOKqV8iWbcsyfjPo6pAuwbnGZT1suaHk oSg6img7hjkxNYelXR+gTDsE1yWYl6EDzkvV/ssqAE9tQPzzh2wW7+rqCmbyTydJiwdj gTozqh7o9714pFs9ix5oBrTK6W4FHRXyl1v3vEAW6BQULdfNbHrVTytYWNkKfxclwggW RN6seG/fm9+WzhgSTgMsdJ6SXxBTbyNaX3Wgf6UcR7g8MSi1o4gSm6PY68N8+e0HCViL 7JFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697826827; x=1698431627; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xk5ybvp+SNZIaZ3pEA3Be7vi8BfZn75lCz2iohEvZyM=; b=vZUZeqYfAClUWzZJOSo9ugk3fcEzNFPzcL+7++DW8sJjuK3rE3v1qS/s5Hvds+4eKg cPTpKi0nSvgKKEPGv0fPUu6m3BorhvuHMlM0baAapODMiGCNvAUUCYoz5r2aI8u4WRzI vdNcrsGHT8Pz5V0tZ3YoRrr7I7iL6E/oGgTefWY8igjVH5fv/cE0qdGZJ0uBWIfbs+3j JY+LeV+jpN+wc5zHoPLM1qQAG2OoppZKtP4GAhdOZThp4IN77BouOWkCuWxRDVXqgPqh k4Sw3E57MNyBZ8HwQhF4NGPAg+WTwCcq0L1Oj5CcgN/rIETDGK7t2rlcsM4Xi9+fp8Ok r62g== X-Gm-Message-State: AOJu0Yzqgyy5Vmzh0gCjG0jzG+AdD5Vo68O2BocuXecfoIUCiLXy+NoL oOfi8lAgcFPTMwJrR4VQvZc= X-Google-Smtp-Source: AGHT+IHGxs5mKwV0SIfJTy/1PmcUSW12rF0lShgSZsg2OVE3kvZusMC/81y+OW2KnKIPoi+leXoxBg== X-Received: by 2002:a05:6a21:78aa:b0:141:69d:8041 with SMTP id bf42-20020a056a2178aa00b00141069d8041mr3176731pzc.48.1697826826732; Fri, 20 Oct 2023 11:33:46 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id z14-20020aa7990e000000b006b4ac8885b4sm1911345pff.14.2023.10.20.11.33.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 11:33:46 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, "Vishal Moola (Oracle)" Subject: [PATCH v3 4/5] mm/khugepaged: Convert alloc_charge_hpage() to use folios Date: Fri, 20 Oct 2023 11:33:30 -0700 Message-Id: <20231020183331.10770-5-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231020183331.10770-1-vishal.moola@gmail.com> References: <20231020183331.10770-1-vishal.moola@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Also remove count_memcg_page_event now that its last caller no longer uses it and reword hpage_collapse_alloc_page() to hpage_collapse_alloc_folio(). This removes 1 call to compound_head() and helps convert khugepaged to use folios throughout. Signed-off-by: Vishal Moola (Oracle) Reviewed-by: Rik van Riel Reviewed-by: Yang Shi --- include/linux/memcontrol.h | 14 -------------- mm/khugepaged.c | 17 ++++++++++------- 2 files changed, 10 insertions(+), 21 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ab94ad4597d0..3126bde982e8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1080,15 +1080,6 @@ static inline void count_memcg_events(struct mem_cgr= oup *memcg, local_irq_restore(flags); } =20 -static inline void count_memcg_page_event(struct page *page, - enum vm_event_item idx) -{ - struct mem_cgroup *memcg =3D page_memcg(page); - - if (memcg) - count_memcg_events(memcg, idx, 1); -} - static inline void count_memcg_folio_events(struct folio *folio, enum vm_event_item idx, unsigned long nr) { @@ -1565,11 +1556,6 @@ static inline void __count_memcg_events(struct mem_c= group *memcg, { } =20 -static inline void count_memcg_page_event(struct page *page, - int idx) -{ -} - static inline void count_memcg_folio_events(struct folio *folio, enum vm_event_item idx, unsigned long nr) { diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 9efd8ff68f06..6a7184cd291b 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -888,16 +888,16 @@ static int hpage_collapse_find_target_node(struct col= lapse_control *cc) } #endif =20 -static bool hpage_collapse_alloc_page(struct page **hpage, gfp_t gfp, int = node, +static bool hpage_collapse_alloc_folio(struct folio **folio, gfp_t gfp, in= t node, nodemask_t *nmask) { - *hpage =3D __alloc_pages(gfp, HPAGE_PMD_ORDER, node, nmask); - if (unlikely(!*hpage)) { + *folio =3D __folio_alloc(gfp, HPAGE_PMD_ORDER, node, nmask); + + if (unlikely(!*folio)) { count_vm_event(THP_COLLAPSE_ALLOC_FAILED); return false; } =20 - folio_prep_large_rmappable((struct folio *)*hpage); count_vm_event(THP_COLLAPSE_ALLOC); return true; } @@ -1064,17 +1064,20 @@ static int alloc_charge_hpage(struct page **hpage, = struct mm_struct *mm, int node =3D hpage_collapse_find_target_node(cc); struct folio *folio; =20 - if (!hpage_collapse_alloc_page(hpage, gfp, node, &cc->alloc_nmask)) + if (!hpage_collapse_alloc_folio(&folio, gfp, node, &cc->alloc_nmask)) { + *hpage =3D NULL; return SCAN_ALLOC_HUGE_PAGE_FAIL; + } =20 - folio =3D page_folio(*hpage); if (unlikely(mem_cgroup_charge(folio, mm, gfp))) { folio_put(folio); *hpage =3D NULL; return SCAN_CGROUP_CHARGE_FAIL; } - count_memcg_page_event(*hpage, THP_COLLAPSE_ALLOC); =20 + count_memcg_folio_events(folio, THP_COLLAPSE_ALLOC, 1); + + *hpage =3D folio_page(folio, 0); return SCAN_SUCCEED; } =20 --=20 2.40.1 From nobody Tue Dec 16 23:40:37 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15EFFC0032E for ; Fri, 20 Oct 2023 18:34:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229922AbjJTSeB (ORCPT ); Fri, 20 Oct 2023 14:34:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37764 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229756AbjJTSdu (ORCPT ); Fri, 20 Oct 2023 14:33:50 -0400 Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 522F0D67 for ; Fri, 20 Oct 2023 11:33:48 -0700 (PDT) Received: by mail-pf1-x433.google.com with SMTP id d2e1a72fcca58-6b5e6301a19so1070790b3a.0 for ; Fri, 20 Oct 2023 11:33:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697826828; x=1698431628; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=shwuj1hNaDkCqNReHb+lM6WWmRCPMvGlFZhV2p81nh8=; b=LIsuR3S32a2PwSp4SAeorZnAZ+GiucSvi/qseMgpfo/GLEHPP4hmdhuddXdQj1MDuL +/KnLW/6eM4+Ryabr0n5D984ZWgNFPeXa9f1wptcPuaeLBAgncm9ILAitxq6H/nM7Yk/ mT9bl9o61Cu54O7HIjk5uF5GnsvZUl+NErG9+MCMF9m0JeGAQRsfNFLOE74RLBSSPGvx mWx/o1rgPPwdk8LKJR5RR7nYPFFwcYwQMFZQXSGBASGwXakh6frzxr1v0EnlC8nyPCKj 4a5NRCzj+AixK3yzH/madUihBRLdSXwVnydlXH5uvV0SQ4ZqkpdKZ+DCTEesJRVEQEBb 8Vfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697826828; x=1698431628; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=shwuj1hNaDkCqNReHb+lM6WWmRCPMvGlFZhV2p81nh8=; b=ExPf8CngRhkpLZz8Y7IZTu9QiOFfE24tpxEhT3/9U33kQ7lC6rh6aHp/GQLsKnKlVL ouFWX1IuitkhzlGPmh6ChvxOzJE0U4uMYxdL7Kf8xZOKqh2jkVoRrtFSGNR5E9b2+jup ym+7dUzEbqeCjInNplNtVDLZIiZ6wKAJb8ygIYqw+N+dQTH4DTwOW8pBW2OSKBJh2cw9 yOlLGxKDnpeifMu0EfHPNxF6Wn96h1PlDmgJz37kNFqdCXiYieh2umd93C9+x+9ue6Ul j/AqcfOSyKlQvz6ZUmJS4FUswa3Fxxsj+t1M6lSSQWofVarnvPLAXr4lqaEOeJcMF/yZ yW6Q== X-Gm-Message-State: AOJu0Yw3EGDK6n64h2JIm/RUWjOx1pIKx12B1jdMXHg+BiNaEvzRNWbI r9kIKTfVSZBqXo3gYaDFipo= X-Google-Smtp-Source: AGHT+IFoA9q8+nleOsVaWYGKJMR5PUXVnKaxoWBcmjk+PdUFvUSMwibx6e8j5iJCbuEQcfjDcMuY/A== X-Received: by 2002:a05:6a00:1ad1:b0:6bd:a7ea:5c7e with SMTP id f17-20020a056a001ad100b006bda7ea5c7emr2751158pfv.11.1697826827696; Fri, 20 Oct 2023 11:33:47 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id z14-20020aa7990e000000b006b4ac8885b4sm1911345pff.14.2023.10.20.11.33.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 11:33:47 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, "Vishal Moola (Oracle)" Subject: [PATCH v3 5/5] mm/khugepaged: Convert collapse_pte_mapped_thp() to use folios Date: Fri, 20 Oct 2023 11:33:31 -0700 Message-Id: <20231020183331.10770-6-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231020183331.10770-1-vishal.moola@gmail.com> References: <20231020183331.10770-1-vishal.moola@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This removes 2 calls to compound_head() and helps convert khugepaged to use folios throughout. Previously, if the address passed to collapse_pte_mapped_thp() corresponded to a tail page, the scan would fail immediately. Using filemap_lock_folio() we get the corresponding folio back and try to operate on the folio instead. Signed-off-by: Vishal Moola (Oracle) Reviewed-by: Rik van Riel Reviewed-by: Yang Shi --- mm/khugepaged.c | 45 ++++++++++++++++++++------------------------- 1 file changed, 20 insertions(+), 25 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6a7184cd291b..bc2d8ff269c7 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1477,7 +1477,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, bool notified =3D false; unsigned long haddr =3D addr & HPAGE_PMD_MASK; struct vm_area_struct *vma =3D vma_lookup(mm, haddr); - struct page *hpage; + struct folio *folio; pte_t *start_pte, *pte; pmd_t *pmd, pgt_pmd; spinlock_t *pml =3D NULL, *ptl; @@ -1510,19 +1510,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, u= nsigned long addr, if (userfaultfd_wp(vma)) return SCAN_PTE_UFFD_WP; =20 - hpage =3D find_lock_page(vma->vm_file->f_mapping, + folio =3D filemap_lock_folio(vma->vm_file->f_mapping, linear_page_index(vma, haddr)); - if (!hpage) + if (IS_ERR(folio)) return SCAN_PAGE_NULL; =20 - if (!PageHead(hpage)) { - result =3D SCAN_FAIL; - goto drop_hpage; - } - - if (compound_order(hpage) !=3D HPAGE_PMD_ORDER) { + if (folio_order(folio) !=3D HPAGE_PMD_ORDER) { result =3D SCAN_PAGE_COMPOUND; - goto drop_hpage; + goto drop_folio; } =20 result =3D find_pmd_or_thp_or_none(mm, haddr, &pmd); @@ -1536,13 +1531,13 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, u= nsigned long addr, */ goto maybe_install_pmd; default: - goto drop_hpage; + goto drop_folio; } =20 result =3D SCAN_FAIL; start_pte =3D pte_offset_map_lock(mm, pmd, haddr, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ - goto drop_hpage; + goto drop_folio; =20 /* step 1: check all mapped PTEs are to the right huge page */ for (i =3D 0, addr =3D haddr, pte =3D start_pte; @@ -1567,7 +1562,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, * Note that uprobe, debugger, or MAP_PRIVATE may change the * page table, but the new page will not be a subpage of hpage. */ - if (hpage + i !=3D page) + if (folio_page(folio, i) !=3D page) goto abort; } =20 @@ -1582,7 +1577,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, * page_table_lock) ptl nests inside pml. The less time we hold pml, * the better; but userfaultfd's mfill_atomic_pte() on a private VMA * inserts a valid as-if-COWed PTE without even looking up page cache. - * So page lock of hpage does not protect from it, so we must not drop + * So page lock of folio does not protect from it, so we must not drop * ptl before pgt_pmd is removed, so uffd private needs pml taken now. */ if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) @@ -1606,7 +1601,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, continue; /* * We dropped ptl after the first scan, to do the mmu_notifier: - * page lock stops more PTEs of the hpage being faulted in, but + * page lock stops more PTEs of the folio being faulted in, but * does not stop write faults COWing anon copies from existing * PTEs; and does not stop those being swapped out or migrated. */ @@ -1615,7 +1610,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, goto abort; } page =3D vm_normal_page(vma, addr, ptent); - if (hpage + i !=3D page) + if (folio_page(folio, i) !=3D page) goto abort; =20 /* @@ -1634,8 +1629,8 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, =20 /* step 3: set proper refcount and mm_counters. */ if (nr_ptes) { - page_ref_sub(hpage, nr_ptes); - add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes); + folio_ref_sub(folio, nr_ptes); + add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes); } =20 /* step 4: remove empty page table */ @@ -1659,14 +1654,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, u= nsigned long addr, maybe_install_pmd: /* step 5: install pmd entry */ result =3D install_pmd - ? set_huge_pmd(vma, haddr, pmd, hpage) + ? set_huge_pmd(vma, haddr, pmd, &folio->page) : SCAN_SUCCEED; - goto drop_hpage; + goto drop_folio; abort: if (nr_ptes) { flush_tlb_mm(mm); - page_ref_sub(hpage, nr_ptes); - add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes); + folio_ref_sub(folio, nr_ptes); + add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes); } if (start_pte) pte_unmap_unlock(start_pte, ptl); @@ -1674,9 +1669,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, spin_unlock(pml); if (notified) mmu_notifier_invalidate_range_end(&range); -drop_hpage: - unlock_page(hpage); - put_page(hpage); +drop_folio: + folio_unlock(folio); + folio_put(folio); return result; } =20 --=20 2.40.1