From nobody Fri Sep 25 05:30:48 2026 Received: from fhigh-b4-smtp.messagingengine.com (fhigh-b4-smtp.messagingengine.com [202.12.124.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3AF481F09A5 for ; Wed, 16 Sep 2026 09:31:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.155 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551119; cv=none; b=YNTajeqU2czOY/k8kmrWVnuSZMw2qO7nhSoBOMamH0WITyiiHj1MUKMyAA6xSH5mxIuGIFlz3c3H1/6qk/jCfnHYbKvQBnhX6U42LZtvoPfu2T/aXsvKknfQLNvSH4ML6RaNXkRH4iCmAIE7rA4yDFfeVteyqLKoKdtB9atwVp4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551119; c=relaxed/simple; bh=RPRWO1cQwfat1Dp+ZIb02Kouxyvk7UB00r0B/YRnvf4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hYMbXhaAjYzCtqJAnRzcjf6qYSM16MNVbHzz/94/KkQX20o14bR4YYDwqq2TlU6+a/E35tjkJaagxlGBil162o4CPDmklU8W5ZB5KptNkGxghd2sDZyAbOtkWVJCRq18UxfWlcFP8Z2Yv3BQiYrO5DOOAixXQWxivjx0NHO8XOk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=Bhn8FHDC; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=kqxh5nkk; arc=none smtp.client-ip=202.12.124.155 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="Bhn8FHDC"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="kqxh5nkk" Received: from phl-compute-02.internal (phl-compute-02.internal [10.202.2.42]) by mailfhigh.stl.internal (Postfix) with ESMTP id 29E827A0089; Wed, 16 Sep 2026 05:31:52 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-02.internal (MEProxy); Wed, 16 Sep 2026 05:31:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789551112; x= 1789637512; bh=JI5Fa516GYJ3tjpfUWQYw4F0sEV0mYVe3RtQG+xuZIc=; b=B hn8FHDCByQ+GdJc0lRviJJLX9D2otwZZQX97ey9dS2ATFtcTTPGO8CVcBlwI5zxm UQXbIy0ul/lGFycowa4JG2GvA7JP4ex256VcW6wuPN9XCeyxkEwCHs4tbjkoxrRM fDw8NxTrNf5q1dyyKsM/Kk3kCUvMpNlHcw5FLD+NvWpXYNDAQa7LFSwAdqSC8mCV IbGSMNngJJRp3yk1ROTY+EmkwFzix7ofcC3Z6fsCLcvgSdPqsxgJnKaaFZDx0EMH Z0JRRvWvYdPeteMp1aNI1Fqm/YU3ofF9+sHtvhwf7pG2MPWbyRj7beWo9xntouun RGAjpUy6exbFm9c82N56g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789551112; x=1789637512; bh=J I5Fa516GYJ3tjpfUWQYw4F0sEV0mYVe3RtQG+xuZIc=; b=kqxh5nkkigRC8Y/hc ATPt7hojrfK36ZFYv2+lrVEJlENTtWvWz3RfQiOWYIqiQu+pqGrnXd+cMLxd5TO0 jT7c5iXWDTEsQ+jw+P3sWBlMcu7jFzreeV+s4j4ADfIiVxK43Rq79THKSP9ZHwJp +Ngp9oEvDlEbVfrXXG2XPUs4rwD9UfRsnp7DBI64AhfWlWOYZqeryf86FofHb2mR +B2btaq3Wptcum88aqgK7v7go7IlOl0Ac+7xHCQW/PLfFjaZjWaPhtOYJ5Muq6gb RFdbIb2CksdIvnSbGsdRyTlJgdCNrg4D/OPyXsFLvFiHZQEntI279eMQAH35R0qR 11itw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEOpyDQEfQMjLyldXBPAs1OiAaYjuycA/zprc2XA0sZE/gLY53LzRl6wisjmJsNcj shXh7L3amtO3W7Dj4gJ1brhV8efHSvbbvvisq8qNCVrU/hVVqWW2fJ8EDoeL9EfJt8J4fa r63UA+S0A+UN3c16swmwWrTj5sK0gOy7RYH1CN+xe4D6TTQ1ncSXBFkweWqSBfk0S52Say Dntsl8DI3TlZwS0igZd1iO+ltDo9n3qHIbwcYcvcbVUlu6Rxag/XK6s48+7ZTTYlI2aJe4 hkL/jN/KxntOAh39VLdXeraBW39brB+PkStvfnz0wMlwqVlAhF3nsCCzTv21LQWeSF/iiS UYsnT0sdXd1+mGrCuKiHZ1L2KKVC614nXF+Mf3lloTxsVreBc7gJjMAQmU8ZV618W/Izmn eSMlfn+hj1WiCgpnCISvQOOarVik4o2xz6ldwlcS7aOzOx+YUz57jrl+PRIDPdYamW4gmU mgMGM7p1wplBpaGjSG5Mup51bicu/FZPGAcyLig8Z1/UJV/8K36pYtigaLEXi6bfXTPZds KqposBmGHUxcr29cTxie9RGUGdtUORuOoZ0EQgZcLjb/l75VAja7SdiS3pvt8WFYx5xvb7 WSnK6J3WaVsI/SCJBTz15wRxePYKVDs2/8S3xa3vW7U5R8VtrfGrbBpRiMYw X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 16 Sep 2026 05:31:51 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v3 01/12] mm/khugepaged: drop redundant mm_struct pin in madvise_collapse() Date: Wed, 16 Sep 2026 10:31:28 +0100 Message-ID: <20260916093145.4022188-2-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260916093145.4022188-1-kirill@shutemov.name> References: <20260916093145.4022188-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" madvise_collapse() holds an mmgrab() reference across its work. It is redundant. Every caller already holds mm_users: - madvise(2) works on current->mm, which lives as long as the task is in the syscall; - process_madvise(2) reaches a remote mm through mm_access(), which takes an mm_users reference and holds it until the syscall returns; - io_uring passes current->mm; - DAMON takes one with get_task_mm() and drops it after the call. Drop the mmgrab()/mmdrop() pair. Assisted-by: LLM Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Signed-off-by: Kiryl Shutsemau (Meta) Acked-by: David Hildenbrand (Arm) --- mm/khugepaged.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index e13d233b9967..75c876acf527 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -3229,7 +3229,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, cc->is_khugepaged =3D false; cc->progress =3D 0; =20 - mmgrab(mm); lru_add_drain_all(); =20 for (addr =3D hstart; addr < hend; addr +=3D HPAGE_PMD_SIZE) { @@ -3285,7 +3284,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, } out_nolock: mmap_assert_locked(mm); - mmdrop(mm); kfree(cc); =20 return thps =3D=3D ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0 --=20 2.54.0 From nobody Fri Sep 25 05:30:48 2026 Received: from fhigh-b4-smtp.messagingengine.com (fhigh-b4-smtp.messagingengine.com [202.12.124.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E00A4A8FD3 for ; Wed, 16 Sep 2026 09:31:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.155 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551132; cv=none; b=DDEM5dgRCCL/ZIBkCzZ2PP+B5+e+X1SA/qGjiK4Hx2M1tp/QpEOhtxJmSifGrOwA8DNc7n3OWOGahq1qk4F3+FwN3i4LzZpDFP+LpAF0zy7Twa9Xd5qgZvIDgVVkbNTcwc/UbVT32xxrtHr8gNNvV3I08S03/cEf/eyT5gRA/vw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551132; c=relaxed/simple; bh=oZSNA87hap+8LZsY08iTLDC7668XQYqV9pSMcwVgaSA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=c4MblTEeUXPpC4KDDsm2y+ifvxP2DH6XichfI27kflzIPg3U9jw4+ZBjU3zmW7s6atjLZ9YZYPRvef2XaD+etAdIE4Ka9Qq9iFX5n7tm2G0lMJaeSZ+0CeSP5kXi/hgF2xsNJYE/zPwc42RsEQ+FOSNqTF1KD4lNeSUmnQW0t9E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=H9elMPvj; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=TunGcNhJ; arc=none smtp.client-ip=202.12.124.155 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="H9elMPvj"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="TunGcNhJ" Received: from phl-compute-06.internal (phl-compute-06.internal [10.202.2.46]) by mailfhigh.stl.internal (Postfix) with ESMTP id 875557A008D; Wed, 16 Sep 2026 05:31:54 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-06.internal (MEProxy); Wed, 16 Sep 2026 05:31:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789551114; x= 1789637514; bh=8kffjeZ0BbAQk8WNkrJMcG1UR27PMTOcUlnehRZ4hss=; b=H 9elMPvjJgiSJuYKab+0Y5yu987oSGjYEACb2aiWsREPZcMO/L24u339TXnzLy0sY BrwD8LAymA8MsPdbmyyGF2d3C1gsfNk963ky5/znSRAEQhTQTYaNp9CNg3YinTXk yu8XZPPYDeltgWfUTkXgpznVTVcGZxZiAUb7IPMCWTbs8/JBWuOSVed2TvyQjSXl N+njdR33I24+71rdDzxL9GLJ2gDbczy7xW5uvhXYz5JW22EwbFWR0Q+9F0TgucX9 /Dfz6yUPGbyyGPn2yQq+qsKPcTa5416T5tdjS4z3PuDf+39UjjWPMi/UFHCpblMn SQVqpxDaA/tcWp1erBM9Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789551114; x=1789637514; bh=8 kffjeZ0BbAQk8WNkrJMcG1UR27PMTOcUlnehRZ4hss=; b=TunGcNhJR4yEJp/xS 6RdkSwFK2RxkFqK0H3jeo0GIGoLEj6v/AmRXZJ0KKKfAW27OEsKMq/zz+SSqTog/ WUGhQbro6FWdR/IVZxmm+wnRhsOd8cCcA+PBgtHXwBHm01Gt6oGu6x/JGdUgYmdk vykveAJ72Q5DxTnuj5wIr4PQeL+TE2/UJsGJ7OIig17YuA+TkBHtLfGp1xdqgu7z CPyEtt3aWoM4RmPmXMq88lvAB5DzuB4E5c49FFMmD8KE5UeQO+NzD8qBCIYuiK+3 AOJ5LCW1j4UM59S8ZC3bXSyubN8fLKGarEyExOYe/eb+rjO5VPE4ijtCas3BcrqR CdzNg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEOpyDQEfQMjLyldXBPAs1OiAaYjuycA/zprc2XA0sZE/gLY53LzRl6wisjmJsNcj shXh7L3amtO3W7Dj4gJ1brhV8efHSvbbvvisq8qNCVrU/hVVqWW2fJ8EDoeL9EfJt8J4fa r63UA+S0A+UN3c16swmwWrTj5sK0gOy7RYH1CN+xe4D6TTQ1ncSXBFkweWqSBfk0S52Say Dntsl8DI3TlZwS0igZd1iO+ltDo9n3qHIbwcYcvcbVUlu6Rxag/XK6s48+7ZTTYlI2aJe4 hkL/jN/KxntOAh39VLdXeraBW39brB+PkStvfnz0wMlwqVlAhF3nsCCzTv21LQWeSF/ipH aSMoXw7Yrzq75z55RCiNMGl8Zjz1OaWOivnaPIGs8vL43WJY+7nhtZqGYOHlPycNBy82DP 8yITUE0MSeu1TrcwigSUkhmilfFGG6a8h46LQ/DEOfydYCycrcQ6fVhus7GOg+CfLPs/gZ SVT0jtv4TBzMKs8Vnlas4qXYQdYYkPaLGWkFHKpFPQ8hjyhbQZdQvMxvs76toKTR60zQFq c9HC++mznhIBchm9YPND7TW+Gt5T2VRACZnh6SClCmaY7wc9WPa4Ycgp3A+M5swF6nr/0E ouHUVqUa7HdcMJiCcGWJTSTBoeMKxCeXXw7GaXguOym/AcnT1kp2xzykiKUg X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 16 Sep 2026 05:31:53 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v3 02/12] mm/khugepaged: count collapses where khugepaged makes them Date: Wed, 16 Sep 2026 10:31:29 +0100 Message-ID: <20260916093145.4022188-3-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260916093145.4022188-1-kirill@shutemov.name> References: <20260916093145.4022188-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" collapse_single_pmd() bumps khugepaged_pages_collapsed for its caller, and tests cc->is_khugepaged to know whether it should: the counter belongs to the daemon, and MADV_COLLAPSE must not touch it. The daemon sees every result of every collapse it asks for, so it can keep its own counter without the shared path testing who called. Assisted-by: LLM Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Signed-off-by: Kiryl Shutsemau (Meta) Acked-by: David Hildenbrand (Arm) --- mm/khugepaged.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 75c876acf527..4979a93e3648 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -2815,10 +2815,8 @@ static enum scan_result collapse_single_pmd(unsigned= long addr, =20 mmap_assert_locked(mm); =20 - if (vma_is_anonymous(vma)) { - result =3D collapse_scan_pmd(mm, vma, addr, lock_dropped, cc); - goto end; - } + if (vma_is_anonymous(vma)) + return collapse_scan_pmd(mm, vma, addr, lock_dropped, cc); =20 file =3D get_file(vma->vm_file); pgoff =3D linear_page_index(vma, addr); @@ -2854,9 +2852,6 @@ static enum scan_result collapse_single_pmd(unsigned = long addr, result =3D SCAN_SUCCEED; mmap_read_unlock(mm); } -end: - if (cc->is_khugepaged && result =3D=3D SCAN_SUCCEED) - ++khugepaged_pages_collapsed; return result; } =20 @@ -2933,6 +2928,8 @@ static void collapse_scan_mm_slot(unsigned int progre= ss_max, =20 *result =3D collapse_single_pmd(khugepaged_scan.address, vma, &lock_dropped, cc); + if (*result =3D=3D SCAN_SUCCEED) + khugepaged_pages_collapsed++; /* move to next address */ khugepaged_scan.address +=3D HPAGE_PMD_SIZE; if (lock_dropped) --=20 2.54.0 From nobody Fri Sep 25 05:30:48 2026 Received: from fout-b8-smtp.messagingengine.com (fout-b8-smtp.messagingengine.com [202.12.124.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2479F486BB5 for ; Wed, 16 Sep 2026 09:32:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.151 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551134; cv=none; b=npMVRKjIGW3D5c/Sjbcl+NylnLsEVNxw7NhSxODi2ZhytBRND+r+wZQYljGGnvoAD66CvIPvKsZEnBoWwCrgqubwX2s4TC7Y5aR0Qtna6jIiH7I5W0GkdXrYXrh5wn/jtWeTRboff7MrXhl5QwhqRaht+NlVIstjpGe8ma3mtcY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551134; c=relaxed/simple; bh=FPRlk/3xXD2hlJWAg1kKXQRCO2yol8H3vn6v3bTyCKc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TDBmJlOuEjYBQ1f2uILUeKRYV9z3hxd4/h3m+hqt3XvG0Flq6NhE7DEa+VB5FT+gv5k4Ub21yUXCzNlJrBC8uin1vVcwjDFl9kDYRzGeDjdXKxD68pEuGO06JhMJEr791oqDps4ge61Rx+ix9Ic40PeCr7EXGQxOFXxA+Jz8ZyM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=YJYr34CC; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=F44WgGMx; arc=none smtp.client-ip=202.12.124.151 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="YJYr34CC"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="F44WgGMx" Received: from phl-compute-07.internal (phl-compute-07.internal [10.202.2.47]) by mailfout.stl.internal (Postfix) with ESMTP id E0E381D000C6; Wed, 16 Sep 2026 05:31:56 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-07.internal (MEProxy); Wed, 16 Sep 2026 05:31:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789551116; x= 1789637516; bh=n4UhpKBn4QA7xtpBWtOXdoBNDcNab8zz8WQs3EPLY2Y=; b=Y JYr34CCNqrZHl7Q6SppINXitbMkf1ZCY/c4ikwCXM4iDuFnpvAs6l1Rcr67jD9by /9ELVNnPHn+j/ziAVMCjQx6mO3ZCxGDn/bkYzgen3oQ4s2M72PP7VK8k6i6slmLQ zHu0wmqaj/TYs0UfOjibMDe5h4t5vULnlatOJu5s9ePMCwyjpnVl+Xr+E3kmW8zQ TNGpW6K26//aKsD7psRsslmfQ99KTRcCPj2NSEJFFLD3JQgoKxhFVrHS6yOIqiu1 wxh5gNN17GmIO43KqQxbUQ6PxRuw59zZQP/t7ZlwfOjhigSnOuC+kXWgQT4viqMa WPPnRihMYKgVQltDDeGmg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789551116; x=1789637516; bh=n 4UhpKBn4QA7xtpBWtOXdoBNDcNab8zz8WQs3EPLY2Y=; b=F44WgGMx9aAyFLuAd DX3sYkckpCW6T+iZKaJ30Vzo+fE5FCIrjlZ56ZuvLWYmxLGD05tCWXB9CwQJ++IK 2iNQYh1Df0PlLP3UR2PN24AB6DRyY0BuqnmWVY6O851JgNjz2aDRMlDLmY1NYNDs WbzTSW7rdfay6p2B9Z4mipH6P2BjIg9zDu/TF9kaJG5KRr5owNTUKKy2ey1i7q6X QuwpHfjYbGxqoRrblImV//Ag340T4vmLd3FwkqiezJimnZvGs7c8ewFcwB6JpjaI T+7JCadGB1Ihcmsuta0c/UcuJzVJMQOGBNkwHZAiLMo8UeVjnEyv/yHXAL2FeQDR I4+wg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEOpyDQEfQMjLyldXBPAs1OiAaYjuycA/zprc2XA0sZE/gLY53LzRl6wisjmJsNcj shXh7L3amtO3W7Dj4gJ1brhV8efHSvbbvvisq8qNCVrU/hVVqWW2fJ8EDoeL9EfJt8J4fa r63UA+S0A+UN3c16swmwWrTj5sK0gOy7RYH1CN+xe4D6TTQ1ncSXBFkweWqSBfk0S52Say Dntsl8DI3TlZwS0igZd1iO+ltDo9n3qHIbwcYcvcbVUlu6Rxag/XK6s48+7ZTTYlI2aJe4 hkL/jN/KxntOAh39VLdXeraBW39brB+PkStvfnz0wMlwqVlAhF3nsCCzTv21LQWeSF/iAB j2t+mXQn775ggvsPtC4n/CdAcpez2fjdkufQ13BoA0ie4gIaTNYLvpt3LO3C6eCly80Oe+ 5R8zcSqP7U+gnF8pyUj+uxBT3U9ceWIQJP0IB2oB5XmmJ1JQkegWkF5kmptEc2b29KTCtv YjEWiAlQ0F9yfOMFkAl3mL0XjabDpETvdumd8hWbEDShtaI7+foqMekZsxpJfWYszrTzrB wgrjD2ilfIfDinaWQgOTVnD+2/QhvZcLmdHcrXaSkRhMjU7RBfePy3+Ur23kphaUVzlgBC moMrk23PR1LJaTeUfHPRh2RGi3a3ZnhUy9KuWiFHCdriFlphP8AnaSUI1ypg X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 16 Sep 2026 05:31:56 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v3 03/12] mm/khugepaged: rename mthp_present_ptes bitmap to eligible_ptes Date: Wed, 16 Sep 2026 10:31:30 +0100 Message-ID: <20260916093145.4022188-4-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260916093145.4022188-1-kirill@shutemov.name> References: <20260916093145.4022188-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" The name says less than the bit means. A set bit means not only that the PTE is present, but also that it passed the other checks: uffd, lazyfree, anonymity, sharing. The PTE can be considered a collapse source. mthp_collapse() then reads the bitmap starting at the PMD order, so the bitmap is not specific to mTHP either. Name it for what a set bit means, and update the comments that named it. No functional change. Assisted-by: LLM Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Signed-off-by: Kiryl Shutsemau (Meta) Acked-by: David Hildenbrand (Arm) --- mm/khugepaged.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 4979a93e3648..081f705cfca2 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -115,8 +115,8 @@ struct collapse_control { /* nodemask for allocation fallback */ nodemask_t alloc_nmask; =20 - /* Each bit represents a single occupied (!none/zero) page. */ - DECLARE_BITMAP(mthp_present_ptes, MAX_PTRS_PER_PTE); + /* Each bit marks a PTE the scan accepted as a collapse source */ + DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); }; =20 /** @@ -627,7 +627,7 @@ static void collapse_control_init_scan(struct collapse_= control *cc) { memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); - bitmap_zero(cc->mthp_present_ptes, MAX_PTRS_PER_PTE); + bitmap_zero(cc->eligible_ptes, MAX_PTRS_PER_PTE); } =20 static void release_pte_folio(struct folio *folio) @@ -1482,15 +1482,15 @@ static unsigned int max_order_from_offset(unsigned = int offset) * mthp_collapse() consumes the bitmap that is generated during * collapse_scan_pmd() to determine what regions and mTHP orders fit best. * - * Each bit in cc->mthp_present_ptes represents a single occupied (!none/z= ero) - * page. We start at the PMD order and check if it is eligible for collaps= e; + * Each bit in cc->eligible_ptes marks a PTE the scan accepted as a collap= se + * source. We start at the PMD order and check if it is eligible for colla= pse; * if not, we check the left and right halves of the PTE page table we are * examining at a lower order. * - * For each of these, we determine how many PTE entries are occupied in the - * range of PTE entries we propose to collapse, then we compare this to a - * threshold number of PTE entries which would need to be occupied for a - * collapse to be permitted at that order (accounting for max_ptes_none). + * For each of these, we count the eligible PTEs in the range we propose to + * collapse, then we compare this to the number of eligible PTEs the range + * would need for a collapse to be permitted at that order (accounting for + * max_ptes_none). * * If a collapse is permitted, we attempt to collapse the PTE range into a * mTHP. @@ -1499,7 +1499,7 @@ static enum scan_result mthp_collapse(struct mm_struc= t *mm, unsigned long address, int referenced, int unmapped, struct collapse_control *cc, unsigned long enabled_orders) { - unsigned int nr_occupied_ptes, nr_ptes, max_ptes_none; + unsigned int nr_eligible_ptes, nr_ptes, max_ptes_none; enum scan_result last_result =3D SCAN_FAIL; int collapsed =3D 0; bool alloc_failed =3D false; @@ -1514,18 +1514,18 @@ static enum scan_result mthp_collapse(struct mm_str= uct *mm, goto next_order; =20 max_ptes_none =3D collapse_max_ptes_none(cc, NULL, order); - nr_occupied_ptes =3D bitmap_weight_from(cc->mthp_present_ptes, offset, + nr_eligible_ptes =3D bitmap_weight_from(cc->eligible_ptes, offset, offset + nr_ptes); =20 /* * Swap PTEs accepted during the scan are counted in @unmapped, - * not in the present-PTE bitmap. Account them for the PMD-order + * not in cc->eligible_ptes. Account them for the PMD-order * candidate. */ if (is_pmd_order(order)) - nr_occupied_ptes +=3D unmapped; + nr_eligible_ptes +=3D unmapped; =20 - if (nr_occupied_ptes >=3D nr_ptes - max_ptes_none) { + if (nr_eligible_ptes >=3D nr_ptes - max_ptes_none) { enum scan_result ret; =20 collapse_address =3D address + offset * PAGE_SIZE; @@ -1731,8 +1731,8 @@ static enum scan_result collapse_scan_pmd(struct mm_s= truct *mm, } } =20 - /* Set bit for occupied pages */ - __set_bit(i, cc->mthp_present_ptes); + /* The scan accepted this PTE as a collapse source */ + __set_bit(i, cc->eligible_ptes); /* * Record which node the original page is from and save this * information to cc->node_load[]. --=20 2.54.0 From nobody Fri Sep 25 05:30:48 2026 Received: from fout-b8-smtp.messagingengine.com (fout-b8-smtp.messagingengine.com [202.12.124.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9BC1E4A64DB for ; Wed, 16 Sep 2026 09:32:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.151 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551136; cv=none; b=mw7vHQW9PXiyFyOScrLDF4wO9tgUe9q7SymvWXLksypOc55VQbqWy8NjQHr0aZjaUC1Q2ZgjXCze5D1G8hbqb/TQg4NOUslj+h8ZHiQ652Zmvl9P07Bj2I7nhXJDpD1BBAh8HWjJhHRM708XenTpjQAG9xLDWphYOpHhPGHytu0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551136; c=relaxed/simple; bh=TKB6T8VtIMFCYymtjKb5LK/4Whb6YCWyLf1ODb2k4yM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sfz3B8cdarBm/fjyF0hiIv+9FwiF8NB72YWxidki7pdhYp1gGZjjSuABv/hqU7K0V6/HZcQwlB68gvb+c+7Y6tTcl5/0g8nN0zRS5sM0wQbdPq5h6TyWodGOzFjoEKMxFhijcl5tVekKlJ5sTeITr4sTexE+ec11j+N8yd4N6GM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=FZrwW60a; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=djeJHwrz; arc=none smtp.client-ip=202.12.124.151 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="FZrwW60a"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="djeJHwrz" Received: from phl-compute-05.internal (phl-compute-05.internal [10.202.2.45]) by mailfout.stl.internal (Postfix) with ESMTP id A1EE61D00084; Wed, 16 Sep 2026 05:31:59 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-05.internal (MEProxy); Wed, 16 Sep 2026 05:32:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789551119; x= 1789637519; bh=avMkfLVeU1+RrFVq7dqGuSLkfIFTUHtIkOYpYaIIbGQ=; b=F ZrwW60a1m5sQGSEnlfYHRzgoLkG3D5P3kbyYZypS4G3D1Y/sjXhpr+7XMk6fPFFc CB4jWSRtzSCkTUPHawA5jvvTbt131TRr573B2NaRjDQxcbLXdfsIAhB3E1DefPle bngTFW6EQS2zIxkMX6HgVaKLMT6uF4JOlDePTF/tK+W5IjXtNjjAG5FHX/c/Y+MP +qKbycwVvjWyhQ3Rw9SR4JwG78QD9Uqa0blpocyFxp6zXe8cx5vHRR/kGKaGYoPf ZwjzbG/oFXjNTSDozQ3OpU5DU0dAYIwpaJbr5RGlXlOZ+lj0lRDYz+T0LDczyUP3 h29BdUPt4biFOn7V9GDtA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789551119; x=1789637519; bh=a vMkfLVeU1+RrFVq7dqGuSLkfIFTUHtIkOYpYaIIbGQ=; b=djeJHwrzEVStD9qyQ ZxZOoZpNzwDjC1N9/9C41R68YCIusyuxXLL62GTE3BfGRGQr+3uCX6Z4STy9KQgZ v1lnlGMWKyIOq5DNRt0o6NO7AJos5JR6akblkt63RY0m2/jk1fOxdl4uYMapOiMS EWpgiMkf5L5wOTAot3xc8PABlHDOn+HNeBCV9fwtC2h+az2Shs6HJTRjkLyWZwtg LNuouYC245nIGuua80XJZ+2L+YWJ82WtIWuT25sxY4bAnKLevX+sQOW+0P6TZbWT e6bnxQlO0rCrOLdht18tOgpWLzFGgINRMWKJzIwDgzVt6cZmEN190Ope9RS+iBBR JTLyg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEOpyDQEfQMjLyldXBPAs1OiAaYjuycA/zprc2XA0sZE/gLY53LzRl6wisjmJsNcj shXh7L3amtO3W7Dj4gJ1brhV8efHSvbbvvisq8qNCVrU/hVVqWW2fJ8EDoeL9EfJt8J4fa r63UA+S0A+UN3c16swmwWrTj5sK0gOy7RYH1CN+xe4D6TTQ1ncSXBFkweWqSBfk0S52Say Dntsl8DI3TlZwS0igZd1iO+ltDo9n3qHIbwcYcvcbVUlu6Rxag/XK6s48+7ZTTYlI2aJe4 hkL/jN/KxntOAh39VLdXeraBW39brB+PkStvfnz0wMlwqVlAhF3nsCCzTv21LQWeSF/iTb MYrnHQOmaofpdATEhW+iKqNMndVUsZkNXro4u1ND3HQ+RLuEHTdkLWeBbFKVfegYNYqNlt X0JHLDrQLD/yJxqXyO2trj9CcN/tXL6neVs2190GRVKkh/LvWIaouJinauIO1qOlOd9xe3 jTDXmMnXkyZwXmnXqPrLOnKd/ybmml26i+ErYaHUo20TmQcmlcVzeB+TjmfO9B95GMfhsm duaCze2HijSX0CfsyWMswWKk99NTMjR5xQp6haMLPl2xKWlu2BgRSz4hIqt2zAWsJuTo3Q asDhRbeixb8zNswBS2kN4z6ROulafxWeWuICeF4opQx4KFDkTKvNGnlsyXYQ X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 16 Sep 2026 05:31:58 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v3 04/12] mm/collapse: add collapse.h for the collapse interface Date: Wed, 16 Sep 2026 10:31:31 +0100 Message-ID: <20260916093145.4022188-5-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260916093145.4022188-1-kirill@shutemov.name> References: <20260916093145.4022188-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" khugepaged.c holds both the users of collapse and the machinery that performs it. The daemon's scan loop, the sysfs tunables, MADV_COLLAPSE's entry point and the collapse itself all sit in one file and reach into each other freely. Nothing marks where a user ends and the engine begins. Start drawing that line. Add mm/collapse.h for what the two sides have to agree on: - enum scan_result - what the engine hands back; - struct collapse_control - the state a request carries. And two constants move with them: - KHUGEPAGED_MAX_PTES_LIMIT -> COLLAPSE_MAX_PTES_LIMIT; - KHUGEPAGED_MIN_MTHP_ORDER -> COLLAPSE_MIN_MTHP_ORDER. Neither is a fact about the daemon, so both lose the KHUGEPAGED_ prefix. No functional change. Assisted-by: LLM Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Signed-off-by: Kiryl Shutsemau (Meta) Acked-by: David Hildenbrand (Arm) --- MAINTAINERS | 1 + mm/collapse.h | 64 +++++++++++++++++++++++++++++++++++++++ mm/khugepaged.c | 79 ++++++++----------------------------------------- 3 files changed, 78 insertions(+), 66 deletions(-) create mode 100644 mm/collapse.h diff --git a/MAINTAINERS b/MAINTAINERS index 35262aa3e67e..36315972156b 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17434,6 +17434,7 @@ F: Documentation/admin-guide/mm/transhuge.rst F: include/linux/huge_mm.h F: include/linux/khugepaged.h F: include/trace/events/huge_memory.h +F: mm/collapse.h F: mm/huge_memory.c F: mm/khugepaged.c F: mm/mm_slot.h diff --git a/mm/collapse.h b/mm/collapse.h new file mode 100644 index 000000000000..b115034d9018 --- /dev/null +++ b/mm/collapse.h @@ -0,0 +1,64 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __MM_COLLAPSE_H +#define __MM_COLLAPSE_H + +#include +#include +#include +#include + +#define COLLAPSE_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1) +#define COLLAPSE_MIN_MTHP_ORDER 2 + +enum scan_result { + SCAN_FAIL, + SCAN_SUCCEED, + SCAN_NO_PTE_TABLE, + SCAN_PMD_MAPPED, + SCAN_EXCEED_NONE_PTE, + SCAN_EXCEED_SWAP_PTE, + SCAN_EXCEED_SHARED_PTE, + SCAN_PTE_NON_PRESENT, + SCAN_PTE_UFFD, + SCAN_PTE_MAPPED_HUGEPAGE, + SCAN_LACK_REFERENCED_PAGE, + SCAN_PAGE_NULL, + SCAN_SCAN_ABORT, + SCAN_PAGE_COUNT, + SCAN_PAGE_LRU, + SCAN_PAGE_LOCK, + SCAN_PAGE_ANON, + SCAN_PAGE_LAZYFREE, + SCAN_PAGE_COMPOUND, + SCAN_ANY_PROCESS, + SCAN_VMA_NULL, + SCAN_VMA_CHECK, + SCAN_ADDRESS_RANGE, + SCAN_DEL_PAGE_LRU, + SCAN_ALLOC_HUGE_PAGE_FAIL, + SCAN_CGROUP_CHARGE_FAIL, + SCAN_TRUNCATED, + SCAN_PAGE_HAS_PRIVATE, + SCAN_STORE_FAILED, + SCAN_COPY_MC, + SCAN_PAGE_FILLED, + SCAN_PAGE_DIRTY_OR_WRITEBACK, +}; + +struct collapse_control { + bool is_khugepaged; + + /* Num pages scanned per node */ + u32 node_load[MAX_NUMNODES]; + + /* Num pages scanned (see khugepaged_pages_to_scan) */ + unsigned int progress; + + /* nodemask for allocation fallback */ + nodemask_t alloc_nmask; + + /* Each bit marks a PTE the scan accepted as a collapse source */ + DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); +}; + +#endif /* __MM_COLLAPSE_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 081f705cfca2..8889f75cf45f 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -26,44 +26,10 @@ #include =20 #include +#include "collapse.h" #include "internal.h" -#include "page_alloc.h" #include "mm_slot.h" - -enum scan_result { - SCAN_FAIL, - SCAN_SUCCEED, - SCAN_NO_PTE_TABLE, - SCAN_PMD_MAPPED, - SCAN_EXCEED_NONE_PTE, - SCAN_EXCEED_SWAP_PTE, - SCAN_EXCEED_SHARED_PTE, - SCAN_PTE_NON_PRESENT, - SCAN_PTE_UFFD, - SCAN_PTE_MAPPED_HUGEPAGE, - SCAN_LACK_REFERENCED_PAGE, - SCAN_PAGE_NULL, - SCAN_SCAN_ABORT, - SCAN_PAGE_COUNT, - SCAN_PAGE_LRU, - SCAN_PAGE_LOCK, - SCAN_PAGE_ANON, - SCAN_PAGE_LAZYFREE, - SCAN_PAGE_COMPOUND, - SCAN_ANY_PROCESS, - SCAN_VMA_NULL, - SCAN_VMA_CHECK, - SCAN_ADDRESS_RANGE, - SCAN_DEL_PAGE_LRU, - SCAN_ALLOC_HUGE_PAGE_FAIL, - SCAN_CGROUP_CHARGE_FAIL, - SCAN_TRUNCATED, - SCAN_PAGE_HAS_PRIVATE, - SCAN_STORE_FAILED, - SCAN_COPY_MC, - SCAN_PAGE_FILLED, - SCAN_PAGE_DIRTY_OR_WRITEBACK, -}; +#include "page_alloc.h" =20 #define CREATE_TRACE_POINTS #include @@ -91,7 +57,6 @@ static DECLARE_WAIT_QUEUE_HEAD(khugepaged_wait); * * Note that these are only respected if collapse was initiated by khugepa= ged. */ -#define KHUGEPAGED_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1) unsigned int khugepaged_max_ptes_none __read_mostly; static unsigned int khugepaged_max_ptes_swap __read_mostly; static unsigned int khugepaged_max_ptes_shared __read_mostly; @@ -101,24 +66,6 @@ static DEFINE_READ_MOSTLY_HASHTABLE(mm_slots_hash, MM_S= LOTS_HASH_BITS); =20 static struct kmem_cache *mm_slot_cache __ro_after_init; =20 -#define KHUGEPAGED_MIN_MTHP_ORDER 2 - -struct collapse_control { - bool is_khugepaged; - - /* Num pages scanned per node */ - u32 node_load[MAX_NUMNODES]; - - /* Num pages scanned (see khugepaged_pages_to_scan) */ - unsigned int progress; - - /* nodemask for allocation fallback */ - nodemask_t alloc_nmask; - - /* Each bit marks a PTE the scan accepted as a collapse source */ - DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); -}; - /** * struct khugepaged_scan - cursor for scanning * @mm_head: the head of the mm list to scan @@ -267,7 +214,7 @@ static ssize_t max_ptes_none_store(struct kobject *kobj, unsigned long max_ptes_none; =20 err =3D kstrtoul(buf, 10, &max_ptes_none); - if (err || max_ptes_none > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_none > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; =20 khugepaged_max_ptes_none =3D max_ptes_none; @@ -292,7 +239,7 @@ static ssize_t max_ptes_swap_store(struct kobject *kobj, unsigned long max_ptes_swap; =20 err =3D kstrtoul(buf, 10, &max_ptes_swap); - if (err || max_ptes_swap > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_swap > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; =20 khugepaged_max_ptes_swap =3D max_ptes_swap; @@ -318,7 +265,7 @@ static ssize_t max_ptes_shared_store(struct kobject *ko= bj, unsigned long max_ptes_shared; =20 err =3D kstrtoul(buf, 10, &max_ptes_shared); - if (err || max_ptes_shared > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_shared > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; =20 khugepaged_max_ptes_shared =3D max_ptes_shared; @@ -378,19 +325,19 @@ static unsigned int collapse_max_ptes_none(struct col= lapse_control *cc, if (is_pmd_order(order)) return max_ptes_none; /* - * for mTHP collapse with the sysctl value set to KHUGEPAGED_MAX_PTES_LIM= IT, + * for mTHP collapse with the sysctl value set to COLLAPSE_MAX_PTES_LIMIT, * scale the maximum number of PTEs to the order of the collapse. */ - if (max_ptes_none =3D=3D KHUGEPAGED_MAX_PTES_LIMIT) + if (max_ptes_none =3D=3D COLLAPSE_MAX_PTES_LIMIT) return (1 << order) - 1; /* - * For mTHP collapse of values other than 0 or KHUGEPAGED_MAX_PTES_LIMIT, + * For mTHP collapse of values other than 0 or COLLAPSE_MAX_PTES_LIMIT, * emit a warning and return 0. */ if (max_ptes_none) pr_warn_once("mTHP collapse does not support max_ptes_none" " values other than 0 or %u, defaulting to 0.\n", - KHUGEPAGED_MAX_PTES_LIMIT); + COLLAPSE_MAX_PTES_LIMIT); return 0; } =20 @@ -476,7 +423,7 @@ int __init khugepaged_init(void) return -ENOMEM; =20 khugepaged_pages_to_scan =3D HPAGE_PMD_NR * 8; - khugepaged_max_ptes_none =3D KHUGEPAGED_MAX_PTES_LIMIT; + khugepaged_max_ptes_none =3D COLLAPSE_MAX_PTES_LIMIT; khugepaged_max_ptes_swap =3D HPAGE_PMD_NR / 8; khugepaged_max_ptes_shared =3D HPAGE_PMD_NR / 2; =20 @@ -1571,8 +1518,8 @@ static enum scan_result mthp_collapse(struct mm_struc= t *mm, * any smaller order enabled. When at the smallest order * we must always move to the next offset. */ - if (order > KHUGEPAGED_MIN_MTHP_ORDER && - (enabled_orders & GENMASK(order - 1, 0))) { + if (order > COLLAPSE_MIN_MTHP_ORDER && + (enabled_orders & GENMASK(order - 1, 0))) { order--; continue; } @@ -1636,7 +1583,7 @@ static enum scan_result collapse_scan_pmd(struct mm_s= truct *mm, * is then checked again in mthp_collapse() for each attempted order. */ if (enabled_orders !=3D BIT(HPAGE_PMD_ORDER)) - max_ptes_none =3D KHUGEPAGED_MAX_PTES_LIMIT; + max_ptes_none =3D COLLAPSE_MAX_PTES_LIMIT; =20 pte =3D pte_offset_map_lock(mm, pmd, start_addr, &ptl); if (!pte) { --=20 2.54.0 From nobody Fri Sep 25 05:30:48 2026 Received: from fout-b8-smtp.messagingengine.com (fout-b8-smtp.messagingengine.com [202.12.124.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E5A0E4A92F1 for ; Wed, 16 Sep 2026 09:32:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.151 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551138; cv=none; b=YMaUPg6lRWEk9MjcRpAdZ8UDdBRG5xD1otvNOKcirtmf5vhkoTBFPiwPQu91srqKEu8EmlGCFStGj92XFPpx+JnXHgpsF/cyfhrdY5xggI+7zz+irjYqYjmJ33hFFN1LWJLezApPRUhxWQCSsaRVO6dZP4c1qqmZtrwUjcBCBTw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551138; c=relaxed/simple; bh=0vsgegkCYa7op9rv9OivvWESbkik5p8dMrwmdH57rbs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ok5X1mYE8aJWTSYv7AA0Kh0USz1/JR64Gfa5M+Q6Yb8/0IHorYYUYy9GEMeqVNBDXFCNZLU07i8jle8s7G/4QQDvJArsINHg6DBQ41Q2T0/q5uMrRZi3Nw9++T8CXtpoGC8gqvFxugs/BRieQoLb/84g8gcEmWAISgulv3k0SeA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=lkTwwmgm; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=IfHcTkeL; arc=none smtp.client-ip=202.12.124.151 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="lkTwwmgm"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="IfHcTkeL" Received: from phl-compute-06.internal (phl-compute-06.internal [10.202.2.46]) by mailfout.stl.internal (Postfix) with ESMTP id 28C0F1D00098; Wed, 16 Sep 2026 05:32:02 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-06.internal (MEProxy); Wed, 16 Sep 2026 05:32:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789551122; x= 1789637522; bh=ICF9OPiue2yxmh8hP/ZUHQcKxFznzKsqrqQwWUt5ISk=; b=l kTwwmgmVxsBasNJphP2tCEw2dpkK2tdbDfRwh1Wn6LKKld4EuTVM8c1XZHRHkm+V THhHH3mh4DUAc1aBNuwdbcOzJDJqjZby1gn80iUHnDXsIm2zWN5kIYakLBKoDdci G9nzvooF83s2B0NlQG6jf62GaJ3DpFfRLOuY/cpCtlcz9Dv6ga7vax8DYx2DXHw5 UcYxqUPBFlrx0bz9Kt6d+HIK+e7R2wy+jYmcNldvYpdpQONVo3tmCiEb6kTNcxlf SBb6Ay9yKpIGFVh5X85F5XzGjrxMKESmxWVofwEWbWJGZIDso9jMhB6xua0qOQkD YzgW0Vj8BKPrWdt6oYtOQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789551122; x=1789637522; bh=I CF9OPiue2yxmh8hP/ZUHQcKxFznzKsqrqQwWUt5ISk=; b=IfHcTkeLQ9ubDaRHX xvvXfbj63w4k4jOZQ31eTezzSh+0fKBJPJOlgqQqeFw1cvxCggrNRs1nnRMUVJlw c/woxe7Cmtp+y77D1khAQRBQG+D8eoVmW1PkmtCiRfaO0ihzKYVMrRnrnKCB06cE R+qVSjSxl0qCWJzjOOD0YUyh98deQRE57vo4upJuuUhT5Dk7gK5f4lIjQZlBK8eI KdrujaJTAXyl283MIN1FfRyLf3OwZJEy7xxvkzbaE6PloPGmNVAtQw4DO2J4Km92 9Nh8yAawStltZzyoS91Er9RZPw+y9C5Cb7YPifrDMrpaXejNkkwhi7NxdxCsvaLY 5qItA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTE31qL+uaIHQKQPvpaICSxxs7QGWvIquuW8cKaqYQWKvt9pKCqy5bgk//TApl3wTl ad4FQSRAknjhMkzVuMAZNh0T12ywqROLvU4AnUXGVwd2kiN78Ar+1NR+fesCkATu1LALh1 4LvX/9qk4pAfO725ECjoAkI82P3z755hZxEkAV0An15e7Wr2vIbgLFM815SfkNmTU/eBmL ORNXO2IPaHkpkCtL9iOfg4dR3lpAdl3Xvq/Y35opAbs/zzbuDBIkBs44NTO3w5x5Bedurt thFl4kzzMjhv7UlAxOBz4vPRXQMYZm6skstaNA0N2ZkZEKrpjBUX5624wA8G4zEpHd89Vf 9T6n7sacdOaicBHl0a9Aiw51JNeA9NYlEYhEhIDx9KYOW0onA68maUQsM43/Hp6DjkU+Nu 72ekwCSo3DXmCWJ6wX+0Gd6ukbM49sUWHHNcWEe2Vr1f863dlft3DSoweFQ83vga9K0ojQ CYUQdYMItjRfsJ/xU+UsSJn4UiKGPcJsiAU3mczm9byTBs3kPOTyf/ZuA2Iqc3vN3ViBj6 gO9wjuvLNdFVs/TsSWTql3QMRjAHfcVzbmQP16qMqIAuCI0bGGcqhqa5juTxSnna4X1a37 vIlEH/Kvcys3ycZrZsrYEZ6/X0mz3FFd+rZMbzP7j/Xp40ENrkR1QO3B00FA X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 16 Sep 2026 05:32:01 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v3 05/12] mm/collapse: state what a collapse may do in the policy Date: Wed, 16 Sep 2026 10:31:32 +0100 Message-ID: <20260916093145.4022188-6-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260916093145.4022188-1-kirill@shutemov.name> References: <20260916093145.4022188-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" Tests scattered through the collapse path decide what a collapse is allowed to do by asking whether khugepaged started it. Between them they settle: - which VMAs are eligible, and how hard to try for a folio; - how many empty, swapped-out or shared PTEs a window may contain, and whether a sub-PMD window is held to a stricter rule than a PMD; - whether a range has to look used, and whether a MADV_FREE'd page is left alone; - whether the PMD is mapped as part of the request, and whether dirty pages are worth writing back and retrying. None of those is a fact about khugepaged. Each is something the caller decided before asking, and the collapse code should not have to look up who called to find out. Add struct collapse_policy for the caller to fill: khugepaged from its own settings, MADV_COLLAPSE from the fact that a user asked explicitly. Every test becomes a read of a field, and cc->is_khugepaged goes, having no reader left. khugepaged fills the policy once per scan pass, MADV_COLLAPSE once per call. That is the one change in behaviour. The max_ptes_* limits and the defrag setting behind the allocation mask are sampled once per pass rather than on every table. A table scanned early in a pass and one scanned late are then treated alike. collapse_file() also drops a NULL check on the collapse_control. It has one call site, reached only from collapse_single_pmd(), which dereferences cc unconditionally, so the check was already dead. Assisted-by: LLM Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Signed-off-by: Kiryl Shutsemau (Meta) --- mm/collapse.h | 31 ++++++++++++- mm/khugepaged.c | 114 ++++++++++++++++++++++++++---------------------- 2 files changed, 93 insertions(+), 52 deletions(-) diff --git a/mm/collapse.h b/mm/collapse.h index b115034d9018..7044dc71c7c2 100644 --- a/mm/collapse.h +++ b/mm/collapse.h @@ -45,8 +45,37 @@ enum scan_result { SCAN_PAGE_DIRTY_OR_WRITEBACK, }; =20 +/* What a collapse is allowed to do, decided by the caller that asks for i= t */ +struct collapse_policy { + /* Limits, stated per PMD; HPAGE_PMD_NR means "no limit" */ + unsigned int max_ptes_none; + unsigned int max_ptes_swap; + unsigned int max_ptes_shared; + + /* Take no swapped-out or shared PTE into a sub-PMD collapse */ + bool strict_sub_pmd; + + /* Leave clean lazyfree folios to reclaim rather than collapse them */ + bool skip_lazyfree; + + /* Refuse a range with no sign of use */ + bool require_referenced; + + /* Map the PMD over a file collapse instead of leaving it to a fault */ + bool install_pmd; + + /* Write dirty pages back and retry once instead of refusing them */ + bool writeback_dirty; + + /* How hard to try for a destination folio */ + gfp_t gfp; + + /* Which VMAs are eligible, as thp_vma_allowable_orders() spells it */ + enum tva_type tva_type; +}; + struct collapse_control { - bool is_khugepaged; + struct collapse_policy policy; =20 /* Num pages scanned per node */ u32 node_load[MAX_NUMNODES]; diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 8889f75cf45f..cb08789b2d38 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -314,15 +314,12 @@ static bool pte_none_or_zero(pte_t pte) static unsigned int collapse_max_ptes_none(struct collapse_control *cc, struct vm_area_struct *vma, unsigned int order) { - const unsigned int max_ptes_none =3D khugepaged_max_ptes_none; + const unsigned int max_ptes_none =3D cc->policy.max_ptes_none; =20 if (vma && userfaultfd_armed(vma)) return 0; - /* for MADV_COLLAPSE, allow any empty/shared zeropage PTEs */ - if (!cc->is_khugepaged) - return HPAGE_PMD_NR; - /* for PMD collapse, respect the user defined maximum */ - if (is_pmd_order(order)) + /* The limit as given, at the PMD order and wherever it is not capped */ + if (is_pmd_order(order) || !cc->policy.strict_sub_pmd) return max_ptes_none; /* * for mTHP collapse with the sysctl value set to COLLAPSE_MAX_PTES_LIMIT, @@ -354,19 +351,12 @@ static unsigned int collapse_max_ptes_shared(struct c= ollapse_control *cc, unsigned int order) { /* - * For MADV_COLLAPSE, do not restrict the number of PTEs that map shared - * anonymous pages. + * A sub-PMD window held to the strict rule takes no shared page at all: + * an mTHP is not worth the CoW-breaking. */ - if (!cc->is_khugepaged) - return HPAGE_PMD_NR; - /* - * for mTHP collapse do not allow collapsing anonymous memory pages that - * are shared between processes. - */ - if (!is_pmd_order(order)) + if (!is_pmd_order(order) && cc->policy.strict_sub_pmd) return 0; - /* for PMD collapse, respect the user defined maximum */ - return khugepaged_max_ptes_shared; + return cc->policy.max_ptes_shared; } =20 /** @@ -382,16 +372,12 @@ static unsigned int collapse_max_ptes_swap(struct col= lapse_control *cc, unsigned int order) { /* - * For MADV_COLLAPSE, do not restrict the number PTEs entries or - * pagecache entries that are non-present. + * A sub-PMD window held to the strict rule takes nothing non-present: + * reading pages back to build an mTHP is not worth the latency. */ - if (!cc->is_khugepaged) - return HPAGE_PMD_NR; - /* for mTHP collapse do not allow any non-present PTEs or pagecache entri= es */ - if (!is_pmd_order(order)) + if (!is_pmd_order(order) && cc->policy.strict_sub_pmd) return 0; - /* for PMD collapse, respect the user defined maximum */ - return khugepaged_max_ptes_swap; + return cc->policy.max_ptes_swap; } =20 int hugepage_madvise(struct vm_area_struct *vma, @@ -678,7 +664,7 @@ static enum scan_result __collapse_huge_page_isolate(st= ruct vm_area_struct *vma, * If the vma has the VM_DROPPABLE flag, the collapse will * preserve the lazyfree property without needing to skip. */ - if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) && + if (cc->policy.skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) && folio_test_lazyfree(folio) && !pte_dirty(pteval)) { result =3D SCAN_PAGE_LAZYFREE; goto out; @@ -767,12 +753,12 @@ static enum scan_result __collapse_huge_page_isolate(= struct vm_area_struct *vma, if (folio_test_large(folio)) list_add_tail(&folio->lru, compound_pagelist); next: - if (cc->is_khugepaged && + if (cc->policy.require_referenced && folio_pte_referenced(folio, vma, addr, pteval)) referenced++; } =20 - if (unlikely(cc->is_khugepaged && !referenced)) { + if (unlikely(cc->policy.require_referenced && !referenced)) { result =3D SCAN_LACK_REFERENCED_PAGE; } else { result =3D SCAN_SUCCEED; @@ -938,9 +924,7 @@ static void khugepaged_alloc_sleep(void) remove_wait_queue(&khugepaged_wait, &wait); } =20 -static struct collapse_control khugepaged_collapse_control =3D { - .is_khugepaged =3D true, -}; +static struct collapse_control khugepaged_collapse_control; =20 static bool collapse_scan_abort(int nid, struct collapse_control *cc) { @@ -976,6 +960,36 @@ static inline gfp_t alloc_hugepage_khugepaged_gfpmask(= void) return khugepaged_defrag() ? GFP_TRANSHUGE : GFP_TRANSHUGE_LIGHT; } =20 +/* khugepaged collapses on its own initiative, so it obeys its own setting= s */ +static void collapse_policy_khugepaged(struct collapse_policy *p) +{ + p->max_ptes_none =3D READ_ONCE(khugepaged_max_ptes_none); + p->max_ptes_swap =3D READ_ONCE(khugepaged_max_ptes_swap); + p->max_ptes_shared =3D READ_ONCE(khugepaged_max_ptes_shared); + p->strict_sub_pmd =3D true; + p->skip_lazyfree =3D true; + p->require_referenced =3D true; + p->install_pmd =3D false; + p->writeback_dirty =3D false; + p->gfp =3D alloc_hugepage_khugepaged_gfpmask(); + p->tva_type =3D TVA_KHUGEPAGED; +} + +/* MADV_COLLAPSE was asked for explicitly, so it is not held to those */ +static void collapse_policy_forced(struct collapse_policy *p) +{ + p->max_ptes_none =3D HPAGE_PMD_NR; + p->max_ptes_swap =3D HPAGE_PMD_NR; + p->max_ptes_shared =3D HPAGE_PMD_NR; + p->strict_sub_pmd =3D false; + p->skip_lazyfree =3D false; + p->require_referenced =3D false; + p->install_pmd =3D true; + p->writeback_dirty =3D true; + p->gfp =3D GFP_TRANSHUGE; + p->tva_type =3D TVA_FORCED_COLLAPSE; +} + #ifdef CONFIG_NUMA static int collapse_find_target_node(struct collapse_control *cc) { @@ -1013,8 +1027,7 @@ static enum scan_result hugepage_vma_revalidate(struc= t mm_struct *mm, unsigned l struct collapse_control *cc, unsigned int order) { struct vm_area_struct *vma; - enum tva_type type =3D cc->is_khugepaged ? TVA_KHUGEPAGED : - TVA_FORCED_COLLAPSE; + enum tva_type type =3D cc->policy.tva_type; =20 if (unlikely(collapse_test_exit_or_disable(mm))) return SCAN_ANY_PROCESS; @@ -1197,8 +1210,7 @@ static enum scan_result __collapse_huge_page_swapin(s= truct mm_struct *mm, static enum scan_result alloc_charge_folio(struct folio **foliop, struct m= m_struct *mm, struct collapse_control *cc, unsigned int order) { - gfp_t gfp =3D (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() : - GFP_TRANSHUGE); + gfp_t gfp =3D cc->policy.gfp; int node =3D collapse_find_target_node(cc); struct folio *folio; =20 @@ -1551,7 +1563,7 @@ static enum scan_result collapse_scan_pmd(struct mm_s= truct *mm, const unsigned int max_ptes_shared =3D collapse_max_ptes_shared(cc, HPAGE= _PMD_ORDER); const unsigned int max_ptes_swap =3D collapse_max_ptes_swap(cc, HPAGE_PMD= _ORDER); unsigned int max_ptes_none =3D collapse_max_ptes_none(cc, vma, HPAGE_PMD_= ORDER); - enum tva_type tva_flags =3D cc->is_khugepaged ? TVA_KHUGEPAGED : TVA_FORC= ED_COLLAPSE; + enum tva_type tva_flags =3D cc->policy.tva_type; pmd_t *pmd; pte_t *pte, *_pte, pteval; int i; @@ -1651,7 +1663,7 @@ static enum scan_result collapse_scan_pmd(struct mm_s= truct *mm, * If the vma has the VM_DROPPABLE flag, the collapse will * preserve the lazyfree property without needing to skip. */ - if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) && + if (cc->policy.skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) && folio_test_lazyfree(folio) && !pte_dirty(pteval)) { result =3D SCAN_PAGE_LAZYFREE; failed_pfn =3D folio_pfn(folio); @@ -1717,13 +1729,13 @@ static enum scan_result collapse_scan_pmd(struct mm= _struct *mm, goto out_unmap; } =20 - if (cc->is_khugepaged && + if (cc->policy.require_referenced && folio_pte_referenced(folio, vma, addr, pteval)) referenced++; } - if (cc->is_khugepaged && - (!referenced || - (unmapped && referenced < HPAGE_PMD_NR / 2))) { + if (cc->policy.require_referenced && + (!referenced || + (unmapped && referenced < HPAGE_PMD_NR / 2))) { result =3D SCAN_LACK_REFERENCED_PAGE; } else { result =3D SCAN_SUCCEED; @@ -2582,11 +2594,11 @@ static enum scan_result collapse_file(struct mm_str= uct *mm, unsigned long addr, xas_unlock_irq(&xas); =20 /* - * Remove pte page tables, so we can re-fault the page as huge. - * If MADV_COLLAPSE, adjust result to call try_collapse_pte_mapped_thp(). + * Remove pte page tables, so we can re-fault the page as huge. A + * caller that wants the PMD mapped now is told to go and do that. */ retract_page_tables(mapping, start); - if (cc && !cc->is_khugepaged) + if (cc->policy.install_pmd) result =3D SCAN_PTE_MAPPED_HUGEPAGE; folio_unlock(new_folio); =20 @@ -2773,11 +2785,8 @@ static enum scan_result collapse_single_pmd(unsigned= long addr, retry: result =3D collapse_scan_file(mm, addr, file, pgoff, cc); =20 - /* - * For MADV_COLLAPSE, when encountering dirty pages, try to writeback, - * then retry the collapse one time. - */ - if (!cc->is_khugepaged && result =3D=3D SCAN_PAGE_DIRTY_OR_WRITEBACK && + /* Dirty pages are worth a writeback and one more try, if asked for */ + if (cc->policy.writeback_dirty && result =3D=3D SCAN_PAGE_DIRTY_OR_WRITEB= ACK && !triggered_wb && mapping_can_writeback(file->f_mapping)) { const loff_t lstart =3D (loff_t)pgoff << PAGE_SHIFT; const loff_t lend =3D lstart + HPAGE_PMD_SIZE - 1; @@ -2794,7 +2803,7 @@ static enum scan_result collapse_single_pmd(unsigned = long addr, result =3D SCAN_ANY_PROCESS; else result =3D try_collapse_pte_mapped_thp(mm, addr, - !cc->is_khugepaged); + cc->policy.install_pmd); if (result =3D=3D SCAN_PMD_MAPPED) result =3D SCAN_SUCCEED; mmap_read_unlock(mm); @@ -2943,6 +2952,9 @@ static void khugepaged_do_scan(struct collapse_contro= l *cc) =20 lru_add_drain_all(); =20 + /* One policy for the whole pass, so every table is treated the same */ + collapse_policy_khugepaged(&cc->policy); + cc->progress =3D 0; while (true) { cond_resched(); @@ -3170,7 +3182,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, cc =3D kmalloc_obj(*cc); if (!cc) return -ENOMEM; - cc->is_khugepaged =3D false; + collapse_policy_forced(&cc->policy); cc->progress =3D 0; =20 lru_add_drain_all(); --=20 2.54.0 From nobody Fri Sep 25 05:30:48 2026 Received: from fhigh-b4-smtp.messagingengine.com (fhigh-b4-smtp.messagingengine.com [202.12.124.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3ACAE493621 for ; Wed, 16 Sep 2026 09:32:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.155 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551142; cv=none; b=PEId5lX4XS2X/62rXQNeN+xy6b+LVlqh+B4vH3vtWSPbdbmkuQ9I5wxoZFzB3dZZvo7Dm7iNflj9Y8nt4IJ35C1uO21rO5sKA5wYMTAb2ZtFvWys61+4U4BoWuVnD1rWyQDoSFxTZmhzTQjojXoTUA9fswRaUvZQMNJR2baTFQM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551142; c=relaxed/simple; bh=pSgO8D8VBlexfqJ0c/bp/gYa9Eq3i5ovQsr5KBYuY4A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jDjOSe3saiRE1lKYP1YASOrDYL3WKOvXDs/bju9MXNoSTzF4h5s0amTzuAyPvI7MpnNhF9FGngKWrW4XqCl0Hr9gZH+6Vns5ZdV4g1gNJA7Do/mJ1jKlq4p23AIe62KZF5LftnEyllHjmbRt4Os53z19gUf6bKXFe5Gz9yASyGA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=mx2F1LbV; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=XmcWBwnG; arc=none smtp.client-ip=202.12.124.155 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="mx2F1LbV"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="XmcWBwnG" Received: from phl-compute-04.internal (phl-compute-04.internal [10.202.2.44]) by mailfhigh.stl.internal (Postfix) with ESMTP id 5BE697A0089; Wed, 16 Sep 2026 05:32:04 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-04.internal (MEProxy); Wed, 16 Sep 2026 05:32:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789551124; x= 1789637524; bh=zPMRtic/KA0rSWQIkttR+MrY+d6BRRufVKkb5veXeh0=; b=m x2F1LbVWRpmYR0FTgvLAxaNn2+T2bxQAAHG6aICs3QArEaVfo5GiLexoOBo62f/V 7iifFU/Wlb3dqTDwTI4jB3k8zO2Bi4K0qTpZBJaleA/GFI/u2nTm/yJVhBnxB09+ bwqEwsYRpZZFQjngVASQB3rt1XXtWQNK6FrFr1z7tnpLOKtGK9dGlrD0X+fUlbma MoYNxWj+odmyakcZGoVfJoEUb5Xl/KSSu9NOwgyLOrXJ+6gTM89AgLuxmD9F3XZ5 x2UBwSyaQ9wXYL0YAnL/ZriuA/hx2GAdylI5wrXkkQaHkwQQxUE+1tQBXPhDpYuY 2T+kxnqLBWFLJ7hed38pQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789551124; x=1789637524; bh=z PMRtic/KA0rSWQIkttR+MrY+d6BRRufVKkb5veXeh0=; b=XmcWBwnGDwu1l7iHr WPdL3luoZRRXDxnJdTlw+a8DWUzav1l88PeG5L5VO/TSqMmntZ4kObtAtL7W2Q5v 8J/llGpCwdyqYoGo3d/COV8fO5pV+jcAa6lh8j8CCvoCoveWglru3zW3LELtfng9 XnOepXKQzG/p3mGTPAxrqYSUuWLT071ftZihjIAlIxx6cgszU5zmQw0qJY4UbDbx oR6AT4qBrEAQMr1jgQxlsQdKzPsGTIyE4EWKgAoGKiXlWV2WoMp5jsLJAqn3Tk0Y syJIDh4BD+zD9+bLZbWkI0DjhoqB5i7O/kq1s7IJPSvp7B1u0NGrcA6W4KrNWH/A g1sLA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEOpyDQEfQMjLyldXBPAs1OiAaYjuycA/zprc2XA0sZE/gLY53LzRl6wisjmJsNcj shXh7L3amtO3W7Dj4gJ1brhV8efHSvbbvvisq8qNCVrU/hVVqWW2fJ8EDoeL9EfJt8J4fa r63UA+S0A+UN3c16swmwWrTj5sK0gOy7RYH1CN+xe4D6TTQ1ncSXBFkweWqSBfk0S52Say Dntsl8DI3TlZwS0igZd1iO+ltDo9n3qHIbwcYcvcbVUlu6Rxag/XK6s48+7ZTTYlI2aJe4 hkL/jN/KxntOAh39VLdXeraBW39brB+PkStvfnz0wMlwqVlAhF3nsCCzTv21LQWeSF/idU l90nh564IAY/gtWrLk+kl+aUclafmy+2TTeoiPxsUwjQtvuB08Yj7GKIp/DEv79v7HQN2z 4Oz0r6NYLbgfbR4716KqKJZYkbkDiTYm1lvYjbArOgZqUc/9QdfhHsq4iQ4GCbUKeFKvRZ 6NVlkYGPjQqi0+SmjTazQrIoAQ+tAE4n9wxGKXs/bv5faKSlnIBkz/UryooHKvu2weMhM9 KdWzoEYl57Ubj2Z2xRHcJZK9oMdIWueTYTTybq1xGHVkdw9j5PlfHhipBuv4iJM6EEzLZZ e+u2F3O9EPCJU+a0r3eoVM63sJU5MRxU3/TdAD63s/H2Bl0X2JFaeTNy/YkA X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 16 Sep 2026 05:32:03 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v3 06/12] mm/collapse: drop the collapse_possible() wrapper Date: Wed, 16 Sep 2026 10:31:33 +0100 Message-ID: <20260916093145.4022188-7-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260916093145.4022188-1-kirill@shutemov.name> References: <20260916093145.4022188-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" collapse_possible() only forwards to collapse_possible_orders() and turns its mask into a bool. Its three callers can test the mask themselves. No functional change. Assisted-by: LLM Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Signed-off-by: Kiryl Shutsemau (Meta) Acked-by: David Hildenbrand (Arm) --- mm/khugepaged.c | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index cb08789b2d38..62bafe509bb7 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -512,17 +512,11 @@ static unsigned long collapse_possible_orders(struct = vm_area_struct *vma, return thp_vma_allowable_orders(vma, vm_flags, tva_flags, orders); } =20 -static bool collapse_possible(struct vm_area_struct *vma, - vm_flags_t vm_flags, enum tva_type tva_flags) -{ - return collapse_possible_orders(vma, vm_flags, tva_flags); -} - void khugepaged_enter_vma(struct vm_area_struct *vma, vm_flags_t vm_flags) { - if (!mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm) && hugepage_enabled() - && collapse_possible(vma, vm_flags, TVA_KHUGEPAGED)) + if (!mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm) && hugepage_enabled() && + collapse_possible_orders(vma, vm_flags, TVA_KHUGEPAGED)) __khugepaged_enter(vma->vm_mm); } =20 @@ -2857,7 +2851,8 @@ static void collapse_scan_mm_slot(unsigned int progre= ss_max, cc->progress++; break; } - if (!collapse_possible(vma, vma->vm_flags, TVA_KHUGEPAGED)) { + if (!collapse_possible_orders(vma, vma->vm_flags, + TVA_KHUGEPAGED)) { cc->progress++; continue; } @@ -3170,7 +3165,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, BUG_ON(vma->vm_start > start); BUG_ON(vma->vm_end < end); =20 - if (!collapse_possible(vma, vma->vm_flags, TVA_FORCED_COLLAPSE)) + if (!collapse_possible_orders(vma, vma->vm_flags, TVA_FORCED_COLLAPSE)) return -EINVAL; =20 hstart =3D ALIGN(start, HPAGE_PMD_SIZE); --=20 2.54.0 From nobody Fri Sep 25 05:30:48 2026 Received: from fout-b8-smtp.messagingengine.com (fout-b8-smtp.messagingengine.com [202.12.124.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EC2764A5C5B for ; Wed, 16 Sep 2026 09:32:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.151 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551142; cv=none; b=ZNVeMtcwt+RBILzKOjl9sxZ7RzeaqRbQLaKhusTuq9E7UoO5oSZcW4ZQYFkrmSZEZ/MeF4aTugMs06AqCf0Je+kOP713H6hmwPfLT8bpflL+CZWbBMjxTWv6YoYHaHMyUUOKLHBcz1v4zb++4DQuN+D99ooeRDh6nPbYybdhmAM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551142; c=relaxed/simple; bh=rm3iqrDU3ML5D5v309pWzhdg4yXMwkstkKS+oIbSZeA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HX7xi7NRpzCPWMLAA06zvBEDFoegcsOitDEGH9loilYOppfp7WANpS/pIZqKeTlZjgIMFthLTRw/ZfCOmAOqfkU6QsLg5zUOfqevBU37Th+XeVH/++Qato37RrDdRs2LyEMQFlaChk/BzoWrRx4XJmretELN2KnHhTBHzLHkXRk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=lYA4tGDu; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=M/lfS48w; arc=none smtp.client-ip=202.12.124.151 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="lYA4tGDu"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="M/lfS48w" Received: from phl-compute-08.internal (phl-compute-08.internal [10.202.2.48]) by mailfout.stl.internal (Postfix) with ESMTP id 9AE901D000C0; Wed, 16 Sep 2026 05:32:06 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-08.internal (MEProxy); Wed, 16 Sep 2026 05:32:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789551126; x= 1789637526; bh=062VynHgCC8xVPC31q5/ECQim7w7ZN4PLXxFdT/y61M=; b=l YA4tGDuaEUXdSshUOBQpg3ITJw+gJfPUgJKndiA5HJok4wVPE0j1y8jR29V1A7xI mTCDuV3oWwpk4mX4BSPv0Pn59ZIb+lnbWWvM981rrrH75Z7PHdBN+c1+3Y0+LWi8 hwqPghOZ0mrukMSgaq6+sKvY6NVtaJfCk4sDUNt52vsKiFdnBq/WhlG2vs8Cp6Zj qAiKbgW5RXlpTgsJbZhV85h+btXre8hOYJimVooZATFVQ3LCH2X1W5uBX9bxCS1R AEN1oRzp0Zb9Xsfs2cmmxWd9ksStBmYf0FyOnLZIAOfVFKNpsAITHtwU9urVL5TT cUxfwG/td7y1LX8MmeRsA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789551126; x=1789637526; bh=0 62VynHgCC8xVPC31q5/ECQim7w7ZN4PLXxFdT/y61M=; b=M/lfS48w1vBKqGkGG Xy0GzfOd7uHnXGTWI1kYRTko4zSYLmvTQ/8JEv1Pdci7dC3U6XZG+tROk8MXxszJ RS/lYFft4QOzek1cPY7Ge7tZmji4DMS6oc0isresHnIVVsNA4Bmn8gQEDAareaDw 3s1jIh1685Zn18D68dhYe8MiqdGrgDHtSyCDuD6/uU1iWkaCZeqLLu+92LhjMaTc QF6nmr9YNxVYZuPKh3qk4Pb+RfKJzOXL4zGu85o5wUr3VqNDLfTeR2RWF2Xjtgie BjeL2QCubnKsImz83eMv+rzY6IssvZGh+1RHxwLmRdQACdK4a+yvoE2mfHj3Rn52 VNWzg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEOpyDQEfQMjLyldXBPAs1OiAaYjuycA/zprc2XA0sZE/gLY53LzRl6wisjmJsNcj shXh7L3amtO3W7Dj4gJ1brhV8efHSvbbvvisq8qNCVrU/hVVqWW2fJ8EDoeL9EfJt8J4fa r63UA+S0A+UN3c16swmwWrTj5sK0gOy7RYH1CN+xe4D6TTQ1ncSXBFkweWqSBfk0S52Say Dntsl8DI3TlZwS0igZd1iO+ltDo9n3qHIbwcYcvcbVUlu6Rxag/XK6s48+7ZTTYlI2aJe4 hkL/jN/KxntOAh39VLdXeraBW39brB+PkStvfnz0wMlwqVlAhF3nsCCzTv21LQWeSF/iJl lc3V5TABAj6mTg4ACFiNaYBUz8Ii5DwjGy2GmhXthAfksiDj7e70MT3WpdoHVOuJI1O8Qm E8NFcOwg77uMOoboAwIwRFbkhNYYpy+NvnBA+GtXFqiw+RNCVT+uCpatG2VsXQn1Z/liYj 9GiuycSPw1uxc/sFOEUyq/kTFTl8tSgLdCzpbE+8A002Mwn9A00DlApggnDWYU1+5Kcjig XVgS45Kf2ceX/eI2JYS6H55MiSAxmF5DMAvd8iiKw0nLeBBVSMANNUgq6MlCA+tk1Iptix DikWe8ZMtvSRRy+crQHb1ebR7xk7I68EDh/Npni1h6aiblqQL5dW1nfQ5F9w X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 16 Sep 2026 05:32:05 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v3 07/12] mm/collapse: name the per-table scan reset for what it resets Date: Wed, 16 Sep 2026 10:31:34 +0100 Message-ID: <20260916093145.4022188-8-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260916093145.4022188-1-kirill@shutemov.name> References: <20260916093145.4022188-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" collapse_control_init_scan() resets what one scan accumulates: the node load, the allocation nodemask and the eligible-PTE bitmap. It runs before every PTE table a scan is given, not once per control, so its name points at the wrong thing. Call it collapse_scan_reset(). Preparation for giving a control a real init and release, run once each for a whole series of scans. Two names a word apart would then stand for two different jobs. No functional change. Assisted-by: LLM Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Signed-off-by: Kiryl Shutsemau (Meta) Acked-by: David Hildenbrand (Arm) --- mm/khugepaged.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 62bafe509bb7..ff1756c1255d 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -550,7 +550,7 @@ void __khugepaged_exit(struct mm_struct *mm) } } =20 -static void collapse_control_init_scan(struct collapse_control *cc) +static void collapse_scan_reset(struct collapse_control *cc) { memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); @@ -1579,7 +1579,7 @@ static enum scan_result collapse_scan_pmd(struct mm_s= truct *mm, goto out; } =20 - collapse_control_init_scan(cc); + collapse_scan_reset(cc); =20 enabled_orders =3D collapse_possible_orders(vma, vma->vm_flags, tva_flags= ); =20 @@ -2657,7 +2657,7 @@ static enum scan_result collapse_scan_file(struct mm_= struct *mm, =20 present =3D 0; swap =3D 0; - collapse_control_init_scan(cc); + collapse_scan_reset(cc); rcu_read_lock(); xas_for_each(&xas, folio, start + HPAGE_PMD_NR - 1) { if (xas_retry(&xas, folio)) --=20 2.54.0 From nobody Fri Sep 25 05:30:48 2026 Received: from fhigh-b4-smtp.messagingengine.com (fhigh-b4-smtp.messagingengine.com [202.12.124.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3590E4A4836 for ; Wed, 16 Sep 2026 09:32:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.155 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551152; cv=none; b=d1XcDJG3msRbQxEgQKxwWISz4jJxyokNznf0NuP8jW1GNbmkKsuNNiD9+jY/EllpDqvYwRG1W7roBIUMzrggmEN6CalCKKJ8D/h0Du/r6LGnvZJcksIA/+OOqhb/F1SQ2CU1dyssX+k3dspa+9tOXnDdB63UOgK89v6pB7AQqPM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551152; c=relaxed/simple; bh=rfs9Lm0CLKJLZKJo7gTkVAb2o23ip3nRMfF8z1fBNDY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=es9979TbLZT7Jo8TSMAKdWNyQKsVIFavt4o1DQ1pOa8QyMvuTsi1bUzL9r3xCHJ5Lmoon0jNx1ipH9vTyyOkf61Hz0YTOV6uWqkBSmo0xqEAEsA0RMdQX+7E0Od9slk9HXpmRpa45YN/MjIXv2lbgyU+T0TSo679JxR72dUvojw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=FKpQmXZB; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=jhs+5HDI; arc=none smtp.client-ip=202.12.124.155 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="FKpQmXZB"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="jhs+5HDI" Received: from phl-compute-04.internal (phl-compute-04.internal [10.202.2.44]) by mailfhigh.stl.internal (Postfix) with ESMTP id EFF897A008A; Wed, 16 Sep 2026 05:32:08 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-04.internal (MEProxy); Wed, 16 Sep 2026 05:32:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789551128; x= 1789637528; bh=Q3XbtEJmWsNT+g02Bw1UcrtZfkVHUUoW2q6DbPGG+78=; b=F KpQmXZB184fo6UZ4QEOsFWMW47cqaVTzpJh6v4ISCKy4NRxLFRX9uv1s+Vy8ShRD W+422RyRjLIiUol4usrTCx14rQ3hDMQy+8Nx1HlIXQvWL4dwhQRPLr2cOUrMpEpL CBm04Jh6Vvgui4qLCv5R95bZLmBMcqTDRrZzBepeONIxO0U6Ja2KvUnG0LLAf9H1 0njnDuJfW+AEq5VGIdWmJWR8wmESiOW6ildk5hcnl+4n6anmDYrEPFuAOoQPnfTm jC5RQzfgitDpQz1tTgU2RLLzAmcr9exMuNoRX6Cp7YkN1EsfjlM+qjNsD/1I0ab4 014sI2tbLtq6QjoYlD71A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789551128; x=1789637528; bh=Q 3XbtEJmWsNT+g02Bw1UcrtZfkVHUUoW2q6DbPGG+78=; b=jhs+5HDIuFHICLQiN ApAxXIoxBGHIQJV35Nc4lvTdPggixWJVh/vMxtHefvDmDgLv72iKzBj6Jm31F/bf HhC17a6FPCpelOaWJWuom0ZJ+7WH9LHiY9yhXc2zAiO6g9k1MlC5vTQ7DEGny9Fi 5LDqGojttj4oCKQEGRIyo4GX8kKEUUao/GbaP91/aFw8rJ8lQYVB0358q2HLB3p+ xklnv5WOaNxLizxm3wK3qUZ09qPLyZe9iQPOCwRKaWe4Krt4O+OqezHxaDukasCI im8GMftcyxahcAAE6q/2ty6DydSkEpPN/lKYcBOvmaTHMzQFgdVVJlWSPoytaba6 aTZ6Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTE31qL+uaIHQKQPvpaICSxxs7QGWvIquuW8cKaqYQWKvt9pKCqy5bgk//TApl3wTl ad4FQSRAknjhMkzVuMAZNh0T12ywqROLvU4AnUXGVwd2kiN78Ar+1NR+fesCkATu1LALh1 4LvX/9qk4pAfO725ECjoAkI82P3z755hZxEkAV0An15e7Wr2vIbgLFM815SfkNmTU/eBmL ORNXO2IPaHkpkCtL9iOfg4dR3lpAdl3Xvq/Y35opAbs/zzbuDBIkBs44NTO3w5x5Bedurt thFl4kzzMjhv7UlAxOBz4vPRXQMYZm6skstaNA0N2ZkZEKrpjBUX5624wA8G4zEpHd89IN i/KDmLvP9t2q/TyXL5za1fYFxo4c+gyrrE4t5l81/2lkILfQ54MPazgd5LGC9+oo2D32LK ZgH4y7AHi4PJxaJtXgsxSllVA4w/pea91FWnyh2eMIshTViwU3sZ0fraEvUw60BjmdV3gd kdNL+FW33NdX9EnfyRI3AAqhm8DOeYNARCQHEZUFjDtsr7O7w8NJFOC9JL+d4XUGLUww0w Ve0jEfV8rN8L/lIHxDHnthdK7FmfioNKt0JMBsuYj8eurCU9RVN31ctkdMTQqGuvQr5bPq MbLd0CkRtCceKCp+kWaCjTbp1V7ttFuhZqhkZBshAQgPahHgTccFTHawGdnA X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 16 Sep 2026 05:32:08 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v3 08/12] mm/collapse: separate scanning a PTE table from collapsing it Date: Wed, 16 Sep 2026 10:31:35 +0100 Message-ID: <20260916093145.4022188-9-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260916093145.4022188-1-kirill@shutemov.name> References: <20260916093145.4022188-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" A collapse is two jobs. One reads a PTE table under mmap_lock and decides whether the range is worth collapsing. The other allocates, isolates, copies and flushes, and wants the lock given up first. collapse_single_pmd() did both, so the boundary between them was somewhere in the middle of a function. Give each half its own function: - collapse_scan_pmd() scans one table and only reads. The anonymous scan that used to carry that name keeps its body as collapse_scan_anon_pmd(), and collapse_scan_pmd() is now the entry that picks the anonymous or the file side. - collapse_run_pmd() does the collapse the scan asked for, and is handed what the scan returned. SCAN_SUCCEED means there is something to collapse. SCAN_PTE_MAPPED_HUGEPAGE means the page cache already holds the PMD folio and only the PTE table is left to retract. Both are work for the run; anything else is why there is nothing to do. collapse_single_pmd() is now the two of them with the mmap_lock drop in between, so its callers see what they saw before. What the scan found and the run needs travels in collapse_control. For an anonymous table that is the orders and the referenced and swapped-out counts, which mthp_collapse() and collapse_huge_page() now read from there instead of taking as arguments. For a file it is the file itself and the offset in it. The file side moves with the anonymous one. collapse_scan_file() used to run with mmap_lock already given up, and called collapse_file() itself when the page cache looked worth it. It now runs under the lock like the anonymous scan and only reads; the run does the collapse. A file collapse works on the page cache and never sees a VMA, so the scan takes the file reference while it still has one and the run gives it back. That changes what a refused file table costs khugepaged. Every file table it scanned used to end its pass over that mm, because the lock had been dropped to scan it; now only a table it goes on to collapse does. Two things on the file side stop being rescanned. When the page cache already holds the PMD folio, the scan says so and the run goes straight to retracting the PTE table. A run that refuses dirty pages and may write them back retries collapse_file() alone. The checks the scan makes ahead of it are ones collapse_file() repeats under the page cache lock. Tracing changes with it. mm_khugepaged_scan_pmd and mm_khugepaged_scan_file used to fire after the collapse, so for an accepted table their status field carried what the collapse made of it. They now fire before it and read SCAN_SUCCEED for an accepted table. What the collapse then made of it is for mm_collapse_huge_page and mm_khugepaged_collapse_file to report. Assisted-by: LLM Reviewed-by: Zi Yan Signed-off-by: Kiryl Shutsemau (Meta) Reviewed-by: Baolin Wang --- mm/collapse.h | 14 ++++ mm/khugepaged.c | 166 +++++++++++++++++++++++++++++++++--------------- 2 files changed, 130 insertions(+), 50 deletions(-) diff --git a/mm/collapse.h b/mm/collapse.h index 7044dc71c7c2..9764011bad86 100644 --- a/mm/collapse.h +++ b/mm/collapse.h @@ -88,6 +88,20 @@ struct collapse_control { =20 /* Each bit marks a PTE the scan accepted as a collapse source */ DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); + + /* + * What a scan found and the run after it needs. Live only between the + * two, and read by nobody else. + * + * The file side takes a reference while it still has the VMA, since a + * file collapse works on the page cache and never sees one; the run is + * what gives it back. + */ + unsigned long scan_orders; + int scan_referenced; + int scan_unmapped; + struct file *scan_file; + pgoff_t scan_pgoff; }; =20 #endif /* __MM_COLLAPSE_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index ff1756c1255d..b6fc2c78e3e2 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1237,8 +1237,8 @@ static enum scan_result alloc_charge_folio(struct fol= io **foliop, struct mm_stru * while allocating a THP, as that could trigger direct reclaim/compaction. * Note that the VMA must be rechecked after grabbing the mmap_lock again. */ -static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned = long start_addr, - int referenced, int unmapped, struct collapse_control *cc, +static enum scan_result collapse_huge_page(struct mm_struct *mm, + unsigned long start_addr, struct collapse_control *cc, unsigned int order) { const unsigned long pmd_addr =3D start_addr & HPAGE_PMD_MASK; @@ -1277,14 +1277,14 @@ static enum scan_result collapse_huge_page(struct m= m_struct *mm, unsigned long s goto out_nolock; } =20 - if (unmapped) { + if (cc->scan_unmapped) { /* * __collapse_huge_page_swapin() will return with mmap_lock * released when it fails. So we jump out_nolock directly in * that case. Continuing to collapse causes inconsistency. */ result =3D __collapse_huge_page_swapin(mm, vma, start_addr, pmd, - referenced, order); + cc->scan_referenced, order); if (result !=3D SCAN_SUCCEED) goto out_nolock; } @@ -1448,9 +1448,8 @@ static unsigned int max_order_from_offset(unsigned in= t offset) * If a collapse is permitted, we attempt to collapse the PTE range into a * mTHP. */ -static enum scan_result mthp_collapse(struct mm_struct *mm, - unsigned long address, int referenced, int unmapped, - struct collapse_control *cc, unsigned long enabled_orders) +static enum scan_result mthp_collapse(struct mm_struct *mm, unsigned long = address, + struct collapse_control *cc) { unsigned int nr_eligible_ptes, nr_ptes, max_ptes_none; enum scan_result last_result =3D SCAN_FAIL; @@ -1463,7 +1462,7 @@ static enum scan_result mthp_collapse(struct mm_struc= t *mm, while (offset < HPAGE_PMD_NR) { nr_ptes =3D 1UL << order; =20 - if (!test_bit(order, &enabled_orders)) + if (!test_bit(order, &cc->scan_orders)) goto next_order; =20 max_ptes_none =3D collapse_max_ptes_none(cc, NULL, order); @@ -1471,19 +1470,18 @@ static enum scan_result mthp_collapse(struct mm_str= uct *mm, offset + nr_ptes); =20 /* - * Swap PTEs accepted during the scan are counted in @unmapped, - * not in cc->eligible_ptes. Account them for the PMD-order - * candidate. + * Swap PTEs accepted during the scan are counted in + * cc->scan_unmapped, not in cc->eligible_ptes. Account them for + * the PMD-order candidate. */ if (is_pmd_order(order)) - nr_eligible_ptes +=3D unmapped; + nr_eligible_ptes +=3D cc->scan_unmapped; =20 if (nr_eligible_ptes >=3D nr_ptes - max_ptes_none) { enum scan_result ret; =20 collapse_address =3D address + offset * PAGE_SIZE; - ret =3D collapse_huge_page(mm, collapse_address, referenced, - unmapped, cc, order); + ret =3D collapse_huge_page(mm, collapse_address, cc, order); =20 switch (ret) { /* Cases where we continue to next collapse candidate */ @@ -1525,7 +1523,7 @@ static enum scan_result mthp_collapse(struct mm_struc= t *mm, * we must always move to the next offset. */ if (order > COLLAPSE_MIN_MTHP_ORDER && - (enabled_orders & GENMASK(order - 1, 0))) { + (cc->scan_orders & GENMASK(order - 1, 0))) { order--; continue; } @@ -1550,14 +1548,14 @@ static enum scan_result mthp_collapse(struct mm_str= uct *mm, return last_result; } =20 -static enum scan_result collapse_scan_pmd(struct mm_struct *mm, - struct vm_area_struct *vma, unsigned long start_addr, - bool *lock_dropped, struct collapse_control *cc) +static enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma, + unsigned long start_addr, struct collapse_control *cc) { const unsigned int max_ptes_shared =3D collapse_max_ptes_shared(cc, HPAGE= _PMD_ORDER); const unsigned int max_ptes_swap =3D collapse_max_ptes_swap(cc, HPAGE_PMD= _ORDER); unsigned int max_ptes_none =3D collapse_max_ptes_none(cc, vma, HPAGE_PMD_= ORDER); enum tva_type tva_flags =3D cc->policy.tva_type; + struct mm_struct *mm =3D vma->vm_mm; pmd_t *pmd; pte_t *pte, *_pte, pteval; int i; @@ -1737,12 +1735,9 @@ static enum scan_result collapse_scan_pmd(struct mm_= struct *mm, out_unmap: pte_unmap_unlock(pte, ptl); if (result =3D=3D SCAN_SUCCEED) { - /* collapse_huge_page() expects the lock to be dropped before calling */ - mmap_read_unlock(mm); - result =3D mthp_collapse(mm, start_addr, referenced, - unmapped, cc, enabled_orders); - /* mmap_lock was released above, set lock_dropped */ - *lock_dropped =3D true; + cc->scan_orders =3D enabled_orders; + cc->scan_referenced =3D referenced; + cc->scan_unmapped =3D unmapped; } out: trace_mm_khugepaged_scan_pmd(mm, failed_pfn, referenced, @@ -2739,45 +2734,86 @@ static enum scan_result collapse_scan_file(struct m= m_struct *mm, else cc->progress +=3D HPAGE_PMD_NR; =20 - if (result =3D=3D SCAN_SUCCEED) { - if (present < HPAGE_PMD_NR - max_ptes_none) { - result =3D SCAN_EXCEED_NONE_PTE; - count_vm_event(THP_SCAN_EXCEED_NONE_PTE); - } else { - result =3D collapse_file(mm, addr, file, start, cc); - } + if (result =3D=3D SCAN_SUCCEED && present < HPAGE_PMD_NR - max_ptes_none)= { + result =3D SCAN_EXCEED_NONE_PTE; + count_vm_event(THP_SCAN_EXCEED_NONE_PTE); } =20 - trace_mm_khugepaged_scan_file(mm, failed_pfn, file, present, swap, result= ); + trace_mm_khugepaged_scan_file(mm, failed_pfn, file, present, swap, + result); return result; } =20 -/* - * Try to collapse a single PMD starting at a PMD aligned addr, and return - * the results. - */ -static enum scan_result collapse_single_pmd(unsigned long addr, - struct vm_area_struct *vma, bool *lock_dropped, - struct collapse_control *cc) +static void collapse_control_init(struct collapse_control *cc) +{ + cc->progress =3D 0; + cc->scan_file =3D NULL; +} + +/* A scan that took a file reference should have been run */ +static void collapse_put_scan_file(struct collapse_control *cc) +{ + if (WARN_ON_ONCE(cc->scan_file)) { + fput(cc->scan_file); + cc->scan_file =3D NULL; + } +} + +static void collapse_control_release(struct collapse_control *cc) +{ + collapse_put_scan_file(cc); +} + +static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, + unsigned long addr, struct collapse_control *cc) { - struct mm_struct *mm =3D vma->vm_mm; - bool triggered_wb =3D false; enum scan_result result; - struct file *file; pgoff_t pgoff; =20 - mmap_assert_locked(mm); + mmap_assert_locked(vma->vm_mm); + /* Whatever the last scan found has to have been run by now */ + collapse_put_scan_file(cc); =20 if (vma_is_anonymous(vma)) - return collapse_scan_pmd(mm, vma, addr, lock_dropped, cc); + return collapse_scan_anon_pmd(vma, addr, cc); =20 - file =3D get_file(vma->vm_file); pgoff =3D linear_page_index(vma, addr); + result =3D collapse_scan_file(vma->vm_mm, addr, vma->vm_file, pgoff, cc); + /* + * SCAN_PTE_MAPPED_HUGEPAGE is work too: the page cache already holds + * the PMD folio, and retracting the PTE table is the run's job. + */ + if (result !=3D SCAN_SUCCEED && result !=3D SCAN_PTE_MAPPED_HUGEPAGE) + return result; =20 - mmap_read_unlock(mm); - *lock_dropped =3D true; + /* + * A file collapse works on the page cache and never sees a VMA, so take + * what it needs from this one while it is still here. + */ + cc->scan_file =3D get_file(vma->vm_file); + cc->scan_pgoff =3D pgoff; + return result; +} + +static enum scan_result collapse_run_pmd(struct mm_struct *mm, + unsigned long addr, enum scan_result result, + struct collapse_control *cc) +{ + struct file *file =3D cc->scan_file; + bool triggered_wb =3D false; + pgoff_t pgoff; + + if (!file) + return mthp_collapse(mm, addr, cc); + + cc->scan_file =3D NULL; + pgoff =3D cc->scan_pgoff; + + /* The scan found the PMD folio in place: nothing to collapse */ + if (result =3D=3D SCAN_PTE_MAPPED_HUGEPAGE) + goto retract; retry: - result =3D collapse_scan_file(mm, addr, file, pgoff, cc); + result =3D collapse_file(mm, addr, file, pgoff, cc); =20 /* Dirty pages are worth a writeback and one more try, if asked for */ if (cc->policy.writeback_dirty && result =3D=3D SCAN_PAGE_DIRTY_OR_WRITEB= ACK && @@ -2789,8 +2825,13 @@ static enum scan_result collapse_single_pmd(unsigned= long addr, triggered_wb =3D true; goto retry; } +retract: fput(file); =20 + /* + * A PMD folio is in the page cache, whether the collapse just put it + * there or found it: retract the PTE table, and map the PMD if asked. + */ if (result =3D=3D SCAN_PTE_MAPPED_HUGEPAGE) { mmap_read_lock(mm); if (collapse_test_exit_or_disable(mm)) @@ -2805,6 +2846,28 @@ static enum scan_result collapse_single_pmd(unsigned= long addr, return result; } =20 +/* + * Try to collapse a single PMD starting at a PMD aligned addr, and return + * the results. + */ +static enum scan_result collapse_single_pmd(unsigned long addr, + struct vm_area_struct *vma, bool *lock_dropped, + struct collapse_control *cc) +{ + struct mm_struct *mm =3D vma->vm_mm; + enum scan_result result; + + result =3D collapse_scan_pmd(vma, addr, cc); + if (result !=3D SCAN_SUCCEED && result !=3D SCAN_PTE_MAPPED_HUGEPAGE) + return result; + + /* The collapse takes its own locks, so give this up */ + mmap_read_unlock(mm); + *lock_dropped =3D true; + + return collapse_run_pmd(mm, addr, result, cc); +} + static void collapse_scan_mm_slot(unsigned int progress_max, enum scan_result *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock) @@ -2947,10 +3010,10 @@ static void khugepaged_do_scan(struct collapse_cont= rol *cc) =20 lru_add_drain_all(); =20 + collapse_control_init(cc); /* One policy for the whole pass, so every table is treated the same */ collapse_policy_khugepaged(&cc->policy); =20 - cc->progress =3D 0; while (true) { cond_resched(); =20 @@ -2981,6 +3044,8 @@ static void khugepaged_do_scan(struct collapse_contro= l *cc) khugepaged_alloc_sleep(); } } + + collapse_control_release(cc); } =20 static bool khugepaged_should_wakeup(void) @@ -3177,8 +3242,8 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, cc =3D kmalloc_obj(*cc); if (!cc) return -ENOMEM; + collapse_control_init(cc); collapse_policy_forced(&cc->policy); - cc->progress =3D 0; =20 lru_add_drain_all(); =20 @@ -3235,6 +3300,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, } out_nolock: mmap_assert_locked(mm); + collapse_control_release(cc); kfree(cc); =20 return thps =3D=3D ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0 --=20 2.54.0 From nobody Fri Sep 25 05:30:48 2026 Received: from fout-b8-smtp.messagingengine.com (fout-b8-smtp.messagingengine.com [202.12.124.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 332704A92DA for ; Wed, 16 Sep 2026 09:32:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.151 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551145; cv=none; b=N9vvgcp7RPbPXXnLQQ4/KKRpRNgx3aFs38PS9423zbIGTd1I+eP9NHrNcw2C7gqnK3BF+AFUxDnfq8hFS46/pcObNZodWmGSEa/IkToJNRhufcGhGBiCJWWnfHgxwUVU8+rszW1K/4+WRT4KpkJTKYLkxSsNhILUVtk39bvo6Rw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551145; c=relaxed/simple; bh=xPfHrLAIT8TrgLbrZTJBVNz5G3dEyFo5cSFn4PiWwQg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=q87roTyg2Tnl5oTIJ1Z0VkPX8SNZBUL8WiECNLvYBsLBtkIgMQIAF3eUjvoBOTBM2k4yRUEWlECSoE40tpWIqa/iQwl6tUjSdFCU6yXnlcHAQLNsR50ywjjSF2bhAOKjkj3njo/m7B2iR4n3mjZo5zxjgfAqBr5KkQkHmSqBTDw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=MLVZXFX/; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=YfyXnvTX; arc=none smtp.client-ip=202.12.124.151 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="MLVZXFX/"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="YfyXnvTX" Received: from phl-compute-05.internal (phl-compute-05.internal [10.202.2.45]) by mailfout.stl.internal (Postfix) with ESMTP id 32B7E1D000CC; Wed, 16 Sep 2026 05:32:11 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-05.internal (MEProxy); Wed, 16 Sep 2026 05:32:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789551131; x= 1789637531; bh=VZjknc0p/C+R8pDyx3LdCEi2GInRJhvEJIZP2SLKlcM=; b=M LVZXFX/Kuiq8X+YYRcBc1YBWna6pgosUG0gRELvtrMeG4CO36/1bvwg+iiqfFAFS hPcJEKcA+snrhq9XvC0wfl44NLIo7scVEQaFjkPI1/+JTzfHJlt3LX3phHIid5/z p2KXrmIh2v/MaRVFhmSUJIxlw4pVmL+z22MtSlEmd/uzMPKdKHcKL0TOAR8EmuqX 0Q2NpculsEpXyz1SMKmcn1kou3ZOxyZRTwFWOBoi5ANwEnxwFI9+T35Q64Uoj/rL ui5EgW4PiI7q5MRfiHel0Pa3fqY+n6JTI+QKrXHnJB23dsUie3YIbZgZEdvodPn8 KmDCLf7GMo/94+LGeZCVA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789551131; x=1789637531; bh=V Zjknc0p/C+R8pDyx3LdCEi2GInRJhvEJIZP2SLKlcM=; b=YfyXnvTXHaUXnUrgK 4/wBUAY/sbRU7rqY+kBe1JI9doG6Pc17SNzv7jRsctmAhlzr8DDyJgZaGB/kkJOP k+vbETcwY9iXFRsC+TNoVBJvyecHVHF54y/QqZogIlVWnULb5vpRP15zMhSt6FmL 5pdRbK5iPSdX0gT//6MJu63l9EKEyVZTKk4tzofoN02iygwbfIza9/zY9AiPeXRX 4cVh2+7H6w//Z/gpvZ3t5CsF6Z841t5GNANqfMZ/fwulDkj0nJpepcQjUlD69RcT /VNyYA8A6qZcltcJAlfGzdsx5eIcWMKDcSYsuMYZHKsrZFNDKZA6xEWqbfWqkeiD vaRhQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTE31qL+uaIHQKQPvpaICSxxs7QGWvIquuW8cKaqYQWKvt9pKCqy5bgk//TApl3wTl ad4FQSRAknjhMkzVuMAZNh0T12ywqROLvU4AnUXGVwd2kiN78Ar+1NR+fesCkATu1LALh1 4LvX/9qk4pAfO725ECjoAkI82P3z755hZxEkAV0An15e7Wr2vIbgLFM815SfkNmTU/eBmL ORNXO2IPaHkpkCtL9iOfg4dR3lpAdl3Xvq/Y35opAbs/zzbuDBIkBs44NTO3w5x5Bedurt thFl4kzzMjhv7UlAxOBz4vPRXQMYZm6skstaNA0N2ZkZEKrpjBUX5624wA8G4zEpHd89m6 KXTYej14sW09tJrBdAp3d4HB9MXT4L4MezAAL6IbBLYswZE69EF55PCj8RxbWAkwuyzzG6 3eV3kRyv9ioa9frL1lMv2Ig9L6oSX+iS1yh3YdrC2oa5iC9wBbBDQVHOkoQLRVhwwYSF9y BaXPplCt7AbFeRMIlUi0FkKnU0DzXNVabvq761FK2+hCChs4TzseTw2OmNJJmyl9uou8TD 41QD65UoEfBiXN3u3JWlvwJ03m9GGUEgFEMAvlEsNYVcGulTcZyg/mLPwqd+pY5wdumqrw 1EhPEiEiI9haXtlPLWdC/ZEmT5if8erQK6qCtu861Z8aH3oXaHHcCuejQe4A X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 16 Sep 2026 05:32:10 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v3 09/12] mm/collapse: open-code collapse_single_pmd() in its two callers Date: Wed, 16 Sep 2026 10:31:36 +0100 Message-ID: <20260916093145.4022188-10-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260916093145.4022188-1-kirill@shutemov.name> References: <20260916093145.4022188-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" collapse_scan_pmd() and collapse_run_pmd() each have a clear locking contract. The scan is called with mmap_lock held for reading and returns with it still held. The collapse is called without it. collapse_single_pmd() kept that boundary inside itself. It dropped the lock on some paths and not others, and reported which by way of a bool its callers had to carry along and then act on. Open-code it in the two callers. Each scans under the lock it already holds and, when the scan found work, gives the lock up before running the collapse. khugepaged's lock_dropped and madvise_collapse()'s mmap_unlocked both go: the code dropping the lock is now the code that wanted to know. khugepaged's walk carries on to the next table while the scan keeps refusing, and ends once a collapse has taken the lock from under it. madvise_collapse() re-finds its VMA after a collapse, which it did before, and now uses a NULL vma to say that it has to. It still reports the drop to its own caller, from the line that does it. The lock is given up and taken again at the same points as before. No functional change. Assisted-by: LLM Reviewed-by: Zi Yan Signed-off-by: Kiryl Shutsemau (Meta) Reviewed-by: Baolin Wang --- mm/khugepaged.c | 103 +++++++++++++++++++++++------------------------- 1 file changed, 50 insertions(+), 53 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index b6fc2c78e3e2..9e6b2af6519e 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -2846,28 +2846,6 @@ static enum scan_result collapse_run_pmd(struct mm_s= truct *mm, return result; } =20 -/* - * Try to collapse a single PMD starting at a PMD aligned addr, and return - * the results. - */ -static enum scan_result collapse_single_pmd(unsigned long addr, - struct vm_area_struct *vma, bool *lock_dropped, - struct collapse_control *cc) -{ - struct mm_struct *mm =3D vma->vm_mm; - enum scan_result result; - - result =3D collapse_scan_pmd(vma, addr, cc); - if (result !=3D SCAN_SUCCEED && result !=3D SCAN_PTE_MAPPED_HUGEPAGE) - return result; - - /* The collapse takes its own locks, so give this up */ - mmap_read_unlock(mm); - *lock_dropped =3D true; - - return collapse_run_pmd(mm, addr, result, cc); -} - static void collapse_scan_mm_slot(unsigned int progress_max, enum scan_result *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock) @@ -2930,7 +2908,7 @@ static void collapse_scan_mm_slot(unsigned int progre= ss_max, VM_BUG_ON(khugepaged_scan.address & ~HPAGE_PMD_MASK); =20 while (khugepaged_scan.address < hend) { - bool lock_dropped =3D false; + unsigned long addr; =20 cond_resched(); if (unlikely(collapse_test_exit_or_disable(mm))) @@ -2940,23 +2918,30 @@ static void collapse_scan_mm_slot(unsigned int prog= ress_max, khugepaged_scan.address + HPAGE_PMD_SIZE > hend); =20 - *result =3D collapse_single_pmd(khugepaged_scan.address, - vma, &lock_dropped, cc); - if (*result =3D=3D SCAN_SUCCEED) - khugepaged_pages_collapsed++; + addr =3D khugepaged_scan.address; /* move to next address */ khugepaged_scan.address +=3D HPAGE_PMD_SIZE; - if (lock_dropped) - /* - * We released mmap_lock so break loop. Note - * that we drop mmap_lock before all hugepage - * allocations, so if allocation fails, we are - * guaranteed to break here and report the - * correct result back to caller. - */ - goto breakouterloop_mmap_lock; - if (cc->progress >=3D progress_max) - goto breakouterloop; + + *result =3D collapse_scan_pmd(vma, addr, cc); + /* Nothing to do here, and the lock is still ours */ + if (*result !=3D SCAN_SUCCEED && + *result !=3D SCAN_PTE_MAPPED_HUGEPAGE) { + if (cc->progress >=3D progress_max) + goto breakouterloop; + continue; + } + + /* + * A collapse takes its own locks and is slow enough + * that a writer should not wait behind it, so give the + * lock up. That ends this walk: vma and the mm are + * whatever the collapse leaves them. + */ + mmap_read_unlock(mm); + *result =3D collapse_run_pmd(mm, addr, *result, cc); + if (*result =3D=3D SCAN_SUCCEED) + khugepaged_pages_collapsed++; + goto breakouterloop_mmap_lock; } } breakouterloop: @@ -3225,7 +3210,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, unsigned long hstart, hend, addr; enum scan_result last_fail =3D SCAN_FAIL; int thps =3D 0; - bool mmap_unlocked =3D false; =20 BUG_ON(vma->vm_start > start); BUG_ON(vma->vm_end < end); @@ -3248,25 +3232,40 @@ int madvise_collapse(struct vm_area_struct *vma, un= signed long start, lru_add_drain_all(); =20 for (addr =3D hstart; addr < hend; addr +=3D HPAGE_PMD_SIZE) { - enum scan_result result =3D SCAN_FAIL; + struct vm_area_struct *found; + enum scan_result result; =20 - if (mmap_unlocked) { + /* + * A collapse gives the lock up, so the VMA has to be found + * again after one: it can shrink while nothing is held. A scan + * that finds nothing to collapse leaves the lock alone, so a + * range that is already collapsed walks on without relocking. + */ + if (!vma) { cond_resched(); mmap_read_lock(mm); - mmap_unlocked =3D false; - *lock_dropped =3D true; - result =3D hugepage_vma_revalidate(mm, addr, false, &vma, + result =3D hugepage_vma_revalidate(mm, addr, false, &found, cc, HPAGE_PMD_ORDER); if (result !=3D SCAN_SUCCEED) { last_fail =3D result; - goto out_nolock; + goto out_locked; } - + vma =3D found; hend =3D min(hend, vma->vm_end & HPAGE_PMD_MASK); } =20 - result =3D collapse_single_pmd(addr, vma, &mmap_unlocked, cc); + result =3D collapse_scan_pmd(vma, addr, cc); + /* Nothing to do here, and the lock is still ours */ + if (result !=3D SCAN_SUCCEED && result !=3D SCAN_PTE_MAPPED_HUGEPAGE) + goto tally; =20 + /* The collapse takes its own locks, so give this up */ + mmap_read_unlock(mm); + *lock_dropped =3D true; + vma =3D NULL; + + result =3D collapse_run_pmd(mm, addr, result, cc); +tally: switch (result) { case SCAN_SUCCEED: case SCAN_PMD_MAPPED: @@ -3288,17 +3287,15 @@ int madvise_collapse(struct vm_area_struct *vma, un= signed long start, default: last_fail =3D result; /* Other error, exit */ - goto out_maybelock; + goto out; } } =20 -out_maybelock: +out: /* Caller expects us to hold mmap_lock on return */ - if (mmap_unlocked) { - *lock_dropped =3D true; + if (!vma) mmap_read_lock(mm); - } -out_nolock: +out_locked: mmap_assert_locked(mm); collapse_control_release(cc); kfree(cc); --=20 2.54.0 From nobody Fri Sep 25 05:30:48 2026 Received: from fhigh-b4-smtp.messagingengine.com (fhigh-b4-smtp.messagingengine.com [202.12.124.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 67D594A92C8 for ; Wed, 16 Sep 2026 09:32:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.155 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551162; cv=none; b=CjfTVxFrqW4N+H2BNBnsnXck4BF7jUGrlXK1dwHmjQQnzQwIIukfydQ19dMsWL8P06xB4U75v98mZcejku1NwiaecwdM5J/sTI4Wj2AzsDtdF7D2MhXzAdtwJ8YCrK0x614wqANJ+poIOq2vlTbFz6gLXr+5zWa1pvh9myX0odM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551162; c=relaxed/simple; bh=/WjhvGACiwNe9/LGX3y5/h7X87Y+zDJYwy+7DsnslnQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BJim/YYCTiqx4T89R6ZWjUwtFGBXcUC9FZfYw667y2GWOTjxIbLMSXMp9lqxPs8LJimEv5pEtzp+YuXqUy7QuyXhfDzQLpDDwAuceQ76/ftRAcN9BvFYJrf/SM3zvIEkd94Af6PJ6bphixb/Zyn0EiGsbaolq3Oylo/F/aWPy9I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=h+nDuZ4L; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=DamH363K; arc=none smtp.client-ip=202.12.124.155 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="h+nDuZ4L"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="DamH363K" Received: from phl-compute-03.internal (phl-compute-03.internal [10.202.2.43]) by mailfhigh.stl.internal (Postfix) with ESMTP id C033B7A007B; Wed, 16 Sep 2026 05:32:13 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-03.internal (MEProxy); Wed, 16 Sep 2026 05:32:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789551133; x= 1789637533; bh=PRWFDwvM0fZr06tYfSZoOVoDNXjF/NbOi9PwgZXG13s=; b=h +nDuZ4L3d7Exo1VnWVHkFTiWmBJ3LgIXQqeNIIwptbCr3rocTTogmeSBRfhparYF 8kgDFCvJL7mBpQZbetk0u9hmVR6g5yZn50P8MISvcSghAJqOQA0voEnJT4/QgyhA GDj831STE2yzmfNUzOohrZzAZZyQzVOEwC7USAAY8Lr6KwgPrJ13zFKFFdQLm8p8 Yb259Q3GPN8AWvUYzvymY2q1ClRPBjA0sILyAiu0+O8zvqqbwC//b17J8h3m4kIC Cr4wjVwJM2U8G//TZoNzgQkHENcWcrEUOxLwrC52SlRwgWsDXJAAn0g8zgyw+dVX QMbQYjvg3wJAXw+3Q2M0Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789551133; x=1789637533; bh=P RWFDwvM0fZr06tYfSZoOVoDNXjF/NbOi9PwgZXG13s=; b=DamH363KmwhSZ5zJu fhU6H/Ey+B7XjK9/lw6VNU7q7qLYVoT4I4fUzQBv/orFWgEdqDdXd+KZurLR/3JV NxsGojIyz8eGDOxEtArmG85Ut+JFBZPugFv1xJSQGTZmWZVQMyhiUldxb+Hh22pt IopaLgTUWX+llHnaqMo1+S7+DAo8YYDHltvyrUOpNQZdViGlTms4zQ5KhalJvaYm YKUzYulz+bw+eZ/ovtFDvl5VjCg5/f2j7NM3Pm0eiJ0wwFgkjhCHKoFh1ml8qd5/ isye1zUR9hdNyHV6WLUSWj2EjEd508yrriVlWflkVEEc06+bZWEhCzx7LmZ43AyL PmG7Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEOpyDQEfQMjLyldXBPAs1OiAaYjuycA/zprc2XA0sZE/gLY53LzRl6wisjmJsNcj shXh7L3amtO3W7Dj4gJ1brhV8efHSvbbvvisq8qNCVrU/hVVqWW2fJ8EDoeL9EfJt8J4fa r63UA+S0A+UN3c16swmwWrTj5sK0gOy7RYH1CN+xe4D6TTQ1ncSXBFkweWqSBfk0S52Say Dntsl8DI3TlZwS0igZd1iO+ltDo9n3qHIbwcYcvcbVUlu6Rxag/XK6s48+7ZTTYlI2aJe4 hkL/jN/KxntOAh39VLdXeraBW39brB+PkStvfnz0wMlwqVlAhF3nsCCzTv21LQWeSF/iHT pyMAuIQkGFQ08rzLATOXG16m6WQQDf0qrhK3S8NG5J8zh2mNM5qcdfG+xe89hRG/m+7INR ADrck8zKABvA0aAWvVdCuEeBi2oCpNFHiAk2Khx6KyEamwPIIWWx1aCyjtcXhsnR1VCXvK jOfLqdyuZxn83ODbS+0qUPAqOyBYa449EcVKzMv4hGVxgKwrNVR4sBWB7R/NCH/wZHyXB7 AWwvIeSxHVuN/dy3GL/fBK1LiTnhq1VJmRsV1IMAH0TLses6J0QUD6xT6cFIZlQl8915KE pMjHkmd7973FH6bjWFDUCbqDKgjeWhNiAp/maWL8JO1SbJWp0+skjlFQ/FUA X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 16 Sep 2026 05:32:12 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v3 10/12] mm/collapse: work out the orders a VMA allows once per VMA Date: Wed, 16 Sep 2026 10:31:37 +0100 Message-ID: <20260916093145.4022188-11-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260916093145.4022188-1-kirill@shutemov.name> References: <20260916093145.4022188-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" The scan asked collapse_possible_orders() for every PTE table, for an answer that is a property of the VMA. Both callers walk a VMA a table at a time, so let them work it out once and pass the mask in. It is only good while the lock that produced it is held, so madvise_collapse() takes it again after every collapse. The mask is then sampled once per VMA rather than once per table. A thp enabled knob written during a walk takes effect one VMA later, and cannot widen a collapse: hugepage_vma_revalidate() tests the order again under the lock the collapse retakes. Assisted-by: LLM Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Signed-off-by: Kiryl Shutsemau (Meta) --- mm/khugepaged.c | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 9e6b2af6519e..12cb67d8df32 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1549,12 +1549,12 @@ static enum scan_result mthp_collapse(struct mm_str= uct *mm, unsigned long addres } =20 static enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma, - unsigned long start_addr, struct collapse_control *cc) + unsigned long start_addr, struct collapse_control *cc, + unsigned long enabled_orders) { const unsigned int max_ptes_shared =3D collapse_max_ptes_shared(cc, HPAGE= _PMD_ORDER); const unsigned int max_ptes_swap =3D collapse_max_ptes_swap(cc, HPAGE_PMD= _ORDER); unsigned int max_ptes_none =3D collapse_max_ptes_none(cc, vma, HPAGE_PMD_= ORDER); - enum tva_type tva_flags =3D cc->policy.tva_type; struct mm_struct *mm =3D vma->vm_mm; pmd_t *pmd; pte_t *pte, *_pte, pteval; @@ -1565,7 +1565,6 @@ static enum scan_result collapse_scan_anon_pmd(struct= vm_area_struct *vma, struct folio *folio =3D NULL; unsigned long failed_pfn =3D -1; unsigned long addr; - unsigned long enabled_orders; spinlock_t *ptl; int node =3D NUMA_NO_NODE, unmapped =3D 0; =20 @@ -1579,8 +1578,6 @@ static enum scan_result collapse_scan_anon_pmd(struct= vm_area_struct *vma, =20 collapse_scan_reset(cc); =20 - enabled_orders =3D collapse_possible_orders(vma, vma->vm_flags, tva_flags= ); - /* * If PMD is the only enabled order, enforce max_ptes_none, otherwise * scan all pages to populate the bitmap for mTHP collapse. The bitmap @@ -2765,7 +2762,8 @@ static void collapse_control_release(struct collapse_= control *cc) } =20 static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, - unsigned long addr, struct collapse_control *cc) + unsigned long addr, struct collapse_control *cc, + unsigned long orders) { enum scan_result result; pgoff_t pgoff; @@ -2775,7 +2773,7 @@ static enum scan_result collapse_scan_pmd(struct vm_a= rea_struct *vma, collapse_put_scan_file(cc); =20 if (vma_is_anonymous(vma)) - return collapse_scan_anon_pmd(vma, addr, cc); + return collapse_scan_anon_pmd(vma, addr, cc, orders); =20 pgoff =3D linear_page_index(vma, addr); result =3D collapse_scan_file(vma->vm_mm, addr, vma->vm_file, pgoff, cc); @@ -2885,15 +2883,17 @@ static void collapse_scan_mm_slot(unsigned int prog= ress_max, =20 vma_iter_init(&vmi, mm, khugepaged_scan.address); for_each_vma(vmi, vma) { - unsigned long hstart, hend; + unsigned long hstart, hend, orders; =20 cond_resched(); if (unlikely(collapse_test_exit_or_disable(mm))) { cc->progress++; break; } - if (!collapse_possible_orders(vma, vma->vm_flags, - TVA_KHUGEPAGED)) { + /* One mask for the whole VMA */ + orders =3D collapse_possible_orders(vma, vma->vm_flags, + cc->policy.tva_type); + if (!orders) { cc->progress++; continue; } @@ -2922,7 +2922,7 @@ static void collapse_scan_mm_slot(unsigned int progre= ss_max, /* move to next address */ khugepaged_scan.address +=3D HPAGE_PMD_SIZE; =20 - *result =3D collapse_scan_pmd(vma, addr, cc); + *result =3D collapse_scan_pmd(vma, addr, cc, orders); /* Nothing to do here, and the lock is still ours */ if (*result !=3D SCAN_SUCCEED && *result !=3D SCAN_PTE_MAPPED_HUGEPAGE) { @@ -3207,14 +3207,16 @@ int madvise_collapse(struct vm_area_struct *vma, un= signed long start, { struct collapse_control *cc; struct mm_struct *mm =3D vma->vm_mm; - unsigned long hstart, hend, addr; + unsigned long hstart, hend, addr, orders; enum scan_result last_fail =3D SCAN_FAIL; int thps =3D 0; =20 BUG_ON(vma->vm_start > start); BUG_ON(vma->vm_end < end); =20 - if (!collapse_possible_orders(vma, vma->vm_flags, TVA_FORCED_COLLAPSE)) + orders =3D collapse_possible_orders(vma, vma->vm_flags, + TVA_FORCED_COLLAPSE); + if (!orders) return -EINVAL; =20 hstart =3D ALIGN(start, HPAGE_PMD_SIZE); @@ -3252,9 +3254,11 @@ int madvise_collapse(struct vm_area_struct *vma, uns= igned long start, } vma =3D found; hend =3D min(hend, vma->vm_end & HPAGE_PMD_MASK); + orders =3D collapse_possible_orders(vma, vma->vm_flags, + cc->policy.tva_type); } =20 - result =3D collapse_scan_pmd(vma, addr, cc); + result =3D collapse_scan_pmd(vma, addr, cc, orders); /* Nothing to do here, and the lock is still ours */ if (result !=3D SCAN_SUCCEED && result !=3D SCAN_PTE_MAPPED_HUGEPAGE) goto tally; --=20 2.54.0 From nobody Fri Sep 25 05:30:48 2026 Received: from fhigh-b4-smtp.messagingengine.com (fhigh-b4-smtp.messagingengine.com [202.12.124.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A684A468C29 for ; Wed, 16 Sep 2026 09:32:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.155 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551154; cv=none; b=SPMhhbGXaylELgIc6vrmk3lenh8oOACGk48UhpN5GcbciFKYPaWDdYFpIBnQRWhx7gOwcryNIcMFoflKKAwtZV+s+hXpruo+5ku1d0I5X0RqsgPMzt8XfsPEKPHiPPb/DkK9iwJlvx/YgU/Pv0k5opSvk6AKn/Gr/vc4VRwOq1E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551154; c=relaxed/simple; bh=s1L0OnGP/FEeB8td6STjQAUsUGT/qmbbu+bKCvB8V1s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=enUqKyWs8W9K9lEalwW9n1GWvMlo1HEeIxGYXvugGVOmBEC7cu/wujhGaxkkDlAmi/ViPJc4pZAQ7u83pG9KQJI00l2/pFX9s7xddsFgkuGzeLKWdqF50L3O7CnfePpbHBk53HMOVpRsQNSnaNh8huFCzeL9MDbLXnpe/8JCQxw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=HIOeenOi; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=QkHXyoxt; arc=none smtp.client-ip=202.12.124.155 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="HIOeenOi"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="QkHXyoxt" Received: from phl-compute-02.internal (phl-compute-02.internal [10.202.2.42]) by mailfhigh.stl.internal (Postfix) with ESMTP id ECC1F7A008D; Wed, 16 Sep 2026 05:32:15 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-02.internal (MEProxy); Wed, 16 Sep 2026 05:32:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789551135; x= 1789637535; bh=+GwJCdLgfVk4XVCX9cK4EkSAFLB/3dv31DwWiTLRd6k=; b=H IOeenOiQbKHmpxeuLJtbAqJZzufrpJTko83KblrjAmmYduhhRnUOzU/qyCsRvO/3 l01f6xf7YAbstF2pHIG2hsxHcZhozHLq4k6IlpbUncv/V6L6g8i9R/vYzgZH+4S9 wDuy8t+ilff7AFjwhgIGvG3XkiLpvLRS5LwC8PI9TZa/7cmmWvaZRWnZRdlVdxm5 K6k99LwMc4wNF1dm8UtI8D/HK0OJB59yi/KPLDvQNIUeh1ZnOvnYT3MZf9t07H6F qXystXzYgY4rVLACl87e1Q45BjWm8EB/EVTAohoKESFVRNU41wkKZCh8SGf4R0S6 gfeeI7RWS9VTR6vyjMmXQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789551135; x=1789637535; bh=+ GwJCdLgfVk4XVCX9cK4EkSAFLB/3dv31DwWiTLRd6k=; b=QkHXyoxta8LOGefHY X+iOI1IPTymo3VpXx3oUoU5VKA1U1KTvX03RT/475gAzHQ2VHLx3ix0+omswYSmc ZnEsjWlvGcWrHfkZFzwnP+ZH3Gn2owyGa3+Mas0xU4pgIBAEf5R5WI+m9CTi0Kgy IWT5/RzzAvQX1SqN4gFAlg0dODkFOl7HVIR/9hTFFSDAl8S3/Zbo5h2bp2uvU3CZ lWGjjUWiReeh6xbNLEA6MCg0eDAYtGNGxpxfDcFntjaHyd+boELjogtmc0bm+xbq zG/jqd8zN4/e61B6YaimF9071l2ZmBgJ2qEv5QKJwSvVmOVjRJuScpqrhJG8PPTU /oMhQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTE31qL+uaIHQKQPvpaICSxxs7QGWvIquuW8cKaqYQWKvt9pKCqy5bgk//TApl3wTl ad4FQSRAknjhMkzVuMAZNh0T12ywqROLvU4AnUXGVwd2kiN78Ar+1NR+fesCkATu1LALh1 4LvX/9qk4pAfO725ECjoAkI82P3z755hZxEkAV0An15e7Wr2vIbgLFM815SfkNmTU/eBmL ORNXO2IPaHkpkCtL9iOfg4dR3lpAdl3Xvq/Y35opAbs/zzbuDBIkBs44NTO3w5x5Bedurt thFl4kzzMjhv7UlAxOBz4vPRXQMYZm6skstaNA0N2ZkZEKrpjBUX5624wA8G4zEpHd89Hy JWBgDmmn5MPLPofWTx3R3L45pyo841HrYYAE+89PVJJU4o+WwYro+D32vc7H4Yx7Q3Vc98 plplj6RkZ9Jv99KgXpVQugI7WQ4LjTf4BuFTOYrbMrU9ih6DpsH+/KcRPPG7HyPOQnRo1Q 724eB8mBiDEJ4UM4zYeRHWV3M9LrtGOxiJ06ryxnaga020c85YShHFNxHD49sQzALMGtT3 BbqdGFWmc9+Nu/oTaALVi+Fv+VIXvWuXxGzVcMSrmXXMWA6jXL/x+AvPp6ZhdAvPovhwZR WikTKhMom3IW4T9M+VciV7O8q7BAufrtyLV015pg1Ukzgyef1cFMh5wPBrYQ X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 16 Sep 2026 05:32:15 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v3 11/12] mm/collapse: declare the collapse interface in collapse.h Date: Wed, 16 Sep 2026 10:31:38 +0100 Message-ID: <20260916093145.4022188-12-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260916093145.4022188-1-kirill@shutemov.name> References: <20260916093145.4022188-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" A collapse takes four calls: - collapse_control_init() - set up the control a caller carries; - collapse_scan_pmd() - scan one PTE table, under mmap_lock; - collapse_run_pmd() - collapse what the scan found, no mmap_lock; - collapse_control_release() - done with the control. All four are static in khugepaged.c, as are collapse_possible_orders(), which says what a VMA allows, and the revalidate a caller needs once a collapse has given the mmap_lock up. No other file can ask for a collapse without them. Declare them in collapse.h, with a comment stating the order they are called in and who holds the lock over each step. Each function says what it needs and what it does where it is defined. hugepage_vma_revalidate() becomes collapse_vma_revalidate(): it is part of what a collapse offers now, not a helper of the daemon. Preparation for implementing MADV_COLLAPSE in madvise.c. No functional change. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) --- mm/collapse.h | 47 ++++++++++++++++++++++++++++++++++++++++++ mm/khugepaged.c | 55 +++++++++++++++++++++++++++++++++++++------------ 2 files changed, 89 insertions(+), 13 deletions(-) diff --git a/mm/collapse.h b/mm/collapse.h index 9764011bad86..e52969965262 100644 --- a/mm/collapse.h +++ b/mm/collapse.h @@ -104,4 +104,51 @@ struct collapse_control { pgoff_t scan_pgoff; }; =20 +/* Which orders a VMA may collapse to, zero when it may not collapse at al= l */ +unsigned long collapse_possible_orders(struct vm_area_struct *vma, + vm_flags_t vm_flags, enum tva_type tva_flags); + +/* + * A caller states what it allows in cc->policy and then hands over one PTE + * table's worth of a VMA at a time: + * + * collapse_control_init(cc) once, before the first table + * collapse_scan_pmd(vma, addr, ...) per table + * collapse_run_pmd(mm, addr, result, cc) when a scan found work + * collapse_control_release(cc) once, when done with the con= trol + * + * The caller holds mmap_lock for reading over the scan and passes an addr= ess + * within @vma, aligned to the PTE table to scan. + * + * The scan returns with that lock still held. It only reads, and almost = every + * table it is offered has nothing to collapse, so a caller walks a whole = VMA + * under the one lock it took to get there. SCAN_SUCCEED means there is + * something to collapse. SCAN_PTE_MAPPED_HUGEPAGE means the page cache + * already holds the PMD folio and only the PTE table is left to retract. + * Both are work for the run, which is handed what the scan returned; anyt= hing + * else is why there is nothing to do. + * + * The run is called without the lock and returns without it, taking what = it + * needs in between: what it does -- allocate, isolate, copy, flush -- is = slow + * enough that a writer would wait behind it. The caller gives the lock up + * first, and with it @vma and anything derived under it, so a caller carr= ying + * on has to look up again with collapse_vma_revalidate(). The run revali= dates + * for itself rather than trusting what the scan saw. + * + * A scan that found something has to be run: the file side takes a refere= nce on + * the file while it still has the VMA to take it from, and the run is what + * gives it back. + */ +void collapse_control_init(struct collapse_control *cc); +void collapse_control_release(struct collapse_control *cc); +enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, + unsigned long addr, struct collapse_control *cc, + unsigned long orders); +enum scan_result collapse_run_pmd(struct mm_struct *mm, unsigned long addr, + enum scan_result result, struct collapse_control *cc); +enum scan_result collapse_vma_revalidate(struct mm_struct *mm, + unsigned long address, bool expect_anon, + struct vm_area_struct **vmap, struct collapse_control *cc, + unsigned int order); + #endif /* __MM_COLLAPSE_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 12cb67d8df32..d4e0674a05d7 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -498,7 +498,7 @@ void __khugepaged_enter(struct mm_struct *mm) * Check what orders are possible based on the vma and collapse type. * This is used to determine if mTHP collapse is a viable option. */ -static unsigned long collapse_possible_orders(struct vm_area_struct *vma, +unsigned long collapse_possible_orders(struct vm_area_struct *vma, vm_flags_t vm_flags, enum tva_type tva_flags) { unsigned long orders; @@ -1011,12 +1011,12 @@ static int collapse_find_target_node(struct collaps= e_control *cc) #endif =20 /* - * If mmap_lock temporarily dropped, revalidate vma - * after taking the mmap_lock again. - * Returns enum scan_result value. + * Find the VMA at @address again once mmap_lock has been given up and tak= en + * back, and check it still allows a collapse of @order there. The VMA ha= s to + * span the whole PMD whatever @order is; with @expect_anon it also has to= be + * anonymous and have an anon_vma. *@vmap is the VMA found, if any. */ - -static enum scan_result hugepage_vma_revalidate(struct mm_struct *mm, unsi= gned long address, +enum scan_result collapse_vma_revalidate(struct mm_struct *mm, unsigned lo= ng address, bool expect_anon, struct vm_area_struct **vmap, struct collapse_control *cc, unsigned int order) { @@ -1264,7 +1264,7 @@ static enum scan_result collapse_huge_page(struct mm_= struct *mm, } =20 mmap_read_lock(mm); - result =3D hugepage_vma_revalidate(mm, pmd_addr, /*expect_anon=3D*/ true, + result =3D collapse_vma_revalidate(mm, pmd_addr, /*expect_anon=3D*/ true, &vma, cc, order); if (result !=3D SCAN_SUCCEED) { mmap_read_unlock(mm); @@ -1299,7 +1299,7 @@ static enum scan_result collapse_huge_page(struct mm_= struct *mm, * mmap_lock. */ mmap_write_lock(mm); - result =3D hugepage_vma_revalidate(mm, pmd_addr, /*expect_anon=3D*/ true, + result =3D collapse_vma_revalidate(mm, pmd_addr, /*expect_anon=3D*/ true, &vma, cc, order); if (result !=3D SCAN_SUCCEED) goto out_up_write; @@ -2741,7 +2741,8 @@ static enum scan_result collapse_scan_file(struct mm_= struct *mm, return result; } =20 -static void collapse_control_init(struct collapse_control *cc) +/* Set up a control before its first scan; cc->policy is the caller's to f= ill */ +void collapse_control_init(struct collapse_control *cc) { cc->progress =3D 0; cc->scan_file =3D NULL; @@ -2756,12 +2757,30 @@ static void collapse_put_scan_file(struct collapse_= control *cc) } } =20 -static void collapse_control_release(struct collapse_control *cc) +/* + * Done with a control. A scan that found something has to have been run = by + * then: the file side takes a reference on the file while it still has the + * VMA to take it from, and the run is what gives it back. + */ +void collapse_control_release(struct collapse_control *cc) { collapse_put_scan_file(cc); } =20 -static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, +/* + * Scan the PTE table of @vma at @addr for a collapse candidate. @addr is + * aligned to the table; @orders is what the caller allows there. + * + * Called with mmap_lock held for reading and returns with it still held. = It + * only reads, and almost every table it is offered has nothing to collaps= e, + * so a caller walks a whole VMA under the one lock it took to get there. + * + * SCAN_SUCCEED means there is something to collapse. SCAN_PTE_MAPPED_HUG= EPAGE + * means the page cache already holds the PMD folio and only the PTE table= is + * left to retract. Both are work for collapse_run_pmd(), which is handed + * what the scan returned; anything else is why there is nothing to do. + */ +enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, unsigned long addr, struct collapse_control *cc, unsigned long orders) { @@ -2793,7 +2812,17 @@ static enum scan_result collapse_scan_pmd(struct vm_= area_struct *vma, return result; } =20 -static enum scan_result collapse_run_pmd(struct mm_struct *mm, +/* + * Collapse the table a scan found work in. @result is what the scan + * returned. + * + * Called without mmap_lock and returns without it, taking what it needs in + * between: what it does -- allocate, isolate, copy, flush -- is slow enou= gh + * that a writer would wait behind it. The caller gives the lock up first, + * and with it the VMA and anything derived under it. The run revalidates= for + * itself rather than trusting what the scan saw. + */ +enum scan_result collapse_run_pmd(struct mm_struct *mm, unsigned long addr, enum scan_result result, struct collapse_control *cc) { @@ -3246,7 +3275,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, if (!vma) { cond_resched(); mmap_read_lock(mm); - result =3D hugepage_vma_revalidate(mm, addr, false, &found, + result =3D collapse_vma_revalidate(mm, addr, false, &found, cc, HPAGE_PMD_ORDER); if (result !=3D SCAN_SUCCEED) { last_fail =3D result; --=20 2.54.0 From nobody Fri Sep 25 05:30:48 2026 Received: from fhigh-b4-smtp.messagingengine.com (fhigh-b4-smtp.messagingengine.com [202.12.124.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 698904A8A2F for ; Wed, 16 Sep 2026 09:32:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.155 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551159; cv=none; b=BSpTkqMmx1fdRdokNnpwES9WZnTFwvRv6NeEP9GAlaVX6IpvfsiuWFERplOyiSFbmW0rYtqdFEYW+kEaVuYdCwReRvC/D93A6x1DSs6ywsyCSffeFZzm8d+Wxrk6yv63cgGSJrKae4UDyyI8j6NCvYEklnADNti/etB1t9ekqas= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551159; c=relaxed/simple; bh=q5k1WOpXlBarlRRA5jzmPq3yHyrZ4F4mo6uJLzOHQP0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZR/S6QfoEvizWg1eo9cCyGJBJ0CcwPPohduX4+guRGjH2N7LQ4SgpSVIYebFhRSP9vg6j/xm1ar2xx3FVW8NUeqZ++cH582Uvp0g74+vItcmtfm3Hl/a2ooBt8EdSkIZWh+8N7jGwkE+E4s2jx2dgZYKAH6eWQ38Ap3N0wIVpFg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=Ea6Pu3zX; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=IeQwcZRT; arc=none smtp.client-ip=202.12.124.155 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="Ea6Pu3zX"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="IeQwcZRT" Received: from phl-compute-08.internal (phl-compute-08.internal [10.202.2.48]) by mailfhigh.stl.internal (Postfix) with ESMTP id 29C877A009C; Wed, 16 Sep 2026 05:32:18 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-08.internal (MEProxy); Wed, 16 Sep 2026 05:32:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789551138; x= 1789637538; bh=VLFqjvsypH6MxOVzXQHLvkpXTaqk9hfZ26wZXIiSNXs=; b=E a6Pu3zXuqqWTRgFc9dCprk1E+1nEaPZGG3gn/vy5mLnXwEoyDTb+owl9F+8Kw4/Z 1IIflXvtN8/4i8e4X9Rh9K5YbXBdvCCtg+sQXVwsoQK+HCSdrbT/cG5WcGSuJlXy VDStdVqz7yUFnXuz/mCv0ypbPxtDgc+53XU2pi1By400Fnep4we/dfWBDafiNg5t Uo31Tf+FXGvOf0cWge2wGfS/r2pGC6MgZ/ynCOxnG6APLpVF95tcxY3O9WHmCg9c ySoMq+XjqcW4/YFWkfrImWRNQSXz+eKiu366NdV3FugdFNkwBkU6qVPBp1G0rhMi L4vXssdMbd5mKKq1MnLIA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789551138; x=1789637538; bh=V LFqjvsypH6MxOVzXQHLvkpXTaqk9hfZ26wZXIiSNXs=; b=IeQwcZRTAPVFcGsQt Zer36Rd5cqawzTROfp4A8YgmjBBfRBDmw5GCCudOkayPzwCOTMl0mpt05eXdaJkp IhPr+vYIAYEQeVbl7TCpf3UeOKAqym8H5np4l1jDpIrPV0fEqb0Ey7u/B+0BEK0k a+SoQHBneBmURAS6He4gaCUbdG7wS4X8ZYZ4mT2Z9c6TdMeX8NRawACSWAVwK3bi 8zFflZQpcW2NcDT4ZQe/s1pl5DxfDaokZMWxD/p5S/4EDK/W78tszdunG+kbEGa7 m6eZTzHmmSu7Wvoh99fyOZaJzG7klLAgcUMt02XC0gYIyRXWk7Mp0IQ6VivPgWVn hdzhA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTE31qL+uaIHQKQPvpaICSxxs7QGWvIquuW8cKaqYQWKvt9pKCqy5bgk//TApl3wTl ad4FQSRAknjhMkzVuMAZNh0T12ywqROLvU4AnUXGVwd2kiN78Ar+1NR+fesCkATu1LALh1 4LvX/9qk4pAfO725ECjoAkI82P3z755hZxEkAV0An15e7Wr2vIbgLFM815SfkNmTU/eBmL ORNXO2IPaHkpkCtL9iOfg4dR3lpAdl3Xvq/Y35opAbs/zzbuDBIkBs44NTO3w5x5Bedurt thFl4kzzMjhv7UlAxOBz4vPRXQMYZm6skstaNA0N2ZkZEKrpjBUX5624wA8G4zEpHd89Fg GFq+lpZWo+FobnIjGR6ga/tKZzqv1DP/krPF6lxtDnv2fFXQnF/Rd3AO3J7hcv9ukN2DGp 6X6cee47q62i28Js3X7EW25mKWln6wdpQFZrygm5UqF3Et9YHPkeUlJ93E5PA4EhBriP3U EpsMUFUs9Fu218oaklx0T0LNWB02N/m51/dfvVtVhJBM7n1n/QsabKldxETUmAMJzmAzUR /CvzoYL1f2qH8D4MbxYT8IHdpFBAqjrnQ6FGrEZhJFfzUk9/ppwqlWpnrHfao3S6uVW9YY HpgZCmZMGM9109dFSfc/zw9cOXhkSEvi+Oh+wc+XAMszgXkC0Q4yRjMa/Ixw X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 16 Sep 2026 05:32:17 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v3 12/12] mm/collapse: implement MADV_COLLAPSE in madvise.c Date: Wed, 16 Sep 2026 10:31:39 +0100 Message-ID: <20260916093145.4022188-13-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260916093145.4022188-1-kirill@shutemov.name> References: <20260916093145.4022188-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" MADV_COLLAPSE is a madvise operation, but its implementation sat in khugepaged.c. The daemon's file therefore also held a syscall's worth of code that has nothing to do with the daemon: the walk over the user's range, the per-PMD loop, and the errno translation. Move it to madvise.c, among the operations it belongs with, along with the errno map and the policy it states for itself. It takes a struct madvise_behavior like every one of those operations, which is where the range, the VMA and the lock-dropped flag it used to be handed separately already live. It stays a caller of the interface khugepaged uses, so nothing about the collapse changes. The !CONFIG_TRANSPARENT_HUGEPAGE stub moves in with it. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) --- include/linux/huge_mm.h | 9 --- mm/khugepaged.c | 158 +------------------------------------ mm/madvise.c | 169 +++++++++++++++++++++++++++++++++++++++- 3 files changed, 169 insertions(+), 167 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index c745f7ad2298..8ca0fa3be2ac 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -510,8 +510,6 @@ change_huge_pud(struct mmu_gather *tlb, struct vm_area_= struct *vma, =20 int hugepage_madvise(struct vm_area_struct *vma, vm_flags_t *vm_flags, int advice); -int madvise_collapse(struct vm_area_struct *vma, unsigned long start, - unsigned long end, bool *lock_dropped); void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start, unsigned long end, struct vm_area_struct *next); spinlock_t *__pmd_trans_huge_lock(pmd_t *pmd, struct vm_area_struct *vma); @@ -715,13 +713,6 @@ static inline int hugepage_madvise(struct vm_area_stru= ct *vma, return -EINVAL; } =20 -static inline int madvise_collapse(struct vm_area_struct *vma, - unsigned long start, - unsigned long end, bool *lock_dropped) -{ - return -EINVAL; -} - static inline void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start, unsigned long end, diff --git a/mm/khugepaged.c b/mm/khugepaged.c index d4e0674a05d7..2a10a30055c6 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -969,21 +969,6 @@ static void collapse_policy_khugepaged(struct collapse= _policy *p) p->tva_type =3D TVA_KHUGEPAGED; } =20 -/* MADV_COLLAPSE was asked for explicitly, so it is not held to those */ -static void collapse_policy_forced(struct collapse_policy *p) -{ - p->max_ptes_none =3D HPAGE_PMD_NR; - p->max_ptes_swap =3D HPAGE_PMD_NR; - p->max_ptes_shared =3D HPAGE_PMD_NR; - p->strict_sub_pmd =3D false; - p->skip_lazyfree =3D false; - p->require_referenced =3D false; - p->install_pmd =3D true; - p->writeback_dirty =3D true; - p->gfp =3D GFP_TRANSHUGE; - p->tva_type =3D TVA_FORCED_COLLAPSE; -} - #ifdef CONFIG_NUMA static int collapse_find_target_node(struct collapse_control *cc) { @@ -2822,9 +2807,8 @@ enum scan_result collapse_scan_pmd(struct vm_area_str= uct *vma, * and with it the VMA and anything derived under it. The run revalidates= for * itself rather than trusting what the scan saw. */ -enum scan_result collapse_run_pmd(struct mm_struct *mm, - unsigned long addr, enum scan_result result, - struct collapse_control *cc) +enum scan_result collapse_run_pmd(struct mm_struct *mm, unsigned long addr, + enum scan_result result, struct collapse_control *cc) { struct file *file =3D cc->scan_file; bool triggered_wb =3D false; @@ -3198,141 +3182,3 @@ bool current_is_khugepaged(void) { return kthread_func(current) =3D=3D khugepaged; } - -static int madvise_collapse_errno(enum scan_result r) -{ - /* - * MADV_COLLAPSE breaks from existing madvise(2) conventions to provide - * actionable feedback to caller, so they may take an appropriate - * fallback measure depending on the nature of the failure. - */ - switch (r) { - case SCAN_ALLOC_HUGE_PAGE_FAIL: - return -ENOMEM; - case SCAN_CGROUP_CHARGE_FAIL: - case SCAN_EXCEED_NONE_PTE: - return -EBUSY; - /* Resource temporary unavailable - trying again might succeed */ - case SCAN_PAGE_COUNT: - case SCAN_PAGE_LOCK: - case SCAN_PAGE_LRU: - case SCAN_DEL_PAGE_LRU: - case SCAN_PAGE_FILLED: - case SCAN_PAGE_HAS_PRIVATE: - case SCAN_PAGE_DIRTY_OR_WRITEBACK: - return -EAGAIN; - /* - * Other: Trying again likely not to succeed / error intrinsic to - * specified memory range. khugepaged likely won't be able to collapse - * either. - */ - default: - return -EINVAL; - } -} - -int madvise_collapse(struct vm_area_struct *vma, unsigned long start, - unsigned long end, bool *lock_dropped) -{ - struct collapse_control *cc; - struct mm_struct *mm =3D vma->vm_mm; - unsigned long hstart, hend, addr, orders; - enum scan_result last_fail =3D SCAN_FAIL; - int thps =3D 0; - - BUG_ON(vma->vm_start > start); - BUG_ON(vma->vm_end < end); - - orders =3D collapse_possible_orders(vma, vma->vm_flags, - TVA_FORCED_COLLAPSE); - if (!orders) - return -EINVAL; - - hstart =3D ALIGN(start, HPAGE_PMD_SIZE); - hend =3D ALIGN_DOWN(end, HPAGE_PMD_SIZE); - - if (hstart >=3D hend) - return 0; - - cc =3D kmalloc_obj(*cc); - if (!cc) - return -ENOMEM; - collapse_control_init(cc); - collapse_policy_forced(&cc->policy); - - lru_add_drain_all(); - - for (addr =3D hstart; addr < hend; addr +=3D HPAGE_PMD_SIZE) { - struct vm_area_struct *found; - enum scan_result result; - - /* - * A collapse gives the lock up, so the VMA has to be found - * again after one: it can shrink while nothing is held. A scan - * that finds nothing to collapse leaves the lock alone, so a - * range that is already collapsed walks on without relocking. - */ - if (!vma) { - cond_resched(); - mmap_read_lock(mm); - result =3D collapse_vma_revalidate(mm, addr, false, &found, - cc, HPAGE_PMD_ORDER); - if (result !=3D SCAN_SUCCEED) { - last_fail =3D result; - goto out_locked; - } - vma =3D found; - hend =3D min(hend, vma->vm_end & HPAGE_PMD_MASK); - orders =3D collapse_possible_orders(vma, vma->vm_flags, - cc->policy.tva_type); - } - - result =3D collapse_scan_pmd(vma, addr, cc, orders); - /* Nothing to do here, and the lock is still ours */ - if (result !=3D SCAN_SUCCEED && result !=3D SCAN_PTE_MAPPED_HUGEPAGE) - goto tally; - - /* The collapse takes its own locks, so give this up */ - mmap_read_unlock(mm); - *lock_dropped =3D true; - vma =3D NULL; - - result =3D collapse_run_pmd(mm, addr, result, cc); -tally: - switch (result) { - case SCAN_SUCCEED: - case SCAN_PMD_MAPPED: - ++thps; - break; - /* Whitelisted set of results where continuing OK */ - case SCAN_NO_PTE_TABLE: - case SCAN_PTE_NON_PRESENT: - case SCAN_PTE_UFFD: - case SCAN_LACK_REFERENCED_PAGE: - case SCAN_PAGE_NULL: - case SCAN_PAGE_COUNT: - case SCAN_PAGE_LOCK: - case SCAN_PAGE_COMPOUND: - case SCAN_PAGE_LRU: - case SCAN_DEL_PAGE_LRU: - last_fail =3D result; - break; - default: - last_fail =3D result; - /* Other error, exit */ - goto out; - } - } - -out: - /* Caller expects us to hold mmap_lock on return */ - if (!vma) - mmap_read_lock(mm); -out_locked: - mmap_assert_locked(mm); - collapse_control_release(cc); - kfree(cc); - - return thps =3D=3D ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0 - : madvise_collapse_errno(last_fail); -} diff --git a/mm/madvise.c b/mm/madvise.c index 963337f93a7a..33bcd390ce43 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -38,6 +38,7 @@ =20 #include "internal.h" #include "swap.h" +#include "collapse.h" =20 #define __MADV_SET_ANON_VMA_NAME (-1) =20 @@ -906,6 +907,171 @@ bool madvise_dontneed_free_valid_vma(struct madvise_b= ehavior *madv_behavior) return true; } =20 +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + +/* MADV_COLLAPSE was asked for explicitly, so it is not held to those */ +static void collapse_policy_forced(struct collapse_policy *p) +{ + p->max_ptes_none =3D HPAGE_PMD_NR; + p->max_ptes_swap =3D HPAGE_PMD_NR; + p->max_ptes_shared =3D HPAGE_PMD_NR; + p->strict_sub_pmd =3D false; + p->skip_lazyfree =3D false; + p->require_referenced =3D false; + p->install_pmd =3D true; + p->writeback_dirty =3D true; + p->gfp =3D GFP_TRANSHUGE; + p->tva_type =3D TVA_FORCED_COLLAPSE; +} + +static int madvise_collapse_errno(enum scan_result r) +{ + /* + * MADV_COLLAPSE breaks from existing madvise(2) conventions to provide + * actionable feedback to caller, so they may take an appropriate + * fallback measure depending on the nature of the failure. + */ + switch (r) { + case SCAN_ALLOC_HUGE_PAGE_FAIL: + return -ENOMEM; + case SCAN_CGROUP_CHARGE_FAIL: + case SCAN_EXCEED_NONE_PTE: + return -EBUSY; + /* Resource temporary unavailable - trying again might succeed */ + case SCAN_PAGE_COUNT: + case SCAN_PAGE_LOCK: + case SCAN_PAGE_LRU: + case SCAN_DEL_PAGE_LRU: + case SCAN_PAGE_FILLED: + case SCAN_PAGE_HAS_PRIVATE: + case SCAN_PAGE_DIRTY_OR_WRITEBACK: + return -EAGAIN; + /* + * Other: Trying again likely not to succeed / error intrinsic to + * specified memory range. khugepaged likely won't be able to collapse + * either. + */ + default: + return -EINVAL; + } +} + +static int madvise_collapse(struct madvise_behavior *madv_behavior) +{ + struct madvise_behavior_range *range =3D &madv_behavior->range; + struct vm_area_struct *vma =3D madv_behavior->vma; + struct mm_struct *mm =3D madv_behavior->mm; + struct collapse_control *cc; + unsigned long hstart, hend, addr, orders; + enum scan_result last_fail =3D SCAN_FAIL; + int thps =3D 0; + + BUG_ON(vma->vm_start > range->start); + BUG_ON(vma->vm_end < range->end); + + orders =3D collapse_possible_orders(vma, vma->vm_flags, + TVA_FORCED_COLLAPSE); + if (!orders) + return -EINVAL; + + hstart =3D ALIGN(range->start, HPAGE_PMD_SIZE); + hend =3D ALIGN_DOWN(range->end, HPAGE_PMD_SIZE); + + if (hstart >=3D hend) + return 0; + + cc =3D kmalloc_obj(*cc); + if (!cc) + return -ENOMEM; + collapse_control_init(cc); + collapse_policy_forced(&cc->policy); + + lru_add_drain_all(); + + for (addr =3D hstart; addr < hend; addr +=3D HPAGE_PMD_SIZE) { + struct vm_area_struct *found; + enum scan_result result; + + /* + * A collapse gives the lock up, so the VMA has to be found + * again after one: it can shrink while nothing is held. A scan + * that finds nothing to collapse leaves the lock alone, so a + * range that is already collapsed walks on without relocking. + */ + if (!vma) { + cond_resched(); + mmap_read_lock(mm); + result =3D collapse_vma_revalidate(mm, addr, false, &found, + cc, HPAGE_PMD_ORDER); + if (result !=3D SCAN_SUCCEED) { + last_fail =3D result; + goto out_locked; + } + vma =3D found; + hend =3D min(hend, vma->vm_end & HPAGE_PMD_MASK); + orders =3D collapse_possible_orders(vma, vma->vm_flags, + cc->policy.tva_type); + } + + result =3D collapse_scan_pmd(vma, addr, cc, orders); + /* Nothing to do here, and the lock is still ours */ + if (result !=3D SCAN_SUCCEED && result !=3D SCAN_PTE_MAPPED_HUGEPAGE) + goto tally; + + /* The collapse takes its own locks, so give this up */ + mmap_read_unlock(mm); + mark_mmap_lock_dropped(madv_behavior); + vma =3D NULL; + + result =3D collapse_run_pmd(mm, addr, result, cc); +tally: + switch (result) { + case SCAN_SUCCEED: + case SCAN_PMD_MAPPED: + ++thps; + break; + /* Whitelisted set of results where continuing OK */ + case SCAN_NO_PTE_TABLE: + case SCAN_PTE_NON_PRESENT: + case SCAN_PTE_UFFD: + case SCAN_LACK_REFERENCED_PAGE: + case SCAN_PAGE_NULL: + case SCAN_PAGE_COUNT: + case SCAN_PAGE_LOCK: + case SCAN_PAGE_COMPOUND: + case SCAN_PAGE_LRU: + case SCAN_DEL_PAGE_LRU: + last_fail =3D result; + break; + default: + last_fail =3D result; + /* Other error, exit */ + goto out; + } + } + +out: + /* Caller expects us to hold mmap_lock on return */ + if (!vma) + mmap_read_lock(mm); +out_locked: + mmap_assert_locked(mm); + collapse_control_release(cc); + kfree(cc); + + return thps =3D=3D ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0 + : madvise_collapse_errno(last_fail); +} + +#else /* CONFIG_TRANSPARENT_HUGEPAGE */ + +static int madvise_collapse(struct madvise_behavior *madv_behavior) +{ + return -EINVAL; +} + +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + static long madvise_dontneed_free(struct madvise_behavior *madv_behavior) { struct mm_struct *mm =3D madv_behavior->mm; @@ -1373,8 +1539,7 @@ static int madvise_vma_behavior(struct madvise_behavi= or *madv_behavior) case MADV_DONTNEED_LOCKED: return madvise_dontneed_free(madv_behavior); case MADV_COLLAPSE: - return madvise_collapse(vma, range->start, range->end, - &madv_behavior->lock_dropped); + return madvise_collapse(madv_behavior); case MADV_GUARD_INSTALL: return madvise_guard_install(madv_behavior); case MADV_GUARD_REMOVE: --=20 2.54.0