From nobody Fri Sep 25 16:54:11 2026 Received: from fout-b8-smtp.messagingengine.com (fout-b8-smtp.messagingengine.com [202.12.124.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4BF9747DFA9 for ; Thu, 10 Sep 2026 12:02:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.151 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041773; cv=none; b=iqDUyDWsgyNHMSOmNXIkOnWTZNq3OZ499Zwd/XQtUDXQI7F1sfcEs0yKvFFup/7HsGd+sLzOv2sCbQQ67KXNk55E1Ke6p+GxjJqPpvXDtwsknyVXTfwZoyuMQ0R5r0jVYMyxIZNcFkT5dX0N+stF3vsw+aLHAGltIi/OqKe3XM8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041773; c=relaxed/simple; bh=RPRWO1cQwfat1Dp+ZIb02Kouxyvk7UB00r0B/YRnvf4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Vwehrbp7+MZJCw9vRsib0fNK9SxWh420R4Ys1lz0r2hnx+Ggkd1NaUSasdgz5nNlH7iGmg3MNmxO68SDhrmA1PprXQLE7d1fqK7OvA/kOVTR0JrhJ+cEOan+Ed3hvpRIIrPGOQBDOB+8G2NoyQ1Cu8pzdmazbbGMW7ALXvV3e3Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=RoNmw2t/; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=pXjiJ4ax; arc=none smtp.client-ip=202.12.124.151 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="RoNmw2t/"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="pXjiJ4ax" Received: from phl-compute-08.internal (phl-compute-08.internal [10.202.2.48]) by mailfout.stl.internal (Postfix) with ESMTP id 3E4E21D00066; Thu, 10 Sep 2026 08:02:47 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-08.internal (MEProxy); Thu, 10 Sep 2026 08:02:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789041767; x= 1789128167; bh=JI5Fa516GYJ3tjpfUWQYw4F0sEV0mYVe3RtQG+xuZIc=; b=R oNmw2t/7fOO/tSm9mOPQlzF/zGd4IY8hpx14GxEtJ8n5ub6w+GRon1KrYeUFRBGG WfZBzLrd6ntIDQLjAjae1jJDFEOJW881bVcTOyhLU7DaSWUDcq3XfgdAZ8tSpT45 yZ6euI/d31PbHULlnNTpcXwRf3z8NTwGo+Q+w9o5+A/7/3pYuQxDHHkFVRUT0e6H jgaFUNRvT/+joNHuExDaMfuB9De2Wh8qBGqTEw2mdV+GTPvCxgxmS2MzoAowsnHx o2BUapLwa7a+z39wClLamUff0u2iAuhDlXfVxr1xCSI7FIMdC4oC/4YrrO1lloUW We+L82oK+W0h+w92P9HlA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789041767; x=1789128167; bh=J I5Fa516GYJ3tjpfUWQYw4F0sEV0mYVe3RtQG+xuZIc=; b=pXjiJ4axMe0kUEfir yIUluhNqtjF3b5GB2g9HF7xhzeC25rvgo80hH5WCBD9MIpzb7GsvcuZsIMxLw3PW 4KTmM64h4qV1aEJLjoSTet8G+9y/yqwD80rcz2TVW2RF7fWyYLGrGAp32p9iCmQU fC1dBbiMRRiKHJDiiiGOZOFInu9FCJG4ICzXBsZ9HZ4tZvgTNnWnd45nBQy27OsO jDnVcAmISuoutwhFViaRNr0tk9fBVJ+wcdK9vs7MoQF7VNRR4XwFidODb67jZQdR dj4uTOYiRJLrctXy+tei3g7Q/NMsnlq0C7kNeLF7LWucCdBw03Hc+TT5QAM9uGx9 AHlAQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEVCHaom43ud29NDpH2Ao3u7HJ/1ECQvA0KKWbvYxhttLa4TfzKR6zF5UuYx3UdSX No90AnuR4y57x1CX4a9AAsMV1JR4eQYCvRkfoTmW6C9Lc6lI8AZbTNwmqz4UTzT3vnk7qg ph+TCI1xI7srBtDO6Tg5MLzI4vqgzAzQGVqRHCjhPRmxizMOs2MvZfOp4fytB88FrNmNEM gORbCSjN6PT3vgIUt7WNrJZsAWFpVymtt3pMoPYb3jF9rU+m42uK7S67tVqaIc+biyrQXs /trDbiCnZfBDKlB4R/Y3z4EcguiWXCyuMp2o4aP4AZn5jbn1xeIsCoHcLAwPh3Pk2Cuooc sGdntzh/yP1mv7gIr2vSfojU8NuPx5fqlOjAGDaKaG8GehyParYQCFRQVBwE8xM1cWN8Fo 9XYhMVUUlRGCVVIjjGKrULoOkvrbLwds8TG3IyuIKPFUAg5H9k2M7VYfpV82EMOFQws8sd O1RPXNeLjDaYUKiXZ07EzB7/z6Zl9pfiotitqbkXDhBkRDECc5IUIMsZGJABqfKWlaZ8Mh 5Px1YXB0tVWvvvdm0Av5Ew0fLm+rKWSOdOPn1dkL4NEarjmDFQk9B5OKzlE9mAI4GjVueL yJQkDpVSg9TiBUpOsfwIKTYEQLszQXZEUTYgVFQxKXlv4ma+qGaBdn6W4/Dw X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Sep 2026 08:02:46 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v2 01/12] mm/khugepaged: drop redundant mm_struct pin in madvise_collapse() Date: Thu, 10 Sep 2026 13:02:21 +0100 Message-ID: <20260910120238.2529819-2-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260910120238.2529819-1-kirill@shutemov.name> References: <20260910120238.2529819-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" madvise_collapse() holds an mmgrab() reference across its work. It is redundant. Every caller already holds mm_users: - madvise(2) works on current->mm, which lives as long as the task is in the syscall; - process_madvise(2) reaches a remote mm through mm_access(), which takes an mm_users reference and holds it until the syscall returns; - io_uring passes current->mm; - DAMON takes one with get_task_mm() and drops it after the call. Drop the mmgrab()/mmdrop() pair. Assisted-by: LLM Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Signed-off-by: Kiryl Shutsemau (Meta) --- mm/khugepaged.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index e13d233b9967..75c876acf527 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -3229,7 +3229,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, cc->is_khugepaged =3D false; cc->progress =3D 0; =20 - mmgrab(mm); lru_add_drain_all(); =20 for (addr =3D hstart; addr < hend; addr +=3D HPAGE_PMD_SIZE) { @@ -3285,7 +3284,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, } out_nolock: mmap_assert_locked(mm); - mmdrop(mm); kfree(cc); =20 return thps =3D=3D ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0 --=20 2.54.0 From nobody Fri Sep 25 16:54:11 2026 Received: from fout-b8-smtp.messagingengine.com (fout-b8-smtp.messagingengine.com [202.12.124.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D20974825DB for ; Thu, 10 Sep 2026 12:02:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.151 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041775; cv=none; b=qImwGjNpln7Cewp/3s50Hs6jj7y6C9EeJaz9mdqcb4CiFDXvrLfkQllLGB1n6yJ8MoXaGlQ3olqH+YNm2FKGPvyJ36EG1VjDk3cmGdOwL+aGN0hQLn0FoGx07m5mY4MTEqltVqqFjnUBowypjfIIwtZlQimRQ5iHJkKSBkzidz0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041775; c=relaxed/simple; bh=oZSNA87hap+8LZsY08iTLDC7668XQYqV9pSMcwVgaSA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=P4VkHdR+qU9hWxO6+gRkYdZcr6V+sRsYJqxa3Y/hyxRJfmhxQqyiUkj0lWsnwQOIp1719AuTV5F7zlZG6VCk2sfPWX4+IBIh3dQ0oEM5ic3MlFIJT1xlIn/bGqZeSBIjY87jK07HGzi3dmPdt4X516R++mVHfyGxouzqcaQexUg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=gAkp2IS3; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=cBn3+Vw8; arc=none smtp.client-ip=202.12.124.151 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="gAkp2IS3"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="cBn3+Vw8" Received: from phl-compute-04.internal (phl-compute-04.internal [10.202.2.44]) by mailfout.stl.internal (Postfix) with ESMTP id 32D381D000AF; Thu, 10 Sep 2026 08:02:50 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-04.internal (MEProxy); Thu, 10 Sep 2026 08:02:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789041770; x= 1789128170; bh=8kffjeZ0BbAQk8WNkrJMcG1UR27PMTOcUlnehRZ4hss=; b=g Akp2IS3NHDVLdMko2d0FV0YB9jFZ8P58J2lDDhdm+9rn2ICnAutLjjqPNBKXg5ji lOI02rQhsVa5mXv3g/m9fV0xldDJQ2Inh16KvSZsdlBpaCdx18L/X6PWmvghqoEn TkljnvzD3nue6abvEgrfgqpzSAa+k34eHKHFdenFppTVK0gWBTYHpals4W/6j+ys zwrHjEh3mfTvMsUghgzLROyM400/foH2+TsWMn3WK6buUEYVBgcW+DNki8hsl0Wg XSmzuwv+hmkJFLXpd/CBy66Cp+lgdWF+0nCtnHJZB8NLNIK0GORKzn5hiun8QHXJ HatFXX6MRpAlEmxYMxkrQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789041770; x=1789128170; bh=8 kffjeZ0BbAQk8WNkrJMcG1UR27PMTOcUlnehRZ4hss=; b=cBn3+Vw8POMp+fium FQ//oEUoWBMKXZdRZ86XmcbWe3fhuloiKaHwzrKiMyDUUPzFJoTAU3VEYgbKeTOK j/tqsnDr5jSq8iJpLC0bWqmyKkX8W7XHXARs/lbBoRy0jErYut2DmM9cTw9gyi6g Fo8Gx9oNLlc1jEbFFgJMc2BqCtK8zTWJj0nJW+JwNE1Gx1ndN08sd9TfJBhKNrMx zwh0jWKP5PTh45SFCtHgVlXDMSswUok9TXoy+ac6iPGNGjc65vvps9KrrCoDGKyk oSy+EDoI1kLKNQmz/GWZQOeczDhEx0llpanWN3rOOhuc8tTNKSwBofM07oLxeOkA TI44g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEVCHaom43ud29NDpH2Ao3u7HJ/1ECQvA0KKWbvYxhttLa4TfzKR6zF5UuYx3UdSX No90AnuR4y57x1CX4a9AAsMV1JR4eQYCvRkfoTmW6C9Lc6lI8AZbTNwmqz4UTzT3vnk7qg ph+TCI1xI7srBtDO6Tg5MLzI4vqgzAzQGVqRHCjhPRmxizMOs2MvZfOp4fytB88FrNmNEM gORbCSjN6PT3vgIUt7WNrJZsAWFpVymtt3pMoPYb3jF9rU+m42uK7S67tVqaIc+biyrQXs /trDbiCnZfBDKlB4R/Y3z4EcguiWXCyuMp2o4aP4AZn5jbn1xeIsCoHcLAwPh3Pk2CuoZA UmuXX4LIsJW7XVhqlJ8zVOJIJ0NZD3NUnvHLW/LU5aYowrVKTasei+Z3d2uPbvnci0030K yz8sOx+VruTDAAU4PwIeDL2+c/v69MtCSMnOR2gZBHe1xmwcFNPzazqUiDd28vHkibOjBl rydmZFXtzNseUJaRa69q6JOEKLHbI//bE8Wmkdx6AhpI6UOOXADahWweL3ANdYpno/6T09 aeZ/Owe5RdNZf5Setq1i2PJ7LMhHyjLJfZELXrv9fOvpeSYe0A5jBN8sv4Sj40ecqCdKBY WMnR4RMhwbxccxq+s/2o8NRXwS9R8hFoxtFyZYufbV+lgI2m3qedCD8KSyag X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Sep 2026 08:02:49 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v2 02/12] mm/khugepaged: count collapses where khugepaged makes them Date: Thu, 10 Sep 2026 13:02:22 +0100 Message-ID: <20260910120238.2529819-3-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260910120238.2529819-1-kirill@shutemov.name> References: <20260910120238.2529819-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" collapse_single_pmd() bumps khugepaged_pages_collapsed for its caller, and tests cc->is_khugepaged to know whether it should: the counter belongs to the daemon, and MADV_COLLAPSE must not touch it. The daemon sees every result of every collapse it asks for, so it can keep its own counter without the shared path testing who called. Assisted-by: LLM Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Signed-off-by: Kiryl Shutsemau (Meta) --- mm/khugepaged.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 75c876acf527..4979a93e3648 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -2815,10 +2815,8 @@ static enum scan_result collapse_single_pmd(unsigned= long addr, =20 mmap_assert_locked(mm); =20 - if (vma_is_anonymous(vma)) { - result =3D collapse_scan_pmd(mm, vma, addr, lock_dropped, cc); - goto end; - } + if (vma_is_anonymous(vma)) + return collapse_scan_pmd(mm, vma, addr, lock_dropped, cc); =20 file =3D get_file(vma->vm_file); pgoff =3D linear_page_index(vma, addr); @@ -2854,9 +2852,6 @@ static enum scan_result collapse_single_pmd(unsigned = long addr, result =3D SCAN_SUCCEED; mmap_read_unlock(mm); } -end: - if (cc->is_khugepaged && result =3D=3D SCAN_SUCCEED) - ++khugepaged_pages_collapsed; return result; } =20 @@ -2933,6 +2928,8 @@ static void collapse_scan_mm_slot(unsigned int progre= ss_max, =20 *result =3D collapse_single_pmd(khugepaged_scan.address, vma, &lock_dropped, cc); + if (*result =3D=3D SCAN_SUCCEED) + khugepaged_pages_collapsed++; /* move to next address */ khugepaged_scan.address +=3D HPAGE_PMD_SIZE; if (lock_dropped) --=20 2.54.0 From nobody Fri Sep 25 16:54:11 2026 Received: from fout-b8-smtp.messagingengine.com (fout-b8-smtp.messagingengine.com [202.12.124.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A20C4825B9 for ; Thu, 10 Sep 2026 12:02:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.151 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041778; cv=none; b=DJ7yQkg0CqZHDahQS9UY6fqEKsp/MdVoiN96j9n25zKq6qgypFrwMmhd+bsHBY+SwdU/wq7VENfjj4RTIUYH3sNwn9a3dMLH2p9MMIPQwdbU385NNqlo3Xh51uNv1MmJr+aj3GoJrODb2TSODBJ0ZRHCyzRI+u/4iaHXwLJfQas= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041778; c=relaxed/simple; bh=FPRlk/3xXD2hlJWAg1kKXQRCO2yol8H3vn6v3bTyCKc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TcDhkbqbsVl+KvWYGxEjcUTEa7i8j/kaOhouz4RZ381IMz/GDICZxYOfjpteXhiiZh1gt4pqdzLsVf8EWBtgGP3SYegmOT8ttdP1wHsgkB0JqWNhGwS99y2l8ohuyZH8NYZiFPu8PHh4cxuNcCPi0klIdBFBYnz5a+rh9ODDBlY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=DQYJaEKc; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=OdIE7we/; arc=none smtp.client-ip=202.12.124.151 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="DQYJaEKc"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="OdIE7we/" Received: from phl-compute-09.internal (phl-compute-09.internal [10.202.2.49]) by mailfout.stl.internal (Postfix) with ESMTP id 139B41D0006F; Thu, 10 Sep 2026 08:02:53 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-09.internal (MEProxy); Thu, 10 Sep 2026 08:02:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789041772; x= 1789128172; bh=n4UhpKBn4QA7xtpBWtOXdoBNDcNab8zz8WQs3EPLY2Y=; b=D QYJaEKcZ1hIz2XR/+D07PnV8T7gpBimlKksRiqPz4pu6xdZWOXgpt8di24rsf1GI 2MeDZT7cKhgxtE0O3wDh/wCzIkeTkpJVt8WofzCNxeRLRmPAAYbWC03nWq27aTdh a4Ey0AucGzQyd+4vNmJrjye01jULfndzP37lAIQWBJBYlCcEgvHJ7gyGiWbK8BbJ c8iNLDIamxFBW3Xi+cKF+EiWAJ780GjR1YtTy6HBGF6XVHpeqr4zskiagoWOR/5z xkMx4G8QG7JmtC9/IC47GVHxiZNuH4f9KGrrbQnCEQyFpA7JhU2xclcCf26ebp86 ONZkCThxCo6JoZZDET3kg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789041772; x=1789128172; bh=n 4UhpKBn4QA7xtpBWtOXdoBNDcNab8zz8WQs3EPLY2Y=; b=OdIE7we/MnoC3LbWk 9ukn3kVN+oHt3PLQ4LUZkGHYu55kH+PzyxRM2jCKqFF4EfLVNC+VF6XkozOMlC/L wtBwROAN1SYB9xmHiOULwSW9iAdJSMDr/nchR3jEbUDKl4K3QE61qOqgLDnZGcph D40XoopUbOm3ct8KvsTw4ruc86yCAIJgTbsMqR6ySJrZZ6QzgFRAMI6N+hQ6u99+ I5xbLKDr0pEJOj2+4CufHwAiy4IEl+08V0wYwR1xkweYcDFNUmCyCYlQModpUeH+ N7h3gcGQJjfEgisomcuqu8GFwneEYA2fzuLaKNjeCqV0kaVFDtD1jaAObsd3csMB W/r5Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEVCHaom43ud29NDpH2Ao3u7HJ/1ECQvA0KKWbvYxhttLa4TfzKR6zF5UuYx3UdSX No90AnuR4y57x1CX4a9AAsMV1JR4eQYCvRkfoTmW6C9Lc6lI8AZbTNwmqz4UTzT3vnk7qg ph+TCI1xI7srBtDO6Tg5MLzI4vqgzAzQGVqRHCjhPRmxizMOs2MvZfOp4fytB88FrNmNEM gORbCSjN6PT3vgIUt7WNrJZsAWFpVymtt3pMoPYb3jF9rU+m42uK7S67tVqaIc+biyrQXs /trDbiCnZfBDKlB4R/Y3z4EcguiWXCyuMp2o4aP4AZn5jbn1xeIsCoHcLAwPh3Pk2CuoWx 8p4Y64DVzNe7hQUlN9olRsCMsh7I+2WdAMmvc4ZwWsdlHLbISDjGhTQqurtvg9FXVBrK6g hFWMxp+oNKj0WJ5sMz+bvWMo4a+Vq6h9DnxgfzlFTz0MXUKTFhYkWK1NX8H9A7Nn62zpk7 xF3otauenLIWJw4+B/KeBQmo6JY/mOELVZxg6fKA59kIiWJk/1ZMgAMpkh8MAGV/oCJAD9 lge/aTFbPneV6plNechjNgQD8OFPfJ+qKOGstpABLeEQsHjsO6ZdsAjJfQ+kHiJQfi91// 8DGmvge6Lc9HhqMV+S+6Fywa+hq3IEjO22LnfnDXEqbkHvpCZNsllHyaVImA X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Sep 2026 08:02:51 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v2 03/12] mm/khugepaged: rename mthp_present_ptes bitmap to eligible_ptes Date: Thu, 10 Sep 2026 13:02:23 +0100 Message-ID: <20260910120238.2529819-4-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260910120238.2529819-1-kirill@shutemov.name> References: <20260910120238.2529819-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" The name says less than the bit means. A set bit means not only that the PTE is present, but also that it passed the other checks: uffd, lazyfree, anonymity, sharing. The PTE can be considered a collapse source. mthp_collapse() then reads the bitmap starting at the PMD order, so the bitmap is not specific to mTHP either. Name it for what a set bit means, and update the comments that named it. No functional change. Assisted-by: LLM Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Signed-off-by: Kiryl Shutsemau (Meta) --- mm/khugepaged.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 4979a93e3648..081f705cfca2 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -115,8 +115,8 @@ struct collapse_control { /* nodemask for allocation fallback */ nodemask_t alloc_nmask; =20 - /* Each bit represents a single occupied (!none/zero) page. */ - DECLARE_BITMAP(mthp_present_ptes, MAX_PTRS_PER_PTE); + /* Each bit marks a PTE the scan accepted as a collapse source */ + DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); }; =20 /** @@ -627,7 +627,7 @@ static void collapse_control_init_scan(struct collapse_= control *cc) { memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); - bitmap_zero(cc->mthp_present_ptes, MAX_PTRS_PER_PTE); + bitmap_zero(cc->eligible_ptes, MAX_PTRS_PER_PTE); } =20 static void release_pte_folio(struct folio *folio) @@ -1482,15 +1482,15 @@ static unsigned int max_order_from_offset(unsigned = int offset) * mthp_collapse() consumes the bitmap that is generated during * collapse_scan_pmd() to determine what regions and mTHP orders fit best. * - * Each bit in cc->mthp_present_ptes represents a single occupied (!none/z= ero) - * page. We start at the PMD order and check if it is eligible for collaps= e; + * Each bit in cc->eligible_ptes marks a PTE the scan accepted as a collap= se + * source. We start at the PMD order and check if it is eligible for colla= pse; * if not, we check the left and right halves of the PTE page table we are * examining at a lower order. * - * For each of these, we determine how many PTE entries are occupied in the - * range of PTE entries we propose to collapse, then we compare this to a - * threshold number of PTE entries which would need to be occupied for a - * collapse to be permitted at that order (accounting for max_ptes_none). + * For each of these, we count the eligible PTEs in the range we propose to + * collapse, then we compare this to the number of eligible PTEs the range + * would need for a collapse to be permitted at that order (accounting for + * max_ptes_none). * * If a collapse is permitted, we attempt to collapse the PTE range into a * mTHP. @@ -1499,7 +1499,7 @@ static enum scan_result mthp_collapse(struct mm_struc= t *mm, unsigned long address, int referenced, int unmapped, struct collapse_control *cc, unsigned long enabled_orders) { - unsigned int nr_occupied_ptes, nr_ptes, max_ptes_none; + unsigned int nr_eligible_ptes, nr_ptes, max_ptes_none; enum scan_result last_result =3D SCAN_FAIL; int collapsed =3D 0; bool alloc_failed =3D false; @@ -1514,18 +1514,18 @@ static enum scan_result mthp_collapse(struct mm_str= uct *mm, goto next_order; =20 max_ptes_none =3D collapse_max_ptes_none(cc, NULL, order); - nr_occupied_ptes =3D bitmap_weight_from(cc->mthp_present_ptes, offset, + nr_eligible_ptes =3D bitmap_weight_from(cc->eligible_ptes, offset, offset + nr_ptes); =20 /* * Swap PTEs accepted during the scan are counted in @unmapped, - * not in the present-PTE bitmap. Account them for the PMD-order + * not in cc->eligible_ptes. Account them for the PMD-order * candidate. */ if (is_pmd_order(order)) - nr_occupied_ptes +=3D unmapped; + nr_eligible_ptes +=3D unmapped; =20 - if (nr_occupied_ptes >=3D nr_ptes - max_ptes_none) { + if (nr_eligible_ptes >=3D nr_ptes - max_ptes_none) { enum scan_result ret; =20 collapse_address =3D address + offset * PAGE_SIZE; @@ -1731,8 +1731,8 @@ static enum scan_result collapse_scan_pmd(struct mm_s= truct *mm, } } =20 - /* Set bit for occupied pages */ - __set_bit(i, cc->mthp_present_ptes); + /* The scan accepted this PTE as a collapse source */ + __set_bit(i, cc->eligible_ptes); /* * Record which node the original page is from and save this * information to cc->node_load[]. --=20 2.54.0 From nobody Fri Sep 25 16:54:11 2026 Received: from fhigh-b4-smtp.messagingengine.com (fhigh-b4-smtp.messagingengine.com [202.12.124.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F3503CAE8F for ; Thu, 10 Sep 2026 12:02:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.155 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041780; cv=none; b=fNZul/Fwp20Rnd1oDkvR6xrX9eKo3IgxN36LEWAEGQn3AKpr6UToSqYw6wbvaHeg+tUsIhWHaCh+I1ijXntYjCqqtJvSngatvxqbZGrJJPllD95OLl2WJeV6/8+x34FkBF8xQlLp2vz5eWn4GgAyImGZABG0WGdoQxKcKfD+91g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041780; c=relaxed/simple; bh=TKB6T8VtIMFCYymtjKb5LK/4Whb6YCWyLf1ODb2k4yM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lTij8TvqYbxK61ibUqGTr8IvqICxkmN26CF/e0CdDlttWpY2cHQenOhjg2OwouxFiOHvnAUbthgsx5BOa+jSs/Z47zMAfmLS4eG2O146G6P7nRZYIjWbuMS/QKu+x14sekVUUhAhyxRXkSbYHVrZiueNq3+ly0yGjLwPiN5N2H0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=T2xJjUL1; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Y8Ku98M2; arc=none smtp.client-ip=202.12.124.155 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="T2xJjUL1"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Y8Ku98M2" Received: from phl-compute-08.internal (phl-compute-08.internal [10.202.2.48]) by mailfhigh.stl.internal (Postfix) with ESMTP id 7B6607A0063; Thu, 10 Sep 2026 08:02:55 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-08.internal (MEProxy); Thu, 10 Sep 2026 08:02:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789041775; x= 1789128175; bh=avMkfLVeU1+RrFVq7dqGuSLkfIFTUHtIkOYpYaIIbGQ=; b=T 2xJjUL1ZJCMdxAKPg3uwAZtQTW68SPh1VtoRK4teQNVLihar9mgoJPDh2apslDeA 1UM2at3b4xJ/2c+9iAKLWFmzd1N6vFSZDxnC4bndyr4q7u/ad++sYlngeR425orY D+Ljclm8KB6q0URN/k2NHOfS3/na4MCYfoKeYamG/VZrvp1r0oJ407MuHIry7FdI pZAL+n3NnpEKJjwttDohUawq6dnMK36cTRDkroJPLbfPHsi/Db7cCa0C9njJcKQR Q1SoW7+Rk/wGjglQSsZ8XmkMWhjKLCrk+o3IpSMfIJH4VimrFQNkRqm7PYM+err0 nJNDFd70gqONHTDdIruhQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789041775; x=1789128175; bh=a vMkfLVeU1+RrFVq7dqGuSLkfIFTUHtIkOYpYaIIbGQ=; b=Y8Ku98M2fH9TsfHUC 3J8yJsl2EJkbzK6Gfgh5GCaY7XPesGxNTFJJK8nc5OfGbYrKKitgz8xIKbeWTTZg IhIgEbvrjhDaGk5ZRVsrdLnmM1i7XBdjH7uSCDFm2ActUo7TjXZGd1D3Y9RSvmp2 Uddob6uTDYV0+RhjSj+AyCdNFimQyIBYebGIWnUsb6iWe6Nkzh9hS0/dYVwccugA D6NdpvpWiQZO2WSyfj8jWEd7ThveOJC4Sx71oG//Ee/nMTXp8s1FbhBvUDsQpIWy NZbFxVbtq2aDzbFSRHBQ/xB9WDP9dr9tsFWT54iV8MXNexIFLXA1PNpN00EhAGvq CxEwQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTGH8GFWwEUBtPxKJ67Rq2x/vdej+Q6nUCGcsDIesBFp0X3hc4jYyZuxmlnDOcFOMj pjMT1hhilwwz+XztIrpfsk+jOgPg51Zko9HtX5FOXtwZM/xuJUIPYajGOB/mCLABcXoBUN tOK7IZxc0LQBxYWwS3D6Gb3+L6Rzatr9ZhNURjEQibWJdyHQU+pHQvTtQJMa086Ym+yd6R IInk1eh8W3seorGmDJU/iV1zGzfJwWvedZT0VKJsUgZ7fACBrQNd9k3L4EzTZNF344phSI j6ov0R0Vo2eJRufAS9ocD/WtRYG80u0cUfKpHkiyow/cp96ftI+MA3MvYDL4cDgvN+YPWI iSSoN3tZcUvge+vs0OnV2TRyl66gqJJm6yyXdLsCQmmv5R9/2dOzXjaaugeZEOo2bKsdLn 9Q6IYpu71Hg/t6ms3VhXiJHEQFznSJfCXNaOL5wkZlE2eD9GJm3ckhTUkCo99iaIo8fFZt Z3yVb0sfuZaVL2lyCOZ1pLGWh2G2T149WHWYELBYlPm3ikACBgYkR2HIDatKODuiSPcH0C rcm8Aj4+jRYJ7o3Sxu+zhnWq7U6r5xV4akeqZzrPPWBQ+8nvmBqEy12KZDbrq45kbjWhWh Wjb9o4DEwpayt5ffXVXUbFhQmM9vRL5Hgpv8+bPDKo3xT37tSMdaJ/LtGCxQ X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Sep 2026 08:02:54 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v2 04/12] mm/collapse: add collapse.h for the collapse interface Date: Thu, 10 Sep 2026 13:02:24 +0100 Message-ID: <20260910120238.2529819-5-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260910120238.2529819-1-kirill@shutemov.name> References: <20260910120238.2529819-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" khugepaged.c holds both the users of collapse and the machinery that performs it. The daemon's scan loop, the sysfs tunables, MADV_COLLAPSE's entry point and the collapse itself all sit in one file and reach into each other freely. Nothing marks where a user ends and the engine begins. Start drawing that line. Add mm/collapse.h for what the two sides have to agree on: - enum scan_result - what the engine hands back; - struct collapse_control - the state a request carries. And two constants move with them: - KHUGEPAGED_MAX_PTES_LIMIT -> COLLAPSE_MAX_PTES_LIMIT; - KHUGEPAGED_MIN_MTHP_ORDER -> COLLAPSE_MIN_MTHP_ORDER. Neither is a fact about the daemon, so both lose the KHUGEPAGED_ prefix. No functional change. Assisted-by: LLM Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Signed-off-by: Kiryl Shutsemau (Meta) --- MAINTAINERS | 1 + mm/collapse.h | 64 +++++++++++++++++++++++++++++++++++++++ mm/khugepaged.c | 79 ++++++++----------------------------------------- 3 files changed, 78 insertions(+), 66 deletions(-) create mode 100644 mm/collapse.h diff --git a/MAINTAINERS b/MAINTAINERS index 35262aa3e67e..36315972156b 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17434,6 +17434,7 @@ F: Documentation/admin-guide/mm/transhuge.rst F: include/linux/huge_mm.h F: include/linux/khugepaged.h F: include/trace/events/huge_memory.h +F: mm/collapse.h F: mm/huge_memory.c F: mm/khugepaged.c F: mm/mm_slot.h diff --git a/mm/collapse.h b/mm/collapse.h new file mode 100644 index 000000000000..b115034d9018 --- /dev/null +++ b/mm/collapse.h @@ -0,0 +1,64 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __MM_COLLAPSE_H +#define __MM_COLLAPSE_H + +#include +#include +#include +#include + +#define COLLAPSE_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1) +#define COLLAPSE_MIN_MTHP_ORDER 2 + +enum scan_result { + SCAN_FAIL, + SCAN_SUCCEED, + SCAN_NO_PTE_TABLE, + SCAN_PMD_MAPPED, + SCAN_EXCEED_NONE_PTE, + SCAN_EXCEED_SWAP_PTE, + SCAN_EXCEED_SHARED_PTE, + SCAN_PTE_NON_PRESENT, + SCAN_PTE_UFFD, + SCAN_PTE_MAPPED_HUGEPAGE, + SCAN_LACK_REFERENCED_PAGE, + SCAN_PAGE_NULL, + SCAN_SCAN_ABORT, + SCAN_PAGE_COUNT, + SCAN_PAGE_LRU, + SCAN_PAGE_LOCK, + SCAN_PAGE_ANON, + SCAN_PAGE_LAZYFREE, + SCAN_PAGE_COMPOUND, + SCAN_ANY_PROCESS, + SCAN_VMA_NULL, + SCAN_VMA_CHECK, + SCAN_ADDRESS_RANGE, + SCAN_DEL_PAGE_LRU, + SCAN_ALLOC_HUGE_PAGE_FAIL, + SCAN_CGROUP_CHARGE_FAIL, + SCAN_TRUNCATED, + SCAN_PAGE_HAS_PRIVATE, + SCAN_STORE_FAILED, + SCAN_COPY_MC, + SCAN_PAGE_FILLED, + SCAN_PAGE_DIRTY_OR_WRITEBACK, +}; + +struct collapse_control { + bool is_khugepaged; + + /* Num pages scanned per node */ + u32 node_load[MAX_NUMNODES]; + + /* Num pages scanned (see khugepaged_pages_to_scan) */ + unsigned int progress; + + /* nodemask for allocation fallback */ + nodemask_t alloc_nmask; + + /* Each bit marks a PTE the scan accepted as a collapse source */ + DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); +}; + +#endif /* __MM_COLLAPSE_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 081f705cfca2..8889f75cf45f 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -26,44 +26,10 @@ #include =20 #include +#include "collapse.h" #include "internal.h" -#include "page_alloc.h" #include "mm_slot.h" - -enum scan_result { - SCAN_FAIL, - SCAN_SUCCEED, - SCAN_NO_PTE_TABLE, - SCAN_PMD_MAPPED, - SCAN_EXCEED_NONE_PTE, - SCAN_EXCEED_SWAP_PTE, - SCAN_EXCEED_SHARED_PTE, - SCAN_PTE_NON_PRESENT, - SCAN_PTE_UFFD, - SCAN_PTE_MAPPED_HUGEPAGE, - SCAN_LACK_REFERENCED_PAGE, - SCAN_PAGE_NULL, - SCAN_SCAN_ABORT, - SCAN_PAGE_COUNT, - SCAN_PAGE_LRU, - SCAN_PAGE_LOCK, - SCAN_PAGE_ANON, - SCAN_PAGE_LAZYFREE, - SCAN_PAGE_COMPOUND, - SCAN_ANY_PROCESS, - SCAN_VMA_NULL, - SCAN_VMA_CHECK, - SCAN_ADDRESS_RANGE, - SCAN_DEL_PAGE_LRU, - SCAN_ALLOC_HUGE_PAGE_FAIL, - SCAN_CGROUP_CHARGE_FAIL, - SCAN_TRUNCATED, - SCAN_PAGE_HAS_PRIVATE, - SCAN_STORE_FAILED, - SCAN_COPY_MC, - SCAN_PAGE_FILLED, - SCAN_PAGE_DIRTY_OR_WRITEBACK, -}; +#include "page_alloc.h" =20 #define CREATE_TRACE_POINTS #include @@ -91,7 +57,6 @@ static DECLARE_WAIT_QUEUE_HEAD(khugepaged_wait); * * Note that these are only respected if collapse was initiated by khugepa= ged. */ -#define KHUGEPAGED_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1) unsigned int khugepaged_max_ptes_none __read_mostly; static unsigned int khugepaged_max_ptes_swap __read_mostly; static unsigned int khugepaged_max_ptes_shared __read_mostly; @@ -101,24 +66,6 @@ static DEFINE_READ_MOSTLY_HASHTABLE(mm_slots_hash, MM_S= LOTS_HASH_BITS); =20 static struct kmem_cache *mm_slot_cache __ro_after_init; =20 -#define KHUGEPAGED_MIN_MTHP_ORDER 2 - -struct collapse_control { - bool is_khugepaged; - - /* Num pages scanned per node */ - u32 node_load[MAX_NUMNODES]; - - /* Num pages scanned (see khugepaged_pages_to_scan) */ - unsigned int progress; - - /* nodemask for allocation fallback */ - nodemask_t alloc_nmask; - - /* Each bit marks a PTE the scan accepted as a collapse source */ - DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); -}; - /** * struct khugepaged_scan - cursor for scanning * @mm_head: the head of the mm list to scan @@ -267,7 +214,7 @@ static ssize_t max_ptes_none_store(struct kobject *kobj, unsigned long max_ptes_none; =20 err =3D kstrtoul(buf, 10, &max_ptes_none); - if (err || max_ptes_none > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_none > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; =20 khugepaged_max_ptes_none =3D max_ptes_none; @@ -292,7 +239,7 @@ static ssize_t max_ptes_swap_store(struct kobject *kobj, unsigned long max_ptes_swap; =20 err =3D kstrtoul(buf, 10, &max_ptes_swap); - if (err || max_ptes_swap > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_swap > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; =20 khugepaged_max_ptes_swap =3D max_ptes_swap; @@ -318,7 +265,7 @@ static ssize_t max_ptes_shared_store(struct kobject *ko= bj, unsigned long max_ptes_shared; =20 err =3D kstrtoul(buf, 10, &max_ptes_shared); - if (err || max_ptes_shared > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_shared > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; =20 khugepaged_max_ptes_shared =3D max_ptes_shared; @@ -378,19 +325,19 @@ static unsigned int collapse_max_ptes_none(struct col= lapse_control *cc, if (is_pmd_order(order)) return max_ptes_none; /* - * for mTHP collapse with the sysctl value set to KHUGEPAGED_MAX_PTES_LIM= IT, + * for mTHP collapse with the sysctl value set to COLLAPSE_MAX_PTES_LIMIT, * scale the maximum number of PTEs to the order of the collapse. */ - if (max_ptes_none =3D=3D KHUGEPAGED_MAX_PTES_LIMIT) + if (max_ptes_none =3D=3D COLLAPSE_MAX_PTES_LIMIT) return (1 << order) - 1; /* - * For mTHP collapse of values other than 0 or KHUGEPAGED_MAX_PTES_LIMIT, + * For mTHP collapse of values other than 0 or COLLAPSE_MAX_PTES_LIMIT, * emit a warning and return 0. */ if (max_ptes_none) pr_warn_once("mTHP collapse does not support max_ptes_none" " values other than 0 or %u, defaulting to 0.\n", - KHUGEPAGED_MAX_PTES_LIMIT); + COLLAPSE_MAX_PTES_LIMIT); return 0; } =20 @@ -476,7 +423,7 @@ int __init khugepaged_init(void) return -ENOMEM; =20 khugepaged_pages_to_scan =3D HPAGE_PMD_NR * 8; - khugepaged_max_ptes_none =3D KHUGEPAGED_MAX_PTES_LIMIT; + khugepaged_max_ptes_none =3D COLLAPSE_MAX_PTES_LIMIT; khugepaged_max_ptes_swap =3D HPAGE_PMD_NR / 8; khugepaged_max_ptes_shared =3D HPAGE_PMD_NR / 2; =20 @@ -1571,8 +1518,8 @@ static enum scan_result mthp_collapse(struct mm_struc= t *mm, * any smaller order enabled. When at the smallest order * we must always move to the next offset. */ - if (order > KHUGEPAGED_MIN_MTHP_ORDER && - (enabled_orders & GENMASK(order - 1, 0))) { + if (order > COLLAPSE_MIN_MTHP_ORDER && + (enabled_orders & GENMASK(order - 1, 0))) { order--; continue; } @@ -1636,7 +1583,7 @@ static enum scan_result collapse_scan_pmd(struct mm_s= truct *mm, * is then checked again in mthp_collapse() for each attempted order. */ if (enabled_orders !=3D BIT(HPAGE_PMD_ORDER)) - max_ptes_none =3D KHUGEPAGED_MAX_PTES_LIMIT; + max_ptes_none =3D COLLAPSE_MAX_PTES_LIMIT; =20 pte =3D pte_offset_map_lock(mm, pmd, start_addr, &ptl); if (!pte) { --=20 2.54.0 From nobody Fri Sep 25 16:54:11 2026 Received: from fout-b8-smtp.messagingengine.com (fout-b8-smtp.messagingengine.com [202.12.124.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AEB5947D95D for ; Thu, 10 Sep 2026 12:02:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.151 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041782; cv=none; b=a5cIR7gMgtAIefaFCR/K9snO1QzCIs2jHIELtambV746oaUGrF1mkKu7dR9omKQtcDSBLWxHUc8zqA/bT4kfYfxrQCx6fipweSDzkos117Fh9ZCIgyjLGeyhsiO5XfF4kVvmR3aYTUv2AN3u2xkKlKY60NHLiKJ/CD0jGS8AKKk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041782; c=relaxed/simple; bh=cG7PZeO4vVIEMitM5ANYNNpZQZT4qWJ/nQtxv2fgw9Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GBaLEna0RklvXQpWivGYP+GFBazCRgIL6UXet4k0s/WvWb0YvxfH4A2oDSTfLfd3dnx5F1GzLaspXSPtpfyP784pYn2yM/7M9DEEmKRO54TZrDMdQsbtZkoNfcm69ql7mYkwxms0bj9VtbcRwnlDGEBTkGWKdXZuYVNBwwMbgWk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=M6QwsyHS; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=EZs4UQag; arc=none smtp.client-ip=202.12.124.151 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="M6QwsyHS"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="EZs4UQag" Received: from phl-compute-02.internal (phl-compute-02.internal [10.202.2.42]) by mailfout.stl.internal (Postfix) with ESMTP id 596D11D00082; Thu, 10 Sep 2026 08:02:58 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-02.internal (MEProxy); Thu, 10 Sep 2026 08:02:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789041778; x= 1789128178; bh=Zi9Br7spViFzIpBHYvoM9/hr0EidxdhOJvgib0JvV/I=; b=M 6QwsyHSzHhQjQyRp9IhCMxwVtmN40c0rvhcww5pPaMc3Q5wjBHVKSnogCng9hQ8h eXMq/ksU948pcoz157P7Y+JdnOTtZC6DQqm9DGzoYHN/64yyHi5IRq9Yb/P0kYv5 RF4F6v7ctLsCc0p+PQxjSuoflhNGJ34ZfG01LJo27DAGSquFHJOaQdXg+nuUcqvN lzB1a9m8uoAhRNcpbHPBEFeMJK6vpEMHS5d4tY4QHY21jRx86BQzUjGVGyzaZAix mzIKQ4AnZwDQqChUndsSGr5QLKx7nX4GvqaMVVT4K6xT9AKc2En+H/0td60bAr28 um6q6XGoP/ADeb20UX03A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789041778; x=1789128178; bh=Z i9Br7spViFzIpBHYvoM9/hr0EidxdhOJvgib0JvV/I=; b=EZs4UQagWnjMRJY8n dwul6dhnRsNVBbOGkbxuB55G8b590Ft2DKL3WEA4wTSlStDdHbSIVhL3OCHo7ij2 ZUaz4EltcBkmDUHA6fT3l+GaufNCbJKr+941BKolRymhpGfUrhA3/ZQKgxWntHbp VO9ZU8/rt2exzldrQrexAJnXvyM8/9vqXpYeP0A6f3Lgpxyt2ZPeeTB9cjs5sVbE iuacJp2TJ9r+CcMmb61HenV8EOy2XybtB0aA60pKJUTFGHoDhs5vrYTPsdTTChZQ 6qftDozxQaT3J2s2YKiV1qg19jkZSSUpEf3wjW3wpNMXlW6VginIgvvSmkPMrDFz CEWlg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEVCHaom43ud29NDpH2Ao3u7HJ/1ECQvA0KKWbvYxhttLa4TfzKR6zF5UuYx3UdSX No90AnuR4y57x1CX4a9AAsMV1JR4eQYCvRkfoTmW6C9Lc6lI8AZbTNwmqz4UTzT3vnk7qg ph+TCI1xI7srBtDO6Tg5MLzI4vqgzAzQGVqRHCjhPRmxizMOs2MvZfOp4fytB88FrNmNEM gORbCSjN6PT3vgIUt7WNrJZsAWFpVymtt3pMoPYb3jF9rU+m42uK7S67tVqaIc+biyrQXs /trDbiCnZfBDKlB4R/Y3z4EcguiWXCyuMp2o4aP4AZn5jbn1xeIsCoHcLAwPh3Pk2CuofT Jefx/QVBBT5LzvkNKCImwRofU2KfKrzkA4sXRxOF4SJL0A+7dONvQ+g1Y7GP6Gq0rb6iub Lq5y978/FBTb820ej2lIE86TqP1eiz+NuLTZEKg8TiQhQL/DDETAtan/Ue1rQ5SofX5Fha OtoWUgjOPaFtfglWmw09XiGyPtRgIg8Udwy64ywQhxVQ1phhpr7lhREjVv4JsCNPiChw79 NpqsbF3hbiDxD3GNnoRt+PEUME8+8lMjxgM/FxKu0ylP3usgudkvQx6n1IwTWjQHqvEPHs guiSvWmdQPYmA6Skyd27PB0qtwtUDhaDCFYKFXCrzfE85+3ErRmW4pa/U/Xg X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Sep 2026 08:02:57 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v2 05/12] mm/collapse: state what a collapse may do in the policy Date: Thu, 10 Sep 2026 13:02:25 +0100 Message-ID: <20260910120238.2529819-6-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260910120238.2529819-1-kirill@shutemov.name> References: <20260910120238.2529819-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" Tests scattered through the collapse path decide what a collapse is allowed to do by asking whether khugepaged started it. Between them they settle: - which VMAs are eligible, and how hard to try for a folio; - how many empty, swapped-out or shared PTEs a window may contain, and whether a sub-PMD window is held to a stricter rule than a PMD; - whether a range has to look used, and whether a MADV_FREE'd page is left alone; - whether the PMD is mapped as part of the request, and whether dirty pages are worth writing back and retrying. None of those is a fact about khugepaged. Each is something the caller decided before asking, and the collapse code should not have to look up who called to find out. Add struct collapse_policy for the caller to fill: khugepaged from its own settings, MADV_COLLAPSE from the fact that a user asked explicitly. Every test becomes a read of a field, and cc->is_khugepaged goes, having no reader left. khugepaged fills the policy once per scan pass, MADV_COLLAPSE once per call. That is the one change in behaviour. The max_ptes_* limits and the defrag setting behind the allocation mask are sampled once per pass rather than on every table. A table scanned early in a pass and one scanned late are then judged alike. collapse_file() also drops a NULL check on the collapse_control. It has one call site, reached only from collapse_single_pmd(), which dereferences cc unconditionally, so the check was already dead. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) Reviewed-by: Baolin Wang Reviewed-by: Zi Yan --- mm/collapse.h | 31 ++++++++++++- mm/khugepaged.c | 114 ++++++++++++++++++++++++++---------------------- 2 files changed, 93 insertions(+), 52 deletions(-) diff --git a/mm/collapse.h b/mm/collapse.h index b115034d9018..7044dc71c7c2 100644 --- a/mm/collapse.h +++ b/mm/collapse.h @@ -45,8 +45,37 @@ enum scan_result { SCAN_PAGE_DIRTY_OR_WRITEBACK, }; =20 +/* What a collapse is allowed to do, decided by the caller that asks for i= t */ +struct collapse_policy { + /* Limits, stated per PMD; HPAGE_PMD_NR means "no limit" */ + unsigned int max_ptes_none; + unsigned int max_ptes_swap; + unsigned int max_ptes_shared; + + /* Take no swapped-out or shared PTE into a sub-PMD collapse */ + bool strict_sub_pmd; + + /* Leave clean lazyfree folios to reclaim rather than collapse them */ + bool skip_lazyfree; + + /* Refuse a range with no sign of use */ + bool require_referenced; + + /* Map the PMD over a file collapse instead of leaving it to a fault */ + bool install_pmd; + + /* Write dirty pages back and retry once instead of refusing them */ + bool writeback_dirty; + + /* How hard to try for a destination folio */ + gfp_t gfp; + + /* Which VMAs are eligible, as thp_vma_allowable_orders() spells it */ + enum tva_type tva_type; +}; + struct collapse_control { - bool is_khugepaged; + struct collapse_policy policy; =20 /* Num pages scanned per node */ u32 node_load[MAX_NUMNODES]; diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 8889f75cf45f..16cb94ea3455 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -314,15 +314,12 @@ static bool pte_none_or_zero(pte_t pte) static unsigned int collapse_max_ptes_none(struct collapse_control *cc, struct vm_area_struct *vma, unsigned int order) { - const unsigned int max_ptes_none =3D khugepaged_max_ptes_none; + const unsigned int max_ptes_none =3D cc->policy.max_ptes_none; =20 if (vma && userfaultfd_armed(vma)) return 0; - /* for MADV_COLLAPSE, allow any empty/shared zeropage PTEs */ - if (!cc->is_khugepaged) - return HPAGE_PMD_NR; - /* for PMD collapse, respect the user defined maximum */ - if (is_pmd_order(order)) + /* The limit as given, at the PMD order and wherever it is not capped */ + if (is_pmd_order(order) || !cc->policy.strict_sub_pmd) return max_ptes_none; /* * for mTHP collapse with the sysctl value set to COLLAPSE_MAX_PTES_LIMIT, @@ -354,19 +351,12 @@ static unsigned int collapse_max_ptes_shared(struct c= ollapse_control *cc, unsigned int order) { /* - * For MADV_COLLAPSE, do not restrict the number of PTEs that map shared - * anonymous pages. + * A sub-PMD window held to the strict rule takes no shared page at all: + * an mTHP is not worth the CoW-breaking. */ - if (!cc->is_khugepaged) - return HPAGE_PMD_NR; - /* - * for mTHP collapse do not allow collapsing anonymous memory pages that - * are shared between processes. - */ - if (!is_pmd_order(order)) + if (!is_pmd_order(order) && cc->policy.strict_sub_pmd) return 0; - /* for PMD collapse, respect the user defined maximum */ - return khugepaged_max_ptes_shared; + return cc->policy.max_ptes_shared; } =20 /** @@ -382,16 +372,12 @@ static unsigned int collapse_max_ptes_swap(struct col= lapse_control *cc, unsigned int order) { /* - * For MADV_COLLAPSE, do not restrict the number PTEs entries or - * pagecache entries that are non-present. + * A sub-PMD window held to the strict rule takes nothing non-present: + * reading pages back to build an mTHP is not worth the latency. */ - if (!cc->is_khugepaged) - return HPAGE_PMD_NR; - /* for mTHP collapse do not allow any non-present PTEs or pagecache entri= es */ - if (!is_pmd_order(order)) + if (!is_pmd_order(order) && cc->policy.strict_sub_pmd) return 0; - /* for PMD collapse, respect the user defined maximum */ - return khugepaged_max_ptes_swap; + return cc->policy.max_ptes_swap; } =20 int hugepage_madvise(struct vm_area_struct *vma, @@ -678,7 +664,7 @@ static enum scan_result __collapse_huge_page_isolate(st= ruct vm_area_struct *vma, * If the vma has the VM_DROPPABLE flag, the collapse will * preserve the lazyfree property without needing to skip. */ - if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) && + if (cc->policy.skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) && folio_test_lazyfree(folio) && !pte_dirty(pteval)) { result =3D SCAN_PAGE_LAZYFREE; goto out; @@ -767,12 +753,12 @@ static enum scan_result __collapse_huge_page_isolate(= struct vm_area_struct *vma, if (folio_test_large(folio)) list_add_tail(&folio->lru, compound_pagelist); next: - if (cc->is_khugepaged && + if (cc->policy.require_referenced && folio_pte_referenced(folio, vma, addr, pteval)) referenced++; } =20 - if (unlikely(cc->is_khugepaged && !referenced)) { + if (unlikely(cc->policy.require_referenced && !referenced)) { result =3D SCAN_LACK_REFERENCED_PAGE; } else { result =3D SCAN_SUCCEED; @@ -938,9 +924,7 @@ static void khugepaged_alloc_sleep(void) remove_wait_queue(&khugepaged_wait, &wait); } =20 -static struct collapse_control khugepaged_collapse_control =3D { - .is_khugepaged =3D true, -}; +static struct collapse_control khugepaged_collapse_control; =20 static bool collapse_scan_abort(int nid, struct collapse_control *cc) { @@ -976,6 +960,36 @@ static inline gfp_t alloc_hugepage_khugepaged_gfpmask(= void) return khugepaged_defrag() ? GFP_TRANSHUGE : GFP_TRANSHUGE_LIGHT; } =20 +/* khugepaged collapses on its own initiative, so it obeys its own setting= s */ +static void collapse_policy_khugepaged(struct collapse_policy *p) +{ + p->max_ptes_none =3D READ_ONCE(khugepaged_max_ptes_none); + p->max_ptes_swap =3D READ_ONCE(khugepaged_max_ptes_swap); + p->max_ptes_shared =3D READ_ONCE(khugepaged_max_ptes_shared); + p->strict_sub_pmd =3D true; + p->skip_lazyfree =3D true; + p->require_referenced =3D true; + p->install_pmd =3D false; + p->writeback_dirty =3D false; + p->gfp =3D alloc_hugepage_khugepaged_gfpmask(); + p->tva_type =3D TVA_KHUGEPAGED; +} + +/* MADV_COLLAPSE was asked for explicitly, so it is not held to those */ +static void collapse_policy_forced(struct collapse_policy *p) +{ + p->max_ptes_none =3D HPAGE_PMD_NR; + p->max_ptes_swap =3D HPAGE_PMD_NR; + p->max_ptes_shared =3D HPAGE_PMD_NR; + p->strict_sub_pmd =3D false; + p->skip_lazyfree =3D false; + p->require_referenced =3D false; + p->install_pmd =3D true; + p->writeback_dirty =3D true; + p->gfp =3D GFP_TRANSHUGE; + p->tva_type =3D TVA_FORCED_COLLAPSE; +} + #ifdef CONFIG_NUMA static int collapse_find_target_node(struct collapse_control *cc) { @@ -1013,8 +1027,7 @@ static enum scan_result hugepage_vma_revalidate(struc= t mm_struct *mm, unsigned l struct collapse_control *cc, unsigned int order) { struct vm_area_struct *vma; - enum tva_type type =3D cc->is_khugepaged ? TVA_KHUGEPAGED : - TVA_FORCED_COLLAPSE; + enum tva_type type =3D cc->policy.tva_type; =20 if (unlikely(collapse_test_exit_or_disable(mm))) return SCAN_ANY_PROCESS; @@ -1197,8 +1210,7 @@ static enum scan_result __collapse_huge_page_swapin(s= truct mm_struct *mm, static enum scan_result alloc_charge_folio(struct folio **foliop, struct m= m_struct *mm, struct collapse_control *cc, unsigned int order) { - gfp_t gfp =3D (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() : - GFP_TRANSHUGE); + gfp_t gfp =3D cc->policy.gfp; int node =3D collapse_find_target_node(cc); struct folio *folio; =20 @@ -1551,7 +1563,7 @@ static enum scan_result collapse_scan_pmd(struct mm_s= truct *mm, const unsigned int max_ptes_shared =3D collapse_max_ptes_shared(cc, HPAGE= _PMD_ORDER); const unsigned int max_ptes_swap =3D collapse_max_ptes_swap(cc, HPAGE_PMD= _ORDER); unsigned int max_ptes_none =3D collapse_max_ptes_none(cc, vma, HPAGE_PMD_= ORDER); - enum tva_type tva_flags =3D cc->is_khugepaged ? TVA_KHUGEPAGED : TVA_FORC= ED_COLLAPSE; + enum tva_type tva_flags =3D cc->policy.tva_type; pmd_t *pmd; pte_t *pte, *_pte, pteval; int i; @@ -1651,7 +1663,7 @@ static enum scan_result collapse_scan_pmd(struct mm_s= truct *mm, * If the vma has the VM_DROPPABLE flag, the collapse will * preserve the lazyfree property without needing to skip. */ - if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) && + if (cc->policy.skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) && folio_test_lazyfree(folio) && !pte_dirty(pteval)) { result =3D SCAN_PAGE_LAZYFREE; failed_pfn =3D folio_pfn(folio); @@ -1717,13 +1729,13 @@ static enum scan_result collapse_scan_pmd(struct mm= _struct *mm, goto out_unmap; } =20 - if (cc->is_khugepaged && + if (cc->policy.require_referenced && folio_pte_referenced(folio, vma, addr, pteval)) referenced++; } - if (cc->is_khugepaged && - (!referenced || - (unmapped && referenced < HPAGE_PMD_NR / 2))) { + if (cc->policy.require_referenced && + (!referenced || + (unmapped && referenced < HPAGE_PMD_NR / 2))) { result =3D SCAN_LACK_REFERENCED_PAGE; } else { result =3D SCAN_SUCCEED; @@ -2582,11 +2594,11 @@ static enum scan_result collapse_file(struct mm_str= uct *mm, unsigned long addr, xas_unlock_irq(&xas); =20 /* - * Remove pte page tables, so we can re-fault the page as huge. - * If MADV_COLLAPSE, adjust result to call try_collapse_pte_mapped_thp(). + * Remove pte page tables, so we can re-fault the page as huge. A + * caller that wants the PMD mapped now is told to go and do that. */ retract_page_tables(mapping, start); - if (cc && !cc->is_khugepaged) + if (cc->policy.install_pmd) result =3D SCAN_PTE_MAPPED_HUGEPAGE; folio_unlock(new_folio); =20 @@ -2773,11 +2785,8 @@ static enum scan_result collapse_single_pmd(unsigned= long addr, retry: result =3D collapse_scan_file(mm, addr, file, pgoff, cc); =20 - /* - * For MADV_COLLAPSE, when encountering dirty pages, try to writeback, - * then retry the collapse one time. - */ - if (!cc->is_khugepaged && result =3D=3D SCAN_PAGE_DIRTY_OR_WRITEBACK && + /* Dirty pages are worth a writeback and one more try, if asked for */ + if (cc->policy.writeback_dirty && result =3D=3D SCAN_PAGE_DIRTY_OR_WRITEB= ACK && !triggered_wb && mapping_can_writeback(file->f_mapping)) { const loff_t lstart =3D (loff_t)pgoff << PAGE_SHIFT; const loff_t lend =3D lstart + HPAGE_PMD_SIZE - 1; @@ -2794,7 +2803,7 @@ static enum scan_result collapse_single_pmd(unsigned = long addr, result =3D SCAN_ANY_PROCESS; else result =3D try_collapse_pte_mapped_thp(mm, addr, - !cc->is_khugepaged); + cc->policy.install_pmd); if (result =3D=3D SCAN_PMD_MAPPED) result =3D SCAN_SUCCEED; mmap_read_unlock(mm); @@ -2943,6 +2952,9 @@ static void khugepaged_do_scan(struct collapse_contro= l *cc) =20 lru_add_drain_all(); =20 + /* One policy for the whole pass, so every table is judged the same */ + collapse_policy_khugepaged(&cc->policy); + cc->progress =3D 0; while (true) { cond_resched(); @@ -3170,7 +3182,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, cc =3D kmalloc_obj(*cc); if (!cc) return -ENOMEM; - cc->is_khugepaged =3D false; + collapse_policy_forced(&cc->policy); cc->progress =3D 0; =20 lru_add_drain_all(); --=20 2.54.0 From nobody Fri Sep 25 16:54:11 2026 Received: from fhigh-b4-smtp.messagingengine.com (fhigh-b4-smtp.messagingengine.com [202.12.124.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C24E4825D2 for ; Thu, 10 Sep 2026 12:03:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.155 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041783; cv=none; b=EYPrnCKXIGsnmoyZhvrE9WWCN4iOoVaY127WrjhHGCe8WYfTul9ctx2P01WIgQ4K/sof2Bevdz5IHI4kwvYQHe3fDtgOdyDkVHsXfuuh+Qp1a4j9P+piGbYE3MQP2wiGCpKvY2z15dL+ce2eg+IMtDhS5QBEu7hModTZh3oFeMU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041783; c=relaxed/simple; bh=n05nd2gOqrE7MnKoi71ylY2ERmggyhsXbG/A0PYB+cQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hx4mQEbYK6xYaEdWxXJlRLZtkOn0tqA6gphfjWCb07n1pyQt7BesO+T0OreRxZ9CtFeH+0oHUDpgGSPq0Lgmx+L08AcocXwGbLltV25pSzz6KoUjb7CNYAk+qvHk6zRGImwVpcZhBzV0zhpgZ+RDetCW8q+083aXLCaMW5GYsFQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=e3XMao9J; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=h6WSVLIR; arc=none smtp.client-ip=202.12.124.155 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="e3XMao9J"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="h6WSVLIR" Received: from phl-compute-04.internal (phl-compute-04.internal [10.202.2.44]) by mailfhigh.stl.internal (Postfix) with ESMTP id 07FD67A0041; Thu, 10 Sep 2026 08:03:01 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-04.internal (MEProxy); Thu, 10 Sep 2026 08:03:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789041780; x= 1789128180; bh=i+eq01ZyefSmr6d5OlM3Mqwv3cqLqDcRLOfX9U2ttRU=; b=e 3XMao9Jtt5cKGQR2StPFOHxjNhnlH+KM78Bk9e8qphJaAiwsiIq/tqXzAzLJcJon R890L+n6Z9KJfRq1GCfBsPFlg41OKgxvTSFB3LvW7h/aZDnGmfPRaRHK8R8MntoL JLx/gPgn1Vy3bGoVtLteJA6XG2lwchbglZ5GEKqtN/W2oqCCHQsgoOaKL0/AAkWW gR+MgNsdMClvl/S4AoUio0Fm6FfqY6ulrPM7DCO5Q9QrwidrmkSEdDjNpG4QQ8D0 t5hHwbKKzyNOQSrfhUM9K05TMYNWDmYO6w3iyUbUHayG0AF17YXc4673mrbJ+5oS Qky2ae+BEq/UBddnqLa6w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789041780; x=1789128180; bh=i +eq01ZyefSmr6d5OlM3Mqwv3cqLqDcRLOfX9U2ttRU=; b=h6WSVLIRkvudhqu7s I1uLv247vGEPvst/A5i2Q+4GDwVl6bFTMpdjvweqShRNzhVPX5c1ALy9eh3oeO/z j+vHCt6eRwcUquM7hxKnFaVXRmYvJ7rwl0vgoWLdvFuqn2GvlMo6obMjwh9P/CcP u2DFYJ7IrXP1TQ8G2xF3eU/yTV7jb2ECP/eaZY/E/NtfJrLWlHxQG59O3HyqO2M7 EvvcC5HNsjEhOpAS55r+ZZybEz8tzeQkZGydfgUwByRe0018zNbiqP5EhtK06k+n mOEl7AmJG7k08qTBvpLGnilACvMnNI5Nh+olzKOYwl1MD1WdUwUWNKjx7eICY0C8 osWcA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTGH8GFWwEUBtPxKJ67Rq2x/vdej+Q6nUCGcsDIesBFp0X3hc4jYyZuxmlnDOcFOMj pjMT1hhilwwz+XztIrpfsk+jOgPg51Zko9HtX5FOXtwZM/xuJUIPYajGOB/mCLABcXoBUN tOK7IZxc0LQBxYWwS3D6Gb3+L6Rzatr9ZhNURjEQibWJdyHQU+pHQvTtQJMa086Ym+yd6R IInk1eh8W3seorGmDJU/iV1zGzfJwWvedZT0VKJsUgZ7fACBrQNd9k3L4EzTZNF344phSI j6ov0R0Vo2eJRufAS9ocD/WtRYG80u0cUfKpHkiyow/cp96ftI+MA3MvYDL4cDgvN+YPPG fRRoz69BGBlITkK/LSke5bri0wWGJeuTSXrBdqgFR2AmcTn0QklmXi9hR9ckuC2cEN5St4 ua3Gjf98LYsRIRagxKu1OvxBNeGrAncStQm1puanRFRCh5145vT6aJEB8yRYFEePWZjpFC 2Wd2atVP/y/YgyZ1OHjajTW+iVe2fwHNhT/VTjz4FTkVYWvS6OAYE8Pkjsr1r2zZ/FP1f8 QEMd++23RwdcOAZEcqBM8kCUcL7jhZDY3s1C6V0em9DHt9GTrEYwhZVexEuOWHMZchUEfg ZUZCLGM30ZvYpJLotneYsm3cxOOLa318xExILXQQQjz2+hBAu7jn5zdj105A X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Sep 2026 08:02:59 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v2 06/12] mm/collapse: drop the collapse_possible() wrapper Date: Thu, 10 Sep 2026 13:02:26 +0100 Message-ID: <20260910120238.2529819-7-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260910120238.2529819-1-kirill@shutemov.name> References: <20260910120238.2529819-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" collapse_possible() only forwards to collapse_possible_orders() and turns its mask into a bool. Its three callers can test the mask themselves. No functional change. Assisted-by: LLM Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Signed-off-by: Kiryl Shutsemau (Meta) --- mm/khugepaged.c | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 16cb94ea3455..2a9f6d8d6695 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -512,17 +512,11 @@ static unsigned long collapse_possible_orders(struct = vm_area_struct *vma, return thp_vma_allowable_orders(vma, vm_flags, tva_flags, orders); } =20 -static bool collapse_possible(struct vm_area_struct *vma, - vm_flags_t vm_flags, enum tva_type tva_flags) -{ - return collapse_possible_orders(vma, vm_flags, tva_flags); -} - void khugepaged_enter_vma(struct vm_area_struct *vma, vm_flags_t vm_flags) { - if (!mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm) && hugepage_enabled() - && collapse_possible(vma, vm_flags, TVA_KHUGEPAGED)) + if (!mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm) && hugepage_enabled() && + collapse_possible_orders(vma, vm_flags, TVA_KHUGEPAGED)) __khugepaged_enter(vma->vm_mm); } =20 @@ -2857,7 +2851,8 @@ static void collapse_scan_mm_slot(unsigned int progre= ss_max, cc->progress++; break; } - if (!collapse_possible(vma, vma->vm_flags, TVA_KHUGEPAGED)) { + if (!collapse_possible_orders(vma, vma->vm_flags, + TVA_KHUGEPAGED)) { cc->progress++; continue; } @@ -3170,7 +3165,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, BUG_ON(vma->vm_start > start); BUG_ON(vma->vm_end < end); =20 - if (!collapse_possible(vma, vma->vm_flags, TVA_FORCED_COLLAPSE)) + if (!collapse_possible_orders(vma, vma->vm_flags, TVA_FORCED_COLLAPSE)) return -EINVAL; =20 hstart =3D ALIGN(start, HPAGE_PMD_SIZE); --=20 2.54.0 From nobody Fri Sep 25 16:54:11 2026 Received: from fhigh-b4-smtp.messagingengine.com (fhigh-b4-smtp.messagingengine.com [202.12.124.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B88BF48592D for ; Thu, 10 Sep 2026 12:03:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.155 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041786; cv=none; b=aFTyExGxRO7dulGUUbHCyYstahTjHOYfz8MWMI4Mv7DAze62dipt6Lt3gp3FseVuusZ1hwzsSapRWhy7hK1PTxhynMbYmB7m2rgjZCceMM0Q0XwdOQzKtcaY4TfUqKSCfi2qqrEUefpRLjCeqrfwRPaJoFXcLrSoORqyIIOlGV4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041786; c=relaxed/simple; bh=FsntK079ojb1NSC5PYEXoKjgxib/Hm4XcgWRqpaOndM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qcVhFqOKTfoHcqpwDC/FgKlgCp1dOwy6X2M3Z193SQ3IZlixsFhxk11D2SqV8mFy6/1pbPa4t89ePY4k5pwt9EV9OXbsrz9xmqBh1B8Ea9wa6ZA/ccI24RYoIki2VFq5ZbAtjLNfKSqRSHAHFMrlx3KGUNT/NfbiKR6wYbS1pLI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=EE4ZmaYn; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=FimdyeX+; arc=none smtp.client-ip=202.12.124.155 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="EE4ZmaYn"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="FimdyeX+" Received: from phl-compute-12.internal (phl-compute-12.internal [10.202.2.52]) by mailfhigh.stl.internal (Postfix) with ESMTP id 9F3887A007F; Thu, 10 Sep 2026 08:03:03 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-12.internal (MEProxy); Thu, 10 Sep 2026 08:03:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789041783; x= 1789128183; bh=3P2qVnrMLPFA6A73AmtwzSkxgpkuNKe+y7ebrjYnAuo=; b=E E4ZmaYn+xzGWUrFAGxtuMzW2lgwc+5yQHzoW3tUnRfFCReRK2pZcN92qd/KdVxme dP3feioWtbpSG2j/mUuul/CIX6lTRTy1jqjZqiIHXMjItrCTLoRk3r3xbAFlNCBf Wc+dyyesEZJtvOPrrDRIcLfl7pLJ3dk5Nby609dVpb+nySy3xIktrteojQ5z8skB XE7+hKRX4ykHnWVBPgd7f4Sw1hOU6A3MHLWjb1BSDKcIeGczic5AQaRJOT5MW1QH 2sBVYhCKvaQobKwXWJrZvXm4vdxzIuIp78jJNZfcYIovJ12S2yVXwNxUK7lqvllA +raLeuq6eBdSAu88GXTig== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789041783; x=1789128183; bh=3 P2qVnrMLPFA6A73AmtwzSkxgpkuNKe+y7ebrjYnAuo=; b=FimdyeX+I8ss92KOM 8oprLKsYgVjsO24m/kSxVTReJ3pLyOhpsEoBfBke0mHEo89/2GPjIh21X3eUfh1W CMhmU4BnBNMpCGXdGnneUhzs3HeauGmd4aWY3jv6cjp6HVfAduY3RYEMpHvs0hc2 3TtHp8GTUxM/DHseXCgI5EfgnQ78WfaaqODpG4wGbA5ucgXC+DHlnoX58ugjPlsA Zpaf3TuB+vGS9a10O4ujldrhv0tPtdG9XXY2gYxFjWzBOJedsX/b7HPdT/xFKPvT QgJaJpIMU5eisXKkJSCojKr3LqdER3TwMQMgbiR1BA/7yy/U1UymWurbc1Q6CXXa twGTw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEVCHaom43ud29NDpH2Ao3u7HJ/1ECQvA0KKWbvYxhttLa4TfzKR6zF5UuYx3UdSX No90AnuR4y57x1CX4a9AAsMV1JR4eQYCvRkfoTmW6C9Lc6lI8AZbTNwmqz4UTzT3vnk7qg ph+TCI1xI7srBtDO6Tg5MLzI4vqgzAzQGVqRHCjhPRmxizMOs2MvZfOp4fytB88FrNmNEM gORbCSjN6PT3vgIUt7WNrJZsAWFpVymtt3pMoPYb3jF9rU+m42uK7S67tVqaIc+biyrQXs /trDbiCnZfBDKlB4R/Y3z4EcguiWXCyuMp2o4aP4AZn5jbn1xeIsCoHcLAwPh3Pk2Cuodv jdvPCsgoell1QwQOyq0lvh52cXi36YL8xROZPqyLVvgeWiebTZZvmdLF57eQz75ZMRoVgY ljlYRhwz6gxNZoz8VK/FA0zpwTwkN521ywySnZx/Vyub2vFHRCS3SsNc4fNTJQkus5W4dZ T6RMH4i7St575scy7WNwXS5FlI7RziOulg6yL8fAjn9BarOT/I7NsStoDPdP8Akd7ladfM lVox66/G2cLMhZYiWREwxfvNJVxLr6APh9MeluH/Fa47cZA0HeyBDs5+Iop73ziWvaYaiw 9CgEi9ST5w0GkoTFe5UjIrTl235lyRYdb63sqaOkJwGIBVe3qTOpwpS2xTpQ X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Sep 2026 08:03:02 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v2 07/12] mm/collapse: name the per-table scan reset for what it resets Date: Thu, 10 Sep 2026 13:02:27 +0100 Message-ID: <20260910120238.2529819-8-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260910120238.2529819-1-kirill@shutemov.name> References: <20260910120238.2529819-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" collapse_control_init_scan() resets what one scan accumulates: the node load, the allocation nodemask and the eligible-PTE bitmap. It runs before every PTE table a scan is given, not once per control, so its name points at the wrong thing. Call it collapse_scan_reset(). Preparation for giving a control a real init and release, run once each for a whole series of scans. Two names a word apart would then stand for two different jobs. No functional change. Assisted-by: LLM Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Signed-off-by: Kiryl Shutsemau (Meta) --- mm/khugepaged.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 2a9f6d8d6695..d38ba030107b 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -550,7 +550,7 @@ void __khugepaged_exit(struct mm_struct *mm) } } =20 -static void collapse_control_init_scan(struct collapse_control *cc) +static void collapse_scan_reset(struct collapse_control *cc) { memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); @@ -1579,7 +1579,7 @@ static enum scan_result collapse_scan_pmd(struct mm_s= truct *mm, goto out; } =20 - collapse_control_init_scan(cc); + collapse_scan_reset(cc); =20 enabled_orders =3D collapse_possible_orders(vma, vma->vm_flags, tva_flags= ); =20 @@ -2657,7 +2657,7 @@ static enum scan_result collapse_scan_file(struct mm_= struct *mm, =20 present =3D 0; swap =3D 0; - collapse_control_init_scan(cc); + collapse_scan_reset(cc); rcu_read_lock(); xas_for_each(&xas, folio, start + HPAGE_PMD_NR - 1) { if (xas_retry(&xas, folio)) --=20 2.54.0 From nobody Fri Sep 25 16:54:11 2026 Received: from fout-b8-smtp.messagingengine.com (fout-b8-smtp.messagingengine.com [202.12.124.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B6EA248595E for ; Thu, 10 Sep 2026 12:03:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.151 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041789; cv=none; b=MUvaWTZmDkFdcKP7vOa4T1KRUsp937xyjXV+ClvJe6Ztr5Nqn0GoTzRXJ4p11h+D/gwNsbdDSgdXZT/B3xHQKbCMoxmG3XTjQXmlOw5isZsKe0WE6xD8saV3K8bDzJipcmioDC1KXO59KyaA7gAH0NRh/l4l9R3sKDo5egOySk4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041789; c=relaxed/simple; bh=7NRubkgTn8Nnkhm969X7TMmPymG/ik7w1EfHwzSDk9E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ulfIXGOfVPIsbwNE6SUqdK6o6k0ZRU6maPPyFERfx1reT6c65Gl7pqCJ4Hg6NGYVbCCVq/b4IEkJEGgK69ksaGqaxkMI7Lnd3UjEj1yssBafMvOtH4vMmu4422IaFElUe7XSzN5UTT9G4jIYvlajIi0USi0c/2jjjOUjMwQ4/bI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=Y5uNEFu9; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=wo/dhbxg; arc=none smtp.client-ip=202.12.124.151 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="Y5uNEFu9"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="wo/dhbxg" Received: from phl-compute-06.internal (phl-compute-06.internal [10.202.2.46]) by mailfout.stl.internal (Postfix) with ESMTP id 827641D00066; Thu, 10 Sep 2026 08:03:06 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-06.internal (MEProxy); Thu, 10 Sep 2026 08:03:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789041786; x= 1789128186; bh=gkbS+/8H8cjRZTrO9cANk9swJlxr7xbA69wrjLxeH1k=; b=Y 5uNEFu9g5zI/9GZci4IlHuvqBZ65sxmy92epLuJiG2aGVsLjyFKl9EFIc092672P xojJI7z4z/P+ZsnRE9/aDGQQF1Wnr6Xix3HHJ2aIxS1BLpbg7LtH9ccBDr0Xk1f/ 1YBSype1bHc6pJGQPY9tmJ8G2NNg7vpf1GjTKZQwi3JuDmJ7dQOpJ79LmYHLGQR/ aAKaiLQ+7NKWryD+2VVnh3yVnFXwiQHGfIY5NS4iUKHjVd9ZRUqH3Gu8bFfn0YQ8 IQq1kj9qlJqLRzpxBZjt+5dgHRqF779mn8RBQ3sD2Zdk5nCAI6Ch7rxkKYotDI5l UKsopU1KAzsO1irtKZH3g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789041786; x=1789128186; bh=g kbS+/8H8cjRZTrO9cANk9swJlxr7xbA69wrjLxeH1k=; b=wo/dhbxgqF0vFveNO w6Krudi5AcFkflQ+6CCl3zTfQ/PTnTik7/ZFiTKIGAmy+6I0iKbf4QY2cOLgAhXN 3Pz7db9VWoEbFiOWTt+LNLLklPdHWplQgeEJDo27RwHNjmYn9yn+H8cmW+ZGmKed bg+Kfw2UpuUgk3DJThY2AFRU/dDAs/LUoZ0blSGJW822gxhgUK8dS9jZraKRdbtY T4mNzWw72xSFycPMGSJzFZxEbQHqhUMrSByCgV8FjnPoFHePUAmzFqgOHdEMxw8O 1WCex/6rtDy3SAXvt9VpqG3qrAO/VIFRfamRy7wuqPHUUEpu0CyTeY1XUZ6lB5JY BN1Aw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEVCHaom43ud29NDpH2Ao3u7HJ/1ECQvA0KKWbvYxhttLa4TfzKR6zF5UuYx3UdSX No90AnuR4y57x1CX4a9AAsMV1JR4eQYCvRkfoTmW6C9Lc6lI8AZbTNwmqz4UTzT3vnk7qg ph+TCI1xI7srBtDO6Tg5MLzI4vqgzAzQGVqRHCjhPRmxizMOs2MvZfOp4fytB88FrNmNEM gORbCSjN6PT3vgIUt7WNrJZsAWFpVymtt3pMoPYb3jF9rU+m42uK7S67tVqaIc+biyrQXs /trDbiCnZfBDKlB4R/Y3z4EcguiWXCyuMp2o4aP4AZn5jbn1xeIsCoHcLAwPh3Pk2Cuoou HO6PwX09gzkHi4Gfrp1fdDra8jIcgYFcUSIPWD8T0ZYcjJ6nwMoBf0TKt047iMECBtffKf rHkHW1XTqgiZy01N2cVx2fZ3gma/bpB2S4iN1pneY8zee/4AdZ4D2swSIHJ4TbOQgQY5/a 7oVvzIcwhVzSBLAGWN0xi7FL+N4peXGFhJXorWyQ/ealT98Z9HQn+ypEiKi/ygbkGx7XTg ev2TK9XWzY2D4TU9M38D26PqkwmnpL6/mEW/CeVCfhgHn68miLohoQNvFETQ+eYzbg3o+6 qhhRpEWSw7furBGYZ3huCTGHg34rPlTAxVnHBXTIHY7oomTW7auHCvNSa8eg X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Sep 2026 08:03:05 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v2 08/12] mm/collapse: separate scanning a PTE table from collapsing it Date: Thu, 10 Sep 2026 13:02:28 +0100 Message-ID: <20260910120238.2529819-9-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260910120238.2529819-1-kirill@shutemov.name> References: <20260910120238.2529819-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" A collapse is two jobs. One reads a PTE table under mmap_lock and decides whether the range is worth collapsing. The other allocates, isolates, copies and flushes, and wants the lock given up first. collapse_single_pmd() did both, so the boundary between them was somewhere in the middle of a function. Give each half its own function: - collapse_scan_pmd() scans one table and only reads. The anonymous scan that used to carry that name keeps its body as collapse_scan_anon_pmd(), and collapse_scan_pmd() is now the entry that picks the anonymous or the file side. - collapse_run_pmd() does the collapse the scan asked for. SCAN_SUCCEED from the scan means there is something to run; anything else is why there is not. collapse_single_pmd() is now the two of them with the mmap_lock drop in between, so its callers see what they saw before. What the scan found and the run needs travels in collapse_control. For an anonymous table that is the orders and the referenced and swapped-out counts. For a file it is the file itself, the offset in it, and whether the PMD folio is already in the page cache. The file side moves with the anonymous one. collapse_scan_file() used to run with mmap_lock already given up, and called collapse_file() itself when the page cache looked worth it. It now runs under the lock like the anonymous scan and only judges; the run does the collapse. A file collapse works on the page cache and never sees a VMA, so the scan takes the file reference while it still has one and the run gives it back. That changes what a refused file table costs khugepaged. Every file table it scanned used to end its pass over that mm, because the lock had been dropped to scan it; now only a table it goes on to collapse does. Two things on the file side stop being rescanned. When the page cache already holds the PMD folio, the scan says so and the run goes straight to retracting the PTE table. A run that refuses dirty pages and may write them back retries collapse_file() alone. The checks the scan makes ahead of it are ones collapse_file() repeats under the page cache lock. Tracing changes with it. mm_khugepaged_scan_pmd and mm_khugepaged_scan_file used to fire after the collapse, so for an accepted table their status field carried what the collapse made of it. They now fire before it and read SCAN_SUCCEED for an accepted table. What the collapse then made of it is for mm_collapse_huge_page and mm_khugepaged_collapse_file to report. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) Reviewed-by: Zi Yan --- mm/collapse.h | 16 ++++++ mm/khugepaged.c | 147 ++++++++++++++++++++++++++++++++++++------------ 2 files changed, 128 insertions(+), 35 deletions(-) diff --git a/mm/collapse.h b/mm/collapse.h index 7044dc71c7c2..346859a2184f 100644 --- a/mm/collapse.h +++ b/mm/collapse.h @@ -88,6 +88,22 @@ struct collapse_control { =20 /* Each bit marks a PTE the scan accepted as a collapse source */ DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); + + /* + * What a scan found and the run after it needs. Live only between the + * two, and read by nobody else. + * + * The file side takes a reference while it still has the VMA, since a + * file collapse works on the page cache and never sees one; the run is + * what gives it back. A scan that found the PMD folio already in the + * cache leaves only the PTE table to retract. + */ + unsigned long scan_orders; + int scan_referenced; + int scan_unmapped; + struct file *scan_file; + pgoff_t scan_pgoff; + bool scan_retract_only; }; =20 #endif /* __MM_COLLAPSE_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index d38ba030107b..c26907300c23 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1550,14 +1550,14 @@ static enum scan_result mthp_collapse(struct mm_str= uct *mm, return last_result; } =20 -static enum scan_result collapse_scan_pmd(struct mm_struct *mm, - struct vm_area_struct *vma, unsigned long start_addr, - bool *lock_dropped, struct collapse_control *cc) +static enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma, + unsigned long start_addr, struct collapse_control *cc) { const unsigned int max_ptes_shared =3D collapse_max_ptes_shared(cc, HPAGE= _PMD_ORDER); const unsigned int max_ptes_swap =3D collapse_max_ptes_swap(cc, HPAGE_PMD= _ORDER); unsigned int max_ptes_none =3D collapse_max_ptes_none(cc, vma, HPAGE_PMD_= ORDER); enum tva_type tva_flags =3D cc->policy.tva_type; + struct mm_struct *mm =3D vma->vm_mm; pmd_t *pmd; pte_t *pte, *_pte, pteval; int i; @@ -1737,12 +1737,9 @@ static enum scan_result collapse_scan_pmd(struct mm_= struct *mm, out_unmap: pte_unmap_unlock(pte, ptl); if (result =3D=3D SCAN_SUCCEED) { - /* collapse_huge_page() expects the lock to be dropped before calling */ - mmap_read_unlock(mm); - result =3D mthp_collapse(mm, start_addr, referenced, - unmapped, cc, enabled_orders); - /* mmap_lock was released above, set lock_dropped */ - *lock_dropped =3D true; + cc->scan_orders =3D enabled_orders; + cc->scan_referenced =3D referenced; + cc->scan_unmapped =3D unmapped; } out: trace_mm_khugepaged_scan_pmd(mm, failed_pfn, referenced, @@ -2739,45 +2736,95 @@ static enum scan_result collapse_scan_file(struct m= m_struct *mm, else cc->progress +=3D HPAGE_PMD_NR; =20 - if (result =3D=3D SCAN_SUCCEED) { - if (present < HPAGE_PMD_NR - max_ptes_none) { - result =3D SCAN_EXCEED_NONE_PTE; - count_vm_event(THP_SCAN_EXCEED_NONE_PTE); - } else { - result =3D collapse_file(mm, addr, file, start, cc); - } + if (result =3D=3D SCAN_SUCCEED && present < HPAGE_PMD_NR - max_ptes_none)= { + result =3D SCAN_EXCEED_NONE_PTE; + count_vm_event(THP_SCAN_EXCEED_NONE_PTE); } =20 - trace_mm_khugepaged_scan_file(mm, failed_pfn, file, present, swap, result= ); + trace_mm_khugepaged_scan_file(mm, failed_pfn, file, present, swap, + result); return result; } =20 -/* - * Try to collapse a single PMD starting at a PMD aligned addr, and return - * the results. - */ -static enum scan_result collapse_single_pmd(unsigned long addr, - struct vm_area_struct *vma, bool *lock_dropped, - struct collapse_control *cc) +static void collapse_control_init(struct collapse_control *cc) +{ + cc->progress =3D 0; + cc->scan_file =3D NULL; +} + +static void collapse_control_release(struct collapse_control *cc) +{ + /* A scan that took a file reference should have been run */ + if (WARN_ON_ONCE(cc->scan_file)) { + fput(cc->scan_file); + cc->scan_file =3D NULL; + } +} + +static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, + unsigned long addr, struct collapse_control *cc) { - struct mm_struct *mm =3D vma->vm_mm; - bool triggered_wb =3D false; enum scan_result result; - struct file *file; pgoff_t pgoff; =20 - mmap_assert_locked(mm); + mmap_assert_locked(vma->vm_mm); + /* Whatever the last scan found has to have been run by now */ + if (WARN_ON_ONCE(cc->scan_file)) { + fput(cc->scan_file); + cc->scan_file =3D NULL; + } =20 if (vma_is_anonymous(vma)) - return collapse_scan_pmd(mm, vma, addr, lock_dropped, cc); + return collapse_scan_anon_pmd(vma, addr, cc); =20 - file =3D get_file(vma->vm_file); pgoff =3D linear_page_index(vma, addr); + result =3D collapse_scan_file(vma->vm_mm, addr, vma->vm_file, pgoff, cc); + switch (result) { + case SCAN_SUCCEED: + cc->scan_retract_only =3D false; + break; + case SCAN_PTE_MAPPED_HUGEPAGE: + /* + * The page cache already holds the PMD folio; what is left is + * to retract the PTE table, which is the run's job. + */ + cc->scan_retract_only =3D true; + result =3D SCAN_SUCCEED; + break; + default: + return result; + } =20 - mmap_read_unlock(mm); - *lock_dropped =3D true; + /* + * A file collapse works on the page cache and never sees a VMA, so take + * what it needs from this one while it is still here. + */ + cc->scan_file =3D get_file(vma->vm_file); + cc->scan_pgoff =3D pgoff; + return result; +} + +static enum scan_result collapse_run_pmd(struct mm_struct *mm, + unsigned long addr, struct collapse_control *cc) +{ + struct file *file =3D cc->scan_file; + bool triggered_wb =3D false; + enum scan_result result; + pgoff_t pgoff; + + if (!file) + return mthp_collapse(mm, addr, cc->scan_referenced, + cc->scan_unmapped, cc, cc->scan_orders); + + cc->scan_file =3D NULL; + pgoff =3D cc->scan_pgoff; + + if (cc->scan_retract_only) { + result =3D SCAN_PTE_MAPPED_HUGEPAGE; + goto retract; + } retry: - result =3D collapse_scan_file(mm, addr, file, pgoff, cc); + result =3D collapse_file(mm, addr, file, pgoff, cc); =20 /* Dirty pages are worth a writeback and one more try, if asked for */ if (cc->policy.writeback_dirty && result =3D=3D SCAN_PAGE_DIRTY_OR_WRITEB= ACK && @@ -2789,8 +2836,13 @@ static enum scan_result collapse_single_pmd(unsigned= long addr, triggered_wb =3D true; goto retry; } +retract: fput(file); =20 + /* + * A PMD folio is in the page cache, whether the collapse just put it + * there or found it: retract the PTE table, and map the PMD if asked. + */ if (result =3D=3D SCAN_PTE_MAPPED_HUGEPAGE) { mmap_read_lock(mm); if (collapse_test_exit_or_disable(mm)) @@ -2805,6 +2857,28 @@ static enum scan_result collapse_single_pmd(unsigned= long addr, return result; } =20 +/* + * Try to collapse a single PMD starting at a PMD aligned addr, and return + * the results. + */ +static enum scan_result collapse_single_pmd(unsigned long addr, + struct vm_area_struct *vma, bool *lock_dropped, + struct collapse_control *cc) +{ + struct mm_struct *mm =3D vma->vm_mm; + enum scan_result result; + + result =3D collapse_scan_pmd(vma, addr, cc); + if (result !=3D SCAN_SUCCEED) + return result; + + /* The collapse takes its own locks, so give this up */ + mmap_read_unlock(mm); + *lock_dropped =3D true; + + return collapse_run_pmd(mm, addr, cc); +} + static void collapse_scan_mm_slot(unsigned int progress_max, enum scan_result *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock) @@ -2947,10 +3021,10 @@ static void khugepaged_do_scan(struct collapse_cont= rol *cc) =20 lru_add_drain_all(); =20 + collapse_control_init(cc); /* One policy for the whole pass, so every table is judged the same */ collapse_policy_khugepaged(&cc->policy); =20 - cc->progress =3D 0; while (true) { cond_resched(); =20 @@ -2981,6 +3055,8 @@ static void khugepaged_do_scan(struct collapse_contro= l *cc) khugepaged_alloc_sleep(); } } + + collapse_control_release(cc); } =20 static bool khugepaged_should_wakeup(void) @@ -3177,8 +3253,8 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, cc =3D kmalloc_obj(*cc); if (!cc) return -ENOMEM; + collapse_control_init(cc); collapse_policy_forced(&cc->policy); - cc->progress =3D 0; =20 lru_add_drain_all(); =20 @@ -3235,6 +3311,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, } out_nolock: mmap_assert_locked(mm); + collapse_control_release(cc); kfree(cc); =20 return thps =3D=3D ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0 --=20 2.54.0 From nobody Fri Sep 25 16:54:11 2026 Received: from fhigh-b4-smtp.messagingengine.com (fhigh-b4-smtp.messagingengine.com [202.12.124.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F40DD4825CB for ; Thu, 10 Sep 2026 12:03:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.155 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041791; cv=none; b=P8afAz38cxPMwIUPKVlgb9xAtAQQZsKgwSB78PyWRAxNTk6PFVVf4eV2yuQAW9kIbnq3t8oj/MJZY1maPC2S2b719AYK3R0MTKYHbmhwhHfhec+UZYsDbaGElPz8cKBEG5PkPwzYvfgF2/3FlRdqmsZN7NMAu/NvmRNWkxk6ENs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041791; c=relaxed/simple; bh=YBHrSEcRxpQBAJBkpCgju4aZIhAPhowSY8zdrX1ZCxk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kBTCfiGulVbST9uz574c97CSa+nErhJLoBvNX/L0IhI1XL0dGhjuhAZgUCLkiAje5B43ff5pI3MzjU+cSVfYhoN1ZlutGBKLu01VwbE0txGIO5HKlpy4s0DSudAVtqvO/sML9QRruPnA4XaDKzndN8rOobaBzbw6tbbWcxphwGc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=csNCK16C; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=bKrmnTez; arc=none smtp.client-ip=202.12.124.155 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="csNCK16C"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="bKrmnTez" Received: from phl-compute-02.internal (phl-compute-02.internal [10.202.2.42]) by mailfhigh.stl.internal (Postfix) with ESMTP id 096237A0063; Thu, 10 Sep 2026 08:03:09 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-02.internal (MEProxy); Thu, 10 Sep 2026 08:03:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789041788; x= 1789128188; bh=kpOrfAtuUuey4lAYKV0xF7Nvz8qXoEbNkfYNAyopBF4=; b=c sNCK16CKEi2s5emKL1cvcwYx6Sa/NQ5ooLbF4lmvC8MhDw2wisf+1Ungv5cOmvlK WYjr3ckTr7NWzAC2gXbDJ6TW8rh3zNh6fd0dFNteb4GJBU0gg142EmvNzCIIUywu ROFZfhRDnyIpols8ofo9cTA53xvwdBa0+dCQCK+Fax9kX3R8GZZk5WfMqIsEb93O QcRR1HQ8dyEebWOtsudne8zY/uhZlDBWIOPJexQpOHSuKiLSr56sf9echxmGiW4o EsLvaBTTKlK77Hqy9sTIv7xhzR6uL7WQ3Ou8y6yIWLeNxugdQnFhGQjOn0D1o4ls XJRfcB4HUjK09XF27afww== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789041788; x=1789128188; bh=k pOrfAtuUuey4lAYKV0xF7Nvz8qXoEbNkfYNAyopBF4=; b=bKrmnTezqC8+vQj/C xhJUHZcCchh1J1z3ldAQOj6PeZkDpbz7tSzdFwC2n67vIXCbEhGhJU8gL0NY93tj Ootw2wkgHaPYUUT6S0efB0T9QJa7wkcsMzxWVcGaKw/BnT4rbczTbhFnzHug97mH eUHiZ9NNL7m14Nja8DBUSGQe8x9zYA7qnVJLWkrkYIQFMLdcpHzivUzd33/F+xnc RIE6ocN965IDbRdLLFII/k71BCNgxibPWUmcq39v1gf4wvliVFa8PCHZHSQAmyIt 5/R6Xdw8B5SRphYaxxs9FNkI8imC1SI/ki3UO0a/ocLso+ejlJJ8HzSgjS+6RQ6V UhfDA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTGH8GFWwEUBtPxKJ67Rq2x/vdej+Q6nUCGcsDIesBFp0X3hc4jYyZuxmlnDOcFOMj pjMT1hhilwwz+XztIrpfsk+jOgPg51Zko9HtX5FOXtwZM/xuJUIPYajGOB/mCLABcXoBUN tOK7IZxc0LQBxYWwS3D6Gb3+L6Rzatr9ZhNURjEQibWJdyHQU+pHQvTtQJMa086Ym+yd6R IInk1eh8W3seorGmDJU/iV1zGzfJwWvedZT0VKJsUgZ7fACBrQNd9k3L4EzTZNF344phSI j6ov0R0Vo2eJRufAS9ocD/WtRYG80u0cUfKpHkiyow/cp96ftI+MA3MvYDL4cDgvN+YPme tp92wygh8UvLM6PCYZaDNz8msbrI5zSvPvb1pIQHvDWn9s3+Yn0I14Z4qCm6i2YJ5WJXO9 Zxa3QdhU5laepvew+dp+2M4dNa1cHgJcn6Yrd+E7A2ncU09Ya/eisbEbE6eBK3l/neISFL XLLv1OBTXWjf/tIgacW38eVhc79/2zgtxZUvViJtIzxSKs9jhdd9qk6Qn6N+IfCie094eb MrWqU1AWU5364KeQICC4uA8zbhIidrwdWihtI9VdYatJ5LhfCnPTnd2t4RV+8VJuxxyiX9 IehlICSYK1zeiTMF/ze/yFdZNVBmxEzCIqhYxb5+M1Oui8PZs4C8Z8TX9hSQ X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Sep 2026 08:03:08 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v2 09/12] mm/collapse: open-code collapse_single_pmd() in its two callers Date: Thu, 10 Sep 2026 13:02:29 +0100 Message-ID: <20260910120238.2529819-10-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260910120238.2529819-1-kirill@shutemov.name> References: <20260910120238.2529819-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" collapse_scan_pmd() and collapse_run_pmd() each have a clear locking contract. The scan is called with mmap_lock held for reading and returns with it still held. The collapse is called without it. collapse_single_pmd() kept that boundary inside itself. It dropped the lock on some paths and not others, and reported which by way of a bool its callers had to carry along and then act on. Open-code it in the two callers. Each scans under the lock it already holds and, on SCAN_SUCCEED, gives the lock up before running the collapse. khugepaged's lock_dropped and madvise_collapse()'s mmap_unlocked both go: the code dropping the lock is now the code that wanted to know. khugepaged's walk carries on to the next table while the scan keeps refusing, and ends once a collapse has taken the lock from under it. madvise_collapse() re-finds its VMA after a collapse, which it did before, and now uses a NULL vma to say that it has to. It still reports the drop to its own caller, from the line that does it. The lock is given up and taken again at the same points as before. No functional change. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) Reviewed-by: Zi Yan --- mm/khugepaged.c | 102 +++++++++++++++++++++++------------------------- 1 file changed, 49 insertions(+), 53 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index c26907300c23..9bdf12128357 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -2857,28 +2857,6 @@ static enum scan_result collapse_run_pmd(struct mm_s= truct *mm, return result; } =20 -/* - * Try to collapse a single PMD starting at a PMD aligned addr, and return - * the results. - */ -static enum scan_result collapse_single_pmd(unsigned long addr, - struct vm_area_struct *vma, bool *lock_dropped, - struct collapse_control *cc) -{ - struct mm_struct *mm =3D vma->vm_mm; - enum scan_result result; - - result =3D collapse_scan_pmd(vma, addr, cc); - if (result !=3D SCAN_SUCCEED) - return result; - - /* The collapse takes its own locks, so give this up */ - mmap_read_unlock(mm); - *lock_dropped =3D true; - - return collapse_run_pmd(mm, addr, cc); -} - static void collapse_scan_mm_slot(unsigned int progress_max, enum scan_result *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock) @@ -2941,7 +2919,7 @@ static void collapse_scan_mm_slot(unsigned int progre= ss_max, VM_BUG_ON(khugepaged_scan.address & ~HPAGE_PMD_MASK); =20 while (khugepaged_scan.address < hend) { - bool lock_dropped =3D false; + unsigned long addr; =20 cond_resched(); if (unlikely(collapse_test_exit_or_disable(mm))) @@ -2951,23 +2929,29 @@ static void collapse_scan_mm_slot(unsigned int prog= ress_max, khugepaged_scan.address + HPAGE_PMD_SIZE > hend); =20 - *result =3D collapse_single_pmd(khugepaged_scan.address, - vma, &lock_dropped, cc); - if (*result =3D=3D SCAN_SUCCEED) - khugepaged_pages_collapsed++; + addr =3D khugepaged_scan.address; /* move to next address */ khugepaged_scan.address +=3D HPAGE_PMD_SIZE; - if (lock_dropped) - /* - * We released mmap_lock so break loop. Note - * that we drop mmap_lock before all hugepage - * allocations, so if allocation fails, we are - * guaranteed to break here and report the - * correct result back to caller. - */ - goto breakouterloop_mmap_lock; - if (cc->progress >=3D progress_max) - goto breakouterloop; + + *result =3D collapse_scan_pmd(vma, addr, cc); + /* Nothing to collapse here, and the lock is still ours */ + if (*result !=3D SCAN_SUCCEED) { + if (cc->progress >=3D progress_max) + goto breakouterloop; + continue; + } + + /* + * A collapse takes its own locks and is slow enough + * that a writer should not wait behind it, so give the + * lock up. That ends this walk: vma and the mm are + * whatever the collapse leaves them. + */ + mmap_read_unlock(mm); + *result =3D collapse_run_pmd(mm, addr, cc); + if (*result =3D=3D SCAN_SUCCEED) + khugepaged_pages_collapsed++; + goto breakouterloop_mmap_lock; } } breakouterloop: @@ -3236,7 +3220,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, unsigned long hstart, hend, addr; enum scan_result last_fail =3D SCAN_FAIL; int thps =3D 0; - bool mmap_unlocked =3D false; =20 BUG_ON(vma->vm_start > start); BUG_ON(vma->vm_end < end); @@ -3259,25 +3242,40 @@ int madvise_collapse(struct vm_area_struct *vma, un= signed long start, lru_add_drain_all(); =20 for (addr =3D hstart; addr < hend; addr +=3D HPAGE_PMD_SIZE) { - enum scan_result result =3D SCAN_FAIL; + struct vm_area_struct *found; + enum scan_result result; =20 - if (mmap_unlocked) { + /* + * A collapse gives the lock up, so the VMA has to be found + * again after one: it can shrink while nothing is held. A scan + * that finds nothing to collapse leaves the lock alone, so a + * range that is already collapsed walks on without relocking. + */ + if (!vma) { cond_resched(); mmap_read_lock(mm); - mmap_unlocked =3D false; - *lock_dropped =3D true; - result =3D hugepage_vma_revalidate(mm, addr, false, &vma, + result =3D hugepage_vma_revalidate(mm, addr, false, &found, cc, HPAGE_PMD_ORDER); if (result !=3D SCAN_SUCCEED) { last_fail =3D result; - goto out_nolock; + goto out_locked; } - + vma =3D found; hend =3D min(hend, vma->vm_end & HPAGE_PMD_MASK); } =20 - result =3D collapse_single_pmd(addr, vma, &mmap_unlocked, cc); + result =3D collapse_scan_pmd(vma, addr, cc); + /* Nothing to collapse here, and the lock is still ours */ + if (result !=3D SCAN_SUCCEED) + goto tally; =20 + /* The collapse takes its own locks, so give this up */ + mmap_read_unlock(mm); + *lock_dropped =3D true; + vma =3D NULL; + + result =3D collapse_run_pmd(mm, addr, cc); +tally: switch (result) { case SCAN_SUCCEED: case SCAN_PMD_MAPPED: @@ -3299,17 +3297,15 @@ int madvise_collapse(struct vm_area_struct *vma, un= signed long start, default: last_fail =3D result; /* Other error, exit */ - goto out_maybelock; + goto out; } } =20 -out_maybelock: +out: /* Caller expects us to hold mmap_lock on return */ - if (mmap_unlocked) { - *lock_dropped =3D true; + if (!vma) mmap_read_lock(mm); - } -out_nolock: +out_locked: mmap_assert_locked(mm); collapse_control_release(cc); kfree(cc); --=20 2.54.0 From nobody Fri Sep 25 16:54:11 2026 Received: from fout-b8-smtp.messagingengine.com (fout-b8-smtp.messagingengine.com [202.12.124.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9663A486639 for ; Thu, 10 Sep 2026 12:03:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.151 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041794; cv=none; b=IolgroVEqvTTL5pELDbc41wOvsMUrxAzJMfbyGQmX6l1sQz5HC7pOzadoC7YHj/wLAlyJa860VkPwJg0YSb/0uK/3LQ4sILdJ6dyrm9PK4YcsK0E2pOQ0TmQsO9G4eKrWMFoZZETv0yC8o3C67deaV1O2XTOs3l46BRmv2EpcuI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041794; c=relaxed/simple; bh=NR/LcEKDkCGT0y0VxK3lPGCDkKuIoiTqcZhAg6L+CaU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sb8LbftWDA7G/UneUQWDwbWTHDGL/2ME0WVqnw4YkhYV9lO6RM1ZgaXDKtNtb9NBcUKf9ycELLNyi8Cxy0McWufCRjWDFU0VoSPpHYtOhIfT3/me+Vk3hhvEHKFOQy5J1DsNiZN5AHPT80Zs4ARWcrJ829mo03XzfvwWSHUxsx0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=AFLreenn; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=F0b2S71Y; arc=none smtp.client-ip=202.12.124.151 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="AFLreenn"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="F0b2S71Y" Received: from phl-compute-03.internal (phl-compute-03.internal [10.202.2.43]) by mailfout.stl.internal (Postfix) with ESMTP id 8661C1D0006F; Thu, 10 Sep 2026 08:03:11 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-03.internal (MEProxy); Thu, 10 Sep 2026 08:03:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789041791; x= 1789128191; bh=q0JyoZbmyblE9u75HKEHqN4ei8zPfvwntibjbDEyo6k=; b=A FLreenn/FKiN76vFFDo7ezg7+h5fUlfERrFlbWnzdgZkxHtyZdLuN9UWj/Xo3TR4 EWI2aVmPgnLHibn9fUnlRdCskXtz4rbSaiJFsRGCdqfBoQus49TX86CZxGN171HS BjhATunS8qCehWn0CPlettdJxcIb6ZnW4+W4itlQHRb7rbo2rwP99RyL9HPwTk0G DlvlQ0JPmXya79GVdL5gW92o5aQc08qzM2aFD2DojBXgMB/L+oqXusvsL0wogcCd o5WeqdNmEpzhBToFHSAYYXN5oBucL5shVq+ZX37ncX0M+iztEAKhYtT0725Ft3Yb Q3LChs9ExYiztWAY77UZA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789041791; x=1789128191; bh=q 0JyoZbmyblE9u75HKEHqN4ei8zPfvwntibjbDEyo6k=; b=F0b2S71Y36jbXPMSH QFQuddPwPMhuJLoaSOwi6ZKPZ7SZoEvhRJUwWEs+7JWjyJClCCyl28BOmQABc9oO IauGAlvOLVF+9BHKVkhURQzEYl/L2dTC2oTYgSwV6cPhvagxjtlGlHVh1owNflId MWyU5D0m77KIhm+7VCXauH9CwPZ9waFJOWPfCL/BIWwID1lXthgYcNntHmhZJCSI 54bH8lNiiIe5+76zcBanpDuA71oFLdrMMc2R+VQzw74wxuutg/eAf/f7avyDzMrU D6HI4tsyRppOc6t+pcRNDI/SmVrwod3F47rWrh+8u1HxNOYPzgcZeViw7XFsYUUB vKrTw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEVCHaom43ud29NDpH2Ao3u7HJ/1ECQvA0KKWbvYxhttLa4TfzKR6zF5UuYx3UdSX No90AnuR4y57x1CX4a9AAsMV1JR4eQYCvRkfoTmW6C9Lc6lI8AZbTNwmqz4UTzT3vnk7qg ph+TCI1xI7srBtDO6Tg5MLzI4vqgzAzQGVqRHCjhPRmxizMOs2MvZfOp4fytB88FrNmNEM gORbCSjN6PT3vgIUt7WNrJZsAWFpVymtt3pMoPYb3jF9rU+m42uK7S67tVqaIc+biyrQXs /trDbiCnZfBDKlB4R/Y3z4EcguiWXCyuMp2o4aP4AZn5jbn1xeIsCoHcLAwPh3Pk2CuoKe z9gEfZbTx/s1NRbY0jw8R+QgBvZ0u9iB6GeqgbbrprwZ22eKYswqdWvNOA3y2DzUhzgifJ L53mMQtzGsi8vSL/IvC+q3HdiZUi9t5dgSeX+YtwXNJa7ADsBMfmglT/I2qmbIDhzGScte RJjv3ehRhULU3HZ3XbvpWYtZsWv6O0Zkxn35AopAZA9dvcjzKlmDJv6kdFgXlqwfKUONgk 7F06cG+mHPizviSr6lePKtukpa6+OGV3R1JWhNCMHndZK+FsrVTG8zWeIbFh0hFzyuoVoq FTfk9M6RLNhEUYFvbPZ8zHbCVbtEsAs1dZzM3yRDwyQgKwmEWN0r2/rzs9zg X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Sep 2026 08:03:10 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v2 10/12] mm/collapse: work out the orders a VMA allows once per VMA Date: Thu, 10 Sep 2026 13:02:30 +0100 Message-ID: <20260910120238.2529819-11-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260910120238.2529819-1-kirill@shutemov.name> References: <20260910120238.2529819-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" The scan asked collapse_possible_orders() for every PTE table, for an answer that is a property of the VMA. Both callers walk a VMA a table at a time, so let them work it out once and pass the mask in. It is only good while the lock that produced it is held, so madvise_collapse() takes it again after every collapse. The mask is then sampled once per VMA rather than once per table. A thp enabled knob written during a walk takes effect one VMA later, and cannot widen a collapse: hugepage_vma_revalidate() tests the order again under the lock the collapse retakes. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) Reviewed-by: Baolin Wang Reviewed-by: Zi Yan --- mm/khugepaged.c | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 9bdf12128357..9e77f71a788d 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1551,12 +1551,12 @@ static enum scan_result mthp_collapse(struct mm_str= uct *mm, } =20 static enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma, - unsigned long start_addr, struct collapse_control *cc) + unsigned long start_addr, struct collapse_control *cc, + unsigned long enabled_orders) { const unsigned int max_ptes_shared =3D collapse_max_ptes_shared(cc, HPAGE= _PMD_ORDER); const unsigned int max_ptes_swap =3D collapse_max_ptes_swap(cc, HPAGE_PMD= _ORDER); unsigned int max_ptes_none =3D collapse_max_ptes_none(cc, vma, HPAGE_PMD_= ORDER); - enum tva_type tva_flags =3D cc->policy.tva_type; struct mm_struct *mm =3D vma->vm_mm; pmd_t *pmd; pte_t *pte, *_pte, pteval; @@ -1567,7 +1567,6 @@ static enum scan_result collapse_scan_anon_pmd(struct= vm_area_struct *vma, struct folio *folio =3D NULL; unsigned long failed_pfn =3D -1; unsigned long addr; - unsigned long enabled_orders; spinlock_t *ptl; int node =3D NUMA_NO_NODE, unmapped =3D 0; =20 @@ -1581,8 +1580,6 @@ static enum scan_result collapse_scan_anon_pmd(struct= vm_area_struct *vma, =20 collapse_scan_reset(cc); =20 - enabled_orders =3D collapse_possible_orders(vma, vma->vm_flags, tva_flags= ); - /* * If PMD is the only enabled order, enforce max_ptes_none, otherwise * scan all pages to populate the bitmap for mTHP collapse. The bitmap @@ -2762,7 +2759,8 @@ static void collapse_control_release(struct collapse_= control *cc) } =20 static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, - unsigned long addr, struct collapse_control *cc) + unsigned long addr, struct collapse_control *cc, + unsigned long orders) { enum scan_result result; pgoff_t pgoff; @@ -2775,7 +2773,7 @@ static enum scan_result collapse_scan_pmd(struct vm_a= rea_struct *vma, } =20 if (vma_is_anonymous(vma)) - return collapse_scan_anon_pmd(vma, addr, cc); + return collapse_scan_anon_pmd(vma, addr, cc, orders); =20 pgoff =3D linear_page_index(vma, addr); result =3D collapse_scan_file(vma->vm_mm, addr, vma->vm_file, pgoff, cc); @@ -2896,15 +2894,17 @@ static void collapse_scan_mm_slot(unsigned int prog= ress_max, =20 vma_iter_init(&vmi, mm, khugepaged_scan.address); for_each_vma(vmi, vma) { - unsigned long hstart, hend; + unsigned long hstart, hend, orders; =20 cond_resched(); if (unlikely(collapse_test_exit_or_disable(mm))) { cc->progress++; break; } - if (!collapse_possible_orders(vma, vma->vm_flags, - TVA_KHUGEPAGED)) { + /* One mask for the whole VMA */ + orders =3D collapse_possible_orders(vma, vma->vm_flags, + cc->policy.tva_type); + if (!orders) { cc->progress++; continue; } @@ -2933,7 +2933,7 @@ static void collapse_scan_mm_slot(unsigned int progre= ss_max, /* move to next address */ khugepaged_scan.address +=3D HPAGE_PMD_SIZE; =20 - *result =3D collapse_scan_pmd(vma, addr, cc); + *result =3D collapse_scan_pmd(vma, addr, cc, orders); /* Nothing to collapse here, and the lock is still ours */ if (*result !=3D SCAN_SUCCEED) { if (cc->progress >=3D progress_max) @@ -3217,14 +3217,16 @@ int madvise_collapse(struct vm_area_struct *vma, un= signed long start, { struct collapse_control *cc; struct mm_struct *mm =3D vma->vm_mm; - unsigned long hstart, hend, addr; + unsigned long hstart, hend, addr, orders; enum scan_result last_fail =3D SCAN_FAIL; int thps =3D 0; =20 BUG_ON(vma->vm_start > start); BUG_ON(vma->vm_end < end); =20 - if (!collapse_possible_orders(vma, vma->vm_flags, TVA_FORCED_COLLAPSE)) + orders =3D collapse_possible_orders(vma, vma->vm_flags, + TVA_FORCED_COLLAPSE); + if (!orders) return -EINVAL; =20 hstart =3D ALIGN(start, HPAGE_PMD_SIZE); @@ -3262,9 +3264,11 @@ int madvise_collapse(struct vm_area_struct *vma, uns= igned long start, } vma =3D found; hend =3D min(hend, vma->vm_end & HPAGE_PMD_MASK); + orders =3D collapse_possible_orders(vma, vma->vm_flags, + cc->policy.tva_type); } =20 - result =3D collapse_scan_pmd(vma, addr, cc); + result =3D collapse_scan_pmd(vma, addr, cc, orders); /* Nothing to collapse here, and the lock is still ours */ if (result !=3D SCAN_SUCCEED) goto tally; --=20 2.54.0 From nobody Fri Sep 25 16:54:11 2026 Received: from fout-b8-smtp.messagingengine.com (fout-b8-smtp.messagingengine.com [202.12.124.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CB8A648663E for ; Thu, 10 Sep 2026 12:03:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.151 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041796; cv=none; b=PX4lOcZrXYFeyDTeW8ibntamXwcJxD2FceSzwhehtiI/XI8lFRP2w0kKILBhR6141eEGEgLWJ/h8BE2t0ShxKtVZfym5LtPOYnSAie5m0NR7EvEKSXCtmjSNfogdxPwKsFUk2gdN4o5+A7XXwzrZ1VkBUILavaT4jiziTHhoGeE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041796; c=relaxed/simple; bh=sB7GXY8tz8WCai8H2INDDIfL/6ubnxNVvg6UUEURYAM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DiPCF9pnXAnlfbbqAi5RsfVyt3CSDEJzleiWAPrm/68BUIdh5kl1SP8ybCCJuk/TUQKzwZPm9loJYq7jKf2ZIjq+mj0mDAd//Wny1EtBKUxqKTdkz4ClD1blO+UnUuP9pS73f8HGl+/4tk0zFi1ty5pxB0HlTPXvQ0+GxH+fChk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=JYuMWBx4; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=XAB7hUSF; arc=none smtp.client-ip=202.12.124.151 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="JYuMWBx4"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="XAB7hUSF" Received: from phl-compute-02.internal (phl-compute-02.internal [10.202.2.42]) by mailfout.stl.internal (Postfix) with ESMTP id B07721D000AF; Thu, 10 Sep 2026 08:03:13 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-02.internal (MEProxy); Thu, 10 Sep 2026 08:03:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789041793; x= 1789128193; bh=U3fJN6p296lB8xQAGr9+sipff9HImTdOJUAonJMUUhM=; b=J YuMWBx4T1Oe8RKUAsbsHTo9RuPm9QxNeyXXP7Kur7rWtkK21LZKwq7Pl6DwcMct0 5s3FS1tKOS8vh8/x1CI8GtIw0E/6Z9QuoZTw5rqLF/ikTvt5ur3h+cZCv1dfgPN2 fVh3UrlWEuuQs/0y13eddIgQe8utwbFytt0wPWL/NOsf+kAUPz8FMpFpskiabVeZ u8rkxQd0VlGS03Fsz5eAOd8ts8cFjJOeGoePwAiK6+WwMFDcD/tvY3iv4zt64bdR 1nRZ9a9Tu42BnT+kMUCIdqMCKdduJfNYCLKe/Ru07GaQUnYmkJ8CUoHuaejSk0FR EVGhzwnuaB/pEzq7nQ6Rw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789041793; x=1789128193; bh=U 3fJN6p296lB8xQAGr9+sipff9HImTdOJUAonJMUUhM=; b=XAB7hUSFpA12cCPJM OMDqYRLFct9E/qNWpFh9WOaKkBgGDBuo0OWLQ0k4cdSWLGEe6I01MZxXtkTHRvyl quHDuLMGAUPV/7a8RT1THTqviv3A/hRciKXzJgXngMUngjH1sDRmpAikw1nvPpHH wlthYfptb5vddwMXqBj3fvOudK8I8FsX1W7IeKhqqg+j3nxLhzpcWMNQ8zUyskIZ 59jitQ/Hfw1XQp60giP7FFJjdgHCLYTc3fvBDub8fRX5XrBkEGBxTq6tXhE26e+8 YY3sRCZXt+je++5NcBa3utZyv2QZTsk3OZFBQshOWcSsAqCY09cUPhlXgiY+XdCB UDZIg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTGUwtybjwPNTLo6R8Djs851CxE4dYABoksc5DOLebDAdjF/7kz2LTXedMZaJHaEsl u+9R36zRT1ZhtRkJBR4Xz/EwBimHSAG+mwOmEGehzEiv3PYSuq6Z2mKTclu6l8ZCGLaRVv RupVEjOuMXLaHhEvmZ7MRnQK4S/jaA4Fn+A/pUOz8kiJx7aYiU9MigP37WKT+MGxNap2wz dRarL4pe9tmhsdpwwUXZj55Ji6mi71ZgLtAGecz1FGZAcB1ieVS0L457vxQ1F9sxIDxLwT QcbTU8EZJEacVnPbAL6aT/vW8eX/g2TSMrM6YxxBPc/XsEEo+/bzW/w+LZ0gFhc/ZZneV+ vBUhDdd1wrc7OOZstiDjdJu3h/8fXkEkTVsJWdfGlzoNfbRCw8ttRXYEMeIWUbm/cXSR4L 0eKZQ8+Dm52lOCDT4lAOt1AIzAq4SRNVZG94DY+1cHz6Wr+rdWScldoDGWCVh0TgCnQvX+ Vsc8iMYu7wteWYcCiBcAtfKxwZV66W02PEgzgOvGxpevqvlGLEyzy8Mr8lBBhpF+WFjwU/ 60RCAixwv13lRN3dbWksHjFMLsqo3WnZ6aV0/CpAK5N13BPRs2YFczS7JV8rfw5eEboqu3 c1ww/CKHpIPKYiFUkb1Cnqv8n2/vWAueO8W2qFX4yGVL8DFNiVEohvoyHhMw X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Sep 2026 08:03:13 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v2 11/12] mm/collapse: declare the collapse interface in collapse.h Date: Thu, 10 Sep 2026 13:02:31 +0100 Message-ID: <20260910120238.2529819-12-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260910120238.2529819-1-kirill@shutemov.name> References: <20260910120238.2529819-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" A collapse takes four calls: - collapse_control_init() - set up the control a caller carries; - collapse_scan_pmd() - scan one PTE table, under mmap_lock; - collapse_run_pmd() - collapse what the scan found, no mmap_lock; - collapse_control_release() - done with the control. All four are static in khugepaged.c, as are collapse_possible_orders(), which says what a VMA allows, and the revalidate a caller needs once a collapse has given the mmap_lock up. No other file can ask for a collapse without them. Declare them in collapse.h, with a comment stating the order they are called in and who holds the lock over each step. hugepage_vma_revalidate() becomes collapse_vma_revalidate(): it is part of what a collapse offers now, not a helper of the daemon. Preparation for implementing MADV_COLLAPSE in madvise.c. No functional change. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) --- mm/collapse.h | 44 ++++++++++++++++++++++++++++++++++++++++++++ mm/khugepaged.c | 18 +++++++++--------- 2 files changed, 53 insertions(+), 9 deletions(-) diff --git a/mm/collapse.h b/mm/collapse.h index 346859a2184f..1ebbbf63fb25 100644 --- a/mm/collapse.h +++ b/mm/collapse.h @@ -106,4 +106,48 @@ struct collapse_control { bool scan_retract_only; }; =20 +/* Which orders a VMA may collapse to, zero when it may not collapse at al= l */ +unsigned long collapse_possible_orders(struct vm_area_struct *vma, + vm_flags_t vm_flags, enum tva_type tva_flags); + +/* + * A caller states what it allows in cc->policy and then hands over one PTE + * table's worth of a VMA at a time: + * + * collapse_control_init(cc) once, before the first table + * collapse_scan_pmd(vma, addr, ...) per table + * collapse_run_pmd(mm, addr, cc) when a scan found work + * collapse_control_release(cc) once, when done with the control + * + * The caller holds mmap_lock for reading over the scan and passes an addr= ess + * within @vma, aligned to the PTE table the scan is to judge. + * + * The scan returns with that lock still held. It only reads, and almost = every + * table it is offered has nothing to collapse, so a caller walks a whole = VMA + * under the one lock it took to get there. SCAN_SUCCEED means there is + * something to collapse; anything else is why there is not. + * + * The run is called without the lock and returns without it, taking what = it + * needs in between: what it does -- allocate, isolate, copy, flush -- is = slow + * enough that a writer would wait behind it. The caller gives the lock up + * first, and with it @vma and anything derived under it, so a caller carr= ying + * on has to look up again with collapse_vma_revalidate(). The run revali= dates + * for itself rather than trusting what the scan saw. + * + * A scan that found something has to be run: the file side takes a refere= nce on + * the file while it still has the VMA to take it from, and the run is what + * gives it back. + */ +void collapse_control_init(struct collapse_control *cc); +void collapse_control_release(struct collapse_control *cc); +enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, + unsigned long addr, struct collapse_control *cc, + unsigned long orders); +enum scan_result collapse_run_pmd(struct mm_struct *mm, unsigned long addr, + struct collapse_control *cc); +enum scan_result collapse_vma_revalidate(struct mm_struct *mm, + unsigned long address, bool expect_anon, + struct vm_area_struct **vmap, struct collapse_control *cc, + unsigned int order); + #endif /* __MM_COLLAPSE_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 9e77f71a788d..77f34aedf0f2 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -498,7 +498,7 @@ void __khugepaged_enter(struct mm_struct *mm) * Check what orders are possible based on the vma and collapse type. * This is used to determine if mTHP collapse is a viable option. */ -static unsigned long collapse_possible_orders(struct vm_area_struct *vma, +unsigned long collapse_possible_orders(struct vm_area_struct *vma, vm_flags_t vm_flags, enum tva_type tva_flags) { unsigned long orders; @@ -1016,7 +1016,7 @@ static int collapse_find_target_node(struct collapse_= control *cc) * Returns enum scan_result value. */ =20 -static enum scan_result hugepage_vma_revalidate(struct mm_struct *mm, unsi= gned long address, +enum scan_result collapse_vma_revalidate(struct mm_struct *mm, unsigned lo= ng address, bool expect_anon, struct vm_area_struct **vmap, struct collapse_control *cc, unsigned int order) { @@ -1264,7 +1264,7 @@ static enum scan_result collapse_huge_page(struct mm_= struct *mm, unsigned long s } =20 mmap_read_lock(mm); - result =3D hugepage_vma_revalidate(mm, pmd_addr, /*expect_anon=3D*/ true, + result =3D collapse_vma_revalidate(mm, pmd_addr, /*expect_anon=3D*/ true, &vma, cc, order); if (result !=3D SCAN_SUCCEED) { mmap_read_unlock(mm); @@ -1299,7 +1299,7 @@ static enum scan_result collapse_huge_page(struct mm_= struct *mm, unsigned long s * mmap_lock. */ mmap_write_lock(mm); - result =3D hugepage_vma_revalidate(mm, pmd_addr, /*expect_anon=3D*/ true, + result =3D collapse_vma_revalidate(mm, pmd_addr, /*expect_anon=3D*/ true, &vma, cc, order); if (result !=3D SCAN_SUCCEED) goto out_up_write; @@ -2743,13 +2743,13 @@ static enum scan_result collapse_scan_file(struct m= m_struct *mm, return result; } =20 -static void collapse_control_init(struct collapse_control *cc) +void collapse_control_init(struct collapse_control *cc) { cc->progress =3D 0; cc->scan_file =3D NULL; } =20 -static void collapse_control_release(struct collapse_control *cc) +void collapse_control_release(struct collapse_control *cc) { /* A scan that took a file reference should have been run */ if (WARN_ON_ONCE(cc->scan_file)) { @@ -2758,7 +2758,7 @@ static void collapse_control_release(struct collapse_= control *cc) } } =20 -static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, +enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, unsigned long addr, struct collapse_control *cc, unsigned long orders) { @@ -2802,7 +2802,7 @@ static enum scan_result collapse_scan_pmd(struct vm_a= rea_struct *vma, return result; } =20 -static enum scan_result collapse_run_pmd(struct mm_struct *mm, +enum scan_result collapse_run_pmd(struct mm_struct *mm, unsigned long addr, struct collapse_control *cc) { struct file *file =3D cc->scan_file; @@ -3256,7 +3256,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, if (!vma) { cond_resched(); mmap_read_lock(mm); - result =3D hugepage_vma_revalidate(mm, addr, false, &found, + result =3D collapse_vma_revalidate(mm, addr, false, &found, cc, HPAGE_PMD_ORDER); if (result !=3D SCAN_SUCCEED) { last_fail =3D result; --=20 2.54.0 From nobody Fri Sep 25 16:54:11 2026 Received: from fout-b8-smtp.messagingengine.com (fout-b8-smtp.messagingengine.com [202.12.124.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2925F486E58 for ; Thu, 10 Sep 2026 12:03:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.151 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041798; cv=none; b=k2Uu8OYHY7XmuFrZnsqHyHLcws25vWA+E90z6N0HhcNOEl1DsAxLH1a2BhefSRgNeCZMRv7iSk8VSydhRcttKb8qacTaayIS1CZd5N2IUtqXHy3aqNrTlb+el9APbr1eYGUiDjblgiSpP1IOPud24X55y5AaEfc3KNIRGifh9pI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789041798; c=relaxed/simple; bh=g8nnLjE/68MHdncarXue+A9bp1V+ExShoGgz0afiaX4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sG82EKYjlASaJy9TzYxM2hoJzMb/CABcISmnhT6suL9+otf3DL5njr5pt1jCIJO2BeLbWN1KxkR4Ucpu9Ghwt+kYq7QEJYfPUqZEqRm8nQSFnSKyxGYwAhrHxlqU0owQfykJpRyE6pumAcIgmZbrwNCFPw64lUYUlrRq/z2FDRE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=Y/IDDIaw; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=hpByUBib; arc=none smtp.client-ip=202.12.124.151 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="Y/IDDIaw"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="hpByUBib" Received: from phl-compute-02.internal (phl-compute-02.internal [10.202.2.42]) by mailfout.stl.internal (Postfix) with ESMTP id 4255A1D0004F; Thu, 10 Sep 2026 08:03:16 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-02.internal (MEProxy); Thu, 10 Sep 2026 08:03:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789041796; x= 1789128196; bh=ZttzZbEWmniFloGWXXa9WcpJ9aZ/lRN/IO7GYTjQ29k=; b=Y /IDDIawoHWwsapkKw6WfeUQ/5ZhVEe130M8sRiq1dHYJgA7nuOCfyb3jyPOAkM7M rPIiJZu3JxA/zcskxOHDEg/vr4YwN7q9sXaFnRI9I1z0pIXld+ykzCipwsUJAc6L Ncu0gqfn/l1Esf8aVfiv9lNK70XN5OHrpj8/BdsiLz569CZjverp2Re1WwOjjypM Fh9dYae9KkB/SYvS2TDDJVhgn9GUuy7vg4R45ZCjcRlVTBR19xuoINVpRsLZ29SD DvdTTjKxEaBS0p3hIBXi8FkPoyrycFLlqGXYa5OFO69d/V8yKKgwQfTkpwurbkUK iSLL0k71z3wUnllHm23eg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789041796; x=1789128196; bh=Z ttzZbEWmniFloGWXXa9WcpJ9aZ/lRN/IO7GYTjQ29k=; b=hpByUBibgZSTSQlt1 0A+OsZkMGGnQF9esFIvBt9DgvbTOiCfur4ZTu5QjuTr5Fl3oTaCcvccQbCbvmmMo TxmR1W15njV63TkCNK66knLEsUKeng2qkj1SRPd0cxbYIyV8WPiEDs46pJSeSOQP z26/UluebBIvIC2edFhjVChSV9BZ1imXauCc2/k7QZPj7hGNYwIJXf/9XML3hjxw mzRmynUvEaYSM/Xj8MQrLcP1cVslzivl4grnggCZC8QXf7EccMPftP0Rd4AmaVJr KDLFGXMJsuk4cD/iaB51Jdlp0hVP2Vsif9GJu+ZJNHPlRTv0K5TK6AHWkCkrniQy tP6ng== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTGUwtybjwPNTLo6R8Djs851CxE4dYABoksc5DOLebDAdjF/7kz2LTXedMZaJHaEsl u+9R36zRT1ZhtRkJBR4Xz/EwBimHSAG+mwOmEGehzEiv3PYSuq6Z2mKTclu6l8ZCGLaRVv RupVEjOuMXLaHhEvmZ7MRnQK4S/jaA4Fn+A/pUOz8kiJx7aYiU9MigP37WKT+MGxNap2wz dRarL4pe9tmhsdpwwUXZj55Ji6mi71ZgLtAGecz1FGZAcB1ieVS0L457vxQ1F9sxIDxLwT QcbTU8EZJEacVnPbAL6aT/vW8eX/g2TSMrM6YxxBPc/XsEEo+/bzW/w+LZ0gFhc/ZZnenC oz1QcdLXM9OiLPZRMrV452RrLNmkTNrinjOdREYRHS2NkkyGueNPNckGvPvq6YZoblNe4k bFSCxQvNlMrOLPybz0XCbF4VEV2g82XKF47qda8jBCJqqo6DE4Jt7dyt7qjGHpXsek1lYr wMLN4eubNCOayGO5XTPuTSp6zMxz7RIv7XbW/rsKqIftVh8MGIg3m2KN63AunHYZiIP5tp /QRV0UtUulhq0K59rVSm7YVyeLPkSeHydjthDNWGrQRUG9VTdcENvWdeFscLeoefHiMs7X O9QxfbHnwjHq3HT2aJxYuRAFT+5/1vfzUIIgKRzqSOv0y1o9qiXJk1ZRkDrg X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Sep 2026 08:03:15 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v2 12/12] mm/collapse: implement MADV_COLLAPSE in madvise.c Date: Thu, 10 Sep 2026 13:02:32 +0100 Message-ID: <20260910120238.2529819-13-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260910120238.2529819-1-kirill@shutemov.name> References: <20260910120238.2529819-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" MADV_COLLAPSE is a madvise operation, but its implementation sat in khugepaged.c. The daemon's file therefore also held a syscall's worth of code that has nothing to do with the daemon: the walk over the user's range, the per-PMD loop, and the errno translation. Move it to madvise.c, among the operations it belongs with, along with the errno map and the policy it states for itself. It takes a struct madvise_behavior like every one of those operations, which is where the range, the VMA and the lock-dropped flag it used to be handed separately already live. It stays a caller of the interface khugepaged uses, so nothing about the collapse changes. The !CONFIG_TRANSPARENT_HUGEPAGE stub moves in with it. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) --- include/linux/huge_mm.h | 9 --- mm/khugepaged.c | 157 +------------------------------------ mm/madvise.c | 169 +++++++++++++++++++++++++++++++++++++++- 3 files changed, 169 insertions(+), 166 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index c745f7ad2298..8ca0fa3be2ac 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -510,8 +510,6 @@ change_huge_pud(struct mmu_gather *tlb, struct vm_area_= struct *vma, =20 int hugepage_madvise(struct vm_area_struct *vma, vm_flags_t *vm_flags, int advice); -int madvise_collapse(struct vm_area_struct *vma, unsigned long start, - unsigned long end, bool *lock_dropped); void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start, unsigned long end, struct vm_area_struct *next); spinlock_t *__pmd_trans_huge_lock(pmd_t *pmd, struct vm_area_struct *vma); @@ -715,13 +713,6 @@ static inline int hugepage_madvise(struct vm_area_stru= ct *vma, return -EINVAL; } =20 -static inline int madvise_collapse(struct vm_area_struct *vma, - unsigned long start, - unsigned long end, bool *lock_dropped) -{ - return -EINVAL; -} - static inline void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start, unsigned long end, diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 77f34aedf0f2..1deb74cf28af 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -969,21 +969,6 @@ static void collapse_policy_khugepaged(struct collapse= _policy *p) p->tva_type =3D TVA_KHUGEPAGED; } =20 -/* MADV_COLLAPSE was asked for explicitly, so it is not held to those */ -static void collapse_policy_forced(struct collapse_policy *p) -{ - p->max_ptes_none =3D HPAGE_PMD_NR; - p->max_ptes_swap =3D HPAGE_PMD_NR; - p->max_ptes_shared =3D HPAGE_PMD_NR; - p->strict_sub_pmd =3D false; - p->skip_lazyfree =3D false; - p->require_referenced =3D false; - p->install_pmd =3D true; - p->writeback_dirty =3D true; - p->gfp =3D GFP_TRANSHUGE; - p->tva_type =3D TVA_FORCED_COLLAPSE; -} - #ifdef CONFIG_NUMA static int collapse_find_target_node(struct collapse_control *cc) { @@ -2802,8 +2787,8 @@ enum scan_result collapse_scan_pmd(struct vm_area_str= uct *vma, return result; } =20 -enum scan_result collapse_run_pmd(struct mm_struct *mm, - unsigned long addr, struct collapse_control *cc) +enum scan_result collapse_run_pmd(struct mm_struct *mm, unsigned long addr, + struct collapse_control *cc) { struct file *file =3D cc->scan_file; bool triggered_wb =3D false; @@ -3179,141 +3164,3 @@ bool current_is_khugepaged(void) { return kthread_func(current) =3D=3D khugepaged; } - -static int madvise_collapse_errno(enum scan_result r) -{ - /* - * MADV_COLLAPSE breaks from existing madvise(2) conventions to provide - * actionable feedback to caller, so they may take an appropriate - * fallback measure depending on the nature of the failure. - */ - switch (r) { - case SCAN_ALLOC_HUGE_PAGE_FAIL: - return -ENOMEM; - case SCAN_CGROUP_CHARGE_FAIL: - case SCAN_EXCEED_NONE_PTE: - return -EBUSY; - /* Resource temporary unavailable - trying again might succeed */ - case SCAN_PAGE_COUNT: - case SCAN_PAGE_LOCK: - case SCAN_PAGE_LRU: - case SCAN_DEL_PAGE_LRU: - case SCAN_PAGE_FILLED: - case SCAN_PAGE_HAS_PRIVATE: - case SCAN_PAGE_DIRTY_OR_WRITEBACK: - return -EAGAIN; - /* - * Other: Trying again likely not to succeed / error intrinsic to - * specified memory range. khugepaged likely won't be able to collapse - * either. - */ - default: - return -EINVAL; - } -} - -int madvise_collapse(struct vm_area_struct *vma, unsigned long start, - unsigned long end, bool *lock_dropped) -{ - struct collapse_control *cc; - struct mm_struct *mm =3D vma->vm_mm; - unsigned long hstart, hend, addr, orders; - enum scan_result last_fail =3D SCAN_FAIL; - int thps =3D 0; - - BUG_ON(vma->vm_start > start); - BUG_ON(vma->vm_end < end); - - orders =3D collapse_possible_orders(vma, vma->vm_flags, - TVA_FORCED_COLLAPSE); - if (!orders) - return -EINVAL; - - hstart =3D ALIGN(start, HPAGE_PMD_SIZE); - hend =3D ALIGN_DOWN(end, HPAGE_PMD_SIZE); - - if (hstart >=3D hend) - return 0; - - cc =3D kmalloc_obj(*cc); - if (!cc) - return -ENOMEM; - collapse_control_init(cc); - collapse_policy_forced(&cc->policy); - - lru_add_drain_all(); - - for (addr =3D hstart; addr < hend; addr +=3D HPAGE_PMD_SIZE) { - struct vm_area_struct *found; - enum scan_result result; - - /* - * A collapse gives the lock up, so the VMA has to be found - * again after one: it can shrink while nothing is held. A scan - * that finds nothing to collapse leaves the lock alone, so a - * range that is already collapsed walks on without relocking. - */ - if (!vma) { - cond_resched(); - mmap_read_lock(mm); - result =3D collapse_vma_revalidate(mm, addr, false, &found, - cc, HPAGE_PMD_ORDER); - if (result !=3D SCAN_SUCCEED) { - last_fail =3D result; - goto out_locked; - } - vma =3D found; - hend =3D min(hend, vma->vm_end & HPAGE_PMD_MASK); - orders =3D collapse_possible_orders(vma, vma->vm_flags, - cc->policy.tva_type); - } - - result =3D collapse_scan_pmd(vma, addr, cc, orders); - /* Nothing to collapse here, and the lock is still ours */ - if (result !=3D SCAN_SUCCEED) - goto tally; - - /* The collapse takes its own locks, so give this up */ - mmap_read_unlock(mm); - *lock_dropped =3D true; - vma =3D NULL; - - result =3D collapse_run_pmd(mm, addr, cc); -tally: - switch (result) { - case SCAN_SUCCEED: - case SCAN_PMD_MAPPED: - ++thps; - break; - /* Whitelisted set of results where continuing OK */ - case SCAN_NO_PTE_TABLE: - case SCAN_PTE_NON_PRESENT: - case SCAN_PTE_UFFD: - case SCAN_LACK_REFERENCED_PAGE: - case SCAN_PAGE_NULL: - case SCAN_PAGE_COUNT: - case SCAN_PAGE_LOCK: - case SCAN_PAGE_COMPOUND: - case SCAN_PAGE_LRU: - case SCAN_DEL_PAGE_LRU: - last_fail =3D result; - break; - default: - last_fail =3D result; - /* Other error, exit */ - goto out; - } - } - -out: - /* Caller expects us to hold mmap_lock on return */ - if (!vma) - mmap_read_lock(mm); -out_locked: - mmap_assert_locked(mm); - collapse_control_release(cc); - kfree(cc); - - return thps =3D=3D ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0 - : madvise_collapse_errno(last_fail); -} diff --git a/mm/madvise.c b/mm/madvise.c index 963337f93a7a..f75a9d139980 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -38,6 +38,7 @@ =20 #include "internal.h" #include "swap.h" +#include "collapse.h" =20 #define __MADV_SET_ANON_VMA_NAME (-1) =20 @@ -906,6 +907,171 @@ bool madvise_dontneed_free_valid_vma(struct madvise_b= ehavior *madv_behavior) return true; } =20 +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + +/* MADV_COLLAPSE was asked for explicitly, so it is not held to those */ +static void collapse_policy_forced(struct collapse_policy *p) +{ + p->max_ptes_none =3D HPAGE_PMD_NR; + p->max_ptes_swap =3D HPAGE_PMD_NR; + p->max_ptes_shared =3D HPAGE_PMD_NR; + p->strict_sub_pmd =3D false; + p->skip_lazyfree =3D false; + p->require_referenced =3D false; + p->install_pmd =3D true; + p->writeback_dirty =3D true; + p->gfp =3D GFP_TRANSHUGE; + p->tva_type =3D TVA_FORCED_COLLAPSE; +} + +static int madvise_collapse_errno(enum scan_result r) +{ + /* + * MADV_COLLAPSE breaks from existing madvise(2) conventions to provide + * actionable feedback to caller, so they may take an appropriate + * fallback measure depending on the nature of the failure. + */ + switch (r) { + case SCAN_ALLOC_HUGE_PAGE_FAIL: + return -ENOMEM; + case SCAN_CGROUP_CHARGE_FAIL: + case SCAN_EXCEED_NONE_PTE: + return -EBUSY; + /* Resource temporary unavailable - trying again might succeed */ + case SCAN_PAGE_COUNT: + case SCAN_PAGE_LOCK: + case SCAN_PAGE_LRU: + case SCAN_DEL_PAGE_LRU: + case SCAN_PAGE_FILLED: + case SCAN_PAGE_HAS_PRIVATE: + case SCAN_PAGE_DIRTY_OR_WRITEBACK: + return -EAGAIN; + /* + * Other: Trying again likely not to succeed / error intrinsic to + * specified memory range. khugepaged likely won't be able to collapse + * either. + */ + default: + return -EINVAL; + } +} + +static int madvise_collapse(struct madvise_behavior *madv_behavior) +{ + struct madvise_behavior_range *range =3D &madv_behavior->range; + struct vm_area_struct *vma =3D madv_behavior->vma; + struct mm_struct *mm =3D madv_behavior->mm; + struct collapse_control *cc; + unsigned long hstart, hend, addr, orders; + enum scan_result last_fail =3D SCAN_FAIL; + int thps =3D 0; + + BUG_ON(vma->vm_start > range->start); + BUG_ON(vma->vm_end < range->end); + + orders =3D collapse_possible_orders(vma, vma->vm_flags, + TVA_FORCED_COLLAPSE); + if (!orders) + return -EINVAL; + + hstart =3D ALIGN(range->start, HPAGE_PMD_SIZE); + hend =3D ALIGN_DOWN(range->end, HPAGE_PMD_SIZE); + + if (hstart >=3D hend) + return 0; + + cc =3D kmalloc_obj(*cc); + if (!cc) + return -ENOMEM; + collapse_control_init(cc); + collapse_policy_forced(&cc->policy); + + lru_add_drain_all(); + + for (addr =3D hstart; addr < hend; addr +=3D HPAGE_PMD_SIZE) { + struct vm_area_struct *found; + enum scan_result result; + + /* + * A collapse gives the lock up, so the VMA has to be found + * again after one: it can shrink while nothing is held. A scan + * that finds nothing to collapse leaves the lock alone, so a + * range that is already collapsed walks on without relocking. + */ + if (!vma) { + cond_resched(); + mmap_read_lock(mm); + result =3D collapse_vma_revalidate(mm, addr, false, &found, + cc, HPAGE_PMD_ORDER); + if (result !=3D SCAN_SUCCEED) { + last_fail =3D result; + goto out_locked; + } + vma =3D found; + hend =3D min(hend, vma->vm_end & HPAGE_PMD_MASK); + orders =3D collapse_possible_orders(vma, vma->vm_flags, + cc->policy.tva_type); + } + + result =3D collapse_scan_pmd(vma, addr, cc, orders); + /* Nothing to collapse here, and the lock is still ours */ + if (result !=3D SCAN_SUCCEED) + goto tally; + + /* The collapse takes its own locks, so give this up */ + mmap_read_unlock(mm); + mark_mmap_lock_dropped(madv_behavior); + vma =3D NULL; + + result =3D collapse_run_pmd(mm, addr, cc); +tally: + switch (result) { + case SCAN_SUCCEED: + case SCAN_PMD_MAPPED: + ++thps; + break; + /* Whitelisted set of results where continuing OK */ + case SCAN_NO_PTE_TABLE: + case SCAN_PTE_NON_PRESENT: + case SCAN_PTE_UFFD: + case SCAN_LACK_REFERENCED_PAGE: + case SCAN_PAGE_NULL: + case SCAN_PAGE_COUNT: + case SCAN_PAGE_LOCK: + case SCAN_PAGE_COMPOUND: + case SCAN_PAGE_LRU: + case SCAN_DEL_PAGE_LRU: + last_fail =3D result; + break; + default: + last_fail =3D result; + /* Other error, exit */ + goto out; + } + } + +out: + /* Caller expects us to hold mmap_lock on return */ + if (!vma) + mmap_read_lock(mm); +out_locked: + mmap_assert_locked(mm); + collapse_control_release(cc); + kfree(cc); + + return thps =3D=3D ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0 + : madvise_collapse_errno(last_fail); +} + +#else /* CONFIG_TRANSPARENT_HUGEPAGE */ + +static int madvise_collapse(struct madvise_behavior *madv_behavior) +{ + return -EINVAL; +} + +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + static long madvise_dontneed_free(struct madvise_behavior *madv_behavior) { struct mm_struct *mm =3D madv_behavior->mm; @@ -1373,8 +1539,7 @@ static int madvise_vma_behavior(struct madvise_behavi= or *madv_behavior) case MADV_DONTNEED_LOCKED: return madvise_dontneed_free(madv_behavior); case MADV_COLLAPSE: - return madvise_collapse(vma, range->start, range->end, - &madv_behavior->lock_dropped); + return madvise_collapse(madv_behavior); case MADV_GUARD_INSTALL: return madvise_guard_install(madv_behavior); case MADV_GUARD_REMOVE: --=20 2.54.0